THE HAGUE, Aug. 17— Sir Humphrey Waldock, president of the International Court of Justice, has died here of a heart attack, the court announced today. He was 77 years old.

As head of the court, Sir Humphrey presided over the United States case against Iran that resulted in a May 1980 order to free the American hostages. Iran ignored the ruling. The United States later withdrew its claim when the seizure was settled in direct negotiations.

Sir Humphrey was elected to the chief judicial arm of the United Nations in 1973 and became president in February 1979 for a threeyear term. Headed Rights Commission

He was formerly president of the European Commission of Human Rights and the European Court of Human Rights, and he served as under secretary in the British Admiralty in World War II.

A court spokesman said Sir Humphrey seemed in good health before he was stricken. He died Saturday night at his home, the spokesman said. A successor on the court must be elected by the United Nations from a list prepared by the Secretary General, Kurt Waldheim. The 15 judges, who serve nine-year terms, elect their own president. He Taught Law at Oxford

Sir Humphrey was born in Colombo, Ceylon, now Sri Lanka. He graduated from Brasenose College, Oxford, in 1927 and taught law at Oxford from 1930 until World War II, when he served in the Admiralty.

He returned to Oxford after the war, holding the Chichele Professor of Public International Law chair from 1947 to 1972. Sir Humphrey was president of the European Commission of Human Rights from 1955 to 1961 and represented the commission before the European Court of Human Rights in Strasbourg, France. He served on numerous United Nations commissions and was a member of the International Law Commission from 1962 to 1971.

He was a judge of the European Court of Human Rights from 1966 to 1974 and served as president in 1971. He was also a member of the Permanent Court of Arbitration, housed with the Court of Justice in the Peace Palace in The Hague. Third Judge to Die in a Year

Sir Humphrey was the third member of the international court to die in the last year. Richard R. Baxter of the United States died last September after a long illness, and Salah El Dine Tarazi of Syria was killed in a traffic accident in October.

One case is currently pending before the court, involving the continental shelf between Tunisia and Libya. Also pending is a request for an advisory opinion involving United Nations administrative rules.

Sir Humphrey is survived by a son and a daughter.