BOSTON, Nov. 6— Despite a nationwide landslide for President Reagan, Lieut. Gov. John F. Kerry, a liberal Democrat, defeated Raymond Shamie, a conservative Republican businessman, today for the Senate seat vacated by Paul E. Tsongas.

At the same time, Democrats won the 10 House seats they held in the last Congress, holding the Republicans to one seat.

''I believe this race gave Massachusetts a real choice,'' Mr. Kerry said. That choice, he said, was that the people of Massachusetts ''emphatically reject the politics of selfishness and the notion that women must be treated as second-class citizens.''

Conceding defeat in a speech to his supporters, Mr. Shamie, a selfmade millionaire, said that ''we can celebrate tonight the larger victory'' achieved by President Reagan, ''which makes up for our disappointment.''

''Only in America,'' Mr. Shamie said, and the crowd shouted ''U.S.A.''

Voter turnout was unusually heavy in Massachusetts, about 85 percent of the state's registered voters, a record, according to Michael Connally, the Secretary of State.

Mr. Kerry will succeed another liberal Democrat. Senator Tsongas announced his retirement earlier this year for health reasons.

With 67 percent of the precincts reporting, the vote was:

Kerry862,304 (56%)Shamie666,549 (44%) A Bitter Contest

The Senate fight was the most bitterly contested race in Massachusetts, which traditionally favors Democrats. Registered Democrats outnumber Republicans here by 4 to 1, and Massachusetts was the only state won by George McGovern, the Democratic Presidential nominee, in 1972.

But conservatives like Mr. Reagan have made deep inroads in recent years among urban working-class and Roman Catholic voters, the traditional backbone of the Democratic Party here. Mr. Shamie sought to capitalize on the popularity of Mr. Reagan's policies and personality by echoing his calls for lower taxes, a strong military and a ban on abortion.

Mr. Shamie, 63 years old, president of a high-technology concern he founded, spent nearly $1 million of his own money on the campaign, much of it for television advertisements. They portrayed him as a smiling, benign figure much like the President. Trading of Accusations

Mr. Shamie repeatedly accused Mr. Kerry, a decorated veteran of the Vietnam War, of lying and charged that he would raise taxes if elected. Mr. Kerry contended that his opponent offered overly simple solutions and cited Mr. Shamie's flirtation with the John Birch Society in the mid-1970's. Mr. Shamie responded by saying the Birch group, a right-wing organization, was part of ''mainstream America.''

Mr. Kerry, a tall, slender 40-year-old man with rugged good looks, sought to cast himself in the mold of Democrats like Senator Gary Hart by saying he would mix liberal compassion with tight budget controls.

A graduate of St. Paul's School and Yale University, Mr. Kerry has a paradoxical heritage, for Massachusetts, of having a father of Irish descent and a mother who is a Forbes, one of the richest Boston families.

Mr. Shamie, who never finished college, lost badly in 1982 to Senator Edward M. Kennedy in his only other attempt to win political office, and politicians here felt he would have run well behind Mr. Kerry but for Mr. Reagan's enormous appeal and the amount of his own money Mr. Shamie spent.

photo of voters