Adele Says It's Not Her Job to 'Validate How People Feel About Their Bodies'

Adele says the uproar over her weight loss was an expected, but frustrating thing to see.

The 33-year-old singer, who lost around 100 lbs. over the last three years, told Oprah Winfrey during the Adele One Night Only special on Sunday that she wasn't surprised by the attention over her body.

"I'm not shocked or even fazed by it because my body has been objectified my entire career," Adele told Winfrey, after the longtime talk show host said that her own fans had felt "upset" and "abandoned" when she lost weight.

"I'm either too big or too small; I'm either hot or I'm not," Adele added.

The 15-time Grammy winner said that her weight loss hasn't changed how she sees her body or anyone else's, and that people shouldn't hold her up as a role model.

For more on Adele's One Night Only concert special and other top stories, listen below to our daily podcast on PEOPLE Every Day.

"I was body positive then and I'm body positive now," she said. "It's not my job to validate how people feel about their bodies. I feel bad if anyone feels horrible about themselves but that's not my job. I'm trying to sort my own life out. I can't add another worry."

Adele explained that her weight loss was largely unintentional — it was a byproduct of the gym time she started putting in three years ago as a way of managing her anxiety.

"I had the most terrifying anxiety attacks after I left my marriage," she said. "They paralyzed me completely, and made me so confused because I wouldn't be able to have any control over my body."

Adele found that working out kept her calm, and she started making daily trips to the gym.

"That really contributed towards me getting my mind right," she said. "… It became my time. Having these sort of pins in my day helped me keep myself together."

RELATED VIDEO: Adele Says She's in a 'Strong Place Now' After Upcoming Album 30 'Was Bloody Hard to Make'

The Easy on Me singer said that as she starts to tour for her new record and won't have as much gym time, she knows she could gain weight again but isn't concerned — she just wants to make sure she maintains her muscle strength for weightlifting. Adele said that her favorite movement is the deadlift, and was able to lift 170 lbs. at one point last year.

"I should've been an athlete," she said. "If only at school I hadn't discovered boys and someone had told me to go and do a bit more PE."

Plus, Adele said, she's also a strong boxer. "I've got a left hook that could kill ya."