Prince Harrysays he and older brotherPrince Williamare on “different paths.”

DuringHarry and Meghan Markle’s interview with Oprah Winfreyon Sunday, the Duke of Sussex said that other members of his family can’t leave the “system” the way he did.

“It’s been really hard,” Harry explained. “I am part of the system with them, I always have been, but I guess — and I’m very aware of this — my brother can’t leave that system, but I have.”

When Oprah asked if William — who is second in line to the throne behind their father,Prince Charles— wanted to leave the royal family, Harry replied, “I don’t know. I can’t speak for him.”

Prince Harry, Duke of Sussex and Prince William, Duke of Cambridge

Prince Harry, Meghan Markle and Oprah Winfrey.Joe Pugliese/Harpo Productions

Prince harry and Megan and Oprah

Harry, 36, said that he “didn’t see a way out” of the royal family before meeting Meghan, 39.

“I was trapped, but I didn’t know I was trapped. Trapped within the system like the rest of the family,” he said. “My father and my brother are trapped. They don’t get to leave, and I have huge compassion for that.”

Get the premiere issue of PEOPLERoyalsfor glamorous new photos and inside stories royals fans haven’t seen or read elsewhere! Subscribe atpeopleroyals.com/launch

Prince Harry, Prince William and Prince Charles.

our-planet-13.jpg

He also described his relationship with his older sibling at the moment as “space.”

“I love William to bits. He’s my brother. We’ve been through hell together,” Harry explained. “But we are on different paths.”

Prince William and Kate Middleton.JUSTIN TALLIS/AFP via Getty Images

Prince William, Duke of Cambridge and Britain’s Catherine, Duchess of Cambridge

On Thursday, Prince William, 38, andKate Middleton, 39, stepped out for the first time since the interview aired. William hit back at racism claims made during the interview, telling a Sky News reporter, “We are very much not a racist family.”

For more onPrince William’s statement, listen below to the episode of PEOPLE Every Day.

When asked if he had spoken to Harry after the Oprah interview, he added, “No, I haven’t spoken to him yet, but I will do.”

Can’t get enough ofPEOPLE’s Royals coverage?Sign up for our free Royals newsletterto get the latest updates onKate Middleton,Meghan Markleand more!

Later in the sit-down, Oprah pressed Harry on the issue, asking him who was behind the racially charged comment. “That conversation I’m never going to share,” he said. “It was awkward. I was a bit shocked.”

source: people.com