Prince Harrysays his brotherPrince William"aired some concerns" about his relationship withMeghan Markle"very early on."

“Some of the things that my brother and sister-in-law — some of the way that they were acting or behaving definitely felt to me as though, unfortunately, that stereotyping was causing a bit of a barrier to them really sort of, you know, introducing or welcoming her in,” Harry, 38, said.

“No, he never tried to dissuade me from marrying Megan, but he aired some concerns, very early on and said, you know, this is gonna be really hard for you,” Harry recalled. “And I still to this day, I don’t truly understand which part of what he was talking about. But maybe, you know, maybe he predicted what the British press’s reaction was gonna be.”

Kate Middleton, Prince William, Meghan Markle and Prince Harry at Christmas 2017.Chris Jackson/Getty Images

Princess Beatrice, Princess Eugenie, Princess Anne, Princess Royal, Prince Andrew, Duke of York, Prince William, Duke of Cambridge, Prince Philip, Duke of Edinburgh, Catherine, Duchess of Cambridge, Meghan Markle and Prince Harry attend Christmas Day Church service at Church of St Mary Magdalene on December 25, 2017 in King’s Lynn

For more on Prince Harry’s memoir, listen below to our daily podcast PEOPLE Every Day.

“The older-younger sort of sibling rivalry as such — now is only really becoming, uh, I guess real to me. Like, so, okay. For instance, I talk about the relationship between William, myself at Eton. Yeah. And the fact that he didn’t really want to know me,” Harry said. “And you know, as a younger brother, that sucks. It’s like, come on. Like you left me at Ludgrove, and now I’m here at Eton. Like, hey, let’s, now we’re the same school, let’s go. Um, and he didn’t want anything to do with me. And that hurt at the time.”

Prince Charles, Prince William, Kate Middleton, Meghan Markle and Prince Harry.Samir Hussein/WireImage

Prince Charles, Prince of Wales, Prince William, Duke of Cambridge and Prince Harry

Prince Harrysaid he is now observing the sibling relationship between his own children, 3-year-old son Archie Harrison and 1-year-old Lilibet Diana.

“But now, well, the gap between me and William was very similar to Archie and Lili,” he said. “And to see Lili obsessed with Archie and Archie like, no, no, Lily, I need my space. I need my space now. I get it. I get the, I get how irritating the younger sibling can be to the older sibling. But in the moment of the time, I didn’t, I didn’t really grasp that. I didn’t really realize it. But yes, I’ve always loved my brother. And I think what would be quite shocking or surprising to people is that after our mother died, we were on different paths. Right? Two individuals who experienced a very similar traumatic experience, but dealt with it in two very different ways.”

Prince William and Prince Harry.Daniel Leavl-Olivas-Pool Getty Images

Prince William and Prince Harry

Can’t get enough ofPEOPLE’s Royals coverage?Sign up for our free Royals newsletterto get the latest updates onKate Middleton,Meghan Markleand more!

According to the report, Harry said the incident began when William, 40, called Meghan “difficult,” “rude” and “abrasive.” Harry replied that his brother was simply “parrot[ing of] the press narrative” about his wife.

The book jacket of Prince Harry’s memoir ‘Spare’.PENGUIN RANDOM HOUSE

Prince Harry book

“He set down (his glass of) water, called me another name, then came at me. It all happened so fast. So very fast,” Harry wrote of the altercation at his London home, according to the report.

source: people.com