WhilePrince Williamis the future king,Prince Harrywas expected to play an important — even if supporting — role in the future of the monarchy.

“That was a lovely picture,” she continued. “Harry was the secret weapon, really.”

However, a schism between the brothers deepened over the past years. Multiple sources previously told PEOPLE that tensions rose back when Harry first told his family he wanted to marryMeghan Markleafter less than a year of dating. An insider says William, who dated college sweetheartKate Middletonfor about eight years before tying the knot,cautioned Harry that things were moving too quickly, leaving the younger sibling angry and hurt.

Prince William and Prince Harry.Shutterstock

Prince William and Prince Harry

PAUL GROVER/AFP/Getty Images

royals-n

The brothers grew up close, especially leaning on each other in the wake of losing their mother Princess Diana at such an early age. However,Prince Harrydealt with being known as “the spare” to his older sibling from his birth.

“It’s not terribly good for your self-confidence,” says Junor. “He was always compared to William: William was the good-looking one, the important one, the clever one. Harry was the also-ran. I think that did take a toll on him.”

Prince Harry, Duke of Sussex and Meghan, Duchess of Sussex

The dream of William and Harry working together despite having their own families and lives proved to be more complex in reality.

“When Meghan came around and she was interested in making changes, he welcomed it,” says a friend.

And when Archie was born, the couple’s focus shifted even more to “doing what’s right for their family,” adds another friend.

Prince Harry Meghan Markle People Magazine Cover

On Monday, the brotherstraveled to Sandringham for a family summitwith grandmotherQueen Elizabethand fatherPrince Charlesto discuss Harry and Meghan’s decision to step down as senior royals.

After a 90-minute discussion, the Queen released the following statement: “Today my family had very constructive discussions on the future of my grandson and his family,” the Queen said in a statement. “My family and I are entirely supportive of Harry and Meghan’s desire to create a new life as a young family. Although we would have preferred them to remain full-time working Members of the Royal Family, we respect and understand their wish to live a more independent life as a family while remaining a valued part of my family.

“Harry and Meghan have made clear that they do not want to be reliant on public funds in their new lives. It has therefore been agreed that there will be a period of transition in which the Sussexes will spend time in Canada and the UK. These are complex matters for my family to resolve, and there is some more work to be done, but I have asked for final decisions to be reached in the coming days.”

Can’t get enough ofPEOPLE‘s Royals coverage?Sign up for our free Royals newsletterto get the latest updates onKate Middleton,Meghan Markleand more!

source: people.com