Mary-Kate and Ashley Olsen turned 21 on Wednesday, and unlike many young stars hitting the milestone, they kept their celebration low-key.

The sisters hosted a cozy dinner party at Hollywood's legendary Chateau Marmont, where guests included their sister Lizzie, 19, and brother Trent, 23.

"It looked like a family dinner," an onlooker tells PEOPLE, "You could tell they were all really close. The girls couldn't stop smiling."

A little after 11 p.m., two red velvet cakes - one covered in white frosting and the other in black - were brought out as the table burst into "Happy Birthday." Their friends cheered as the girls closed their eyes and blew out the candles.