Outside of his decades-long career, Ted Danson finds comfort in his family.

While speaking with PEOPLE exclusively at the NBCU Emmy Kick-Off Luncheon on Friday, April 11, the A Man on the Inside actor, 77, opened up about how he finds newness and joy outside of work.

Danson, who’s married to fellow actress Mary Steenburgen, says he’s been spending his free time in a “little house” surrounded by his family, which includes his stepson Charlie McDowell, his wife Lily Collins, 36, and their newborn daughter, Tove. Nearby their home, Danson tells PEOPLE, he also lives near his daughter, who also has a new baby.

“It’s a joy,” he says. “I mean, you’re jammed into one little house, so you take care. But yeah, no, it was brilliant.”

“I think we were all crying when they left,” he adds. “It was such a rare sweet to be able to be there.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.  

The Good Place star welcomed his two daughters Kate and Alexis Danson in 1979 and 1985, respectively, with his second wife, Casey Coates. Two years after his split from Coates in 1993, Danson married Steenburgen after they reconnected on the set of Pontiac Moon, making him a stepfather to her kids, Lilly and Charlie. The blended family has supported each other over the years.

Unlike his relationship with Steenburgen, Ted rarely discusses his children in interviews or features them on social media, though they both forged their own creative paths like their father. Kate has pursued a career onstage and on-screen as an actress, while Alexis has made music under several pseudonyms.

During an appearance on Jimmy Kimmel Live! on Thursday, Nov. 14, 2024, Danson spoke about being a grandfather. He also shared his granddaughter’s hilarious thoughts when meeting her baby brother for the first time.

“We have three granddaughters. Magnificent. Our first grandson was born and his name is Sunny. And it’s a great name,” Danson began. “Funny little story about the youngest granddaughter when she found out my daughter was pregnant with Sunny-to-be. [She] was not happy because her place as the baby — the younger person, the one who got the most attention — was at risk.”

Danson goes on to say that his granddaughter “really vocalized it” but noted that eventually, she started to suck it up.

“But then when Sunny was born, there was this, you know, the photo. Five minutes old, buck naked on his back. And it was texted to the family,” he said. “And it went to her family, the granddaughter’s family, and the youngest stood over there and wouldn’t have anything to do with it.”

“And everyone was going, ‘Oh my god, look look look.’ Finally, she rolled her eyes and came over and sat down next to him and looked at him and went, ‘That penis is massive,’ ” Danson shared, prompting Kimmel to laugh.