The talk show host - who memorably declared "I'm F---king Ben Affleck" in a music video Sunday night - cannot stop singing the actor's praises.

"Ben could not have been more sporting," he told PEOPLE of his video costar. "He told us, 'Listen, if we're going to do this, we've got to do it right. I don't want you guys to worry about my image.' "

In the video Affleck and Kimmel declare their love while backed by an A-list gospel choir including Cameron Diaz, Robin Williams, Ashlee Simpson and Don Cheadle.

There are also non-singing cameos by Brad Pitt and Harrison Ford, who blows the new couple a kiss before driving off in his convertible - revealing a "Honk if you're f---ing Ben Affleck" bumper sticker.

"That was his real car," Kimmel says of the legendary Indiana Jones star. "He's still got that [sticker] on there!"

So what did Silverman think of the new song? "I suspected Jimmy was on the down low for a long time now," she deadpanned to the New York Daily News. "I wish I could say I was surprised by this."