When he and Gil are talking in the studio, Frasier closes his briefcase twice.
Niles says that Allistair is the first gay friend of Frasier's who thought Frasier was also gay. Given how reacted to them, it seems unlikely he'd forget the events of The Matchmaker (1994), where just that did indeed previously happen.