Note how he doesn't give him a proper reason as to why he doesn't want to say it. Like
"It's a word with a very harmful history, and whenever someone like me said such a word in the past, it was preceeded with something like 'lets lynch that' and I don't want to be associated with that kind of person"
or
"It is not my place to use such a word that was used to make you feel less than human. I respect you too much as a human being and as a fantastic actor to use such a vile word in your presence and it is a word I would never feel comfortable with using to describe anyone because of its loaded history"
His reason for not saying it is because he doesn't want to get into trouble. This right here is a slice of today's 'colorblind' america. White people who really dont understand racism and think it boils down to just not saying racial slurs and jokes out in public