I often take what fighters say with a grain of salt when they get asked these types of questions.
Their egos get in the way. I remember Mayweather saying years ago that Emanuel Burton Augustus was his toughest fight, and not Jose Luis Castillo. A lot of people (not me, but the fight was close) thought Castillo won the first fight. Where as Augustus may have bloodied Mayweather's nose, but he also got the crap kicked out of him and won maybe 1 or 2 rounds.
James Toney says Mike McCallum was his best opponent. A guy whom Toney went 2-0-1 against, and not Roy Jones, who arguably won all 12 rounds when he faced Toney.
Then you got other guys like Roberto Duran and George Foreman who are just hilarious with their excuses
"Leonard didn't beat me, the IRS did"
"I was about to knock Ali out but I got distracted by my disloyal friend in the audience who was rooting for Ali. By the time I realized he betrayed me by rooting for Ali, the time had passed and Ali recovered"