Yeah, he faded after "And Then There Was X", RR didn't change or adapt the formula at all, and X's rawness had a familiar quality by then, that was overlooked for Jay, Jadakiss, Fab, and Nas. He was still a heavy weight in sales, but his music quality was down, and he just wasn't progressing as an artist, I would believe his drug and alcohol addiction really wore on him around these years, after that 97-2000 era. By 2004 he was being arrested frequently for more and more bizarre and drug addled behavior.
Regarding "Party Up", that song was major when it came out, among my generation, 8th grade at the time, and it wasn't "white friendly" at the time, so much, that came later. That song was at the 8th grade dance, two girls trying to dance on me lol but it was considered a dope song, "And Then There Was X" def. was a shift to more commercial singles, "What These bytches Want" and "Party Up", the latter became something of a joke, but the former holds up.
By 2003, X was finished musically, and professionally. The shift in the game, his inability to progress, his drug addiction, and enablers, are all factors.