I think 50 blowin up so huge as he did in 03 was part of the reason. He probably doubted himself and his capability to be the torchbearer of hiphop.
This brings me to the next point, his bruised ego and confidence. The Nas battle didn't turn out to be the overwhelming victory he expected it to be. A lot of people felt he lost that battle, and regardless of his own conviction that he had won, that must have hurt him. Add also the fact that he had just dropped BP2, which was considered weak and underwhelming.
The third reason is spelled Damon Dash. I find it very possible Jay was already planning on how to dissolve the Roc and distance himself from Dame and Biggs. Perhaps this "retirement" was part of that plan, on some "Im a pretend to quit rap when really Im just gonna quit fukkin wit you". We seen the footage from Choke No Joke how Dame was already distanced from what Jay was doin and uninformed about the marketing plans for the Black Album.