Can't say it has reached that points but I'm not exactly a neat freak.
Only I'd draw the lines is with the gross: fungus, unflush toilets. You can throw dirty underwear into the mix.
Essentially, when it reaches a point that shows a lack of respect or no pride.