I'm from Cali but I've lived in the south almost a decade now. My biggest annoyance is the culture. It's like the rest of the country is 50 years ahead of the south. A lot of people do well (college, career focused, travel, etc) but many cap out at a job paying $15 with benefits. Family and community is valued down here more than individuality and autonomy. The subtle racism gets old, the close mindedness gets old, the "city slickers" comments get old, etc.
The cost of living keeps me. But the segregation is still very prevalent but most don't know anything else. I grew up in a completely different culture so I struggled to settle into this one. It's like taking a trip back in time.