Ah there's your problem. They're not just white. They're also older. How old are you OP?
If they were younger, I don't see why it would be difficult to hang with them
I've gone on brunches and ski trips with my white coworkers. But most are around my age so it's a lot easier.
Best bet is it to keep it generic and put out some feelers to see their interests.
Stay away from religion and politics.
Keep it simple "gas prices, taxes, mortgages, places to travel, places to get good food, things to do on a weekend, new technology, how much everything is going up, furniture, hitting the gym, diet" eventually u get a feel of what the person likes