Last thing you need to do is avoid enjoying your home. Spend time in your yard. Go for walks. Chill in the garage. The only way to deal with passive aggressive racism is to put it on the guilty/innocent scale. You aren't breaking any laws, you aren't a criminal, you live in a new town in a place you bought/rent/whatever. You aren't guilty of anything so why feel guilty? There's a liberating feeling to knowing you...did nothing wrong. So act like it man.
When I was in college, I used to work at my dad's business on the weekends when it was closed. I didn't live far away so sometimes I'd just walk there. After I was doing work, say around 3PM, I'd leave. Now this business was on a relatively busy street, so when I'd walk out and lock the door, I'd be visible to every car driving by, anyone walking by, businesses to the left and right side of the road, etc. I'd lock the door, walk down the stairs, head towards the sidewalk and then make my way home. Yea being a black man in that situation, I knew people were looking at me because I certainly stood out as a black guy going in/out of a nice office building. I'm sure if a cop drove by and saw me they might pull into the lot and question me. But I'd casually lock up and leave every time no matter who was staring because I knew I was doing nothing wrong. Yea part of me was concerned about the eyeballs looking at me. But I wasn't stealing, I had my wallet, my last name matched the last name on the sign, etc.
One time as I locked up and started walking towards the sidewalk, I noticed a white woman talking on the phone. She was staring directly at me as I walked towards her. I could tell she must have been walking by as I came out the office and locked up. I knew she was calling the cops. But I kept walking, gave her a nod and said hi, and walked by. She never stopped staring as I passed her on the sidewalk. Cops never came - and mind you, my walk home was at least 5 miles or so. Plenty of time for cops to find my ass, if they were coming. Didn't break a sweat because I knew I didn't do anything wrong.