On a first date... I don't get annoyed by it, I mean, dude's just there to get to know me, he doesn't owe me anything. I'm just assuming he isn't interested... probably not gonna bring up a second date unless he expressly asks, ya know?
But if we've been going out for some time, then it's just weird. Like, I'm a touchy-feely person after I'm comfortable with someone - I initiate hugs, handholding, kissing, etc, and if the guy is motionless, I'm out.