Real talk, I see a fair amount of big dudes with some eye-catching women. Funny thing is, more times than not, when I see an unattractive chick she's usually with a really skinny dude.
I think dudes tend to assume women look at the men through the same lens that they look at women. So, they would think that women generally react to a certain look the same way they do, and it doesn't work like that....or at least not as much as with men. A dude's confidence and personality and status is like a woman's breasts, hips and butt.
At least from what I see, big dudes are not behind when it comes to getting women....even attractive, in-shape women.
Peace