Don't sht where you eat. It's foolish to jeopardize your income, the literal source of the food in your stomach, over some random woman.
If drama, conflict, or me2 erupt; the woman is normally fine but the dude is sent to the unemployment line. The offense would have to be horribly egregious for the woman to be culpable, irregardless of who messes it up.
The only wise way to do this is if you work at a large employer. In that instance, a woman from a different, unrelated department is safer. Think of a pharmacist in a large hospital ignoring his fellow pharmacists and lab techs for a pediatric icu nurse, or a dude from accounting ignoring his entire floor to go holler at women in sales dept. Still risky, but less so.