Herpes enters the body through mucous membranes and establish latency in nerve ganglia, which are clusters of nerve cells outside the brain and spinal cord.
Think of nerve ganglia as electric power strips, with different parts of the body attached to different power strips. This is why people who are infected always have their outbreaks in a specific region of their body.
It is spread via physical contact with an infected area. So if a woman with cold sores can pass it to your genitals via a blowjob, even if she's not having a breakout. It can also infect the fingertip, nail bed or eyes. Herpes is the leading cause of blindness.
Herpes can lay dormant in the body for years and an initial outbreak can eventually be triggered by moments of high stress.
So it's possible to literally fukk around and not find out.
EDIT: The most common way that people develop cold sores (oral herpes) is as a baby or child, when a loved one gives them kisses and passes it to them. It's not always a sexual thing, especially since cold sores isn't really stigmatized