No you can't. It goes beyond just tax benefits. You can't name a non spouse as your beneficiary for life insurance, 401k, etc. You can't make appointments, plans, changes, for a non-spouse. You can't add your non-spouse to your health insurance. If something happens to your significant other, you have absolutely no say in what happens to him/her. If your woman gets pregnant and something happens during birth, you have no say in what happens to your kids or your woman. It's the same reason gay people are fighting for marriage rights. It's not just about being "married". I'm not gonna look for it now, but there was a list of over 100 rights that you don't get if you are not married.
If you plan to raise a family you have to get married, that's if you want to be able to lead and manage your family effectively. As for exit penalties, they have prenups for that, and you should be marrying someone in a similar financial bracket, so alimony would be a moot point anyway.