I'm 164-166 and 5'10 and my caloric intake for the day is about 2500 with light exercise 1-3 days a week for about a hour to maintain the size I am now. If I want to lose weight its a measly 1950

So Op if you have a desk job or a job where you barely walk or do physical labor your probably at around 2100 to maintain the size your at now. Which I have to assume your eating wayyyyy over like most people.
1st thing you need to determine is if you want to bulk up or cut down. They require to different approaches, and bulking up is usually easier because you can still eat what you want to a certain extent while trying to cultivate muscle growth. Cutting is like going on a diet, but its pretty much a change of life, don't look at it as a diet but what you need to do to maintain a certain look that you want. If you look at cutting like a diet then it wont last long.
Once you figure out if you want to get bigger or tone up then you have to make it a point to hit the gym, I still waste time in the gym because I freestyle it which holds my progress back severally. Find or make good weekly plan of target areas for you weekly and stick to it! 3 days a week minimum is what you want to spend at the gym. Anything less than that and its going to be hard to see benefits unless your going crazy hard.
If you drink cokes and super sweet drink, my 1st suggestion is to cut those out of your diet. You dont have to ever drink them, but they should be a treat. Something here and there not often, once I realized that 250 calories out of 1900 was just gone for a drink I couldn't rationalize it in my head. And I love drinking cokes and things... but 250 cal plus for a drink....nah. I'd rather put that toward some junk food if im just waste calories like that.
Anyway, take it slow man. Do your research and dont try to do it all at once, it can be over whelming.