Former coffee drinker here.
Like you I didn't drink much coffee until graduating college and starting work. (It was Free, why not?)
For about two years I used to drink two cups a day until I noticed it affecting my ability to fall asleep. Additionally, on the weekends - when I didn't drink coffee - I'd get withdrawal headaches.
So, I stopped drinking it, slowly at first by mixing with decaf. I'm now able to fall asleep normally and have the loveliest of dreams.
Another thing that bothered me about drinking coffee...By using it, I felt like I was essentially taking a drug to get me through the day. This felt...wrong. And it led me to reorganize aspects of my day to diminish my "need" of caffeine.