don't get discouraged and do something you'll regret
CompSci and EE complement one another well. Assembly, c/++, python, and java are some programs I saw being used along with the more higher level languages like matlab.
Also, because when you're an EE you have to take plenty of physics courses for the degree, he may be 1 or 2 physics courses away from a B.A. in physics depending on the school.
Mess around and end up with 3 degrees which is a good thing.
The math can be handled. Just do the problems at the back of your book until you are sick of them. I remember taking calc.3 and drawing so many cylinders and quadric surfaces that I was not only tired of seeing the damn shapes but could see an equation and immediately figure out the shape.
Don't be afraid of the math and surely don't listen to the kids talking about how hard something is because chances are, you probably will find it simple.