Honestly while it could be the siding/insulation what I'm thinking now is that the most likely culprit is the number of downstairs windows and doors that haven't been reinsulated or replaced in the 30+ years since it was built.
On my first floor alone I have like 13 windows, two separate sliding glass doors and two doors leading either directly outside or to my uninsulated garage that also has three different points of exit.
I say this because my second floor doesn't seem to have half as many issues but I also have half as many windows and zero doors on that level.
Hmmm, check the weatherstripping around the doors. Easiest way would be to use an incense stick and wave it around the frame of the doors on a windy day. If you see the smoke being blown away, replace the weatherstripping. You can also do that with the windows and sliding glass doors. There are weatherstripping kits for them. Caulk also does wonders. Another thing you wanna check are the areas around the frames of the doors and windows. Typically, in my experience, the insulation around them is either terribad or non-existent, and they aren't air-sealed. What this means is that around each portal (window, door, etc.) there is a 6-10" (or larger) hole letting heat/AC out into the neighborhood. Multiply that by the amount of windows and doors and you can see the problem, mathematically.
Is there any way you can see what your attic looks like as far as insulation?