Chill dude, he didn't fire you.
Are you the only one available? We're you the only one he called? We're you the only that declined?
Even if you answered yes to all of that, i still find it hard to believe he would fire you without due process, for something so petty. No warning or write up?
I'll update. He may be pissed for the next few weeks and not schedule me