In some countries it's literally part of how things go, and they know you're a tourist so they probably already giving you a higher price. That being said don't be an a$$hole, if you're a Westerner visiting a non-Western country it's like 80% chances you're more priviliged than almost everyone in that country (and CLEARLY than street vendors) so just give the extra dollar/euro/pound. You'll survive.
I'm more pissed off at him recording the whole thing. It already pisses me off when it's between Westerners but this gives me big colonial vibes. Locals aren't there for your little social experience.