Castles can be done with Yasuke, but you really gotta study the layout carefully.
First thing is to find every bell tower and shoot them shyts down. Yasuke can handle a castle but wave after wave of Guardians can fukk all that up in minutes.
Biggest challenge is lack of eagle vision, so he can't identify enemy locations as easily. You gotta sneak to each mini-section of the castle....figure out how many enemies are around and then take out them out quickly. Sometimes you can stealth if there's heavy foliage but mostly it's just getting them all to fight you at once and mowing them down as fast as possible. Work outside-inside, so take out the whole perimeter and start with archers or people on elevated platforms.
Daisho/daimyo can be annoying as fukk because some are often isolated with only one or two soldiers nearby but others will have an entire garden full of nikkas just watching them. Monks, samurai, brutes, teppo wielders etc. Worse is they'll be in a tightly confined space, so you gotta fight 10, 12 dudes but in cramped quarters and Yasuke can't elevate to get away from all that......but he's GOAT at taking down bosses and high level targets.