I think the French's colonialism in North Africa can be directly traced to the Islamic Caliphate conquering and colonizing what is now Portugal and Spain and encroaching on the Kingdom's borders. Therefore what the French did in those colonies were justified.
Can you imagine if Hindus in India just funded attacks in Mecca during the Hajj because Islam came in and colonized a civilization that existed for thousands of years before the religion conquered the subcontinent, the effects of which can still be seen today? People would rightfully laugh at such logic.
This logic also falls apart when you consider the Philippines and a group like Abu Sayyaf. Who did the Philippines colonize exactly?