I'm so pleased you decided to write about this. What a superb essay.
As a descendent of those colonisers, I'm heartily ashamed of that ignominious swathe of British history.
I worked at a hospital for a while and often overheard comments about doctors, regarding how good their spoken English was. Some of those doctors had English as their first language, others their second (or other) - but each and every one of them had a skin tone which was Other Than White. Conversely, we also had doctors from Poland and other European countries, who also spoke English as a Second Or Other Language (ESOL)....yet I never once heard similar comments about them. Nor did I tend to hear complaints about accents marring comprehension aimed at the 'foreign but white' contingent. It boiled my blood to be honest. I was born and bred in this area - believe me when I say that all of those doctors spoke (and understood) English better than rather a lot of my fellow 'locals'.