Faction Changed, Still Have Champion's Light

Yeah. I Faction changed to Alliance after doing the Blood Elf Heritage Armor quest on my rogue. Lor’Themar is following me in Ghostlands. No way to remove this ‘buff’.

I wish he’d follow me around