Phylarch won't leave my Lumber Mill

I had placed Phylarch in the work slot in my Lumber Mill, but as I was deactivating higher level followers in order to level lower ones, I removed him from the slot and deactivated him. He’s still been in my Lumber Mill for almost a week offering his quest. He shows as deactivated, and I’ve waited days but he still stays. There was even an update downloaded. I’m not sure why he’s still there or how to remove him.