More specifically, the Huojin philosophy is one of more action, less contemplation. The Tushui go the other way. And the Horde team that went to the Wandering Isle seemed more aligned with the Huojin way, so the Huojin joined the Horde.
Garrosh just happened to be Warchief at the time. They have no personal attachment to him. Even less so, given what happened to good old Ji in the Siege of Orgrimmar.