The Amani were pushed out of their territory as Humans expanded out of Tirisfal and Dwarves expanded out of Dun Morogh. They got pushed up into the Northern Eastern Kingdoms, and then the High Elves showed up. But many Forest Troll tribes continue to live throughout the Eastern Kingdoms.
The Gurubashi empire went on for a while too. Even after the civil war, where they overcame/expelled followers of Hakkar, they continued up until they got their hands on the Tide Stone, which ticked off Neptulon, who made it his goal to end the Empire. And he did.
Zul’Farrak dates back to pre-Sundering times, and while the people have adapted to their former jungle enviroment becoming a desert, they never became Empire builders. Then Goblins, Ogres, Pirates, Dragons, and others moved in, and Zul’Farrak became a common desitnation for mercenaries looking for treasure.
Drakkari were doing okay too, until the Lich King showed up. Things only went downhill from there.
I also think there’s a mischaricterization here about what consitutes “civilization falling apart”. There’s still plenty of Troll civilizations, with religion, art, language, social structure, etc. They just don’t operate under an Imperial governmental structure or all build giant stone temple cities like the Zandalari did/do. They also don’t live in “thrown-together villages of straw huts”. The overwhelming majority of Troll residential buildings -even on Zandalar- are made of wood with thatched/hide roofs, and are in large part influenced by a Pacific Islander aesthetic. This was/is even the case with the Zandalari.
It might also have something to do with the fact that the big stone cities are first and foremost be religious sites that also seem to double as places for people to gather and trade. Like Drazar’alor, with its prominent port, shrines, throne rooms, market places, etc, doesn’t seem to feature any actual residential areas. So even when a city with significant religious/cultural meaning becomes ruins, it doesn’t necessarily lose it’s sacredness and cultural signifigance and the stone sructures were never buildings in which people really “lived” to begin with.
But the real reason why is because when conceptualizing trolls, the folks at Blizzard drew quite a bit from turn the century adventure stories about European explorers finding “lost civilizations” in Africa/Asia/South America. And a lot of the way in which WoW trolls have been portrayed is in service to them playing the same role as indigenous peoples in these tales… So yeah.