Server switching lag?

I've encountered a very odd issue lately. I have characters on a lot of servers and normally when I switch server to the character listing it takes a couple of seconds to show them. But lately this is not the case. It'll take ~20 seconds just to display my characters. The second I get through the "logging in to game server" it's like it's loading or something? What in the world is causing this and why has this *never* been an issue until 8.0 dropped.

<edit> Clearing the \cache folder fixes this. Why is this? I shouldn't need to clear the cache every time I start the game since 8.0, I hope?
Btw this is *still* an issue for me and 8.0 is the blame. I've never had this problem before. It's something that slowly crawls up during the day and if I don't clear the cache I'll be spending ~20 seconds just waiting on the loading in the character display after doing a server hop.

After I clear the cache the loading is 1-2 seconds.
I quickly googled this issue today since I've also recently been having this problem, I found this post along with one from November of last year, and a handful of others. Posting to say I've had this problem for the last few weeks or so and you're right about it only getting worse the longer it goes on, which is something I didn't initially notice but then started to pay attention to. I will try clearing my cahce in the meantime, but I hope blizzard takes notice of this problem.