I thought it was my PC at first but turned out that even my clan mates experienced the same problem with me. Those who play DI on PC generally experience a long loading time to enter the Raid/Dungeon and Vault Defense instances.
When I switched to the phone to play DI, the loading time for these instances was normal.