Killed a realmwalker and that apparently completed my whispers so I went to the tree to hand in before going into the portal. In the two minutes (ish) it took to go to the tree, stash the loot from the whisper and return the portal was gone.
Is that a bug? Do the despawn immediately if you port away? Do the despawn on a timer, and if so what is it? Must be damned short. 