You don’t want the portal to Valdrakken, as that isn’t the Broken Isles, it’s the Dragon Isles. 
As Vralok mentioned you want the portal to Aszuna, which is on the opposite side from the Valdrakken portal and one down. It seems that one can’t be interacted with, which seems odd, and may be related to the other issue we’re having with the Valdrakken portal for some folks.
I’ve already forwarded a report to have it looked into.