Seems like I only broke the first portal, I was able to continue the quest after that.
Now I can fly!