When zoning, even inside a dungeon from one part to another, the Primordial Binding buff count will lose any contributions from Cosmic Anomalies around you.
For example, I have 5 anomalies around me (buff count is 5) and I run inside a nightmare dungeon between one section to another. When the section text appears (e.g. in Sunken Library, moving from Acquisitions to Forbidden Archive), the Primordial Bindings buff count disappears and my movement speed returns to base.
Casting my skill that Cosmic Anomaly is bound to, blowing them up will cause the Primordial Binding buffs to start accumulating again as the anomalies respawn. This is the only work-around I’ve found.
I guess the Primordial Bindings buff is only incremented on anomaly (or any conjuration) spawn. Probably just how that passive works, as Sorc doesn’t have permanent pets, so this case hasn’t cropped up yet. Same behavior when spawning a Familiar or Hydra and immediately zoning/transitioning to another intra-zone area. All stacks of the buff are lost.