When the initial Manifested Timeways enemy for the affix in an M+ dungeon is killed, its spawns (or the spawns’ spawns) regularly fall through the world or to other levels if you are near walls, corners, or on a place that is over another level (such as the bridges in CoS). The Black Rook spiral stairs are another example of a bad location for these.
If they spawn inside terrain, they will usually evade bug if you walk away and then snap to you. The ones that spawn under the world or on another level, however, never snap and so their potential buffs are lost.