Don’t worry. You have still this quest. Only some quest items drop sometimes off if you leave the area