Very Low FPS in 6v6 game mode (especially when Backfilled)

I have really low fps (20-40) in 6v6, but i see it sometimes the FPS go back to normal.
But when backfilled in qp, i see that the low fps state , stays for much longer in match (actually the whole match).
I have AMD gpu