But why the hell does it go down when I’ve done jack all except play the game?
Endorsement rating will decay after a number of games played. If you are not consistent enough with earning more endorsements it will drop to a lower level.
Me, sitting on Endorsement-level 5 for years: What? 

Are you still earning endorsements as you play?
I just wish the endorsement system actually served its intended purpose.
Anyone ever explained why there’s a minor built in punishment for finding people who want to stay together as a team cause you’re having fun?
Like yeah we’re groovin and all… but now I’m racking up games with no endorsements except the occasionally good sportsmanship from the other team.
You’d think stacking would at the very least pause your decay because your endorsement possibilities reduce significantly if you play several games with the same people.
I swear the system broke or got more strict. I keep falling to endorsement level 3 and feeling like crap. What am I doing wrong? Maybe I don’t get 3 or more endorsements every match, due in part to an endorsements being limited to 3 to use per match. It’s gotten to the point where I’m not even happy to see it rise to level 4 anymore. Because I know in just a couple matches it’s going to drop back down to 3 regardless of endorsements. It’s like salty people are reporting me in Quick Play games and it’s damaging my endorsement level or something. Now I had level 5 for a very long time due to how Lucio Ball matches work. But even after Lucio Ball left I maintained a level 5 rating for an extremely long time afterward. Now I seem to be bobbing up and down endorsement levels 3 and 4. I’m sorry but if this is intentional I think it’s really crappy. I’m playing with the team; I’m having fun. Heck I even make callouts to the few in voice chat in Quick Play. It’s just a slap in the face to see my endorsement level fall again and again and again. And it was happening repeatedly while I was playing the game regularly and frequently. Particularly during the Winter Wonderland event when I was super active.
This whole “not consistent enough with earning more endorsements” thing is either stupid, or it’s too picky. I don’t know about you guys, but I hand my 3 out of 5 teammate endorsements out at borderline random sometimes. Sometimes I wish I could endorse the whole team. It can be hard to decide who I think deserves them. The system as it is now doesn’t seem to reflect how much of a polite chap someone really is. I am positive that unless they change endorsements for Lucio Ball or how they’re recorded that I will spike up to level 5 again. Only to lose it months later and then start teetering between 4 and 3 again. I love Overwatch but when my endorsement level randomly drops to 3 after a match, even with endorsements; it doesn’t feel like the game loves me back. It doesn’t feel like not enough players appreciating me; it feels like a system demoting me for not meeting a quota.