For non-participation players may report you if they feel you may be /afk or not participating. When enough reports are received in a match you will usually receive a temporary debuff that encourages you to get into combat. If you do not, you are usually removed from the match.
Those reports may also trigger an investigation, but none of that is automated. A member or members of a specific team are tasked for these kinds of things and will look into the reports and your in-game behavior to determine if there is evidence of participation in the Battleground or not.
Some may report if you are not participating in the manner they feel you should, but that doesn’t necessarily do anything. As I said, the application of those penalties are not automated. They are based on your actions, and the data your actions provide.
With that said, it is important to keep in mind that a Battleground is a team event. You aren’t there solo, so to a degree, it is often necessary to work with your team. If you are going off by yourself to defend a lone tower, without the support of your team, and you never really encounter anyone else to defend against, well, that really isn’t participating.