It means “removal of a 10th”, where 1 in every 10 soldiers from a unit was executed. I don’t think it was every used against “populations” outside of the Military for mass punishment for units [who fled battle].
To that end, any general reduction in “power” or “force” is now referred to as decimation.
Actually it literally means “tenth” (deci) + “kill” (mate). The “mat” root shows up for example in the phrase “check mate” and in the Spanish word matar which means “to kill”.