Mace mastery causes damage to go down?

So playing a barb now at lvl 17. I have a Bloodrise and Stoutnail. Using double swing. Putting points into mace mastery causes the damage shown on the character sheet to go down instead of up. For both weapons. By one point. For both this level and the last. What’s going on?