OP and several other people already said. They clog up the ground and make it hard to pick up items. I was gem capped and can confirm not only is that annoying, it can also make it annoying to even move and attack.
Frankly instead of raising the cap you should be able to pick them up anyway with no effect, or have them converted to a pittance in gold.