I know I can assign this to a assigned number and press and hold but why cant I use the left mouse key and hold it down on the number to press the assigned number?
Just not meant to, at least for now. If you have side buttons on your mouse, you might be able to reorganize them to do what you want though.