Macro conditions are parsed individually, at the time you push the button, with the first condition to be satisfied determining what is done. As such the macro you’ve written cannot possibly do anything other than cast Rank 1. Allow me to explain.
When you push the button, the first condition is [nomod] - just that one condition. So if you push the button without a modifier, it will satisfy that requirement in full and cast Rank 1. If you hold down a mod, this is no longer satisfied and it moves onto the next condition: [@mouseover, help, nodead]. If you have a living mouseover target this condition is satisfied in full and you will cast a Rank 1; importantly this condition does not look for a modifier. If one of those three is not valid it moves onto the next condition, which is the [] catch-all condition, which again will result in Rank 1 being cast no matter what the exact situation is. The rest of the macro is even never considered as a result, even if you’re holding a modifier.
So you need to rework your conditions to satisfy your requirements, making sure they’re in the correct order (and eliminating all but one catch-all condition). Consider the following:
/cast [nomod][nomod, @mouseover, help, nodead] Greater Heal(Rank 1); [mod:shift][mod:shift, @mouseover, help, nodead] Greater Heal(Rank 2); [mod:alt][mod:alt, @mouseover, help, nodead][] Greater Heal(Rank 5)
That first looks for nomod, then for nomod and a living, friendly mouseover target to cast Rank 1. If neither of those are satisfied it goes on to look for mod:shift, then mod:shift in addition to a living, friendly mouseover target for Rank 3. Failing that, it looks for mod:alt, then mod:alt with a living, friendly mouseover target, and also has a catch-all condition for Rank 5. By reversing the conditions, such as [nomod, @mouseover, help, nodead][nomod] you can prioritise either your active target or your mouseover target.
You can, of course, still use separate /cast lines instead of the semicolons separating the spells. It doesn’t change the functionality - it’s strictly a formatting preference, which I use to save a few characters on longer macros.
As per Elocin’s suggestion, however, you should really move this to the UI and Macro forum. That way you’re talking to people focussed on this sort of thing, instead of hoping someone randomly pops in 20 hours later with an answer.
*ed: Found a typo that would stop the macro working at all - an extra square bracket. Should be good now. Cheers, Tratt, for bringing me back here to double-check what I wrote!