Can't refund Paragon Point that is located in the middle

For example, on the first paragon board. If you have the 5 will power on the left, then the 5 dex on the left 5 intel. If you have three in a row like this you are unable to refund any of them. Due to breaking the node that connects them together.

I am unsure if I am stupid and missing something or its a bug.