Custom games gets bugged out sometimes when trying to play with friends leading to some frustrating experiences.
You should allow friends to join first before than making the game public.
Joining through friends list should work as well but it does not.
Sometimes the games just cannot be joined.