As you have learned, the custom nozzle stiffness input works only for those vessel/nozzle combinations that define a plane. Your vertical (top) nozzle on a vertical vessel does not define such a plane so in-plane and out-plane are not defined.
I suggest you revert to the older style of input - use a set of springs each using a Node/CNode pair. A set of 6 restraints will do it.
I have requested a program change where the "custom" nozzle definition will also accept global input (X, Y, Z) rather than just local input (a vessel reference vector).