Hello Venket
I cannot see all the axes in your snap.
Assuming that Vessel Lateral Direction is Y Axes,
You can leave Y & Z Stiffness blank as CAESAR II will take the default value for them and you are correct about RX = Torsional Rotational Stiffness.
Now,
In plane and out plane rotational stiffness should be
RY = 4212978432. mm. N /deg [Inplane rotational Stiffness]
RZ = 1448742400. mm. N /deg [Outplane rotational Stiffness]
I think the attached image can help you in this regard if i am wrong.
Have a good day !!