The quick answer to your question is 1.125" bolts have a root dia 0.963" when the threading is taken off. Use this to calculate the "core area" of the bolts and you get 8.740 in².
Without going into the "nubbin or serration" question your incomplete details raise other questions.
The 12no 1.125" bolts implies an 8" 600# RTJ flange.
Yet you give gasket diameters of 7.623" outer and 6.291" inner. From this the width is 0.666" [as you calculate] and a pitch 6.957" [the average].
When we look up in ASME B16.20 the nearest ring no is R-50 w 0.625" P 10.625". But an R-50 must be used with 8" 1500# where the bolts would be too small. I go through the same exercise with RX and BX rings both in B16.20 again but there is no match. It may be that the information you are inputting to the flange solver is doolaly [useless].