What is the optimum number of ground vias to be used in coaxial via geometry?
Optimum number, like with most other things, depends.
It will depend on via size. It will depend on frequency. It will depend on available room. Could even depend on whether it’s laser drilled or mechanical drilled, how many layers, thickness. As with anything else there’s no “ONE” magic number.
The important thing is to try to understand the effects, the trade-offs. Even the fact it’s an option. Then use the understanding you have to answer for YOUR board, using YOUR materials, on YOUR project.
As always, depends. For vias in thin material of the same size, 6 is ideal. A large hole in thin material could have as many as 200 which is really more than you need but the same size hole in thicker material the most you could have is 106.
So to get a true answer you have to specify material, material thickness, hole size, via size and type of via at a minimum.