How to fit fisheye image distortion coefficients using deep learning?

Knowing the one-to-one correspondence between the coordinates of the many distorted and non-distorted pixel points of a fisheye image, how should I fit the 4 distortion coefficients of the fisheye parameters(MappingCoefficients) by deep learning? My program works fine but does not converge, I don't know what's wrong, if you find a problem or a better suggestion, thank you very much!

In the process, I have tried hard to optimize the global/optimization toolbox of lsqonlin, fminsearch, paticleswarm built-in functions and other algorithms cannot solve a valid solution.

The programs and images are all attached. The entry script is test3.m

matlab version: R2022a

test3

References:

[1] undistortFisheyePoints function How does it work?

[2] How can the parameters of the calibrated fisheye camera model in matlab and opencv be converted to each other?

[cui,xingxing]

After trying, matlab distortion model coefficients (Scaramuzza fisheye model) is too complex, recommend matlab to distort the function directly generate C code to use, rather than their direct coefficient conversion,for example,This "undistortFisheyeImage" function support C/C++ code generation.

[Bjorn Gustavsson]

I am terribly sorry for having to phrase this so bluntly but I don't know how to get the message across in a different way. Looking at fish-eye optics and images taken with such lenses from a "distortion-undistortion" perspective is to constrain the thinking in a way that is simultaneously oversimplified a overcomplicated. The wall in you image might not be flat and vertical, as far as I know it might very well be built by Gaudi and bend this way or that way. The supposedly flat checker-board might be a funnily patterened beach-ball. The point is that this task of camera calibration should be explicitly done to find the pair of functions mapping pixels (u,v) to spherical angles (azimuth and polar angles relative to the camera and its optical axis):

and its functional inverse:

and the camera position and rotation parameters.

If one cuts the problem there the camera-calibration task becomes far cleaner, and the "undistortion"-operations also becomes far simpler in terms of projection-mapping operations of/on the images.