It seems from this last comment that you re making some progress. There are some points I should make.
First, when you convert to a polar form, remember that a tool like cart2pol works around the origin. So if your data is offset, then you would need to translate it, so the origin is directly in the center. A mean shift is probably good much of the time, but if that translation is not perfect, then you would probably see a sinusoidal bias introduced into the expression r(theta).
Next, in the case of the relationship you show, a simple model is probably not too bad. Essentially, you would be creating a simple Fourrier series approximation. But if that is inadequate, you might be forced to use other model forms, perhaps regression spline based models.
The problem here, is that a simple fit of the form y=f(x) must fail, because this is not a single valued relationship.
After a polar transformation, we would see:
[theta,R] = cart2pol(px,py);
And this is now a single valued function, thus R = f(theta). So in theory, things can be done.
We might now wish to fit this data using what is essentially a fourier series approximation. There will be a DC bias in there, so a constant term. The model would be of the general form:
R(theta) = a0 + a1*sin(b1*theta) + a2*cos(b2*theta) + a3*sin(b3*theta) + a4*cos(b4*theta) + ...
You could use as many terms there as you can estimate. Good starting values would be important of course. Is all good now? NO! You still have a significant problem.
At least in this data, we see cusps at every peak. That corresponds to essential singularities in the derivative, but still singularities. Can a functional form on the one I showed above represent a function with singularities well? Sadly, no. No simple trig series can represent a function with singulariites in it without infinitely many terms. Those cusps become a problem. And sadly, any other tool you use, such as a regression spline will also see a problem, since most functions do not handle singularities well, even if they are only in the derivatives.