R*2 is not remotely meaningful here, unless you find a way to intelligently extend it to this situation.
I would probably find the closest point on the curve for each data point. What is the distance to that curve, for each point. Now you might compute the sum of squares of those distances. Call that A. Finally, compute the sum of squares of the distances for each point to the mean of all of your data. Call that B.
Finally, compute the parameter
P = 1 - A/B
Note that P might be considered to be somewhat analogous to an R^2 parameter. It will be 1 when the polygon passes exactly through the points. It will be 0 for the simple polygon that lies exactly at the mean of your data, thus just a single point.