Here is the doc for the NumLevels parameter:
Number of gray levels, specified as an integer. For example, if NumLevels is 8, graycomatrix scales the values in I so they are integers between 1 and 8. The number of gray-levels determines the size of the gray-level co-occurrence matrix (glcm).
And here is the code fragment inside graycomatrix.m that performs the scaling:
slope = NL / (GL(2) - GL(1));
intercept = 1 - (slope*(GL(1)));
SI = floor(imlincomb(slope,I,intercept,'double'));