If I understood you right this could do the trick and is about 2.5x faster on my machine.
[row,col] = ind2sub([size2, size2],1:size2*size2);
blockRow = ceil(row/gran);
blockCol = ceil(col/gran);
idx = sub2ind([size2/gran, size2/gran], blockRow, blockCol);
A = ones(size2, size2);
A(1:size2*size2) = exp(+1i*X(idx));
Important to not is that size2 needs to be a multiple of gran to for this solution to work.