With M = 8, you have log2(M) == 3 bits per symbol. Later you generate 1000 bits, but this is not an integer number of symbols at 3 bits/symbol (it is an integer at 4 bits per symbol when M = 16, so you don't get that error in that case).
It seems like you want 1000 symbols (i.e., numSymPerFrame = 1000), not 1000 bits, so you have to generate numSymPerFrame*k bits (i.e., 3000 bits when M == 8, or 4000 bits when M == 16).
Also, you need to specify OutputType='bit' in pskdemod in order to use biterr to compare bit sequences.
% Modulation order M
M = 8;
k = log2(M); % Bits per symbol
EbNoVec = (5:15); % Eb/No values (dB)
numSymPerFrame = 1000; % Number of PSK symbols per frame
berEst = zeros(size(EbNoVec));
for n = 1:length(EbNoVec)
% Convert Eb/No to SNR
snrdB = EbNoVec(n) + 10*log10(k);
% Reset the error and bit counters
numErrs = 0;
numBits = 0;
while numErrs < 200 && numBits < 1e7
% Generate binary data and convert to symbols
data_in = randi([0 1],numSymPerFrame*k,1);
%----------------------------------------------------------------
% Your modulator here:
modSig = pskmod(data_in,M,InputType='bit'); % PSK Modulation
% Pass through AWGN channel:
rxSig = awgn(modSig, snrdB); % Additive White Gaussian Noise with
% increasing snrdB
% Your demodulator here:
data_out = pskdemod(rxSig,M,OutputType='bit'); % PSK Demodulation
%----------------------------------------------------------------
% Calculate the number of bit errors
nErrors = biterr(data_in,data_out);
numErrs = numErrs + nErrors;
numBits = numBits + numSymPerFrame*k;
end
% Estimate the BER
berEst(n) = numErrs/numBits;
fprintf("snrdB: %.6f berEst: %.6f " + ...
" \n \t numErrs: %d numBits: %d\n\n", snrdB, berEst(n), numErrs, numBits);
end
