Hi Jurrien,
your code looks more or less good. Although note Matlab operates in radians, so it can be clearer to work in angular frequency (just throw in 2*pi's).
clear all, close all, clc
Fs=10; %Sampling frequency
t=0:1/Fs:1; %timebase
F=5 %central frequency
x=cos(2*pi*F*t); %signal 1
%x=cos(2*pi*F*t+pi/2); %signal 2
% x=sin(2*pi*F*t).*cos(2*pi*F*t); %signal 3
% performing the HILBERT TRANSFORM
hx = hilbert(x);
% calculating the INSTANTANEOUS AMPLITUDE (ENVELOPE)
inst_amp = abs(hx);
% calculating the INSTANTANEOUS FREQUENCY
inst_freq = diff(unwrap(angle(hx)))/((1/Fs)*2*pi);
subplot(3,1,1), plot(t,x), title('Modulated signal')
subplot(3,1,2), plot(t,inst_amp), title('Instantaneous amplitude')
subplot(3,1,3), plot(t(1:(length(t)-1)), inst_freq),title('Instantaneous frequency')
If you're ever unsure of your results go back to basics. Here, for signal 1 I put in a single 5Hz cosine wave, this has a period of 0.2s. By the Nquist sampling criteria one needs a minimum sampling rate of T/2 (2*Fs) to capture the waveform.
You see that it generates a triangle wave as the sampling points occur only at the crest and the tough of the wave (signal 1). If you shift the wave by 90deg (signal 2) the samples are generated at the zero-crossing points of the wave. This gives you essentially a flat DC signal.
Play with the sampling frequency, and the length of the timebase and see how that affects signals 1 & 2.
Finally, read off the frequency of the sin(x)cos(x) from the plot and compare it to the computed result.
Spoiler: higher sampling frequencies and longer samples give better results.
