How can i compare 2 audio files (for lipsync)

Basically i want to compare 2 audio files from a singer singing at 2 different perfomances and i wanna check if the singer lipsyncs at one of them by comparing the audio of both of them! Is this possible? i am thinking about cross correlation but i am not exactly sure where to begin with and what are the criteria to have an accurate answer about the % of similarities between the 2 files

Answer 1

[Nikhil Reddy Pottanigari]

Hi,

I Implemented the code for checking two audio files are in sync. If not using Euclidean Distance of Spectral Features as

measure of differentiator and representing them as confusion matrix.

% Create an audioDatastore that points to two input audio files
% Audio Toolbox.
function [] = CompareForSync(file1,file2)
ads = audioDatastore({file1,file2});
numFiles = numel(ads.Files);

% Create a preprocessing pipeline so that all of the audio is mono, 16
% kHz, and normalized such that the max absolute value is 1.
desiredFs = 16e3;
adsMono = transform(ads,@(x)mean(x,2));
adsMono16 = transform(adsMono,@(x,info)resample(x,desiredFs,info.SampleRate),'IncludeInfo',true);
adsMono16Normalized = transform(adsMono16,@(x)x/max(abs(x)));

% Create an audioFeatureExtractor object to extract the spectral centroid,
% rolloff, and flux for 30 ms windows and 10 ms hops.
afe = audioFeatureExtractor('SampleRate',desiredFs, ...
    'Window',hann(round(desiredFs*0.03),'periodic'), ...
    'OverlapLength',round(desiredFs*0.02), ...
    'spectralCentroid',true, ...
    'spectralRolloffPoint',true, ...
    'spectralFlux',true);

% In a loop, read a file from the datastore, extract features, and then
% average the features across the analyzed file.
featuresPerFile = zeros(numFiles,3);
for ii = 1:numFiles
    audioIn = read(adsMono16Normalized);
    features = extract(afe,audioIn);
    figure('Name',' audio ');
    plot(features);
    featuresPerFile(ii,:) = mean(movmedian(features,10),1,'omitnan');
end

% Calculate the Euclidean distance between each pair.
distanceMatrix = zeros(size(featuresPerFile,1));
for ii = 1:size(featuresPerFile,1)
    distanceMatrix(:,ii) = sqrt(sum((featuresPerFile(ii,:) - featuresPerFile).^2,2));
end

% Create a heatmap to inspect simalarity.
filenames = extractBetween(ads.Files,'samples\','-');

try
figure('Name','confusion matrix');
heatmap(filenames,filenames,distanceMatrix)
title('Euclidean Distance of Spectral Features')
disp('both are not in sync')
catch 
    close 'confusion matrix'
    disp('Both are in sync')
end

I guess you can work ahead on this to solve your usecase.