Quickly finding maximum value of a large (~8,000,000) vector
1 vue (au cours des 30 derniers jours)
Afficher commentaires plus anciens
I was trying to see if I could speed up one of my functions so I ran it through profiler. The script loads big files and writes out smaller portions of the data for me to use later. Before writing however, I check to see if the data has any abnormally large values and if it does, I skip it. After looking at the profiler output, I saw that checking to see if there are any abnormally large values is what takes the most time by far (for 70 files, about 450 seconds out of 550 seconds). I never would have thought would have been the slowest part of the code so I'm wondering if I'm missing something that I could easily do to speed it up.
So here's the abridged version of the code:
for i = 1:nFiles % loop over files
Data = load_data(file(i)); % data is a structure with metadata and the actual
% data is in Data.trace which is 8640001x10x3
for s = 1:10 % loop over stations
badStation = 0;
for c = 1:3 % loop over components
% if one component is bad, skip the station
if max(abs(Data.trace(:,s,c))) > badThreshold
badStation = 1;
break;
end
end
if ~badStation
for c = 1:3 % write loop
write_data(Data.trace(:,s,c);
end
end
end
end
I've checked that memory isn't an issue as there's still 9 GB free during run time. Anything simple I can do to improve this?
Thanks in advance.
0 commentaires
Réponse acceptée
Plus de réponses (2)
Walter Roberson
le 22 Jan 2013
You can vectorize a bit:
for a = 1 : 10
if max( reshape( abs(Data.trace(:,s,1:3)), [], 1) <= badThreshold
for c = 1 : 3
write_data(data.trace(:,s,c));
end
end
end
Jan
le 24 Jan 2013
Modifié(e) : Jan
le 24 Jan 2013
See FEX: anyExceed : This can test the values of the array on the fly, such that abs(data) is not created as a temporary variable. The testing stops early, when any element outside the wanted range is found. Therefore a positive test can be found without searching the complete array.
Voir également
Catégories
En savoir plus sur Data Type Identification dans Help Center et File Exchange
Produits
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!