How to plot billions of points efficiently?
42 vues (au cours des 30 derniers jours)
Afficher commentaires plus anciens
I have 3 billions of 2D points to be plotted, which requires a lot of memory. Thus, I can only do this on a server, which has about 1TB memory. But the server does not have a decent graphics card, thus export of figures uses CPU to render and takes more than 5 hours. But this procedure needs to be done many times, because I need to change the scale of axes according to the shape of the scatter diagrams. My desktop has a decent graphics card. Could I utilize the desktop's ability when data can not fit into its memory?
Here is an example of my figures.
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/191660/image.png)
6 commentaires
Stephen23
le 4 Juil 2018
Modifié(e) : Stephen23
le 4 Juil 2018
" In linear scale, it can be easily done by using round(). "
Yes, I also thought of using round, or some kind of tolerance.
"In linear scale, it can be easily done by using round(). But in log scale, I have no clean way to do this. Do you have any idea?"
Convert to linear scale, round to whatever precision, get the unique X-Y pairs, use the indices to plot a subset of the data. I think with a few billion points this might be possible with the memory that you have available, but you would have to try.
Réponse acceptée
Stephen23
le 4 Juil 2018
Modifié(e) : Stephen23
le 4 Juil 2018
Here is one way to subsample the data to produce almost identical plots:
% Fake data:
X = 10.^randn(2e4,1);
Y = 10.^randn(2e4,1);
figure()
scatter(X,Y,'filled')
set(gca,'xscale','log','yscale','log','title','AllData')
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/189743/image.png)
% Merge data points:
Xb = log10(X);
Yb = log10(Y);
Xf = 0.05; % adjust factor to suit
Yf = 0.05; % adjust factor to suit
Xb = Xf*round(Xb/Xf);
Yb = Yf*round(Yb/Yf);
[~,idx] = unique([Xb,Yb],'rows');
figure()
scatter(X(idx),Y(idx),'filled')
set(gca,'xscale','log','yscale','log','title','SubData')
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/189744/image.png)
The number of points plotted:
>> numel(X) % AllData
ans = 20000
>> nnz(idx) % SubData
ans = 6653
You can also see that all extrema are still clearly visible.
PS: you might be able to save some memory by putting the merging onto one line:
[~,idx] = unique([Xf*round(log10(X)/Xf),Yf*round(log10(Y)/Yf)],'rows');
0 commentaires
Plus de réponses (2)
Steven Lord
le 5 Juil 2018
Consider storing your data as a tall array and using the tall visualization capabilities introduced in release R2017b.
Voir également
Catégories
En savoir plus sur Line Plots dans Help Center et File Exchange
Produits
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!