kmeans for dataset that's too big for memory

Hi,

I'd like to do kmeans for a dataset that is too big to be loaded in memory. What options do I have? I can load a part of the dataset to workspace by using MATFILE but general kmeans function requires loading entire dataset, which is not possible for my computer. Please help.

Answer 1

[Jan]

The obvious strategy is to install more RAM.

If you start with the claim, that the function you want to use requires the complete data set to be loaded at once, there is no way to escape from the need to load the complete data set at once. If this is not possible for your computer, upgrade it or run the code on a more powerful computer.

Sorry, I know that this answer is trivial. But it is the only possible answer matching your question exactly. Of course you could apply a data compression in the RAM or condense the data at first. But this is not exactly what you are asking for, and in addition, it would be much more expensive most likely.

[Jeff]

Thanks Jan. On Mathworks webpage there's a section titled 'Big data with MATLAB' where it says the kmeans function available in statistic toolbox offers some solution for big data but I don't see any detail in the matlab help window. I hope someone might have experienced this before.

[Jan]

And does "Big data" mean that it is more than matches into the RAM? How big are your data?

[Jeff]

Yes, big data here refers to dataset that's too large for the memory. The dataset I'm using is about 90GB in size.

[Dominique]

Although the question is outdated, I hope my answer is still helpful to others. My needs may be specific in that I have a dataset of 100000 samples, but I need to do a kmeans run of maybe ~10000 level as the number of clusters, so tens of GB of memory usage is very common for me on matlab.

So I compared kmeans runs on matlab and sklearn, and found that the former is <0.5x as fast as the latter. but the memory consumption is completely ahead on sklearn, probably only a few hundred MB. if you use sklearnex, an Intel chip based acceleration of sklearn, the memory consumption is slightly lower, but the acceleration is simply unrivaled.

As an example, I ran 10,000 samples for a 20,000 clustering, on matlab it was 60s, ~35,000MB, on sklearn it was 130s, ~470MB, and sklearnex accelerated, it was 17s, ~450MB. so I think the answer to this one is probably obvious.