kmeans for dataset that's too big for memory

Hi,

I'd like to do kmeans for a dataset that is too big to be loaded in memory. What options do I have? I can load a part of the dataset to workspace by using MATFILE but general kmeans function requires loading entire dataset, which is not possible for my computer. Please help.

Answer 1

[Jan]

The obvious strategy is to install more RAM.

If you start with the claim, that the function you want to use requires the complete data set to be loaded at once, there is no way to escape from the need to load the complete data set at once. If this is not possible for your computer, upgrade it or run the code on a more powerful computer.

Sorry, I know that this answer is trivial. But it is the only possible answer matching your question exactly. Of course you could apply a data compression in the RAM or condense the data at first. But this is not exactly what you are asking for, and in addition, it would be much more expensive most likely.

[Jeff]

Thanks Jan. On Mathworks webpage there's a section titled 'Big data with MATLAB' where it says the kmeans function available in statistic toolbox offers some solution for big data but I don't see any detail in the matlab help window. I hope someone might have experienced this before.

[Jan]

And does "Big data" mean that it is more than matches into the RAM? How big are your data?

[Jeff]

Yes, big data here refers to dataset that's too large for the memory. The dataset I'm using is about 90GB in size.