Working with Matlab for a 15! x 30 matrix

Is there any way to have matlab run a matrix that is 15! x 15, a matrix comprised of all permutations of the integers 1:15.

I get an error saying that

"Requested 479001600x12 (42.8GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and

cause MATLAB to become unresponsive. See array size limit or preference panel for more information."

which makes sense but I am curious if this is a possible task and how I would go about running computations with this much information. I tried taking of the limit to the maximum array size, but as you might imagine my computer still could not process command.

Thank you for any tips or advice you might have.

[Image Analyst]

Why do you need the whole 1,307,674,368,000-by-15 matrix in memory at the same time? Can't you process just chunks of it? Anyway, how many minutes, hours, or days do you think this 20 trillion element matrix will take to process?

[Spencer Giglio]

The problem that I see with chuncking it is to generate all the permutations of the numbers 1:15 I am using the function

perms(n)

for n = 15 and without using this Matlab function I am not sure how else I would ensure I covered all 1.3*10^12 permutations. I understand it is an unreasonably large data set but I am wondering if it is even possible.

Answer 1

[Cris LaPierre]

That many rows qualifies this as a Tall Array. Perhaps consider looking into how to analyze Big Data in MATLAB.

Answer 2

[Steven Lord]

To speak to some of the questions Image Analyst asked, how large a contiguous block of memory would storing this matrix require?

>> numRows = factorial(15);
>> numCols = 15;
>> numElements = numRows*numCols;
>> bytes = 8*numElements;
>> terabytes = bytes/(1024^4)
terabytes =
          142.718749120831

Let's say for sake of argument you were able to store this. How long would it take to operate on these combinations, assuming you could process 1000 combinations per second?

>> processingTime = years(seconds(numRows)/1000)
processingTime =

          41.4385510996119

So to be finished now, you would have to have started in the 1970s.

>> datetime('today')-years(processingTime)
ans = 
  datetime
   19-Dec-1978 21:07:12

If you started now:

>> datetime('today')+years(processingTime)
ans = 
  datetime
   04-Nov-2061 02:52:48

Let's hope you weren't running these calculations for something related to Halley's Comet -- it reached perihelion in July.

[Spencer Giglio]

Haha well shoot, looks like I am going to need a different strategy.

[Steven Lord]

If you describe how you were hoping to use this very large array we might be able to suggest some alternate approaches.