Increasing Java Heap Memory will not help in that kind of situation. Increased Java Heap memory helps only in situations such as the editor not being able to handle large files, or perhaps the Variable Browser having problems with a large array. Or, in some cases, creating large variables at the Java level such as if you used the older XML parser that built java objects.
The Java Heap memory is reserved and taken away from general memory available for computation. Unless you have an overhead (such as editor) task that is eating Java memory, you should not increase the Java Heap, and might even consider reducing it.
You might want to consider enabling the setting that prevents allocation of arrays larger than a particular size. Doing so will not enable your code to run, but it might fail in a slightly more controlled manner. The setting is mostly there to prevent allocating huge arrays that require the operating system to start swapping to disk (which is very slow.)
Chances are that you will not be able to run your desired code without either reducing resolution (some how), or possibly reducing number of iterations, or by adding more RAM. Increasing Virtual Memory (swapping to disk) might perhaps allow the code to run, but if so then it would likely be rather slow.
