Predict Computing Resources Needed to train Neural Network?

I'm working on implementing and optimizing a medium sized forward feed neural network. Right now it has as input a 400-element vector and outputting a 36 element vector (regression fit to training data), although this may be scaled up to (2500,1000) respectively.

My question is, given different structures (multiple layers, nodes/layer) and different amounts of training data (100000 tests, 10000 tests?) how can I predict what resources I will need/how long it will take to run on these resources.

I am going to be running the training on a shared cluster and I need to ask for a number of cpu cores (up to 32), RAM (up to 1TB), and time. Since I can't run the training on my personal computer, obviously, I don;t know how to predict how long the problem will take...

Any help or pointers to other resources would be very appreciated.