vertical layers, problem with mpi run

[edith]

i am having a trouble with idealized analytical case that i try to run.
domain is rectangular basin, depth changing from 5-15m to offshore direction and fresh water inflow from point source in one location.

when i try to add vertical (to use 30 layers.. worked nicely with 20layers both on parallel and serial) model works when running serial but dont work on parallel (tried couple of times to be sure that i did not changed anything in model on the way).

the same number of layers works on serial but dont work on parallel (segmentation error) even when parallel worked well with smaller number of layers (20 seams to be maximum i can get).
Anybody have any idea what could cause such a thing? what is it that i am doing wrong?

[kate]

You are asking for more memory. What are the system limits for memory? You can have "ulimit -a" or "limit" depending on your shell (it's a built-in shell command).

[edith]

thank you for the reply.
i have been trying to figure out if it is memory issue.
my domain size is not too big 128x256x30 and to run on 8 cluster. how much memory should one have to perform such a run? i set as many of parameters in ulimit as high as i can. is there anything else i should be able to do (except getting new computer)? can i modify something in roms run so it would take less memory (i turned off writing avr and as few switches as possible)?

ulimit -a gives:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 16383
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

could this error have something to do with how sigma coordinates are divided? shallowest place in domain is 5m deep and maybe i have set something so it cant divide it to more the 20 layers?

[kate]

It's not that - we have 60 layers in 10 m of water. You say it works in serial mode, but not parallel. Could it be a parallel bug? Perhaps in the sources? Is your code from before or after Hernan's latest update to the sources code? If before, you could try updating.

[edith]

if it is problem in parallel i cant imagine what kind of problem. as i said exactly same run with 20layers works fine on parallel.

[kate]

We don't know, so you will have to do some detective work. Can you debug it via a debugger or via print statements? Where is the seg fault happening?

[edith]

error is on the first step of model run. debud dont help, thats how i got seg error, parallel error was even more useless .
i think it is most likely to be memory problem cos same run that runs on 8clusters refuses to run on any less number of clusters so i suppose there is nothing much i can do about it.
thanks Kate for help and nice holidays to everybody.