application called MPI_Abort(MPI_COMM_WORLD, 821) - process 570

[J-shel]

Hello,
My wind input dimensions are (81, 1440, 721), which is time, longitude and latitude.
ww3_shel run successfully using 256 and 512 MPI processes, while it called MPI_Abort when using 1024 MPI processes.

Error message in the log file

shel start
Abort(821) on node 570 (rank 570 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 821) - process 570
shel end

Error message in ww3_shel.out

...
Type 5 : Nesting data
-----------------------------------------
From : 2022/08/04 12:00:00 UTC
To : 2022/08/14 12:00:00 UTC
Interval : 01:00:00

   Wave model ...

w3servmd MPI_ABORT, IEXIT= 821
w3servmd UNIT missing
w3servmd MSG missing
w3servmd FILE missing
w3servmd LINE missing
w3servmd COMM missing

I wonder if WW3 has a upper limit of the processes used and how to decide it?
Thank you!

[aliabdolali]

The upper limit with the carddeck decomposition is NSPEC= number of frequency bins * number of directions

[J-shel]

Hi, I didn't find any parameters named like "number of frequency bins" and "number of directions". Is it related to the workload, i.e. ,the shape of the input data? Could you explain it a little bit?

[aliabdolali]

https://github.com/NOAA-EMC/WW3/blob/develop/model/inp/ww3_grid.inp#L15
25 and 24 in this example are the number of freq and direction, therefore user cannot use more than 25*24 cores. Check your setting and you can calculate the upper limit.

[J-shel]

Cool! Thank you very much! O(∩_∩)O