What is ArrayTrans

ArrayTransformer<T,rs,rm,al> (ArrayTrans) is an operator that handles trans-rank MPI copying of a ParArray<T,rs,rm,al>.

ArrayTrans is the foundation of communication during repeated ghost point updating, which is the primary communication pattern of a spacial decomposed parallel PDE solver in DNDS.

ArrayTrans Bench

Written in Python, the benchmark calls DNDS modules and builds Array<real, vdim>, where vdim is a positive size.

Code is here.

To mimic standard workload, use 3-D block (c-indexed) each process.

Each process has size of $32\times32\times32\times\text{vdim}$.

Finds neighbor ranks: upper, lower, front, back, left, right.

Sizes:

• Volume points: $32^3=32768$
• Face points to be pulled: $32^2\times6=6144$, which is \(18.75\%\)

Tested vdim:

• 6
• 20
• 120

Pitfall Using mpirun -np X python xxx.py might be dangerous If np=64, and OMP_NUM_THREADS (or other thread number control) is not set, and something in python (like NumPy) decides to launch 64 threads, you have 4096 copies of NumPy need to load, which could cause IO system break or memory drain on some systems. See this discussion on stack overflow. On THTJ (thcp1), need to set export OMP_NUM_THREADS=1 before launch.

Machines

Results

Bandwidth results: bytes/s (per rank / total)