Run-Warp3D-on-Oakley.md

+* [Serial Run Instructions](#serial)
+* [~~Parallel Run Instructions~~](#parallel)
+
+### <a name="serial"></a>Serial (OpenMP only)
+
+Using the serial version limits you to running this job on one node (12 cores at Oakley). Requesting more threads than 12 on Oakley will be counterproductive.
+
+1. Go into your work directory with the job you want to run.
+
+  ```bash
+  $ cd /path/to/warp/job
+  ```
+
+2. Create the PBS batch script `job.pbs` in your work directory given as below.
+
+  ```bash
+  #PBS -N warp_job
+  #PBS -l walltime=01:00:00
+  #PBS -l nodes=1:ppn=12
+  #PBS -j oe
+  #PBS -q @oak-batch.osc.edu
+  #PBS -S /bin/bash
+
+  module load warp3d/17.5.3
+  
+  cd $PBS_O_WORKDIR
+
+  warp3d.omp 12 < input.wrp > output_file
+  ```
+
+  Modify `walltime=hh:mm:ss` with an overestimate of the time needed to run the job. The `12` is the number of threads `warp3d.omp` will be using (in this case all the cores on a single node on Oakley). Also don't forget to modify `input.wrp` to be the correct input file read in by Warp3D
+
+3. Run the batch job
+
+  ```bash
+  $ qsub job.pbs
+  ```
+
+4. Check on the status of the batch job
+
+  ```bash
+  $ qstat -u <username>
+  ```
+
+### <a name="parallel"></a>~~Parallel (OpenMP + MPI)~~
+
+Before running in parallel you will need to decide how many MPI ranks you will be using for the job. This number is usually specified as the number of domains in the geometry.
+
+This will in turn constrain the number of threads you can do per MPI rank on Oakley (12 cores per node)...
+
+```c
+((ranks * threads) % 12 == 0) && (12 % threads == 0)
+```
+
+* The first statement gives you an integer number of total nodes.
+* The second statement gives an integer number of MPI processes per node.
+
+For example, if there are 4 domains in the geometry file (input.wrp) then you will be using 4 MPI ranks.
+
+MPI Ranks = 4
+
+* 3 threads per rank (1 node, 4 processes per node)
+* 6 threads per rank (2 nodes, 2 processes per node) = example below
+* 12 threads per rank (4 nodes, 1 process per node)
+
+It is best to choose the number of threads per rank that mirrors the size of the domains.
+
+Example:
+
+1. Go into your work directory with the job you want to run.
+
+  ```bash
+  $ cd /path/to/warp/job
+  ```
+
+2. Create the PBS batch script `job.pbs` in your work directory given as below.
+
+  ```bash
+  #PBS -N warp_job
+  #PBS -l walltime=01:00:00
+  #PBS -l nodes=2:ppn=12
+  #PBS -j oe
+  #PBS -q @oak-batch.osc.edu
+  #PBS -S /bin/bash
+
+  module load warp3d/17.5.3
+  
+  cd $PBS_O_WORKDIR
+
+  # Uses 4 domains (MPI ranks = 4) with 6 threads per rank
+  export OMP_NUM_THREADS=6
+  export NUM_PERHOST=$((12/$OMP_NUM_THREADS))
+
+  # Env variable for Intel MPI runtime
+  export I_MPI_FABRICS=shm:dapl
+
+  echo "*input from 'input.wrp'" | mpiexec -npernode ${NUM_PERHOST} warp3d.mpi_omp > output_file
+  ```
+
+  Note, `mpiexec` requires you to feed in a small command from `stdin` instead of a large file ([see below](#issue-bigfile))
+
+3. ...
+
+4. ...
+
+### Issues
+
+1. <a name="issue-bigfile"></a>`mpiexec` has a file size limit of exactly 377 KiB from `stdin`.
+  * Doug has mentioned he won't fix this as it is easier to just work around it. The work around is shown in the batch script.
\ No newline at end of file