SuperNu - IMC-DDMC Radiation Transport Software for Explosive Outflows
----------------------------------------------------------------------
Copyright (c) 2013-2019 Ryan Wollaeger and Daniel van Rossum.  All rights reserved.

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, version 3 of the License.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    Please find a copy of the GNU General Public License in the COPYING
    file provided along with this program, or see <http://www.gnu.org/licenses/>.

LANL_COPYING and LANL_README
----------------------------
Modifications made by Ryan Wollaeger while at Los Alamos National Laboratory (LANL), from January 2015 to the present, fall under a LANS copyright assertion.
See LANL_COPYING for the copyright assertion and LANL_README for a summary of the corresponding modifications made by Ryan
Wollaeger while at LANL.

If you use SuperNu, we ask you to appropriately acknowledge SuperNu and cite the papers in which the methods are described.
Thank you,
the authors

[email protected]
[email protected]


SUPERNU METHOD PAPERS
=====================
Wollaeger, van Rossum et al. 2013, ApJS 209, 37
Wollaeger & van Rossum 2014, ApJS 214, 28
van Rossum & Wollaeger 2015, in preparation


SUMMARY OF SUPERNU PHYSICS
==========================
SuperNu is a program for simulating time-dependent radiation transport in local thermodynamic equilibrium with matter.  It applies the methods of Implicit Monte Carlo (IMC) [1] and Discrete Diffusion Monte Carlo (DDMC) [2] for static or homologously expanding spatial grids.  The radiation field affects material temperature but does not affect the motion of the fluid.  SuperNu may be applied to simulate radiation transport for supernovae with ejecta velocities that are not affected by radiation momentum [3].  The physical opacity calculation includes elements from Hydrogen up to Cobalt [3].  SuperNu is motivated by the ongoing research into the effect of variation in the structure of progenitor star explosions on observables: the brightness and shape of light curves and the temporal evolution of the spectra.  Consequently, the code may be used to post-process data from hydrodynamic simulations.  SuperNu does not include any capabilities or methods that allow for non-trivial hydrodynamics.

[1] JA Fleck Jr, JD Cummings Jr, "An implicit Monte Carlo scheme for calculating time and frequency dependent nonlinear radiation transport", (1971)
[2] JD Densmore, TJ Urbatsch, TM Evans, MW Buksas, "A hybrid transport-diffusion method for Monte Carlo radiative-transfer simulations", (2007)
[3] RT Wollaeger, DR van Rossum, "Radiation Transport for Explosive Outflows: Opacity Regrouping", (2014)


CODE FEATURES
=============
The time-dependent equations for photon transport and diffusion are solved with Monte Carlo (MC) by discretizing the radiation energy in the system in the form of MC packets (particles).  These particles are transported over the computational domain (spatial grid) where they interact with the gas according to multi-group opacities (groups).  Particles that leave the domain are tallied per timestep and flux wavelength bin, and written to disk.

Particles:
- MC Particles are generated in locations where energy sources are non-zero.
- The number of particles generated locally is determined by:
  - a power law of the local energy source,
  - the fraction of the local energy source to the total energy source,
  - a minimal number of particles per grid cell with non-zero energy source.
- Energy sources include:
  - analytic boundary and volume sources,
  - manufactured-solution sources,
  - energy deposition by radioactive decay of supernova explosion products.
- Particles that get censused at the end of a timestep are stored in the particle array.
  - particle direction is stored in the lab frame,
  - particle energy/wavelength is stored in the comoving frame.

Transport:
— Particle transport methods are:
  — Implicit Monte Carlo (IMC),
  — Discrete Diffusion Monte Carlo (DDMC).
— DDMC is used in optically thick regions of a domain.
— Geometries:
  — 3D spherical,
  — 2D cylindrical,
  — 3D cartesian.
— Assumptions:
  — local thermodynamic equilibrium (LTE),
  — material either static or in homologous expansion.
- particle transport is parallelized using a hybrid of MPI and OpenMP

Spatial grid:
— The grid is the domain over which MC particles are tracked.
— The grid can either be a spatial or a velocity mesh.
— If the grid is physically moving:
  — the grid has units of velocity,
  — the velocity grid is not affected by the radiation radiation field,
  — the velocity grid is constant.
— If the grid is physically static:
  — the grid is in units of length,
  — the domain is static (no velocity field).
- The grid array is geometry-agnostic
— The grid array is compressed by not storing regions of zero opacity (zero mass).
- The grid array only stores quantities that are needed for the transport step

Gas:
— Gas properties are domain decomposed.
- Gas properties are updated after transport steps.
— Gas properties include:
  - chemical composition,
  — density,
  — material temperature,
  — heat capacity,
  — opacity: Thomson scattering, and multi-group absorption.
— Leakage (DDMC) and Planck opacities are calculated from scattering and absorption opacities.
— Mutli-group absorption opacity includes bound-bound (bb), bound-free (bf), and free-free (ff) data for Hydrogen to Cobalt:
  — line data for bound-bound opacities are taken from http://kurucz.harvard.edu/atoms.html.

Groups:
— Opacity frequency dependence is discretized with multi-group.
— The frequency-resolved opacities are averaged within each group.
— The code may use a constant multi-group grid in units of either:
  — wavelength,
  — non-dimensional wavelength (wavelength multiplied by material temperature).

IO:
- input is divided in two categories:
  - model specific input files are named input.* and (see the Input/ directory):
    - input.par: runtime parameters,
    - input.str: velocity-density-composition structure on the computational domain.
  - model independent data files are named data.* (see the Data/ directory):
    - data.bf_verner: bound-free cross section data (Verner et al. 1996, ApJ 465, 487)
    - data.ff_sutherland: free-free gaunt factor data (Sutherland 1998, MNRAS 300, 321)
    - data.ion: atomic level data (from http://kurucz.harvard.edu/atoms.html)
    - Atoms/data.atom.* bound-bound transition data
- output files are named output.*
  - stdout is written to output.log unless disabled in by an input parameter.
  - flux variables are saved as output.flx_*
  - grid variables are saved as output.grd_*
  - total (integrated over the domain) energy budget numbers are saved as output.tot_energy


SUPERNU OUTPUT
==============
Flux:
SuperNu writes flux output in ascii (output.flx_luminos) as a sequence of spectra, one spectrum per line, one line per viewing angle, repeated in each time step.  So the number of columns equals the number of wl-bins, and the number of rows equals nmu*nphi*ntimestep, where mu=cos(theta).  The luminos values are in units [erg/s].  The 'output.flx_grid' file describes the wavelength, viewing angle, and time bins.

Energy totals:
SuperNu writes output of energy totals in ascii (output.tot_energy) each time step, with each column corresponding to a particular energy value.
The first column is energy conservation error.  These values can be used to find unintended energy leaks or sources in simulations.

Grid-based variables:
Using parameter in_io_nogriddump, SuperNu optionally writes grid-based variables in ascii (output.grd_*).
Grid variables include material temperature (output.grd_temp), radiation energy density (output.grd_eraddens), and Planck opacity (output.grd_capgrey).
The grid variables can be mapped to spatial cells (with an (i,j,k) index) with output.grd_grid.
As headers, output.grd_grid has (in row order) the geometry index (grd_igeom), the number of spatial cells along each dimension (grd_nx,grd_ny,grd_nz),
and the total number of array cells used to yield an optimal row size (ncpr), column size (nrow) to minimize padding cells per write
(the final header has nrow*ncpr,nrow,ncpr).
The next 3 rows are space or velocity values at the cell edges in the x,y,z dimensions.
The following integers are the cell indices of grid variables for spatial index locations (i,j,k), where the column index is the x-dimension (i),
and the row index is the serialized index for the y (j) and z (k) dimensions (the row index is j+(k-1)*ny).
Thus, the cell padding information can be used to remove extra padding values from the grid output, and the grid index mapping to the (i,j,k)
spatial cell index can used to reconstruct spatial profiles of the padding-stripped grid data.


CODING STANDARDS
================
- Source files with the .f suffix adhere to fortran fixed format standard
  - the default indentation width is 1
- Source files with the .f90 suffix adhere to fortran free format standard
  - the default indentation width is 3
- Each subroutine/function contains a comment section that explicitly states its purpose
- Each new paragraph in a source file starts with a comment line detailing the purpose of the following paragraph
- All public variables in a module have a threeletter_ prefix that is unique for that module.
- All variables/subroutines/functions/intrinsics etc. have lowercase names.
  - the only exeception is a USE statement for modules that are defined in the same file in which they are used.


SETUP INSTRUCTIONS
==================
The following procedure produces two supernu executables, one serial and one MPI-parallel version.

# mkdir ~/supernu
# cd ~/supernu
# git clone [email protected]:supernu/supernu.git src

-- SERIAL --
# mkdir bin
# cd bin
# lndir ../src
  - Tools/lndir.sh can be used instead of lndir if lndir is not available on the system)
  - make sure that no .mod, .o, or .a files were linked from the source directory
# ln -sf ./mpimod_ser.f ./mpimod.f
# cp -f ./System/Makefile.compiler.gfortran ./Makefile.compiler
  - modify the compiler flags to your liking (e.g. optimization, debug flags etc.)
# make -j
# cd ../../supernu

-- MPI --
# mkdir bin-mpi
# cd bin-mpi
# lndir ../src
# cp -f ./System/Makefile.compiler.gfortran ./Makefile.compiler
  - update the ./Makefile.compiler to something like: FC=mpif90
  - modify the compiler flags to your liking (e.g. optimization, debug flags etc.)
# ln -sf ./mpimod_mpi.f ./mpimod.f
# make -j


USE INSTRUCTIONS
================
SuperNu requires the above specified input files (input.* and data.* files) to run.
The following procedure sets up and runs an example SuperNu simulation:

##-- prepare sim directory
# mkdir -p ~/sim-supernu/test/run001
# cd ~/sim-supernu/test/run001
# cp ~/supernu/bin/supernu .
# ln -sf ~/supernu/src/Data/* .
# ln -sf /home/Data/Atoms.20120801 ./Atoms
##-- setup simulation
# cp -f ~/supernu/src/Input/input.str_r64 input.str
# cp -f ~/supernu/src/Input/input.w7.par input.par
# echo "in_name = 'w7_11r64'" >> input.par
# echo "in_comment = 'test simulation 1D'" >> input.par
##-- run simulation
# ./supernu