Xingjian (Ken) Yan

Xingjian (Ken) Yan

PhD Student, MIT Department of Earth, Atmospheric, and Planetary Sciences

MPAS-A Notes

Notes for Running MODEL FOR PREDICTION ACROSS SCALES - ATMOSPHERE, MIT EAPS, 2025

Notes for building everything MPAS from this tutorial.

TL;DR

Slide Show here!

Allocate Rescource

qsub -I -l select=1:ncpus=32:mem=128GB -q main -l walltime=8:00:00 -A <Project id>

Run

On Derecho

kenyan@dec2435:~> module swap intel gcc/12.4.0

Due to MODULEPATH changes, the following have been reloaded:
  1) cray-mpich/8.1.29     2) hdf5/1.12.3     3) ncarcompilers/1.0.0     4) netcdf/4.9.2

kenyan@dec2435:~> module unload netcdf 


kenyan@dec2435:~> module load   hdf5-mpi/1.12.3 netcdf-mpi/4.9.3 parallel-netcdf/1.14.0 parallelio/2.6.6
kenyan@dec2435:~> module load cdo ncl ncview
kenyan@dec2435:~> module list

Currently Loaded Modules:
  1) ncarenv/24.12 (S)   3) gcc/12.4.0            5) libfabric/1.15.2.0   7) hdf5-mpi/1.12.3    9) parallel-netcdf/1.14.0  11) cdo/2.4.4  13) ncview/2.1.9
  2) craype/2.7.31       4) ncarcompilers/1.0.0   6) cray-mpich/8.1.29    8) netcdf-mpi/4.9.3  10) parallelio/2.6.6        12) ncl/6.6.2

  Where:
   S:  Module is Sticky, requires --force to unload or purge

On Dolma

module load apps/miniconda/3.6 
micromamba activate mpas_toolchain

## Setups
link to model: /net/flood/data2/users/x_yan/mpas_toolchain/mpas
different meshes (60-3km being x20): /net/flood/data2/users/x_yan/mpas_toolchain/meshes
convert to lat-lon toolbox: /net/flood/data2/users/x_yan/mpas_toolchain/convert_mpas
create init mesh: /net/flood/data2/users/x_yan/mpas_toolchain/MPAS-Tools/mesh_tools/grid_rotate
WPS_GEOG is at: /net/flood/data2/users/x_yan/WPS_ncar

We get the static Puerto Rico Mesh right

# 1) Copy and link
mkdir /net/flood/data2/users/x_yan/mpas_runs/240-48km_variable
cd /net/flood/data2/users/x_yan/mpas_runs/240-48km_variable
ln -s /net/flood/data2/users/x_yan/mpas_toolchain/meshes/x20.835586.grid.nc .
cp /net/flood/data2/users/x_yan/mpas_toolchain/MPAS-Tools/mesh_tools/grid_rotate/namelist.input .

# 2) Rewrite a minimal, clean namelist with a final newline + blank line
cat > namelist.input << 'EOF'
&input
  config_original_latitude_degrees  = 0.0
  config_original_longitude_degrees = 0.0
  config_new_latitude_degrees       = 18.25
  config_new_longitude_degrees      = -66.50
  config_birdseye_rotation_counter_clockwise_degrees = 0.0
/
 
EOF

# 3) Ensure Unix line endings (harmless if already OK)
dos2unix namelist.input 2>/dev/null || true

# 4) Verify that the last line is truly blank (there should be output here)
tail -n 2 -v namelist.input

# 5) Run the rotation
/net/flood/data2/users/x_yan/mpas_toolchain/MPAS-Tools/mesh_tools/grid_rotate/grid_rotate x20.835586.grid.nc PR.grid.nc

We should see /net/flood/data2/users/x_yan/mpas_runs/240-48km_variable/PR.grid.nc 569ed333871385abbc73cf47c1745ce8

Then we do 3.2 Static, terrestrial field processing

ln -s /net/flood/data2/users/x_yan/mpas_toolchain/mpas/init_atmosphere_model .
cp /net/flood/data2/users/x_yan/mpas_toolchain/mpas/namelist.init_atmosphere .
cp /net/flood/data2/users/x_yan/mpas_toolchain/mpas/streams.init_atmosphere .

## CHANGE the 2 files copied!
'''
In namelist.init_atmosphere, for nhyd_model - config_init_case
2 = ideal baroclinic wave
4 = ideal squall line
5 = ideal supercell
6 = ideal mountain wave
7 = real-data initialization
8 = surface update file creation (like sst_update=1 in WRF)
9 = lateral boundary conditions



&preproc_stages
    config_static_interp = true
    config_native_gwd_static = true
    config_vertical_grid = false
    config_met_interp = false
    config_input_sst = false
    config_frac_seaice = false
/



'''

./init_atmosphere_model

We should be able to see the static field

micromamba run -n ncl_66 bash -lc \
'export FNAME="/net/flood/data2/users/x_yan/mpas_runs/240-48km_variable/PR.static.nc"; \
ncl /net/flood/data2/users/x_yan/mpas_toolchain/mpas_tutorial/ncl_scripts/plot_terrain.ncl'

image

Then we Interpolating real-data initial conditions as in section 3.4

ln -s /net/flood/data2/users/x_yan/mpas_toolchain/mpas_tutorial/met_data/GFS:2014-09-10_00 .

## CHANGE namelist.init_atmosphere
## CHANGE streams.init_atmosphere

<immutable_stream name="input"
                  type="input"
                  filename_template="PR.static.nc"
                  input_interval="initial_only" />

<immutable_stream name="output"
                  type="output"
		  io_type="pnetcdf, cdsf5"
                  filename_template="PR.init.nc"
                  packages="initial_conds"
                  output_interval="initial_only" />

./init_atmosphere_model

Then we Integrate as in section 3.5

ln -s /net/flood/data2/users/x_yan/mpas_toolchain/mpas/atmosphere_model . 
ln -s /net/flood/data2/users/x_yan/mpas_toolchain/mpas/src/core_atmosphere/physics/physics_wrf/files/* .

cp /net/flood/data2/users/x_yan/mpas_toolchain/mpas/namelist.atmosphere .
cp /net/flood/data2/users/x_yan/mpas_toolchain/mpas/streams.atmosphere .
cp /net/flood/data2/users/x_yan/mpas_toolchain/mpas/stream_list.atmosphere.* .

## CHANGE namelist.atmosphere -
## config_dt — the model timestep (delta-t) in seconds.
## config_len_disp — the length-scale for explicit horizontal diffusion, in meters.

ln -s /net/flood/data2/users/x_yan/mpas_toolchain/meshes/x20.835586.graph.info.part.144 .

## if running in parallel, needs to match the grid decomposition file prefix
mpiexec -n 144 ./atmosphere_model

We should be able to see the output

micromamba run -n ncl_66 bash -lc \
'export GNAME="/net/flood/data2/users/x_yan/mpas_runs/mimic_mem2_May_2001/PR.init.nc"; \
export FNAME="/net/flood/data2/users/x_yan/mpas_runs/mimic_mem2_May_2001/diag.2001-05-01_00.00.00.nc"; \
ncl /net/flood/data2/users/x_yan/mpas_toolchain/mpas_tutorial/ncl_scripts/latlon_cells.ncl'

image

Prepare LENS2 to GFS-like IC inputs

Use pywinter and cam-se-regrid

What we need: image

TL; DR: We run 2 scripts:

## On Derecho, run
/glade/u/home/kenyan/mpas_lens/process_ihesp_data.csh


# We should see in /glade/derecho/scratch/kenyan/iHESP_stage/2001-04-30_06Z/:
atmos_ta.nc     atmos_hus.nc   atmos_ua.nc   atmos_va.nc
atmos_z3.nc     atmos_ps.nc    atmos_base.nc
atmos_tos_1.nc  atmos_ts_1.nc  atmos_snw_1.nc atmos_ps2.nc
atmos_tsl_1.nc  atmos_mrlsl_1.nc
atmos_zsfc.nc -> (symlink to invariant)
atmos_lmask.nc -> (symlink to invariant)


## On Dolma, run with pangeos3_ken (needs > 200 GB memory)
/net/flood/data2/users/x_yan/mpas_lens/edit_nc_files.py

# Should see
/glade/derecho/scratch/kenyan/MPAS/intermediate/iHESP/mem2/2001/FILE:2001-04-30_06

# To Watch:
qstat -u $USER

Postprocessing

There is tutorial here Include the file “target_domain” to remap at smaller domain and finer resolution

By default, the diag files contain:

RH, T, height, winds @ 200, 250, 500, 700, 850, 925 hPa CAPE, CIN, LCL, LFC, updraft helicity U10, V10, T2, Q2
Simulated radar reflectivity PMSL
Surface, 1 km AGL, 6 km AGL winds (various other 2-d fields)

By default, the "history" files contain:
q_v, q_c, q_r, ...
theta
zonal, meridional wind
vertical velocity
full pressure
dry density
accumulated rain (cumulus and microphysics)
soil moisture, soil temperature
(various other fields)
Full mesh information (vertical and horizontal)

Visualize by convert to lat-lon grid - outputnamde latlon.nc

cd /net/flood/data2/users/x_yan/mpas_runs/mimic_mem2_May_2001

cp /net/flood/data2/users/x_yan/mpas_runs/mem2_1_May_2001_PR_60_3km_circ_12h/target_domain .

/net/flood/data2/users/x_yan/mpas_toolchain/convert_mpas/convert_mpas history*nc

# or
convert_mpas history.2014-09-15_00.00.00.nc

convert_mpas PR.init.nc diag*nc

# or
/net/flood/data2/users/x_yan/mpas_toolchain/convert_mpas/convert_mpas history.2014-09-15_00.00.00.nc

Errors and Fixes

Thompson microphysics tables for clouds

subroutine thompson_init: failure opening MP_THOMPSON_QRacrQG.DBL

# 1) build the tables (writes MP_THOMPSON_* files in the CWD)
cd /net/flood/data2/users/x_yan/mpas_toolchain/mpas
./build_tables

# 2) put the generated tables in your run directory
cd /net/flood/data2/users/x_yan/mpas_runs/mimic_mem2_May_2001/

# EITHER: make absolute symlinks
ln -s /net/flood/data2/users/x_yan/mpas_toolchain/mpas/MP_THOMPSON_*.DBL .

# OR: copy the actual files (bulletproof)
cp -av /net/flood/data2/users/x_yan/mpas_toolchain/mpas/MP_THOMPSON_*.DBL .

Side stuff on HPC without setenv - use export

cp /net/flood/data2/users/x_yan/mpas_toolchain/mpas_tutorial/ncl_scripts/mesh_resolution.ncl .
export FNAME=/net/flood/data2/users/x_yan/mpas_runs/240-48km_variable/PR.grid.nc
ncl mesh_resolution.ncl