Intro to NCL (on HPC and PC)

This is a quick intro on how to use NCL to process raw grib files downloaded from ECMWF using MARS api requests.

An IMPORTANT reminder is that the official ECMWF python script for converting from model level hybrid scales to height / pressure levels has significant errors and runs very slowly. See the forum link for details.

Instead, the CDO command line interface of the ml2pl and ml2hl functions would come to rescue!

Install NCl and CDO in the conda env

1. For Windows users, install Ubuntu from the microsoft store, then proceed to step 2.
2. For Linux users, module load your conda setup via module load anaconda, or, for MIT Engaging users, use: module load miniforge/24.3.0-0.
3. Create a conda env with specifying the desired installed package: conda create -n ncl_stable -c conda-forge ncl
4. Then conda install cdo
5. If the above is not working, try following the conda forge commannds in this link or the offical tutorial
6. The conda forge for cdo may not be working, so you can try the Max-Plack Institut link here, if your HPC supports docker…

Convert from hybrid model levels (1-137) to pressure levels

1. cd to your desired path where input and output files are (will be) located.
2. cdo ml2pl,90000,85000,80000,75000,70000,65000,60000,55000,50000,45000,40000,35000,30000,25000,20000,15000,10000 1998_pt_lapse.grib 1998_pt_lapse_pl.grib

Something to notice is that ml2pl function may not work well on date ranges containing more than 2 time stamps, so it would be better to download the raw data in daily segment, and then use a batch process scirpt to convert.

1. download raw data and load the above sh script
2. chmod +x convert_grib_files.sh
3. dos2unix convert_grib_files.sh
4. ./convert_grib_files.sh
5. you should see something like
6. To concat the daily grib into a monthly grib, use regex cdo command: cat 1998-09-*_pressure_level_pt_lapse.grib > 1998-09_pressure_level_pt_lapse.grib

Convert from fc data type (time + step; init + lead) to an data type (time; valid_time)

Do the following in a python jupyter notebook, GIVEN YOU HAVE DOWNLOADED STEP IN 3/6/9/12:

1) Rename coords

ds_pl = ds_pl.rename(time=”init_time”, step=”lead_time”)

2) Stack over (init_time, lead_time)

ds_pl_stacked = ds_pl.stack(time_step=(“init_time”, “lead_time”))

3) Assign a new ‘time’ coordinate from ‘valid_time’

new_time = ds_pl_stacked[“valid_time”].values.ravel()

ds_pl_stacked = ds_pl_stacked.assign_coords( time=(“time_step”, new_time) )

4) Swap out the stacked dimension so that everything is indexed by new time

ds_pl_stacked = ds_pl_stacked.swap_dims({“time_step”: “time”})

5) (Optional) drop ‘valid_time’ and tidy up

ds_pl_stacked = ds_pl_stacked.drop_vars(“valid_time”) ds_pl_stacked = ds_pl_stacked.sortby(“time”)

6) (Optional) Change from 0 - 360 to -180 to 180 to put Euro-Atlantic in the middle

ds_pl_stacked[‘longitude’] = ((ds_pl_stacked[‘longitude’] + 180) % 360) - 180 ds_pl_stacked = ds_pl_stacked.sortby(‘longitude’)

ds_pl_stacked

Convert from spherical harmonics coord to gridded resolution

1. Convert from “reduced Gaussian” to “regular Gaussian”: cdo setgridtype,regular T639_19980915_19980930_330isentropic_pv.grib T639_19980915_19980930_330isentropic_pv_gauss.grib
2. cdo invertlat in out: cdo -f nc -sp2gpl /mnt/c/Users/yanxi/Downloads/T639_19980915_19980930_330isentropic_uvvo.grib /mnt/c/Users/yanxi/Downloads/T639_19980915_19980930_330isentropic_uvvo_gauss.nc
3. cdo merge T639_19980915_19980930_330isentropic_pv.nc T639 _19980915_19980930_330isentropic_uvvo_gauss.nc T639_19980915_19980930_330isentropic_concat.nc

Spatial Derivatives

Please email Ken for code.

Mean-Perturbation Decompositions

Please email Ken for code.