TraCE T31 to T85

* Note this is unsupported

Converting TraCE T31 to T85 for Stand-alone CAM3.1 simulations using repeating 12 month climatology forcing- expert users only

This page describes a step-by-step process for converting the 22,000 year TraCE simulation from it's original resolution (T31) to a higher resolution (T85) in order to run the stand-alone version of CAM3.1 and simulate high-resolution time slices forced by the TraCE T31 climate.  Note, this simulation is forced with a 12 month repeating annual climatology.

Steps

  1. Download the TraCE Roadmap.pdf for reference.
  2. Download the development logfile for detailed notes.
  3. Choose the TraCE time slice or period of interest (POI) you want to use (e.g., 14.5ka). 
    1. Identifiy the case name (CASEID) that corresponds to the time period you have chosen.  In this case, the TraCE Roadmap shows that there are two TraCE timeslices that bookend the years 14.5:  b30.14_9kaDVT and b30.14_35kaDVT.  You would use the b30.14_9kaDVT simulation because as the model runs forward in time, it counts down toward the present day through year 14.5. 
    2. Identify the years within this time slice that correlate to your POI.  e.g., 14.5ka = 14.9yr0 + 400y = 14.9yr400 = 14.5ka
  4. Notify NCAR (nanr@ucar.edu) with an email that identifies your caseid (b30.14_9kaDVT), and start year (in this case, 400) so that she can pre-stage the required restart, initial, and history files.
  5. mirage0:  Download 20y of atm history files from the HPSS.  I recommend taking 20y around your POI.  e.g., years 390-409 from b30.14_9kaDVT. 
    1. create a working directory to store the large number of atm history files:
      1. If you are on mirage0, use this path:  /glade/user/<USER>/TRACE/b30.14_9kaDVT
    2. cd to your new working directory:  /glade/user/<USER>/TRACE/b30.14_9kaDVT
    3. download the atm history files files: 
      1. hsi -P 'get /CCSM/csm/b30.14_9kaDVT/atm/hist/b30.14_9kaDVT.cam2.h0.039[0-9].tar'
      2. hsi -P 'get /CCSM/csm/b30.14_9kaDVT/atm/hist/b30.14_9kaDVT.cam2.h0.040[0-9].tar'
    4. untar the atm history files.
  6. Create a 20y SST and sea ice climatology using TS and ICEFRAC from the atm history files as proxies for SST and sea ice (Table 1).
    1. machine:  mirage0
    2. srcDir:  /T31-T85/source/
    3. script:  mk_SST_climo_from_histFiles.pl  
    4. infile:
    5. outfile:
    6. You no longer need the history files and can delete them if you need the space.
  7. Interpolate the monthly SST climatology from T31 to T85.
    1. machine:  mirage0
    2. srcDir:  /T31-T85/source/
    3. script: interpolate_sst_t31-t85.ncl
    4. input:  b30.14_9kaDVT_ic_48x96_climo.531-550.<DATE>.nc
    5. output:  sst_ic_128x256_b30.14_9kaDVT.531-550.<DATE>.nc
  8. Create a new topography file by adding Peltier ICE-5G ice sheets for your POI.  Be sure to choose the Peltier period that PRECEDES the time slice you are interested in.  For example, the TraCE simulation for 14.9ka uses the Peltier ICE-5G distribution at 15ka B.P.
    1. machine:  mirage0
    2. srcDir:  /T31-T85/source/
    3. script:   mkTopo.10min.ncl
    4. input1:  ice5g_v1.2_15.0k_10min.ice.nc
    5. input2:  ice5g_v1.2_15.0k_10min.topo.nc
    6. output: topo_15.0ka_10min.<DATE>.nc
  9. Create the T85 CAM3.1 initial conditions.  This step interpolates your 10min topography onto the T85 grid.
    1. machine:  mirage0
    2. srcDir:  /T31-T85/source/
    3. script:  SA_t85_cami_create_trace_script.ncl
    4. input1:  b30.14_9kaDVT.cam2.i.531-01-01-00000.nc
    5. input2: topo_15.0ka_10min.<DATE>.nc
    6. input3:  sst_ic_128x256_b30.14_9kaDVT.<FYR>-<LYR>.<DATE>.nc
    7. output: b30.14_9kaDVT_T85.cam2.i.<RESTART-YEAR>-01-01-00000_SA.nc
  10. Build a 1 day simulation.  This 1-day run creates a surface dataset and the initial conditions for the land model (clm2.i).
    1. machine:  bluefire
    2. srcDir:  /T31-T85/cam_runs/scripts/
    3. script:  bld-1day.csh
    4. usage1.  bld-1day.csh
    5. input1:  b30.14_9kaDVT_T85.cam2.i.<RESTART-YEAR>-01-01-00000_SA.nc
    6. input2:  sst_ic_128x256_b30.14_9kaDVT.531-550.<DATE>.nc
    7. input3:  mksrf_pft_ice5g_v1.2_15.0k.nc
    8. input4:  mksrf_lanwat_ice5g_v1.2_15.0k.nc
    9. input5:  mksrf_glacier_ice5g_v1.2_15.0k.nc
    10. output1:  surface-dataset.  /ptmp/USER/b30.14_9kaDVT/cam_b30.14_9kaDVT_T31/rundir/surface-data.096x048.nc
  11. Run your 1-day simulation and save the output
    1. machine:  bluefire
    2. srcDir:  /T31-T85/cam_runs/AGCM-T85-14.37ka/cam_b30.14_37kaDVT_T85-1day/
    3. usage:  bsub < run-cam.csh
    4. Save the new surface dataset from the one-day simulation.
      1. path: /ptmp/USER/b30.14_9kaDVT/cam_b30.14_9kaDVT_T31/rundir/surface-data.096x048.nc  
      2. rename the surface dataset to reflect your simulation deta  e.g., surface-data.b30.14_9kaDVTd.256x128.111205.nc
      3. copy your new surface dataset file to the SourceMods directory under your cam_runs directory: 
        • e.g.,/T31-T85/cam_runs/AGCM-T85-14.37ka/SourceMods/surface-data.b30.14_9kaDVTd.256x128.111205.nc
  12. Interpolate the spunup land model carbon pools from the T31 TraCE run and interpolate them to the new T85 grid.  Interpinic is a tool that will interpolate the pools and create a new land initial condition file (clm.i).  Run interpinic on the clm.i file created by your one day run.
    1. machine:  bluefire
    2. srcDir:  /T31-T85/interpinic-DGVM
    3. script:  interpinic
    4. You need restart files for this step.  Create a directory to store your restart files:  /ptmp/<USER>/trace_restart/
    5. Choose the TraCE restart file that you want to use:
      • hsi -P 'ls /CCSM/csm/b30.14_9kaDVT/restart.tars/'
    6. download the restart file from the HPSS.
      • hsi -P 'get /CCSM/csm/b30.14_9kaDVT/restart.tars/b30.14_9kaDVT.ccsm.r.0531-01-01-00000.080728-131811.tar'
    7. cd to /T31-T85/interpinic-DGVM/
    8. build the executable (if required):  gmake (be sure to optimize for bluefire)
      1. gmake SMP=TRUE OPT=TRUE -j 64
      2. setenv OMP_NUM_THREADS 64
    9. copy the TraCE T31 source clm.i file to the 'in' directory:  /T31-T85/interpinic-DGVM/in
    10. copy the new T85 clm2.i file to the 'out' directory.
    11. Run interpinic to transfer the spunup carbon pools from the T31 grid to the newly created T85 grid.
      1. interpinic -i in/T31.clm2.i -o out/T85.clm.i
    12. rename our T85.clm2.i file to reflect it's origins:
      1. e.g.:  mv cam_b30.14_37kaDVT_T85-1day.clm2.i.0531-01-02-00000.nc b30.14_9kaDVT.clm2.i.0531-01-01-00000.IP.nc
  13. Extract the greenhouse gases for your time period
    1. machine:  bluefire
    2. srcDir:  /T31-T85/cam_runs/scripts
    3. script:  calc_ghg.ncl
    4. modify script for your time period and run script. 
    5. output file:  ghg_14.37kaBP.txt
  14. Set up and run the production simulation.
    1. machine:  bluefire
    2. srcDir:  /T31-T85/cam_runs/scripts/
    3. script:  bld-production.csh
    4. modify the build script to point to your newly created forcing files.
    5. build your production case:  ./bld-production.csh
    6. run your production case:  bsub < run-cam.csh
  15. Archive history files using an automated script: 
    1. source:   ~nanr/tools/archiveFiles.pl
    2. modify script for your CASEID.
    3. copy into your run directory and manually start the script:
      1. usage:  ./archiveFiles.pl
    4. Note:  The script first checks for the history files on the HPSS, then writes all history/rest/init files to the HPSS, then moves the local copy to your local /ptmp/user/archive/CASEID directory.  You must delete them from there once you are satisfied the archiver is working properly to store your files on the HPSS.
  16. Alternate:  Archive history files to HPSS (manually):
    1. cd /ptmp/<USER>/<my-run-path>/rundir/
    2. log onto the HPSS (this will put you in your home directory: e.g., /home/nanr/):  hsi -P   
    3. create a new case directory:  (e.g., mkdir cam_b30.14_37kaDVT_T85)
    4. cd cam_b30.14_37kaDVT_T85/
    5. create land and atm subdirectories:  mkdir [lnd,atm]/[hist,rest,init].  e.g.:
      1. mkdir lnd
      2. mkdir lnd/hist
      3. mkdir lnd/rest
      4. mkdir lnd/init
    6. cd to those directories and put your files there: e.g.:
      1. cd lnd/rest
      2. put CASEID.clm2.r.0001-01.nc

Notes

  • Customize the "user modification" section for each source code for your time period (POI) and the filenames of the new files you create.  Look for CASEID or <CASEID> or <POI> for places you need to modify the code.  Most steps of the code will automatically look for the input files and place the output files in directories relative to the src (or src-low2high) directory.
  • POI = "Period of Interest"
  • CASEID = TraCE name (e.g., b30.14_9kaDVT)
  • perl script usage:  ./mk_SST_climo_from_histFiles.pl
  • ncl script usage:  ncl mkTopo.10min.ncl

  • Download a set of detailed step-by-step instruction here.

  • Download a cartoon of T31 to T85 directory structure here.

  • Download a cartoon that outlines the steps required to convert from T31 to T85 here.

  • Download a copy of Feng's thesis here.

ORNL Notes for retrieving restart files:

hsi:   cd /home/fenghe/CCSM/b30.14_9kaDVT/restart.tars

hsi:  [/home/fenghe/CCSM/b30.14_9kaDVT/restart.tars]: get b30.14_9kaDVT.ccsm.r.0401-01-01-00000.080724-122437.tar

Trace Local Path: 

  • silver:/datalocal/ccpi/nanr

Todo

  • Optimize T85 (CAM3.1) - unlikely to get help with this.  Run in 6 month increments to fit in 6hr submission window
  • Automate HPSS archiving - add to submission script

Table 1

Machine Source Directory Source Code Description
mirage0.ucar.edu source/ mk_SST_climo_from_histFiles.pl Create a 20y monthly SST and sea ice climotology using TS and ICEFRAC from TraCE atmosphere history files.
mirage0.ucar.edu source/ interpolate_sst_t31-t85.ncl Convert the SST climatology from T31 to T85.  These SST/ice files will force the AGCM.
mirage0.ucar.edu source/ mkTopo.10min.ncl Add Peltier ICE5G ice sheets to the 10min USGS base map.
mirage0.ucar.edu source/ SA_t85_cami_create_script.ncl Create T85 CAM (atmosphere) initial conditions
bluefire.ucar.edu cam_runs/scripts/ bld-1day.csh This creates a 1 day simulation at T85 that is required to create new land (CLM) files (e.g., initial condition and surface dataset) that conform to the new land ice and sea level distributions.
bluefire.ucar.edu interpinic-DGVM/ interpinic-DGVM Interpolate the T31 TraCE initial conditions for the land model from T31 (TraCE) to T85. 
bluefire.ucar.edu cam_runs/scripts/ bld-production.csh Production run at T85

Land ice from ICE5G (Peltier)

NOTE:  POI = "Period of Interest"

1.  Create high resolution topography (10min)

Code:  mkTopo.10min.ncl

  • input:  USGS 10min modern topography
  • input2:  ICE5G/10min/POI.ice.nc
  • input3:  ICE5G/10min/POI.topo.nc

This code uses the default USGS topography everywhere except under new, glacial ice.  These areas are converted to land (landfract=1 and landmask=1).

2.  Create global proxy sea surface temperautres (SST) by averaging TS and ICEFRAC the monthly atmospheric history files from the TraCE simulation.

Original source code

  • /glade/proj2/cgd/ccr/paleo/TRACE/
  • cp -pR T31-T85 to myPath