NCO 4.0.9 User’s Guide

Foreword

NCO is the result of software needs that arose while I worked on projects funded by NCAR, NASA, and ARM. Thinking they might prove useful as tools or templates to others, it is my pleasure to provide them freely to the scientific community. Many users (most of whom I have never met) have encouraged the development of NCO. Thanks espcially to Jan Polcher, Keith Lindsay, Arlindo da Silva, John Sheldon, and William Weibel for stimulating suggestions and correspondence. Your encouragment motivated me to complete the NCO User’s Guide. So if you like NCO, send me a note! I should mention that NCO is not connected to or officially endorsed by Unidata, ACD, ASP, CGD, or Nike.

Charlie Zender
May 1997
Boulder, Colorado

Major feature improvements entitle me to write another Foreword. In the last five years a lot of work has been done to refine NCO. NCO is now an open source project and appears to be much healthier for it. The list of illustrious institutions that do not endorse NCO continues to grow, and now includes UCI.

Charlie Zender
October 2000
Irvine, California

The most remarkable advances in NCO capabilities in the last few years are due to contributions from the Open Source community. Especially noteworthy are the contributions of Henry Butowsky and Rorik Peterson.

Charlie Zender
January 2003
Irvine, California

NCO was generously supported from 2004–2008 by US National Science Foundation (NSF) grant IIS-0431203. This support allowed me to maintain and extend core NCO code, and others to advance NCO in new directions: Gayathri Venkitachalam helped implement MPI; Harry Mangalam improved regression testing and benchmarking; Daniel Wang developed the server-side capability, SWAMP; and Henry Butowsky, a long-time contributor, developed ncap2. This support also led NCO to debut in professional journals and meetings. The personal and professional contacts made during this evolution have been immensely rewarding.

Charlie Zender
March 2008
Grenoble, France

Summary

This manual describes NCO, which stands for netCDF Operators. NCO is a suite of programs known as operators. Each operator is a standalone, command line program executed at the shell-level like, e.g., ls or mkdir. The operators take netCDF files (including HDF5 files constructed using the netCDF API) as input, perform an operation (e.g., averaging or hyperslabbing), and produce a netCDF file as output. The operators are primarily designed to aid manipulation and analysis of data. The examples in this documentation are typical applications of the operators for processing climate model output. This stems from their origin, though the operators are as general as netCDF itself.

1. Introduction

1.1 Availability

The complete NCO source distribution is currently distributed as a compressed tarfile from http://sf.net/projects/nco and from http://dust.ess.uci.edu/nco/nco.tar.gz. The compressed tarfile must be uncompressed and untarred before building NCO. Uncompress the file with ‘gunzip nco.tar.gz’. Extract the source files from the resulting tarfile with ‘tar -xvf nco.tar’. GNU tar lets you perform both operations in one step with ‘tar -xvzf nco.tar.gz’.

The documentation for NCO is called the NCO User’s Guide. The User’s Guide is available in Postscript, HTML, DVI, TeXinfo, and Info formats. These formats are included in the source distribution in the files ‘nco.ps’, ‘nco.html’, ‘nco.dvi’, ‘nco.texi’, and ‘nco.info*’, respectively. All the documentation descends from a single source file, ‘nco.texi’ (1). Hence the documentation in every format is very similar. However, some of the complex mathematical expressions needed to describe ncwa can only be displayed in DVI, Postscript, and PDF formats.

A complete list of papers and publications on/about NCO is available on the NCO homepage. Most of these are freely available. The primary refereed publications are fxm ZeM06 and fxm Zen07. These contain copyright restrictions which limit their redistribution, but they are freely available in preprint form from the NCO.

If you want to quickly see what the latest improvements in NCO are (without downloading the entire source distribution), visit the NCO homepage at http://nco.sf.net. The HTML version of the User’s Guide is also available online through the World Wide Web at URL http://nco.sf.net/nco.html. To build and use NCO, you must have netCDF installed. The netCDF homepage is http://www.unidata.ucar.edu/packages/netcdf.

New NCO releases are announced on the netCDF list and on the nco-announce mailing list http://lists.sf.net/mailman/listinfo/nco-announce.

1.2 Operating systems compatible with NCO

NCO has been successfully ported and tested and is known to work on the following 32- and 64-bit platforms: IBM AIX 4.x, 5.x, FreeBSD 4.x, GNU/Linux 2.x, LinuxPPC, LinuxAlpha, LinuxARM, LinuxSparc64, SGI IRIX 5.x and 6.x, MacOS X 10.x, NEC Super-UX 10.x, DEC OSF, Sun SunOS 4.1.x, Solaris 2.x, Cray UNICOS 8.x–10.x, and MS Windows95 and all later versions. If you port the code to a new operating system, please send me a note and any patches you required.

The major prerequisite for installing NCO on a particular platform is the successful, prior installation of the netCDF library (and, as of 2003, the UDUnits library). Unidata has shown a commitment to maintaining netCDF and UDUnits on all popular UNIX platforms, and is moving towards full support for the Microsoft Windows operating system (OS). Given this, the only difficulty in implementing NCO on a particular platform is standardization of various C-language API system calls. NCO code is tested for ANSI compliance by compiling with C compilers including those from GNU (‘gcc -std=c99 -pedantic -D_BSD_SOURCE -D_POSIX_SOURCE’ -Wall) (2), Comeau Computing (‘como --c99’), Cray (‘cc’), HP/Compaq/DEC (‘cc’), IBM (‘xlc -c -qlanglvl=extc99’), Intel (‘icc -std=c99’), NEC (‘cc’), PathScale (QLogic) (‘pathcc -std=c99’), PGI (‘pgcc -c9x’), SGI (‘cc -c99’), and Sun (‘cc’). NCO (all commands and the libnco library) and the C++ interface to netCDF (called libnco_c++) comply with the ISO C++ standards as implemented by Comeau Computing (‘como’), Cray (‘CC’), GNU (‘g++ -Wall’), HP/Compaq/DEC (‘cxx’), IBM (‘xlC’), Intel (‘icc’), NEC (‘c++’), PathScale (Qlogic) (‘pathCC’), PGI (‘pgCC’), SGI (‘CC -LANG:std’), and Sun (‘CC -LANG:std’). See ‘nco/bld/Makefile’ and ‘nco/src/nco_c++/Makefile.old’ for more details and exact settings.

Until recently (and not even yet), ANSI-compliant has meant compliance with the 1989 ISO C-standard, usually called C89 (with minor revisions made in 1994 and 1995). C89 lacks variable-size arrays, restricted pointers, some useful printf formats, and many mathematical special functions. These are valuable features of C99, the 1999 ISO C-standard. NCO is C99-compliant where possible and C89-compliant where necessary. Certain branches in the code are required to satisfy the native SGI and SunOS C compilers, which are strictly ANSI C89 compliant, and cannot benefit from C99 features. However, C99 features are fully supported by modern AIX, GNU, Intel, NEC, Solaris, and UNICOS compilers. NCO requires a C99-compliant compiler as of NCO version 2.9.8, released in August, 2004.

The most time-intensive portion of NCO execution is spent in arithmetic operations, e.g., multiplication, averaging, subtraction. These operations were performed in Fortran by default until August, 1999. This was a design decision based on the relative speed of Fortran-based object code vs. C-based object code in late 1994. C compiler vectorization capabilities have dramatically improved since 1994. We have accordingly replaced all Fortran subroutines with C functions. This greatly simplifies the task of building NCO on nominally unsupported platforms. As of August 1999, NCO built entirely in C by default. This allowed NCO to compile on any machine with an ANSI C compiler. In August 2004, the first C99 feature, the restrict type qualifier, entered NCO in version 2.9.8. C compilers can obtain better performance with C99 restricted pointers since they inform the compiler when it may make Fortran-like assumptions regarding pointer contents alteration. Subsequently, NCO requires a C99 compiler to build correctly (3).

In January 2009, NCO version 3.9.6 was the first to link to the GNU Scientific Library (GSL). GSL must be version 1.4 or later. NCO, in particular ncap2, uses the GSL special function library to evaluate geoscience-relevant mathematics such as Bessel functions, Legendre polynomials, and incomplete gamma functions (see section GSL special functions).

In June 2005, NCO version 3.0.1 began to take advantage of C99 mathematical special functions. These include the standarized gamma function (called tgamma() for “true gamma”). NCO automagically takes advantage of some GNU Compiler Collection (GCC) extensions to ANSI C.

As of July 2000 and NCO version 1.2, NCO no longer performs arithmetic operations in Fortran. We decided to sacrifice executable speed for code maintainability. Since no objective statistics were ever performed to quantify the difference in speed between the Fortran and C code, the performance penalty incurred by this decision is unknown. Supporting Fortran involves maintaining two sets of routines for every arithmetic operation. The USE_FORTRAN_ARITHMETIC flag is still retained in the ‘Makefile’. The file containing the Fortran code, ‘nco_fortran.F’, has been deprecated but a volunteer (Dr. Frankenstein?) could resurrect it. If you would like to volunteer to maintain ‘nco_fortran.F’ please contact me.

1.2.1 Compiling NCO for Microsoft Windows OS

NCO has been successfully ported and tested on most Microsoft Windows operating systems including: 95/98/NT/2000/XP/Vista. The switches necessary to accomplish this are included in the standard distribution of NCO. Using the freely available Cygwin (formerly gnu-win32) development environment (4), the compilation process is very similar to installing NCO on a UNIX system. Set the PVM_ARCH preprocessor token to WIN32. Note that defining WIN32 has the side effect of disabling Internet features of NCO (see below). NCO should now build like it does on UNIX.

The least portable section of the code is the use of standard UNIX and Internet protocols (e.g., ftp, rcp, scp, sftp, getuid, gethostname, and header files ‘<arpa/nameser.h>’ and ‘<resolv.h>’). Fortunately, these UNIX-y calls are only invoked by the single NCO subroutine which is responsible for retrieving files stored on remote systems (see section Accessing Remote Files). In order to support NCO on the Microsoft Windows platforms, this single feature was disabled (on Windows OS only). This was required by Cygwin 18.x—newer versions of Cygwin may support these protocols (let me know if this is the case). The NCO operators should behave identically on Windows and UNIX platforms in all other respects.

1.3 Symbolic Links

NCO relies on a common set of underlying algorithms. To minimize duplication of source code, multiple operators sometimes share the same underlying source. This is accomplished by symbolic links from a single underlying executable program to one or more invoked executable names. For example, ncea and ncrcat are symbolically linked to the ncra executable. The ncra executable behaves slightly differently based on its invocation name (i.e., ‘argv[0]’), which can be ncea, ncra, or ncrcat. Logically, these are three different operators that happen to share the same executable.

For historical reasons, and to be more user friendly, multiple synonyms (or pseudonyms) may refer to the same operator invoked with different switches. For example, ncdiff is the same as ncbo and ncpack is the same as ncpdq. We implement the symbolic links and synonyms by the executing the following UNIX commands in the directory where the NCO executables are installed.

ln -s -f ncbo ncdiff    # ncbo --op_typ='+'
ln -s -f ncra ncecat    # ncra --pseudonym='ncecat'
ln -s -f ncra ncrcat    # ncra --pseudonym='ncrcat'
ln -s -f ncbo ncadd     # ncbo --op_typ='+'
ln -s -f ncbo ncsubtract # ncbo --op_typ='-'
ln -s -f ncbo ncmultiply # ncbo --op_typ='*'
ln -s -f ncbo ncdivide   # ncbo --op_typ='/'
ln -s -f ncpdq ncpack    # ncpdq
ln -s -f ncpdq ncunpack  # ncpdq --unpack
# NB: Cygwin executable (and link) names have an '.exe' suffix, e.g.,
ln -s -f ncbo.exe ncdiff.exe
...

The imputed command called by the link is given after the comment. As can be seen, some these links impute the passing of a command line argument to further modify the behavior of the underlying executable. For example, ncdivide is a pseudonym for ncbo --op_typ='/'.

1.4 Libraries

Like all executables, the NCO operators can be built using dynamic linking. This reduces the size of the executable and can result in significant performance enhancements on multiuser systems. Unfortunately, if your library search path (usually the LD_LIBRARY_PATH environment variable) is not set correctly, or if the system libraries have been moved, renamed, or deleted since NCO was installed, it is possible NCO operators will fail with a message that they cannot find a dynamically loaded (aka shared object or ‘.so’) library. This will produce a distinctive error message, such as ‘ld.so.1: /usr/local/bin/ncea: fatal: libsunmath.so.1: can't open file: errno=2’. If you received an error message like this, ask your system administrator to diagnose whether the library is truly missing (5), or whether you simply need to alter your library search path. As a final remedy, you may re-compile and install NCO with all operators statically linked.

1.5 netCDF2/3/4 and HDF4/5 Support

netCDF version 2 was released in 1993. NCO (specifically ncks) began soon after this in 1994. netCDF 3.0 was released in 1996, and we were eager to reap the performance advantages of the newer netCDF implementation. One netCDF3 interface call (nc_inq_libvers) was added to NCO in January, 1998, to aid in maintainance and debugging. In March, 2001, the final conversion of NCO to netCDF3 was completed (coincidentally on the same day netCDF 3.5 was released). NCO versions 2.0 and higher are built with the -DNO_NETCDF_2 flag to ensure no netCDF2 interface calls are used.

However, the ability to compile NCO with only netCDF2 calls is worth maintaining because HDF version 4 (6) (available from HDF) supports only the netCDF2 library calls (see http://hdf.ncsa.uiuc.edu/UG41r3_html/SDS_SD.fm12.html#47784). Note that there are multiple versions of HDF. Currently HDF version 4.x supports netCDF2 and thus NCO version 1.2.x. If NCO version 1.2.x (or earlier) is built with only netCDF2 calls then all NCO operators should work with HDF4 files as well as netCDF files (7). The preprocessor token NETCDF2_ONLY exists in NCO version 1.2.x to eliminate all netCDF3 calls. Only versions of NCO numbered 1.2.x and earlier have this capability. The NCO 1.2.x branch will be maintained with bugfixes only (no new features) until HDF begins to fully support the netCDF3 interface (which is employed by NCO 2.x). If, at compilation time, NETCDF2_ONLY is defined, then NCO version 1.2.x will not use any netCDF3 calls and, if linked properly, the resulting NCO operators will work with HDF4 files. The ‘Makefile’ supplied with NCO 1.2.x is written to simplify building in this HDF capability. When NCO is built with make HDF4=Y, the ‘Makefile’ sets all required preprocessor flags and library links to build with the HDF4 libraries (which are assumed to reside under /usr/local/hdf4, edit the ‘Makefile’ to suit your installation).

HDF version 5 became available in 1999, but did not support netCDF (or, for that matter, Fortran) as of December 1999. By early 2001, HDF5 did support Fortran90. In 2004, Unidata and NCSA began a project to implement the HDF5 features necessary to support the netCDF API. NCO version 3.0.3 added support for reading/writing netCDF4-formatted HDF5 files in October, 2005. See Selecting Output File Format for more details.

HDF support for netCDF was completed with HDF5 version version 1.8 in 2007. The netCDF front-end that uses this HDF5 back-end was completed and released soon after as netCDF version 4. Download it from the netCDF4 website.

NCO version 3.9.0, released in May, 2007, added support for all netCDF4 atomic data types except NC_STRING. Support for NC_STRING, including ragged arrays of strings, was finally added in version 3.9.9, released in June, 2009. Support for additional netCDF4 features has been incremental. We add one netCDF4 feature at a time. You must build NCO with netCDF4 to obtain this support.

The main netCDF4 features that NCO currently supports are the new atomic data types, Lempel-Ziv compression (deflation), and chunking. The new atomic data types are NC_UBYTE, NC_USHORT, NC_UINT, NC_INT64, and NC_UINT64. Eight-byte integer support is an especially useful improvement from netCDF3. All NCO operators support these types, e.g., ncks copies and prints them, ncra averages them, and ncap2 processes algebraic scripts with them. ncks prints compression information, if any, to screen.

NCO version 3.9.1 (June, 2007) added support for netCDF4 Lempel-Ziv deflation. Lempel-Ziv deflation is a lossless compression technique. See Deflation for more details.

NCO version 3.9.9 (June, 2009) added support for netCDF4 chunking in ncks and ncecat. NCO version 4.0.4 (September, 2010) completed support for netCDF4 chunking in the remaining operators. See Chunking for more details.

netCDF4-enabled NCO handles netCDF3 files without change. In addition, it automagically handles netCDF4 (HDF5) files: If you feed NCO netCDF3 files, it produces netCDF3 output. If you feed NCO netCDF4 files, it produces netCDF4 output. Use the handy-dandy ‘-4’ switch to request netCDF4 output from netCDF3 input, i.e., to convert netCDF3 to netCDF4. See Selecting Output File Format for more details.

As of 2010, netCDF4 is still relatively new software. Problems with netCDF4 and HDF libraries are still being fixed. Binary NCO distributions shipped as RPMs use the netCDF4 library, while debs use the netCDF3 library, because of upstream requirements.

One must often build NCO from source to obtain netCDF4 support. Typically, one specifies the root of the netCDF4 installation directory. Do this with the NETCDF4_ROOT variable. Then use your preferred NCO build mechanism, e.g.,

export NETCDF4_ROOT=/usr/local/netcdf4 # Set netCDF4 location
cd ~/nco;./configure --enable-netcdf4  # Configure mechanism -or-
cd ~/nco/bld;./make NETCDF4=Y allinone # Old Makefile mechanism

We carefully track the netCDF4 releases, and keep the netCDF4 atomic type support and other features working. Our long term goal is to utilize more of the extensive new netCDF4 feature set. The next major netCDF4 feature we are likely to utilize is parallel I/O. We will enable this in the MPI netCDF operators.

1.6 Help Requests and Bug Reports

We generally receive three categories of mail from users: help requests, bug reports, and feature requests. Notes saying the equivalent of "Hey, NCO continues to work great and it saves me more time everyday than it took to write this note" are a distant fourth.

There is a different protocol for each type of request. The preferred etiquette for all communications is via NCO Project Forums. Do not contact project members via personal e-mail unless your request comes with money or you have damaging information about our personal lives. Please use the Forums—they preserve a record of the questions and answers so that others can learn from our exchange. Also, since NCO is government-funded, this record helps us provide program officers with information they need to evaluate our project.

Before posting to the NCO forums described below, you might first register your name and email address with SourceForge.net or else all of your postings will be attributed to "nobody". Once registered you may choose to "monitor" any forum and to receive (or not) email when there are any postings including responses to your questions. We usually reply to the forum message, not to the original poster.

If you want us to include a new feature in NCO, check first to see if that feature is already on the TODO list. If it is, why not implement that feature yourself and send us the patch? If the feature is not yet on the list, then send a note to the NCO Discussion forum.

Read the manual before reporting a bug or posting a help request. Sending questions whose answers are not in the manual is the best way to motivate us to write more documentation. We would also like to accentuate the contrapositive of this statement. If you think you have found a real bug the most helpful thing you can do is simplify the problem to a manageable size and then report it. The first thing to do is to make sure you are running the latest publicly released version of NCO.

Once you have read the manual, if you are still unable to get NCO to perform a documented function, submit a help request. Follow the same procedure as described below for reporting bugs (after all, it might be a bug). That is, describe what you are trying to do, and include the complete commands (run with ‘-D 5’), error messages, and version of NCO (with ‘-r’). Post your help request to the NCO Help forum.

If you think you used the right command when NCO misbehaves, then you might have found a bug. Incorrect numerical answers are the highest priority. We usually fix those within one or two days. Core dumps and sementation violations receive lower priority. They are always fixed, eventually.

How do you simplify a problem that reveal a bug? Cut out extraneous variables, dimensions, and metadata from the offending files and re-run the command until it no longer breaks. Then back up one step and report the problem. Usually the file(s) will be very small, i.e., one variable with one or two small dimensions ought to suffice. Run the operator with ‘-r’ and then run the command with ‘-D 5’ to increase the verbosity of the debugging output. It is very important that your report contain the exact error messages and compile-time environment. Include a copy of your sample input file, or place one on a publically accessible location, of the file(s). Post the full bug report to the NCO Project buglist.

Build failures count as bugs. Our limited machine access means we cannot fix all build failures. The information we need to diagnose, and often fix, build failures are the three files output by GNU build tools, ‘nco.config.log.${GNU_TRP}.foo’, ‘nco.configure.${GNU_TRP}.foo’, and ‘nco.make.${GNU_TRP}.foo’. The file ‘configure.eg’ shows how to produce these files. Here ${GNU_TRP} is the "GNU architecture triplet", the chip-vendor-OS string returned by ‘config.guess’. Please send us your improvements to the examples supplied in ‘configure.eg’. The regressions archive at http://dust.ess.uci.edu/nco/rgr contains the build output from our standard test systems. You may find you can solve the build problem yourself by examining the differences between these files and your own.

2. Operator Strategies

2.1 Philosophy

The main design goal is command line operators which perform useful, scriptable operations on netCDF files. Many scientists work with models and observations which produce too much data to analyze in tabular format. Thus, it is often natural to reduce and massage this raw or primary level data into summary, or second level data, e.g., temporal or spatial averages. These second level data may become the inputs to graphical and statistical packages, and are often more suitable for archival and dissemination to the scientific community. NCO performs a suite of operations useful in manipulating data from the primary to the second level state. Higher level interpretive languages (e.g., IDL, Yorick, Matlab, NCL, Perl, Python), and lower level compiled languages (e.g., C, Fortran) can always perform any task performed by NCO, but often with more overhead. NCO, on the other hand, is limited to a much smaller set of arithmetic and metadata operations than these full blown languages.

Another goal has been to implement enough command line switches so that frequently used sequences of these operators can be executed from a shell script or batch file. Finally, NCO was written to consume the absolute minimum amount of system memory required to perform a given job. The arithmetic operators are extremely efficient; their exact memory usage is detailed in Memory Requirements.

2.2 Climate Model Paradigm

NCO was developed at NCAR to aid analysis and manipulation of datasets produced by General Circulation Models (GCMs). Datasets produced by GCMs share many features with all gridded scientific datasets and so provide a useful paradigm for the explication of the NCO operator set. Examples in this manual use a GCM paradigm because latitude, longitude, time, temperature and other fields related to our natural environment are as easy to visualize for the layman as the expert.

2.3 Temporary Output Files

NCO operators are designed to be reasonably fault tolerant, so that if there is a system failure or the user aborts the operation (e.g., with C-c), then no data are lost. The user-specified output-file is only created upon successful completion of the operation (8). This is accomplished by performing all operations in a temporary copy of output-file. The name of the temporary output file is constructed by appending .pid<process ID>.<operator name>.tmp to the user-specified output-file name. When the operator completes its task with no fatal errors, the temporary output file is moved to the user-specified output-file. Note the construction of a temporary output file uses more disk space than just overwriting existing files “in place” (because there may be two copies of the same file on disk until the NCO operation successfully concludes and the temporary output file overwrites the existing output-file). Also, note this feature increases the execution time of the operator by approximately the time it takes to copy the output-file. Finally, note this feature allows the output-file to be the same as the input-file without any danger of “overlap”.

Other safeguards exist to protect the user from inadvertently overwriting data. If the output-file specified for a command is a pre-existing file, then the operator will prompt the user whether to overwrite (erase) the existing output-file, attempt to append to it, or abort the operation. However, in processing large amounts of data, too many interactive questions slows productivity. Therefore NCO also implements two ways to override its own safety features, the ‘-O’ and ‘-A’ switches. Specifying ‘-O’ tells the operator to overwrite any existing output-file without prompting the user interactively. Specifying ‘-A’ tells the operator to attempt to append to any existing output-file without prompting the user interactively. These switches are useful in batch environments because they suppress interactive keyboard input.

2.4 Appending Variables

Adding variables from one file to another is often desirable. This is referred to as appending, although some prefer the terminology merging (9) or pasting. Appending is often confused with what NCO calls concatenation. In NCO, concatenation refers to splicing a variable along the record dimension. The length along the record dimension of the output is the sum of the lengths of the input files. Appending, on the other hand, refers to copying a variable from one file to another file which may or may not already contain the variable (10). NCO can append or concatenate just one variable, or all the variables in a file at the same time.

In this sense, ncks can append variables from one file to another file. This capability is invoked by naming two files on the command line, input-file and output-file. When output-file already exists, the user is prompted whether to overwrite, append/replace, or exit from the command. Selecting overwrite tells the operator to erase the existing output-file and replace it with the results of the operation. Selecting exit causes the operator to exit—the output-file will not be touched in this case. Selecting append/replace causes the operator to attempt to place the results of the operation in the existing output-file, See section ncks netCDF Kitchen Sink.

The simplest way to create the union of two files is

ncks -A fl_1.nc fl_2.nc

This puts the contents of ‘fl_1.nc’ into ‘fl_2.nc’. The ‘-A’ is optional. On output, ‘fl_2.nc’ is the union of the input files, regardless of whether they share dimensions and variables, or are completely disjoint. The append fails if the input files have differently named record dimensions (since netCDF supports only one), or have dimensions of the same name but different sizes.

2.5 Simple Arithmetic and Interpolation

Users comfortable with NCO semantics may find it easier to perform some simple mathematical operations in NCO rather than higher level languages. ncbo (see section ncbo netCDF Binary Operator) does file addition, subtraction, multiplication, division, and broadcasting. ncflint (see section ncflint netCDF File Interpolator) does file addition, subtraction, multiplication and interpolation. Sequences of these commands can accomplish simple but powerful operations from the command line.

2.6 Averagers vs. Concatenators

The most frequently used operators of NCO are probably the averagers and concatenators. Because there are so many permutations of averaging (e.g., across files, within a file, over the record dimension, over other dimensions, with or without weights and masks) and of concatenating (across files, along the record dimension, along other dimensions), there are currently no fewer than five operators which tackle these two purposes: ncra, ncea, ncwa, ncrcat, and ncecat. These operators do share many capabilities (11), but each has its unique specialty. Two of these operators, ncrcat and ncecat, are for concatenating hyperslabs across files. The other two operators, ncra and ncea, are for averaging hyperslabs across files (12). First, let’s describe the concatenators, then the averagers.

2.6.1 Concatenators ncrcat and ncecat

Joining independent files together along a record dimension is called concatenation. ncrcat is designed for concatenating record variables, while ncecat is designed for concatenating fixed length variables. Consider five files, ‘85.nc’, ‘86.nc’, … ‘89.nc’ each containing a year’s worth of data. Say you wish to create from them a single file, ‘8589.nc’ containing all the data, i.e., spanning all five years. If the annual files make use of the same record variable, then ncrcat will do the job nicely with, e.g., ncrcat 8?.nc 8589.nc. The number of records in the input files is arbitrary and can vary from file to file. See section ncrcat netCDF Record Concatenator, for a complete description of ncrcat.

However, suppose the annual files have no record variable, and thus their data are all fixed length. For example, the files may not be conceptually sequential, but rather members of the same group, or ensemble. Members of an ensemble may have no reason to contain a record dimension. ncecat will create a new record dimension (named record by default) with which to glue together the individual files into the single ensemble file. If ncecat is used on files which contain an existing record dimension, that record dimension is converted to a fixed-length dimension of the same name and a new record dimension (named record) is created. Consider five realizations, ‘85a.nc’, ‘85b.nc’, … ‘85e.nc’ of 1985 predictions from the same climate model. Then ncecat 85?.nc 85_ens.nc glues the individual realizations together into the single file, ‘85_ens.nc’. If an input variable was dimensioned [lat,lon], it will have dimensions [record,lat,lon] in the output file. A restriction of ncecat is that the hyperslabs of the processed variables must be the same from file to file. Normally this means all the input files are the same size, and contain data on different realizations of the same variables. See section ncecat netCDF Ensemble Concatenator, for a complete description of ncecat.

ncpdq makes it possible to concatenate files along any dimension, not just the record dimension. First, use ncpdq to convert the dimension to be concatenated (i.e., extended with data from other files) into the record dimension. Second, use ncrcat to concatenate these files. Finally, if desirable, use ncpdq to revert to the original dimensionality. As a concrete example, say that files ‘x_01.nc’, ‘x_02.nc’, … ‘x_10.nc’ contain time-evolving datasets from spatially adjacent regions. The time and spatial coordinates are time and x, respectively. Initially the record dimension is time. Our goal is to create a single file that contains joins all the spatially adjacent regions into one single time-evolving dataset.

for idx in 01 02 03 04 05 06 07 08 09 10; do # Bourne Shell
  ncpdq -a x,time x_${idx}.nc foo_${idx}.nc # Make x record dimension
done
ncrcat foo_??.nc out.nc       # Concatenate along x
ncpdq -a time,x out.nc out.nc # Revert to time as record dimension

Note that ncrcat will not concatenate fixed-length variables, whereas ncecat concatenates both fixed-length and record variables along a new record variable. To conserve system memory, use ncrcat where possible.

2.6.2 Averagers ncea, ncra, and ncwa

The differences between the averagers ncra and ncea are analogous to the differences between the concatenators. ncra is designed for averaging record variables from at least one file, while ncea is designed for averaging fixed length variables from multiple files. ncra performs a simple arithmetic average over the record dimension of all the input files, with each record having an equal weight in the average. ncea performs a simple arithmetic average of all the input files, with each file having an equal weight in the average. Note that ncra cannot average fixed-length variables, but ncea can average both fixed-length and record variables. To conserve system memory, use ncra rather than ncea where possible (e.g., if each input-file is one record long). The file output from ncea will have the same dimensions (meaning dimension names as well as sizes) as the input hyperslabs (see section ncea netCDF Ensemble Averager, for a complete description of ncea). The file output from ncra will have the same dimensions as the input hyperslabs except for the record dimension, which will have a size of 1 (see section ncra netCDF Record Averager, for a complete description of ncra).

2.6.3 Interpolator ncflint

ncflint can interpolate data between or two files. Since no other operators have this ability, the description of interpolation is given fully on the ncflint reference page (see section ncflint netCDF File Interpolator). Note that this capability also allows ncflint to linearly rescale any data in a netCDF file, e.g., to convert between differing units.

2.7 Large Numbers of Files

Occasionally one desires to digest (i.e., concatenate or average) hundreds or thousands of input files. Unfortunately, data archives (e.g., NASA EOSDIS) may not name netCDF files in a format understood by the ‘-n loop’ switch (see section Specifying Input Files) that automagically generates arbitrary numbers of input filenames. The ‘-n loop’ switch has the virtue of being concise, and of minimizing the command line. This helps keeps output file small since the command line is stored as metadata in the history attribute (see section History Attribute). However, the ‘-n loop’ switch is useless when there is no simple, arithmetic pattern to the input filenames (e.g., ‘h00001.nc’, ‘h00002.nc’, … ‘h90210.nc’). Moreover, filename globbing does not work when the input files are too numerous or their names are too lengthy (when strung together as a single argument) to be passed by the calling shell to the NCO operator (13). When this occurs, the ANSI C-standard argc-argv method of passing arguments from the calling shell to a C-program (i.e., an NCO operator) breaks down. There are (at least) three alternative methods of specifying the input filenames to NCO in environment-limited situations.

The recommended method for sending very large numbers (hundreds or more, typically) of input filenames to the multi-file operators is to pass the filenames with the UNIX standard input feature, aka stdin:

# Pipe large numbers of filenames to stdin
/bin/ls | grep ${CASEID}_'......'.nc | ncecat -o foo.nc

This method avoids all constraints on command line size imposed by the operating system. A drawback to this method is that the history attribute (see section History Attribute) does not record the name of any input files since the names were not passed on the command line. This makes determining the data provenance at a later date difficult. To remedy this situation, multi-file operators store the number of input files in the nco_input_file_number global attribute and the input file list itself in the nco_input_file_list global attribute (see section File List Attributes). Although this does not preserve the exact command used to generate the file, it does retains all the information required to reconstruct the command and determine the data provenance.

A second option is to use the UNIX xargs command. This simple example selects as input to xargs all the filenames in the current directory that match a given pattern. For illustration, consider a user trying to average millions of files which each have a six character filename. If the shell buffer can not hold the results of the corresponding globbing operator, ‘??????.nc’, then the filename globbing technique will fail. Instead we express the filename pattern as an extended regular expression, ‘......\.nc’ (see section Subsetting Variables). We use grep to filter the directory listing for this pattern and to pipe the results to xargs which, in turn, passes the matching filenames to an NCO multi-file operator, e.g., ncecat.

# Use xargs to transfer filenames on the command line
/bin/ls | grep ${CASEID}_'......'.nc | xargs -x ncecat -o foo.nc

The single quotes protect the only sensitive parts of the extended regular expression (the grep argument), and allow shell interpolation (the ${CASEID} variable substitution) to proceed unhindered on the rest of the command. xargs uses the UNIX pipe feature to append the suitably filtered input file list to the end of the ncecat command options. The -o foo.nc switch ensures that the input files supplied by xargs are not confused with the output file name. xargs does, unfortunately, have its own limit (usually about 20,000 characters) on the size of command lines it can pass. Give xargs the ‘-x’ switch to ensure it dies if it reaches this internal limit. When this occurs, use either the stdin method above, or the symbolic link presented next.

Even when its internal limits have not been reached, the xargs technique may not be sophisticated enough to handle all situations. A full scripting language like Perl can handle any level of complexity of filtering input filenames, and any number of filenames. The technique of last resort is to write a script that creates symbolic links between the irregular input filenames and a set of regular, arithmetic filenames that the ‘-n loop’ switch understands. For example, the following Perl script a monotonically enumerated symbolic link to up to one million ‘.nc’ files in a directory. If there are 999,999 netCDF files present, the links are named ‘000001.nc’ to ‘999999.nc’:

# Create enumerated symbolic links
/bin/ls | grep \.nc | perl -e \
'$idx=1;while(<STDIN>){chop;symlink $_,sprintf("%06d.nc",$idx++);}'
ncecat -n 999999,6,1 000001.nc foo.nc
# Remove symbolic links when finished
/bin/rm ??????.nc

The ‘-n loop’ option tells the NCO operator to automatically generate the filnames of the symbolic links. This circumvents any OS and shell limits on command line size. The symbolic links are easily removed once NCO is finished. One drawback to this method is that the history attribute (see section History Attribute) retains the filename list of the symbolic links, rather than the data files themselves. This makes it difficult to determine the data provenance at a later date.

2.8 Large Datasets

Large datasets are those files that are comparable in size to the amount of random access memory (RAM) in your computer. Many users of NCO work with files larger than 100 MB. Files this large not only push the current edge of storage technology, they present special problems for programs which attempt to access the entire file at once, such as ncea and ncecat. If you work with a 300 MB files on a machine with only 32 MB of memory then you will need large amounts of swap space (virtual memory on disk) and NCO will work slowly, or even fail. There is no easy solution for this. The best strategy is to work on a machine with sufficient amounts of memory and swap space. Since about 2004, many users have begun to produce or analyze files exceeding 2 GB in size. These users should familiarize themselves with NCO’s Large File Support (LFS) capabilities (see section Large File Support). The next section will increase your familiarity with NCO’s memory requirements. With this knowledge you may re-design your data reduction approach to divide the problem into pieces solvable in memory-limited situations.

If your local machine has problems working with large files, try running NCO from a more powerful machine, such as a network server. Certain machine architectures, e.g., Cray UNICOS, have special commands which allow one to increase the amount of interactive memory. On Cray systems, try to increase the available memory with the ilimit command. If you get a memory-related core dump (e.g., ‘Error exit (core dumped)’) on a GNU/Linux system, try increasing the process-available memory with ulimit.

The speed of the NCO operators also depends on file size. When processing large files the operators may appear to hang, or do nothing, for large periods of time. In order to see what the operator is actually doing, it is useful to activate a more verbose output mode. This is accomplished by supplying a number greater than 0 to the ‘-D debug-level’ (or ‘--debug-level’, or ‘--dbg_lvl’) switch. When the debug-level is nonzero, the operators report their current status to the terminal through the stderr facility. Using ‘-D’ does not slow the operators down. Choose a debug-level between 1 and 3 for most situations, e.g., ncea -D 2 85.nc 86.nc 8586.nc. A full description of how to estimate the actual amount of memory the multi-file NCO operators consume is given in Memory Requirements.

2.9 Memory Requirements

Many people use NCO on gargantuan files which dwarf the memory available (free RAM plus swap space) even on today’s powerful machines. These users want NCO to consume the least memory possible so that their scripts do not have to tediously cut files into smaller pieces that fit into memory. We commend these greedy users for pushing NCO to its limits!

This section describes the memory NCO requires during operation. The required memory is based on the underlying algorithms. The description below is the memory usage per thread. Users with shared memory machines may use the threaded NCO operators (see section OpenMP Threading). The peak and sustained memory usage will scale accordingly, i.e., by the number of threads. Memory consumption patterns of all operators are similar, with the exception of ncap2.

2.9.1 Single and Multi-file Operators

The multi-file operators currently comprise the record operators, ncra and ncrcat, and the ensemble operators, ncea and ncecat. The record operators require much less memory than the ensemble operators. This is because the record operators operate on one single record (i.e., time-slice) at a time, wherease the ensemble operators retrieve the entire variable into memory. Let MS be the peak sustained memory demand of an operator, FT be the memory required to store the entire contents of all the variables to be processed in an input file, FR be the memory required to store the entire contents of a single record of each of the variables to be processed in an input file, VR be the memory required to store a single record of the largest record variable to be processed in an input file, VT be the memory required to store the largest variable to be processed in an input file, VI be the memory required to store the largest variable which is not processed, but is copied from the initial file to the output file. All operators require MI = VI during the initial copying of variables from the first input file to the output file. This is the initial (and transient) memory demand. The sustained memory demand is that memory required by the operators during the processing (i.e., averaging, concatenation) phase which lasts until all the input files have been processed. The operators have the following memory requirements: ncrcat requires MS <= VR. ncecat requires MS <= VT. ncra requires MS = 2FR + VR. ncea requires MS = 2FT + VT. ncbo requires MS <= 3VT (both input variables and the output variable). ncflint requires MS <= 3VT (both input variables and the output variable). ncpdq requires MS <= 2VT (one input variable and the output variable). ncwa requires MS <= 8VT (see below). Note that only variables that are processed, e.g., averaged, concatenated, or differenced, contribute to MS. Variables which do not appear in the output file (see section Subsetting Variables) are never read and contribute nothing to the memory requirements.

ncwa consumes between two and seven times the memory of a variable in order to process it. Peak consumption occurs when storing simultaneously in memory one input variable, one tally array, one input weight, one conformed/working weight, one weight tally, one input mask, one conformed/working mask, and one output variable. When invoked, the weighting and masking features contribute up to three-sevenths and two-sevenths of these requirements apiece. If weights and masks are not specified (i.e., no ‘-w’ or ‘-a’ options) then ncwa requirements drop to MS <= 3VT (one input variable, one tally array, and the output variable).

The above memory requirements must be multiplied by the number of threads thr_nbr (see section OpenMP Threading). If this causes problems then reduce (with ‘-t thr_nbr’) the number of threads.

2.9.2 Memory for ncap2

ncap2 has unique memory requirements due its ability to process arbitrarily long scripts of any complexity. All scripts acceptable to ncap2 are ultimately processed as a sequence of binary or unary operations. ncap2 requires MS <= 2VT under most conditions. An exception to this is when left hand casting (see section Left hand casting) is used to stretch the size of derived variables beyond the size of any input variables. Let VC be the memory required to store the largest variable defined by left hand casting. In this case, MS <= 2VC.

ncap2 scripts are complete dynamic and may be of arbitrary length. A script that contains many thousands of operations, may uncover a slow memory leak even though each single operation consumes little additional memory. Memory leaks are usually identifiable by their memory usage signature. Leaks cause peak memory usage to increase monotonically with time regardless of script complexity. Slow leaks are very difficult to find. Sometimes a malloc() (or new[]) failure is the only noticeable clue to their existance. If you have good reasons to believe that a memory allocation failure is ultimately due to an NCO memory leak (rather than inadequate RAM on your system), then we would be very interested in receiving a detailed bug report.

2.10 Performance

An overview of NCO capabilities as of about 2006 is in Zender, C. S. (2008), “Analysis of Self-describing Gridded Geoscience Data with netCDF Operators (NCO)”, Environ. Modell. Softw., doi:10.1016/j.envsoft.2008.03.004. This paper is also available at http://dust.ess.uci.edu/ppr/ppr_Zen08_ems.pdf.

NCO performance and scaling for arithmetic operations is described in Zender, C. S., and H. J. Mangalam (2007), “Scaling Properties of Common Statistical Operators for Gridded Datasets”, Int. J. High Perform. Comput. Appl., 21(4), 485-498, doi:10.1177/1094342007083802. This paper is also available at http://dust.ess.uci.edu/ppr/ppr_ZeM07_ijhpca.pdf.

It is helpful to be aware of the aspects of NCO design that can limit its performance:

1. No data buffering is performed during nc_get_var and nc_put_var operations. Hyperslabs too large too hold in core memory will suffer substantial performance penalties because of this.
2. Since coordinate variables are assumed to be monotonic, the search for bracketing the user-specified limits should employ a quicker algorithm, like bisection, than the two-sided incremental search currently implemented.
3. C_format, FORTRAN_format, signedness, scale_format and add_offset attributes are ignored by ncks when printing variables to screen.
4. In the late 1990s it was discovered that some random access operations on large files on certain architectures (e.g., UNICOS) were much slower with NCO than with similar operations performed using languages that bypass the netCDF interface (e.g., Yorick). This may have been a penalty of unnecessary byte-swapping in the netCDF interface. It is unclear whether such problems exist in present day (2007) netCDF/NCO environments, where unnecessary byte-swapping has been reduced or eliminated.

3. NCO Features

Many features have been implemented in more than one operator and are described here for brevity. The description of each feature is preceded by a box listing the operators for which the feature is implemented. Command line switches for a given feature are consistent across all operators wherever possible. If no “key switches” are listed for a feature, then that particular feature is automatic and cannot be controlled by the user.

3.1 Internationalization

NCO support for internationalization of textual input and output (e.g., Warning messages) is nascent. We hope to produce foreign language string catalogues in 2004.

3.2 Metadata Optimization

NCO supports padding headers to improve the speed of future metadata operations. Use the ‘--hdr_pad’ and ‘--header_pad’ switches to request that hdr_pad bytes be inserted into the metadata section of the output file. Future metadata expansions will not incur the performance penalty of copying the entire output file unless the expansion exceeds the amount of header padding exceeded. This can be beneficial when it is known that some metadata will be added at a future date.

This optimization exploits the netCDF library nc__enddef() function, which behaves differently with different versions of netCDF. It will improve speed of future metadata expansion with CLASSIC and 64bit netCDF files, but not necessarily with NETCDF4 files, i.e., those created by the netCDF interface to the HDF5 library (see section Selecting Output File Format).

3.3 OpenMP Threading

NCO supports shared memory parallelism (SMP) when compiled with an OpenMP-enabled compiler. Threads requests and allocations occur in two stages. First, users may request a specific number of threads thr_nbr with the ‘-t’ switch (or its long option equivalents, ‘--thr_nbr’, ‘--threads’, and ‘--omp_num_threads’). If not user-specified, OpenMP obtains thr_nbr from the OMP_NUM_THREADS environment variable, if present, or from the OS, if not.

NCO may modify thr_nbr according to its own internal settings before it requests any threads from the system. Certain operators contain hard-code limits to the number of threads they request. We base these limits on our experience and common sense, and to reduce potentially wasteful system usage by inexperienced users. For example, ncrcat is extremely I/O-intensive so we restrict thr_nbr <= 2 for ncrcat. This is based on the notion that the best performance that can be expected from an operator which does no arithmetic is to have one thread reading and one thread writing simultaneously. In the future (perhaps with netCDF4), we hope to demonstrate significant threading improvements with operators like ncrcat by performing multiple simultaneous writes.

Compute-intensive operators (ncap, ncwa and ncpdq) benefit most from threading. The greatest increases in throughput due to threading occur on large datasets where each thread performs millions, at least, of floating point operations. Otherwise, the system overhead of setting up threads probably outweighs the speed enhancements due to SMP parallelism. However, we have not yet demonstrated that the SMP parallelism scales well beyond four threads for these operators. Hence we restrict thr_nbr <= 4 for all operators. We encourage users to play with these limits (edit file ‘nco_omp.c’) and send us their feedback.

Once the initial thr_nbr has been modified for any operator-specific limits, NCO requests the system to allocate a team of thr_nbr threads for the body of the code. The operating system then decides how many threads to allocate based on this request. Users may keep track of this information by running the operator with dbg_lvl > 0.

By default, threaded operators attach one global attribute, nco_openmp_thread_number, to any file they create or modify. This attribute contains the number of threads the operator used to process the input files. This information helps to verify that the answers with threaded and non-threaded operators are equal to within machine precision. This information is also useful for benchmarking.

3.4 Command Line Options

NCO achieves flexibility by using command line options. These options are implemented in all traditional UNIX commands as single letter switches, e.g., ‘ls -l’. For many years NCO used only single letter option names. In late 2002, we implemented GNU/POSIX extended or long option names for all options. This was done in a backward compatible way such that the full functionality of NCO is still available through the familiar single letter options. In the future, however, some features of NCO may require the use of long options, simply because we have nearly run out of single letter options. More importantly, mnemonics for single letter options are often non-intuitive so that long options provide a more natural way of expressing intent.

Extended options, also called long options, are implemented using the system-supplied ‘getopt.h’ header file, if possible. This provides the getopt_long function to NCO (14).

The syntax of short options (single letter options) is -key value (dash-key-space-value). Here, key is the single letter option name, e.g., ‘-D 2’.

The syntax of long options (multi-letter options) is --long_name value (dash-dash-key-space-value), e.g., ‘--dbg_lvl 2’ or --long_name=value (dash-dash-key-equal-value), e.g., ‘--dbg_lvl=2’. Thus the following are all valid for the ‘-D’ (short version) or ‘--dbg_lvl’ (long version) command line option.

ncks -D 3 in.nc        # Short option
ncks --dbg_lvl=3 in.nc # Long option, preferred form
ncks --dbg_lvl 3 in.nc # Long option, alternate form

The last example is preferred for two reasons. First, ‘--dbg_lvl’ is more specific and less ambiguous than ‘-D’. The long option form makes scripts more self documenting and less error prone. Often long options are named after the source code variable whose value they carry. Second, the equals sign = joins the key (i.e., long_name) to the value in an uninterruptible text block. Experience shows that users are less likely to mis-parse commands when restricted to this form.

GNU implements a superset of the POSIX standard which allows any unambiguous truncation of a valid option to be used.

ncks -D 3 in.nc        # Short option
ncks --dbg_lvl=3 in.nc # Long option, full form
ncks --dbg=3 in.nc     # Long option, unambiguous truncation
ncks --db=3 in.nc      # Long option, unambiguous truncation
ncks --d=3 in.nc       # Long option, ambiguous truncation

The first four examples are equivalent and will work as expected. The final example will exit with an error since ncks cannot disambiguate whether ‘--d’ is intended as a truncation of ‘--dbg_lvl’, of ‘--dimension’, or of some other long option.

NCO provides many long options for common switches. For example, the debugging level may be set in all operators with any of the switches ‘-D’, ‘--debug-level’, or ‘--dbg_lvl’. This flexibility allows users to choose their favorite mnemonic. For some, it will be ‘--debug’ (an unambiguous truncation of ‘--debug-level’, and other will prefer ‘--dbg’. Interactive users usually prefer the minimal amount of typing, i.e., ‘-D’. We recommend that scripts which are re-usable employ some form of the long options for future maintainability.

This manual generally uses the short option syntax. This is for historical reasons and to conserve space. The remainder of this manual specifies the full long_name of each option. Users are expected to pick the unambiguous truncation of each option name that most suits their taste.

3.5 Specifying Input Files

It is important that users be able to specify multiple input files without typing every filename in full, often a tedious task even by graduate student standards. There are four different ways of specifying input files to NCO: explicitly typing each, using UNIX shell wildcards, and using the NCO ‘-n’ and ‘-p’ switches (or their long option equivalents, ‘--nintap’ or ‘--pth’ and ‘--path’, respectively). Techniques to augment these methods to specify arbitrary numbers (e.g., thousands) and patterns of filenames are discussed separately (see section Large Numbers of Files).

To illustrate these methods, consider the simple problem of using ncra to average five input files, ‘85.nc’, ‘86.nc’, … ‘89.nc’, and store the results in ‘8589.nc’. Here are the four methods in order. They produce identical answers.

ncra 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncra 8[56789].nc 8589.nc
ncra -p input-path 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncra -n 5,2,1 85.nc 8589.nc

The first method (explicitly specifying all filenames) works by brute force. The second method relies on the operating system shell to glob (expand) the regular expression 8[56789].nc. The shell passes valid filenames which match the expansion to ncra. The third method uses the ‘-p input-path’ argument to specify the directory where all the input files reside. NCO prepends input-path (e.g., ‘/data/usrname/model’) to all input-files (but not to output-file). Thus, using ‘-p’, the path to any number of input files need only be specified once. Note input-path need not end with ‘/’; the ‘/’ is automatically generated if necessary.

The last method passes (with ‘-n’) syntax concisely describing the entire set of filenames (15). This option is only available with the multi-file operators: ncra, ncrcat, ncea, and ncecat. By definition, multi-file operators are able to process an arbitrary number of input-files. This option is very useful for abbreviating lists of filenames representable as alphanumeric_prefix+numeric_suffix+‘.’+filetype where alphanumeric_prefix is a string of arbitrary length and composition, numeric_suffix is a fixed width field of digits, and filetype is a standard filetype indicator. For example, in the file ‘ccm3_h0001.nc’, we have alphanumeric_prefix = ‘ccm3_h’, numeric_suffix = ‘0001’, and filetype = ‘nc’.

NCO is able to decode lists of such filenames encoded using the ‘-n’ option. The simpler (3-argument) ‘-n’ usage takes the form -n file_number,digit_number,numeric_increment where file_number is the number of files, digit_number is the fixed number of numeric digits comprising the numeric_suffix, and numeric_increment is the constant, integer-valued difference between the numeric_suffix of any two consecutive files. The value of alphanumeric_prefix is taken from the input file, which serves as a template for decoding the filenames. In the example above, the encoding -n 5,2,1 along with the input file name ‘85.nc’ tells NCO to construct five (5) filenames identical to the template ‘85.nc’ except that the final two (2) digits are a numeric suffix to be incremented by one (1) for each successive file. Currently filetype may be either be empty, ‘nc’, ‘cdf’, ‘hdf’, or ‘hd5’. If present, these filetype suffixes (and the preceding ‘.’) are ignored by NCO as it uses the ‘-n’ arguments to locate, evaluate, and compute the numeric_suffix component of filenames.

Recently the ‘-n’ option has been extended to allow convenient specification of filenames with “circular” characteristics. This means it is now possible for NCO to automatically generate filenames which increment regularly until a specified maximum value, and then wrap back to begin again at a specified minimum value. The corresponding ‘-n’ usage becomes more complex, taking one or two additional arguments for a total of four or five, respectively: -n file_number,digit_number,numeric_increment[,numeric_max[,numeric_min]] where numeric_max, if present, is the maximum integer-value of numeric_suffix and numeric_min, if present, is the minimum integer-value of numeric_suffix. Consider, for example, the problem of specifying non-consecutive input files where the filename suffixes end with the month index. In climate modeling it is common to create summertime and wintertime averages which contain the averages of the months June–July–August, and December–January–February, respectively:

ncra -n 3,2,1 85_06.nc 85_0608.nc
ncra -n 3,2,1,12 85_12.nc 85_1202.nc
ncra -n 3,2,1,12,1 85_12.nc 85_1202.nc

The first example shows that three arguments to the ‘-n’ option suffice to specify consecutive months (06, 07, 08) which do not “wrap” back to a minimum value. The second example shows how to use the optional fourth and fifth elements of the ‘-n’ option to specify a wrap value to NCO. The fourth argument to ‘-n’, if present, specifies the maximum integer value of numeric_suffix. In this case the maximum value is 12, and will be formatted as ‘12’ in the filename string. The fifth argument to ‘-n’, if present, specifies the minimum integer value of numeric_suffix. The default minimum filename suffix is 1, which is formatted as ‘01’ in this case. Thus the second and third examples have the same effect, that is, they automatically generate, in order, the filenames ‘85_12.nc’, ‘85_01.nc’, and ‘85_02.nc’ as input to NCO.

3.6 Specifying Output Files

NCO commands produce no more than one output file, fl_out. Traditionally, users specify fl_out as the final argument to the operator, following all input file names. This is the positional argument method of specifying input and ouput file names. The positional argument method works well in most applications. NCO also supports specifying fl_out using the command line switch argument method, ‘-o fl_out’.

Specifying fl_out with a switch, rather than as a positional argument, allows fl_out to precede input files in the argument list. This is particularly useful with multi-file operators for three reasons. Multi-file operators may be invoked with hundreds (or more) filenames. Visual or automatic location of fl_out in such a list is difficult when the only syntactic distinction between input and output files is their position. Second, specification of a long list of input files may be difficult (see section Large Numbers of Files). Making the input file list the final argument to an operator facilitates using xargs for this purpose. Some alternatives to xargs are very ugly and undesirable. Finally, many users are more comfortable specifying output files with ‘-o fl_out’ near the beginning of an argument list. Compilers and linkers are usually invoked this way.

Users should specify fl_out using either but not both methods. If fl_out is specified twice (once with the switch and once as the last positional argument), then the positional argument takes precedence.

3.7 Accessing Remote Files

All NCO operators can retrieve files from remote sites as well as from the local file system. A remote site can be an anonymous FTP server, a machine on which the user has rcp, scp, or sftp privileges, or NCAR’s Mass Storage System (MSS), or an OPeNDAP server. Examples of each are given below, following a brief description of the particular access protocol.

To access a file via an anonymous FTP server, supply the remote file’s URL. FTP is an intrinsically insecure protocol because it transfers passwords in plain text format. Users should access sites using anonymous FTP when possible. Some FTP servers require a login/password combination for a valid user account. NCO allows these transactions so long as the required information is stored in the ‘.netrc’ file. Usually this information is the remote machine name, login, and password, in plain text, separated by those very keywords, e.g.,

machine dust.ess.uci.edu login zender password bushlied

Eschew using valuable passwords for FTP transactions, since ‘.netrc’ passwords are potentially exposed to eavesdropping software (16).

SFTP, i.e., secure FTP, uses SSH-based security protocols that solve the security issues associated with plain FTP. NCO supports SFTP protocol access to files specified with a homebrew syntax of the form

sftp://machine.domain.tld:/path/to/filename

Note the second colon following the top-level-domain (tld). This syntax is a hybrid between an FTP URL and a standard remote file syntax.

To access a file using rcp or scp, specify the Internet address of the remote file. Of course in this case you must have rcp or scp privileges which allow transparent (no password entry required) access to the remote machine. This means that ‘~/.rhosts’ or ‘~/ssh/authorized_keys’ must be set accordingly on both local and remote machines.

To access a file on a High Performance Storage System (HPSS) (such as that at NCAR, ECMWF, LANL, DKRZ, LLNL) specify the full HPSS pathname of the remote file. NCO will attempt to detect whether the local machine has direct (synchronous) HPSS access. In this case, NCO attempts to use the Hierarchical Storage Interface (HSI) command hsi get (17).

The following examples show how one might analyze files stored on remote systems.

ncks -l . ftp://dust.ess.uci.edu/pub/zender/nco/in.nc
ncks -l . sftp://dust.ess.uci.edu:/home/ftp/pub/zender/nco/in.nc
ncks -l . dust.ess.uci.edu:/home/zender/nco/data/in.nc
ncks -l . /ZENDER/nco/in.nc
ncks -l . /home/zender/nco/in.nc
ncks -l . http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in.nc 

The first example works verbatim if your system is connected to the Internet and is not behind a firewall. The second example works if you have sftp access to the machine dust.ess.uci.edu. The third example works if you have rcp or scp access to the machine dust.ess.uci.edu. The fourth and fifth examples work on NCAR computers with local access to the HPSS hsi get command (18). The sixth command works if your local version of NCO is OPeNDAP-enabled (this is fully described in OPeNDAP), or if the remote file is accessible via wget. The above commands can be rewritten using the ‘-p input-path’ option as follows:

ncks -p ftp://dust.ess.uci.edu/pub/zender/nco -l . in.nc
ncks -p sftp://dust.ess.uci.edu:/home/ftp/pub/zender/nco -l . in.nc
ncks -p dust.ess.uci.edu:/home/zender/nco -l . in.nc
ncks -p /ZENDER/nco -l . in.nc
ncks -p /home/zender/nco -l . in.nc # HPSS
ncks -p http://motherlode.ucar.edu:8080/thredds/dodsC/testdods \ 
     -l . in.nc

Using ‘-p’ is recommended because it clearly separates the input-path from the filename itself, sometimes called the stub. When input-path is not explicitly specified using ‘-p’, NCO internally generates an input-path from the first input filename. The automatically generated input-path is constructed by stripping the input filename of everything following the final ‘/’ character (i.e., removing the stub). The ‘-l output-path’ option tells NCO where to store the remotely retrieved file and the output file. Often the path to a remotely retrieved file is quite different than the path on the local machine where you would like to store the file. If ‘-l’ is not specified then NCO internally generates an output-path by simply setting output-path equal to input-path stripped of any machine names. If ‘-l’ is not specified and the remote file resides on the NCAR HPSS system, then the leading character of input-path, ‘/’, is also stripped from output-path. Specifying output-path as ‘-l ./’ tells NCO to store the remotely retrieved file and the output file in the current directory. Note that ‘-l .’ is equivalent to ‘-l ./’ though the latter is recommended as it is syntactically more clear.

3.7.1 OPeNDAP

The Distributed Oceanographic Data System (DODS) provides useful replacements for common data interface libraries like netCDF. The DODS versions of these libraries implement network transparent access to data via a client-server data access protocol that uses the HTTP protocol for communication. Although DODS-technology originated with oceanography data, it applyies to virtually all scientific data. In recognition of this, the data access protocol underlying DODS (which is what NCO cares about) has been renamed the Open-source Project for a Network Data Access Protocol, OPeNDAP. We use the terms DODS and OPeNDAP interchangeably, and often write OPeNDAP/DODS for now. In the future we will deprecate DODS in favor of DAP or OPeNDAP, as appropriate (19).

NCO may be DAP-enabled by linking NCO to the OPeNDAP libraries. This is described in the OPeNDAP documentation and automagically implemented in NCO build mechanisms (20). The ‘./configure’ mechanism automatically enables NCO as OPeNDAP clients if it can find the required OPeNDAP libraries (21). in the usual locations. The $DODS_ROOT environment variable may be used to override the default OPeNDAP library location at NCO compile-time. Building NCO with ‘bld/Makefile’ and the command make DODS=Y adds the (non-intuitive) commands to link to the OPeNDAP libraries installed in the $DODS_ROOT directory. The file ‘doc/opendap.sh’ contains a generic script intended to help users install OPeNDAP before building NCO. The documentation at the OPeNDAP Homepage is voluminous. Check there and on the DODS mail lists. to learn more about the extensive capabilities of OPeNDAP (22).

Once NCO is DAP-enabled the operators are OPeNDAP clients. All OPeNDAP clients have network transparent access to any files controlled by a OPeNDAP server. Simply specify the input file path(s) in URL notation and all NCO operations may be performed on remote files made accessible by a OPeNDAP server. This command tests the basic functionality of OPeNDAP-enabled NCO clients:

% ncks -O -o ~/foo.nc -C -H -v one -l /tmp \
  -p http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc
% ncks -H -v one ~/foo.nc
one = 1

The one = 1 outputs confirm (first) that ncks correctly retrieved data via the OPeNDAP protocol and (second) that ncks created a valid local copy of the subsetted remote file. With minor changes to the above command, netCDF4 can be used as both the input and output file format:

% ncks -4 -O -o ~/foo.nc -C -H -v one -l /tmp \
  -p http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in_4.nc
% ncks -H -v one ~/foo.nc
one = 1

And, of course, OPeNDAP-enabled NCO clients continue to support other, orthogonal features such as UDUnits (see section UDUnits Support):

% ncks -u -C -H -v wvl -d wvl,'0.4 micron','0.7 micron' \
  -p http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in_4.nc
% wvl[0]=5e-07 meter

The next command is a more advanced example which demonstrates the real power of OPeNDAP-enabled NCO clients. The ncwa client requests an equatorial hyperslab from remotely stored NCEP reanalyses data of the year 1969. The NOAA OPeNDAP server (hopefully!) serves these data. The local ncwa client then computes and stores (locally) the regional mean surface pressure (in Pa).

ncwa -C -a lat,lon,time -d lon,-10.,10. -d lat,-10.,10. -l /tmp -p \
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.dailyavgs/surface \
  pres.sfc.1969.nc ~/foo.nc

All with one command! The data in this particular input file also happen to be packed (see section Methods and functions), although this is completely transparent to the user since NCO automatically unpacks data before attempting arithmetic.

NCO obtains remote files from the OPeNDAP server (e.g., ‘www.cdc.noaa.gov’) rather than the local machine. Input files are first copied to the local machine, then processed. The OPeNDAP server performs data access, hyperslabbing, and transfer to the local machine. This allows the I/O to appear to NCO as if the input files were local. The local machine performs all arithmetic operations. Only the hyperslabbed output data are transferred over the network (to the local machine) for the number-crunching to begin. The advantages of this are obvious if you are examining small parts of large files stored at remote locations.

3.8 Retaining Retrieved Files

In order to conserve local file system space, files retrieved from remote locations are automatically deleted from the local file system once they have been processed. Many NCO operators were constructed to work with numerous large (e.g., 200 MB) files. Retrieval of multiple files from remote locations is done serially. Each file is retrieved, processed, then deleted before the cycle repeats. In cases where it is useful to keep the remotely-retrieved files on the local file system after processing, the automatic removal feature may be disabled by specifying ‘-R’ on the command line.

Invoking -R disables the default printing behavior of ncks. This allows ncks to retrieve remote files without automatically trying to print them. See ncks netCDF Kitchen Sink, for more details.

Note that the remote retrieval features of NCO can always be used to retrieve any file, including non-netCDF files, via SSH, anonymous FTP, or msrcp. Often this method is quicker than using a browser, or running an FTP session from a shell window yourself. For example, say you want to obtain a JPEG file from a weather server.

ncks -R -p ftp://weather.edu/pub/pix/jpeg -l . storm.jpg

In this example, ncks automatically performs an anonymous FTP login to the remote machine and retrieves the specified file. When ncks attempts to read the local copy of ‘storm.jpg’ as a netCDF file, it fails and exits, leaving ‘storm.jpg’ in the current directory.

If your NCO is DAP-enabled (see section OPeNDAP), then you may use NCO to retrieve any files (including netCDF, HDF, etc.) served by an OPeNDAP server to your local machine. For example,

ncks -R -l . -p \
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.dailyavgs/surface \
  pres.sfc.1969.nc

Note that NCO is never the preffered way to transport files from remote machines. For large jobs, that is best handled by FTP, SSH, or wget. It may occasionally be useful to use NCO to transfer files when your other preferred methods are not available locally.

3.9 Selecting Output File Format

All NCO operators support (read and write) all three (or four, depending on how one counts) file formats supported by netCDF4. The default output file format for all operators is the input file format. The operators listed under “Availability” above allow the user to specify the output file format independent of the input file format. These operators allow the user to convert between the various file formats. (The operators ncatted and ncrename do not support these switches so they always write the output netCDF file in the same format as the input netCDF file.)

netCDF supports four types of files: CLASSIC, 64BIT, NETCDF4, and NETCDF4_CLASSIC, The CLASSIC format is the traditional 32-bit offset written by netCDF2 and netCDF3. As of 2005, most netCDF datasets are in CLASSIC format. The 64BIT format was added in Fall, 2004.

The NETCDF4 format uses HDF5 as the file storage layer. The files are (usually) created, accessed, and manipulated using the traditional netCDF3 API (with numerous extensions). The NETCDF4_CLASSIC format refers to netCDF4 files created with the NC_CLASSIC_MODEL mask. Such files use HDF5 as the back-end storage format (unlike netCDF3), though they incorporate only netCDF3 features. Hence NETCDF4_CLASSIC files are perfectly readable by applications which use only the netCDF3 API and library. NCO must be built with netCDF4 to write files in the new NETCDF4 and NETCDF4_CLASSIC formats, and to read files in the new NETCDF4 format. Users are advised to use the default CLASSIC format or the NETCDF4_CLASSIC format until netCDF4 is more widespread. Widespread support for NETCDF4 format files is not expected for a few more years, 2013–2014, say. If performance or coolness are issues, then use NETCDF4_CLASSIC instead of CLASSIC format files.

As mentioned above, all operators write use the input file format for output files unless told otherwise. Toggling the long option ‘--64bit’ switch (or its key-value equivalent ‘--fl_fmt=64bit’) produces the netCDF3 64-bit offset format named 64BIT. NCO must be built with netCDF 3.6 or higher to produce a 64BIT file. Using the ‘-4’ switch (or its long option equivalents ‘--4’ or ‘--netcdf4’), or setting its key-value equivalent ‘--fl_fmt=netcdf4’ produces a NETCDF4 file (i.e., HDF). Casual users are advised to use the default (netCDF3) CLASSIC format until netCDF 3.6 and netCDF 4.0 are more widespread. Conversely, operators given the ‘-3’ (or ‘--3’) switch without arguments will (attempt to) produce netCDF3 CLASSIC output, even from netCDF4 input files.

These examples demonstrate converting a file from any netCDF format into any other netCDF format (subject to limits of the format):

ncks --fl_fmt=classic in.nc foo_3c.nc # netCDF3 classic
ncks --fl_fmt=64bit in.nc foo_364.nc # netCDF3 64bit
ncks --fl_fmt=netcdf4_classic in.nc foo_4c.nc # netCDF4 classic
ncks --fl_fmt=netcdf4 in.nc foo_4.nc # netCDF4 
ncks -3 in.nc foo_3c.nc # netCDF3 classic
ncks --3 in.nc foo_3c.nc # netCDF3 classic
ncks -4 in.nc foo_4.nc # netCDF4 
ncks --4 in.nc foo_4.nc # netCDF4 
ncks --64 in.nc foo364.nc # netCDF3 64bit

Of course since most operators support these switches, the “conversions” can be done at the output stage of arithmetic or metadata processing rather than requiring a separate step. Producing (netCDF3) CLASSIC or 64BIT files from NETCDF4_CLASSIC files will always work. However, producing netCDF3 files from NETCDF4 files will only work if the output files are not required to contain netCDF4-specific features.

Note that NETCDF4 and NETCDF4_CLASSIC are the same binary format. The latter simply causes a writing application to fail if it attempts to write a NETCDF4 file that cannot be completely read by the netCDF3 library. Conversely, NETCDF4_CLASSIC indicates to a reading application that all of the file contents are readable with the netCDF3 library. As of October, 2005, NCO writes no netCDF4-specific data structures and so always succeeds at writing NETCDF4_CLASSIC files.

There are at least three ways to discover the format of a netCDF file, i.e., whether it is a classic (32-bit offset) or newer 64-bit offset netCDF3 format, or is netCDF4 format. Each method returns the information using slightly different terminology that becomes easier to understand with practice.

First, examine the end of the first line of global metadata output by ‘ncks -M’:

% ncks -M foo_3c.nc
Opened file foo_3c.nc: dimensions = 19, variables = 261, global atts. = 4, 
  id = 65536, type = NC_FORMAT_CLASSIC 
% ncks -M foo_364.nc
Opened file foo_364.nc: dimensions = 19, variables = 261, global atts. = 4, 
  id = 65536, type = NC_FORMAT_64BIT
% ncks -M foo_4c.nc
Opened file foo_4c.nc: dimensions = 19, variables = 261, global atts. = 4,
  id = 65536, type = NC_FORMAT_NETCDF4_CLASSIC
% ncks -M foo_4.nc
Opened file foo_4.nc: dimensions = 19, variables = 261, global atts. = 4,
  id = 65536, type = NC_FORMAT_NETCDF4

This method requires a netCDF4-enabled NCO version 3.9.0+ (i.e., from 2007 or later).

Second, query the file with ‘ncdump -k’:

% ncdump -k foo_3.nc
classic
% ncdump -k foo_364.nc
64-bit-offset
% ncdump -k foo_4c.nc
netCDF-4 classic model
% ncdump -k foo_4.nc
netCDF-4

This method requires a netCDF4-enabled netCDF 3.6.2+ (i.e., from 2007 or later).

The third option uses the POSIX-standard od (octal dump) command:

% od -An -c -N4 foo_3c.nc
   C   D   F 001
% od -An -c -N4 foo_364.nc
   C   D   F 002
% od -An -c -N4 foo_4c.nc
 211   H   D   F
% od -An -c -N4 foo_4.nc
 211   H   D   F

This option works without NCO and ncdump. Values of ‘C D F 001’ and ‘C D F 002’ indicate 32-bit (classic) and 64-bit netCDF3 formats, respectively, while values of ‘211 H D F’ indicate the newer netCDF4 file format.

3.10 Large File Support

NCO has Large File Support (LFS), meaning that NCO can write files larger than 2 GB on some 32-bit operating systems with netCDF libraries earlier than version 3.6. If desired, LFS support must be configured when both netCDF and NCO are installed. netCDF versions 3.6 and higher support 64-bit file addresses as part of the netCDF standard. We recommend that users ignore LFS support which is difficult to configure and is implemented in NCO only to support netCDF versions prior to 3.6. This obviates the need for configuring explicit LFS support in applications (such as NCO) which now support 64-bit files directly through the netCDF interface. See Selecting Output File Format for instructions on accessing the different file formats, including 64-bit files, supported by the modern netCDF interface.

If you are still interesting in explicit LFS support for netCDF versions prior to 3.6, know that LFS support depends on a complex, interlocking set of operating system (23) and netCDF suppport issues. The netCDF LFS FAQ at http://my.unidata.ucar.edu/content/software/netcdf/faq-lfs.html describes the various file size limitations imposed by different versions of the netCDF standard. NCO and netCDF automatically attempt to configure LFS at build time.

3.11 Subsetting Variables

Subsetting variables refers to explicitly specifying variables to be included or excluded from operator actions. Subsetting is implemented with the ‘-v var[,…]’ and ‘-x’ options. A list of variables to extract is specified following the ‘-v’ option, e.g., ‘-v time,lat,lon’. Not using the ‘-v’ option is equivalent to specifying all variables. The ‘-x’ option causes the list of variables specified with ‘-v’ to be excluded rather than extracted. Thus ‘-x’ saves typing when you only want to extract fewer than half of the variables in a file.

Variables explicitly specified for extraction with ‘-v var[,…]’ must be present in the input file or an error will result. Variables explicitly specified for exclusion with ‘-x -v var[,…]’ need not be present in the input file. Remember, if averaging or concatenating large files stresses your systems memory or disk resources, then the easiest solution is often to use the ‘-v’ option to retain only the most important variables (see section Memory Requirements).

Due to its special capabilities, ncap2 interprets the ‘-v’ switch differently (see section ncap2 netCDF Arithmetic Processor). For ncap2, the ‘-v’ switch takes no arguments and indicates that only user-defined variables should be output. ncap2 neither accepts nor understands the -x switch.

As of NCO 2.8.1 (August, 2003), variable name arguments of the ‘-v’ switch may contain extended regular expressions. As of NCO 3.9.6 (January, 2009), variable names arguments to ncatted may contain extended regular expressions. For example, ‘-v '^DST'’ selects all variables beginning with the string ‘DST’. Extended regular expressions are defined by the GNU egrep command. The meta-characters used to express pattern matching operations are ‘^$+?.*[]{}|’. If the regular expression pattern matches any part of a variable name then that variable is selected. This capability is called wildcarding, and is very useful for sub-setting large data files.

Because of its wide availability, NCO uses the POSIX regular expression library regex. Regular expressions of arbitary complexity may be used. Since netCDF variable names are relatively simple constructs, only a few varieties of variable wildcards are likely to be useful. For convenience, we define the most useful pattern matching operators here:

‘^’

Matches the beginning of a string

‘$’

Matches the end of a string

‘.’

Matches any single character

The most useful repetition and combination operators are

‘?’

The preceding regular expression is optional and matched at most once

‘*’

The preceding regular expression will be matched zero or more times

‘+’

The preceding regular expression will be matched one or more times

‘|’

The preceding regular expression will be joined to the following regular expression. The resulting regular expression matches any string matching either subexpression.

To illustrate the use of these operators in extracting variables, consider a file with variables Q, Q01–Q99, Q100, QAA–QZZ, Q_H2O, X_H2O, Q_CO2, X_CO2.

ncks -v 'Q.?' in.nc              # Variables that contain Q
ncks -v '^Q.?' in.nc             # Variables that start with Q
ncks -v '^Q+.?.' in.nc           # Q, Q0--Q9, Q01--Q99, QAA--QZZ, etc.
ncks -v '^Q..' in.nc             # Q01--Q99, QAA--QZZ, etc.
ncks -v '^Q[0-9][0-9]' in.nc     # Q01--Q99, Q100
ncks -v '^Q[[:digit:]]{2}' in.nc # Q01--Q99
ncks -v 'H2O$' in.nc             # Q_H2O, X_H2O 
ncks -v 'H2O$|CO2$' in.nc        # Q_H2O, X_H2O, Q_CO2, X_CO2 
ncks -v '^Q[0-9][0-9]$' in.nc    # Q01--Q99
ncks -v '^Q[0-6][0-9]|7[0-3]' in.nc # Q01--Q73, Q100
ncks -v '(Q[0-6][0-9]|7[0-3])$' in.nc # Q01--Q73
ncks -v '^[a-z]_[a-z]{3}$' in.nc # Q_H2O, X_H2O, Q_CO2, X_CO2

Beware—two of the most frequently used repetition pattern matching operators, ‘*’ and ‘?’, are also valid pattern matching operators for filename expansion (globbing) at the shell-level. Confusingly, their meanings in extended regular expressions and in shell-level filename expansion are significantly different. In an extended regular expression, ‘*’ matches zero or more occurences of the preceding regular expression. Thus ‘Q*’ selects all variables, and ‘Q+.*’ selects all variables containing ‘Q’ (the ‘+’ ensures the preceding item matches at least once). To match zero or one occurence of the preceding regular expression, use ‘?’. Documentation for the UNIX egrep command details the extended regular expressions which NCO supports.

One must be careful to protect any special characters in the regular expression specification from being interpreted (globbed) by the shell. This is accomplish by enclosing special characters within single or double quotes

ncra -v Q?? in.nc out.nc   # Error: Shell attempts to glob wildcards
ncra -v '^Q+..' in.nc out.nc # Correct: NCO interprets wildcards
ncra -v '^Q+..' in*.nc out.nc # Correct: NCO interprets, Shell globs 

The final example shows that commands may use a combination of variable wildcarding and shell filename expansion (globbing). For globbing, ‘*’ and ‘?’ have nothing to do with the preceding regular expression! In shell-level filename expansion, ‘*’ matches any string, including the null string and ‘?’ matches any single character. Documentation for bash and csh describe the rules of filename expansion (globbing).

3.12 Subsetting Coordinate Variables

By default, coordinates variables associated with any variable appearing in the input-file will be placed in the output-file, even if they are not explicitly specified, e.g., with the ‘-v’ switch. Thus variables with a latitude coordinate lat always carry the values of lat with them into the output-file. This feature can be disabled with ‘-C’, which causes NCO to not automatically add coordinates to the variables appearing in the output-file. However, using ‘-C’ does not preclude the user from including some coordinates in the output files simply by explicitly selecting the coordinates with the -v option. The ‘-c’ option, on the other hand, is a shorthand way of automatically specifying that all coordinate variables in the input-files should appear in the output-file. Thus ‘-c’ allows the user to select all the coordinate variables without having to know their names. As of NCO version 3.9.6 (January, 2009) both ‘-c’ and ‘-C’ honor the CF coordinates convention described in CF Conventions. As of NCO version 4.0.8 (April, 2011) both ‘-c’ and ‘-C’ honor the CF bounds convention described in CF Conventions.

3.13 C and Fortran Index conventions

The ‘-F’ switch changes NCO to read and write with the Fortran index convention. By default, NCO uses C-style (0-based) indices for all I/O. In C, indices count from 0 (rather than 1), and dimensions are ordered from slowest (inner-most) to fastest (outer-most) varying. In Fortran, indices count from 1 (rather than 0), and dimensions are ordered from fastest (inner-most) to slowest (outer-most) varying. Hence C and Fortran data storage conventions represent mathematical transposes of eachother. Note that record variables contain the record dimension as the most slowly varying dimension. See ncpdq netCDF Permute Dimensions Quickly for techniques to re-order (including transpose) dimensions and to reverse data storage order.

Consider a file ‘85.nc’ containing 12 months of data in the record dimension time. The following hyperslab operations produce identical results, a June-July-August average of the data:

ncra -d time,5,7 85.nc 85_JJA.nc
ncra -F -d time,6,8 85.nc 85_JJA.nc

Printing variable three_dmn_var in file ‘in.nc’ first with the C indexing convention, then with Fortran indexing convention results in the following output formats:

% ncks -v three_dmn_var in.nc
lat[0]=-90 lev[0]=1000 lon[0]=-180 three_dmn_var[0]=0 
...
% ncks -F -v three_dmn_var in.nc
lon(1)=0 lev(1)=100 lat(1)=-90 three_dmn_var(1)=0 
...

3.14 Hyperslabs

A hyperslab is a subset of a variable’s data. The coordinates of a hyperslab are specified with the -d dim,[min][,[max][,[stride]]] short option (or with the same arguments to the ‘--dimension’ or ‘--dmn’ long options). At least one hyperslab argument (min, max, or stride) must be present. The bounds of the hyperslab to be extracted are specified by the associated min and max values. A half-open range is specified by omitting either the min or max parameter. The separating comma must be present to indicate the omission of one of these arguments. The unspecified limit is interpreted as the maximum or minimum value in the unspecified direction. A cross-section at a specific coordinate is extracted by specifying only the min limit and omitting a trailing comma. Dimensions not mentioned are passed with no reduction in range. The dimensionality of variables is not reduced (in the case of a cross-section, the size of the constant dimension will be one). If values of a coordinate-variable are used to specify a range or cross-section, then the coordinate variable must be monotonic (values either increasing or decreasing). In this case, command-line values need not exactly match coordinate values for the specified dimension. Ranges are determined by seeking the first coordinate value to occur in the closed range [min,max] and including all subsequent values until one falls outside the range. The coordinate value for a cross-section is the coordinate-variable value closest to the specified value and must lie within the range or coordinate-variable values.

Coordinate values should be specified using real notation with a decimal point required in the value, whereas dimension indices are specified using integer notation without a decimal point. This convention serves only to differentiate coordinate values from dimension indices. It is independent of the type of any netCDF coordinate variables. For a given dimension, the specified limits must both be coordinate values (with decimal points) or dimension indices (no decimal points). The stride option, if any, must be a dimension index, not a coordinate value. See section Stride, for more information on the stride option.

User-specified coordinate limits are promoted to double precision values while searching for the indices which bracket the range. Thus, hyperslabs on coordinates of type NC_BYTE and NC_CHAR are computed numerically rather than lexically, so the results are unpredictable.

The relative magnitude of min and max indicate to the operator whether to expect a wrapped coordinate (see section Wrapped Coordinates), such as longitude. If min > max, the NCO expects the coordinate to be wrapped, and a warning message will be printed. When this occurs, NCO selects all values outside the domain [max < min], i.e., all the values exclusive of the values which would have been selected if min and max were swapped. If this seems confusing, test your command on just the coordinate variables with ncks, and then examine the output to ensure NCO selected the hyperslab you expected (coordinate wrapping is currently only supported by ncks).

Because of the way wrapped coordinates are interpreted, it is very important to make sure you always specify hyperslabs in the monotonically increasing sense, i.e., min < max (even if the underlying coordinate variable is monotonically decreasing). The only exception to this is when you are indeed specifying a wrapped coordinate. The distinction is crucial to understand because the points selected by, e.g., -d longitude,50.,340., are exactly the complement of the points selected by -d longitude,340.,50..

Not specifying any hyperslab option is equivalent to specifying full ranges of all dimensions. This option may be specified more than once in a single command (each hyperslabbed dimension requires its own -d option).

3.15 Stride

All data operators support specifying a stride for any and all dimensions at the same time. The stride is the spacing between consecutive points in a hyperslab. A stride of 1 picks all the elements of the hyperslab, and a stride of 2 skips every other element, etc.. ncks multislabs support strides, and are more powerful than the regular hyperslabs supported by the other operators (see section Multislabs). Using the stride option for the record dimension with ncra and ncrcat makes it possible, for instance, to average or concatenate regular intervals across multi-file input data sets.

The stride is specified as the optional fourth argument to the ‘-d’ hyperslab specification: -d dim,[min][,[max][,[stride]]]. Specify stride as an integer (i.e., no decimal point) following the third comma in the ‘-d’ argument. There is no default value for stride. Thus using ‘-d time,,,2’ is valid but ‘-d time,,,2.0’ and ‘-d time,,,’ are not. When stride is specified but min is not, there is an ambiguity as to whether the extracted hyperslab should begin with (using C-style, 0-based indexes) element 0 or element ‘stride-1’. NCO must resolve this ambiguity and it chooses element 0 as the first element of the hyperslab when min is not specified. Thus ‘-d time,,,stride’ is syntactically equivalent to ‘-d time,0,,stride’. This means, for example, that specifying the operation ‘-d time,,,2’ on the array ‘1,2,3,4,5’ selects the hyperslab ‘1,3,5’. To obtain the hyperslab ‘2,4’ instead, simply explicitly specify the starting index as 1, i.e., ‘-d time,1,,2’.

For example, consider a file ‘8501_8912.nc’ which contains 60 consecutive months of data. Say you wish to obtain just the March data from this file. Using 0-based subscripts (see section C and Fortran Index conventions) these data are stored in records 2, 14, … 50 so the desired stride is 12. Without the stride option, the procedure is very awkward. One could use ncks five times and then use ncrcat to concatenate the resulting files together:

for idx in 02 14 26 38 50; do # Bourne Shell
  ncks -d time,${idx} 8501_8912.nc foo.${idx}
done
foreach idx (02 14 26 38 50) # C Shell
  ncks -d time,${idx} 8501_8912.nc foo.${idx}
end
ncrcat foo.?? 8589_03.nc
rm foo.??

With the stride option, ncks performs this hyperslab extraction in one operation:

ncks -d time,2,,12 8501_8912.nc 8589_03.nc

See section ncks netCDF Kitchen Sink, for more information on ncks.

Applying the stride option to the record dimension in ncra and ncrcat makes it possible, for instance, to average or concatenate regular intervals across multi-file input data sets.

ncra -F -d time,3,,12 85.nc 86.nc 87.nc 88.nc 89.nc 8589_03.nc
ncrcat -F -d time,3,,12 85.nc 86.nc 87.nc 88.nc 89.nc 8503_8903.nc

3.16 Multislabs

A multislab is a union of one or more hyperslabs. One defines multislabs by chaining together hyperslab commands, i.e., -d options (see section Hyperslabs). Support for specifying a multi-hyperslab or multislab for any variable was first added to ncks in late 2002. The other operators received MSA capabilities in April 2008. Sometimes multi-slabbing is referred to by the acronym MSA, which stands for “Multi-Slabbing Algorithm”.

Multislabs overcome some restraints that limit hyperslabs. A single -d option can only specify a contiguous and/or a regularly spaced multi-dimensional data array. Multislabs are constructed from multiple -d options and may therefore have non-regularly spaced arrays. For example, suppose it is desired to operate on all longitudes from 10.0 to 20.0 and from 80.0 to 90.0 degrees. The combined range of longitudes is not selectable in a single hyperslab specfication of the form ‘-d dimension,min,max’ or ‘-d dimension,min,max,stride’ because its elements are irregularly spaced in coordinate space (and presumably in index space too). The multislab specification for obtaining these values is simply the union of the hyperslabs specifications that comprise the multislab, i.e.,

ncks -d lon,10.,20. -d lon,80.,90. in.nc out.nc
ncks -d lon,10.,15. -d lon,15.,20. -d lon,80.,90. in.nc out.nc

Any number of hyperslabs specifications may be chained together to specify the multislab.

Users may specify redundant ranges of indices in a multislab, e.g.,

ncks -d lon,0,4 -d lon,2,9,2 in.nc out.nc

This command retrieves the first five longitudes, and then every other longitude value up to the tenth. Elements 0, 2, and 4 are specified by both hyperslab arguments (hence this is redundant) but will count only once if an arithmetic operation is being performed. This example uses index-based (not coordinate-based) multislabs because the stride option only supports index-based hyper-slabbing. See section Stride, for more information on the stride option.

Multislabs are more efficient than the alternative of sequentially performing hyperslab operations and concatenating the results. This is because NCO employs a novel multislab algorithm to minimize the number of I/O operations when retrieving irregularly spaced data from disk. The NCO multislab algorithm retrieves each element from disk once and only once. Thus users may take some shortcuts in specifying multislabs and the algorithm will obtain the intended values. Specifying redundant ranges is not encouraged, but may be useful on occasion and will not result in unintended consequences.

A final example shows the real power of multislabs. Suppose the Q variable contains three dimensional arrays of distinct chemical constituents in no particular order. We are interested in the NOy species in a certain geographic range. Say that NO, NO2, and N2O5 are elements 0, 1, and 5 of the species dimension of Q. The multislab specification might look something like

ncks -d species,0,1 -d species,5 -d lon,0,4 -d lon,2,9,2 in.nc out.nc

Multislabs are powerful because they may be specified for every dimension at the same time. Thus multislabs obsolete the need to execute multiple ncks commands to gather the desired range of data.

3.17 Wrapped Coordinates

A wrapped coordinate is a coordinate whose values increase or decrease monotonically (nothing unusual so far), but which represents a dimension that ends where it begins (i.e., wraps around on itself). Longitude (i.e., degrees on a circle) is a familiar example of a wrapped coordinate. Longitude increases to the East of Greenwich, England, where it is defined to be zero. Halfway around the globe, the longitude is 180 degrees East (or West). Continuing eastward, longitude increases to 360 degrees East at Greenwich. The longitude values of most geophysical data are either in the range [0,360), or [-180,180). In either case, the Westernmost and Easternmost longitudes are numerically separated by 360 degrees, but represent contiguous regions on the globe. For example, the Saharan desert stretches from roughly 340 to 50 degrees East. Extracting the hyperslab of data representing the Sahara from a global dataset presents special problems when the global dataset is stored consecutively in longitude from 0 to 360 degrees. This is because the data for the Sahara will not be contiguous in the input-file but is expected by the user to be contiguous in the output-file. In this case, ncks must invoke special software routines to assemble the desired output hyperslab from multiple reads of the input-file.

Assume the domain of the monotonically increasing longitude coordinate lon is 0 < lon < 360. ncks will extract a hyperslab which crosses the Greenwich meridian simply by specifying the westernmost longitude as min and the easternmost longitude as max. The following commands extract a hyperslab containing the Saharan desert:

ncks -d lon,340.,50. in.nc out.nc
ncks -d lon,340.,50. -d lat,10.,35. in.nc out.nc

The first example selects data in the same longitude range as the Sahara. The second example further constrains the data to having the same latitude as the Sahara. The coordinate lon in the output-file, ‘out.nc’, will no longer be monotonic! The values of lon will be, e.g., ‘340, 350, 0, 10, 20, 30, 40, 50’. This can have serious implications should you run ‘out.nc’ through another operation which expects the lon coordinate to be monotonically increasing. Fortunately, the chances of this happening are slim, since lon has already been hyperslabbed, there should be no reason to hyperslab lon again. Should you need to hyperslab lon again, be sure to give dimensional indices as the hyperslab arguments, rather than coordinate values (see section Hyperslabs).

3.18 Auxiliary Coordinates

Utilize auxiliary coordinates specified in values of the coordinate variable’s standard_name attributes, if any, when interpreting hyperslab and multi-slab options. Also ‘--auxiliary’. This switch supports hyperslabbing cell-based grids over coordinate ranges. This works on datasets that associate coordinate variables to grid-mappings using the CF-convention (see section CF Conventions) coordinates and standard_name attributes described here. Currently, NCO understands auxiliary coordinate variables pointed to by the standard_name attributes for latitude and longitude. Cells that contain a value within the user-specified range [lon_min,lon_max,lat_min,lat_max] are included in the output hyperslab.

A cell-based grid collapses the horizontal spatial information (latitude and longitude) and stores it along a one-dimensional coordinate that has a one-to-one mapping to both latitude and longitude coordinates. Rectangular (in longitude and latitude) horizontal hyperslabs cannot be selected using the typical procedure (see section Hyperslabs) of separately specifying ‘-d’ arguments for longitude and latitude. Instead, when the ‘-X’ is used, NCO learns the names of the latitude and longitude coordinates by searching the standard_name attribute of all variables until it finds the two variables whose standard_name’s are “latitude” and “longitude”, respectively. This standard_name attribute for latitude and longitude coordinates follows the CF-convention (see section CF Conventions).

Putting it all together, consider a variable gds_3dvar output from simulations on a cell-based geodesic grid. Although the variable contains three dimensions of data (time, latitude, and longitude), it is stored in the netCDF file with only two dimensions, time and gds_crd.

% ncks -m -C -v gds_3dvar ~/nco/data/in.nc
gds_3dvar: type NC_FLOAT, 2 dimensions, 4 attributes, chunked? no, compressed? no, packed? no, ID = 41
gds_3dvar RAM size is 10*8*sizeof(NC_FLOAT) = 80*4 = 320 bytes
gds_3dvar dimension 0: time, size = 10 NC_DOUBLE, dim. ID = 20 (CRD)(REC)
gds_3dvar dimension 1: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
gds_3dvar attribute 0: long_name, size = 17 NC_CHAR, value = Geodesic variable
gds_3dvar attribute 1: units, size = 5 NC_CHAR, value = meter
gds_3dvar attribute 2: coordinates, size = 15 NC_CHAR, value = lat_gds lon_gds
gds_3dvar attribute 3: purpose, size = 64 NC_CHAR, value = Test auxiliary coordinates like those that define geodesic grids

The coordinates attribute lists the names of the latitude and longitude coordinates, lat_gds and lon_gds, respectively. The coordinates attribute is recommended though optional. With it, the user can immediately identify which variables contain the latitude and longitude coordinates. Without a coordinates attribute it would be unclear at first glance whether a variable resides on a cell-based grid. In this example, time is a normal record dimension and gds_crd is the cell-based dimension.

The cell-based grid file must contain two variables whose standard_name attributes are “latitude”, and “longitude”:

% ncks -m -C -v lat_gds,lon_gds ~/nco/data/in.nc
lat_gds: type NC_DOUBLE, 1 dimensions, 4 attributes, chunked? no, compressed? no, packed? no, ID = 37
lat_gds RAM size is 8*sizeof(NC_DOUBLE) = 8*8 = 64 bytes
lat_gds dimension 0: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
lat_gds attribute 0: long_name, size = 8 NC_CHAR, value = Latitude
lat_gds attribute 1: standard_name, size = 8 NC_CHAR, value = latitude
lat_gds attribute 2: units, size = 6 NC_CHAR, value = degree
lat_gds attribute 3: purpose, size = 62 NC_CHAR, value = 1-D latitude coordinate referred to by geodesic grid variables

lon_gds: type NC_DOUBLE, 1 dimensions, 4 attributes, chunked? no, compressed? no, packed? no, ID = 38
lon_gds RAM size is 8*sizeof(NC_DOUBLE) = 8*8 = 64 bytes
lon_gds dimension 0: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
lon_gds attribute 0: long_name, size = 9 NC_CHAR, value = Longitude
lon_gds attribute 1: standard_name, size = 9 NC_CHAR, value = longitude
lon_gds attribute 2: units, size = 6 NC_CHAR, value = degree
lon_gds attribute 3: purpose, size = 63 NC_CHAR, value = 1-D longitude coordinate referred to by geodesic grid variables

In this example lat_gds and lon_gds represent the latitude or longitude, respectively, of cell-based variables. These coordinates (must) have the same single dimension (gds_crd, in this case) as the cell-based variables. And the coordinates must be one-dimensional—multidimensional coordinates will not work.

This infrastructure allows NCO to identify, interpret, and process (e.g., hyperslab) the variables on cell-based grids as easily as it works with regular grids. To time-average all the values between zero and 180 degrees longitude and between plus and minus 30 degress latitude, we use

ncra -O -X 0.,180.,-30.,30. -v gds_3dvar in.nc out.nc

NCO accepts multiple ‘-X’ arguments for cell-based grid multi-slabs, just as it accepts multiple ‘-d’ arguments for multi-slabs of regular coordinates.

ncra -O -X 0.,180.,-30.,30. -X 270.,315.,45.,90. in.nc out.nc

The arguments to ‘-X’ are always interpreted as floating point numbers, i.e., as coordinate values rather than dimension indices so that these two commands produce identical results

ncra -X 0.,180.,-30.,30. in.nc out.nc
ncra -X 0,180,-30,30 in.nc out.nc

In contrast, arguments to ‘-d’ require decimal places to be recognized as coordinates not indices (see section Hyperslabs). We recommend always using decimal points with ‘-X’ arguments to avoid confusion.

3.19 UDUnits Support

There is more than one way to hyperskin a cat. The UDUnits package provides a library which, if present, NCO uses to translate user-specified physical dimensions into the physical dimensions of data stored in netCDF files. Unidata provides UDUnits under the same terms as netCDF, so sites should install both. Compiling NCO with UDUnits support is currently optional but may become required in a future version of NCO.

Two examples suffice to demonstrate the power and convenience of UDUnits support. First, consider extraction of a variable containing non-record coordinates with physical dimensions stored in MKS units. In the following example, the user extracts all wavelengths in the visible portion of the spectrum in terms of the units very frequently used in visible spectroscopy, microns:

% ncks -C -H -v wvl -d wvl,"0.4 micron","0.7 micron" in.nc
wvl[0]=5e-07 meter

The hyperslab returns the correct values because the wvl variable is stored on disk with a length dimension that UDUnits recognizes in the units attribute. The automagical algorithm that implements this functionality is worth describing since understanding it helps one avoid some potential pitfalls. First, the user includes the physical units of the hyperslab dimensions she supplies, separated by a simple space from the numerical values of the hyperslab limits. She encloses each coordinate specifications in quotes so that the shell does not break the value-space-unit string into separate arguments before passing them to NCO. Double quotes ("foo") or single quotes ('foo') are equally valid for this purpose. Second, NCO recognizes that units translation is requested because each hyperslab argument contains text characters and non-initial spaces. Third, NCO determines whether the wvl is dimensioned with a coordinate variable that has a units attribute. In this case, wvl itself is a coordinate variable. The value of its units attribute is meter. Thus wvl passes this test so UDUnits conversion is attempted. If the coordinate associated with the variable does not contain a units attribute, then NCO aborts. Fourth, NCO passes the specified and desired dimension strings (microns are specified by the user, meters are required by NCO) to the UDUnits library. Fifth, the UDUnits library that these dimension are commensurate and it returns the appropriate linear scaling factors to convert from microns to meters to NCO. If the units are incommensurate (i.e., not expressible in the same fundamental MKS units), or are not listed in the UDUnits database, then NCO aborts since it cannot determine the user’s intent. Finally, NCO uses the scaling information to convert the user-specified hyperslab limits into the same physical dimensions as those of the corresponding cooridinate variable on disk. At this point, NCO can perform a coordinate hyperslab using the same algorithm as if the user had specified the hyperslab without requesting units conversion.

The translation and dimensional innterpretation of time coordinates shows a more powerful, and probably more common, UDUnits application. In this example, the user prints all data between 4 PM and 7 PM on December 8, 1999, from a variable whose time dimension is hours since the year 1900:

% ncks -u -H -C -v time_udunits -d time_udunits,"1999-12-08 \
  16:00:0.0","1999-12-08 19:00:0.0" in.nc
time_udunits[1]=876018 hours since 1900-01-01 00:00:0.0

Here, the user invokes the stride (see section Stride) capability to obtain every other timeslice. This is possible because the UDUnits feature is additive, not exclusive—it works in conjunction with all other hyperslabbing (see section Hyperslabs) options and in all operators which support hyperslabbing. The following example shows how one might average data in a time period spread across multiple input files

ncra -d time,"1939-09-09 12:00:0.0","1945-05-08 00:00:0.0" \
  in1.nc in2.nc in3.nc out.nc

Note that there is no excess whitespace before or after the individual elements of the ‘-d’ argument. This is important since, as far as the shell knows, ‘-d’ takes only one command-line argument. Parsing this argument into its component dim,[min][,[max][,[stride]]] elements (see section Hyperslabs) is the job of NCO. When unquoted whitespace is present between these elements, the shell passes NCO arugment fragments which will not parse as intended.

NCO implemented support for the UDUnits2 library with version 3.9.2 (August, 2007). The UDUnits2 package supports non-ASCII characters and logarithmic units. We are interested in user-feedback on these features.

One aspect that deserves mention is that UDUnits, and thus NCO, supports run-time definition of the location of the relevant UDUnits databases. With UDUnits version 1, users may specify the directory which contains the UDUnits database, ‘udunits.dat’, via the UDUNITS_PATH environment variable. With UDUnits version 2, users may specify the UDUnits database file itself, ‘udunits2.xml’, via the UDUNITS2_XML_PATH environment variable.

export UDUNITS_PATH='/nonstandard/location/share/udunits' # UDUnits1
export UDUNITS2_XML_PATH='/nonstandard/location/share/udunits/udunits2.xml' # UDUnits2

This run-time flexibility can enable the full functionality of pre-built binaries on machines with libraries in different locations.

The UDUnits package documentation describes the supported formats of time dimensions. Among the metadata conventions which adhere to these formats are the Climate and Forecast (CF) Conventions and the Cooperative Ocean/Atmosphere Research Data Service (COARDS) Conventions. The following ‘-d arguments’ extract the same data using commonly encountered time dimension formats:

-d time,"1918-11-11 11:00:0.0","1939-09-09 00:00:0.0"

All of these formats include at least one dash - in a non-leading character position (a dash in a leading character position is a negative sign). NCO assumes that a non-leading dash in a limit string indicates that a UDUnits date conversion is requested.

As of version 4.0.0 (January, 2010), NCO supports some calendar attributes specified by the CF conventions.

Supported types:

"365_day"/"noleap", "360_day", "gregorian", "standard"

Unsupported types:

"366_day"/"all_leap","proleptic_gregorian","julian","none"

Unsupported types default to mixed Gregorian/Julian as defined by UDUnits.

An Example: Consider the following netCDF variable

variables:
  double lon_cal(lon_cal) ;
    lon_cal:long_name = "lon_cal" ;
    lon_cal:units = "days since 1964-2-28 0:0:0" ;
    lon_cal:calendar = "365_day" ;
data:
  lon_cal = 1,2,3,4,5,6,7,8,9,10;

So the command 
"ncks -v lon_cal -d lon_cal,'1964-3-1 0:00:0.0','1964-3-4 00:00:0.0' in.nc out.nc"
Results in the hyperslab lon_cal=1,2,3,4   

netCDF variables should always be stored with MKS (i.e., God’s) units, so that application programs may assume MKS dimensions apply to all input variables. The UDUnits feature is intended to alleviate some of the NCO user’s pain when handling MKS units. It connects users who think in human-friendly units (e.g., miles, millibars, days) to extract data which are always stored in God’s units, MKS (e.g., meters, Pascals, seconds). The feature is not intended to encourage writers to store data in esoteric units (e.g., furlongs, pounds per square inch, fortnights).

3.20 Rebasing Time Coordinate

Time re-basing is performed automatically when UDUnits is installed. Time rebasing is invoked when numerous files share a common time-based record coordinate, and the record coordinate is in the same time units in each file, but the date offset differs among files. For example suppose the time coordinate is in hours and each day in January is stored in its own daily file. Each daily file records the temperature variable tpt(time) with an (unadjusted) time coordinate value between 0–23 hours, and uses the units attribute to advance the base time:

file01.nc time:units="hours since 1990-1-1"   
file02.nc time:units="hours since 1990-1-2"   
...
file31.nc time:units="hours since 1990-1-31"   

// Find mean noontime temperature in January
ncra -v tpt -d time,"1990-1-1 12:00:00","1990-1-31 23:59:59",24 \
      file??.nc noon.nc    

// Concatenate day2 noon through day3 noon records
ncrcat -v tpt -d time,"1990-1-2 12:00:00","1990-1-3 11:59:59" \ 
      file01.nc file02.nc file03.nc noon.nc    

// Results: time is "re-based" to the time units in "file01.nc"
time=36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52,
     53, 54, 55, 56, 57, 58, 59 ;
  
// If we repeat the above command but with only two input files...
ncrcat -v tpt -d time,"1990-1-2 12:00:00","1990-1-3 11:59:59" \
      file02.nc file03 noon.nc    

// ...then the output time coordinate is based on the time units in "file02.nc"
time = 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28,
    29, 30, 31, 32, 33, 34, 35 ;

3.21 Missing values

The phrase missing data refers to data points that are missing, invalid, or for any reason not intended to be arithmetically processed in the same fashion as valid data. The NCO arithmetic operators attempt to handle missing data in an intelligent fashion. There are four steps in the NCO treatment of missing data:

1. Identifying variables that may contain missing data.

   NCO follows the convention that missing data should be stored with the _FillValue specified in the variable’s _FillValue attributes. The only way NCO recognizes that a variable may contain missing data is if the variable has a _FillValue attribute. In this case, any elements of the variable which are numerically equal to the _FillValue are treated as missing data.

   NCO adopted the behavior that the default attribute name, if any, assumed to specify the value of data to ignore is _FillValue with version 3.9.2 (August, 2007). Prior to that, the missing_value attribute, if any, was assumed to specify the value of data to ignore. Supporting both of these attributes simultaneously is not practical. Hence the behavior NCO once applied to missing_value it now applies to any _FillValue. NCO now treats any missing_value as normal data (24).

   It has been and remains most advisable to create both _FillValue and missing_value attributes with identical values in datasets. Many legacy datasets contain only missing_value attributes. NCO can help migrating datasets between these conventions. One may use ncrename (see section ncrename netCDF Renamer) to rename all missing_value attributes to _FillValue:

   ncrename -a .missing_value,_FillValue inout.nc

   Alternatively, one may use ncatted (see section ncatted netCDF Attribute Editor) to add a _FillValue attribute to all variables

   ncatted -O -a _FillValue,,o,f,1.0e36 inout.nc

2. Converting the _FillValue to the type of the variable, if neccessary.

   Consider a variable var of type var_type with a _FillValue attribute of type att_type containing the value _FillValue. As a guideline, the type of the _FillValue attribute should be the same as the type of the variable it is attached to. If var_type equals att_type then NCO straightforwardly compares each value of var to _FillValue to determine which elements of var are to be treated as missing data. If not, then NCO converts _FillValue from att_type to var_type by using the implicit conversion rules of C, or, if att_type is NC_CHAR (25), by typecasting the results of the C function strtod(_FillValue). You may use the NCO operator ncatted to change the _FillValue attribute and all data whose data is _FillValue to a new value (see section ncatted netCDF Attribute Editor).

3. Identifying missing data during arithmetic operations.

   When an NCO arithmetic operator processes a variable var with a _FillValue attribute, it compares each value of var to _FillValue before performing an operation. Note the _FillValue comparison imposes a performance penalty on the operator. Arithmetic processing of variables which contain the _FillValue attribute always incurs this penalty, even when none of the data are missing. Conversely, arithmetic processing of variables which do not contain the _FillValue attribute never incurs this penalty. In other words, do not attach a _FillValue attribute to a variable which does not contain missing data. This exhortation can usually be obeyed for model generated data, but it may be harder to know in advance whether all observational data will be valid or not.

4. Treatment of any data identified as missing in arithmetic operators.

   NCO averagers (ncra, ncea, ncwa) do not count any element with the value _FillValue towards the average. ncbo and ncflint define a _FillValue result when either of the input values is a _FillValue. Sometimes the _FillValue may change from file to file in a multi-file operator, e.g., ncra. NCO is written to account for this (it always compares a variable to the _FillValue assigned to that variable in the current file). Suffice it to say that, in all known cases, NCO does “the right thing”.

   It is impossible to determine and store the correct result of a binary operation in a single variable. One such corner case occurs when both operands have differing _FillValue attributes, i.e., attributes with different numerical values. Since the output (result) of the operation can only have one _FillValue, some information may be lost. In this case, NCO always defines the output variable to have the same _FillValue as the first input variable. Prior to performing the arithmetic operation, all values of the second operand equal to the second _FillValue are replaced with the first _FillValue. Then the arithmetic operation proceeds as normal, comparing each element of each operand to a single _FillValue. Comparing each element to two distinct _FillValue’s would be much slower and would be no likelier to yield a more satisfactory answer. In practice, judicious choice of _FillValue values prevents any important information from being lost.

3.22 Chunking

All netCDF4-enabled NCO operators that define variables support a plethora of chunksize options. Chunking can significantly accelerate or degrade read/write access to large datasets. Dataset chunking issues are described in detail here.

The NCO chunking implementation is designed to be flexible. Users control three aspects of the chunking implementation. These are known as the chunking policy, chunking map, and chunksize. The first two are high-level mechanisms that apply to an entire file, while the third allows per-dimension specification of parameters. The implementation is a hybrid of the ncpdq packing policies (see section ncpdq netCDF Permute Dimensions Quickly), and the hyperslab specifications (see section Hyperslabs). Each aspect is intended to have a sensible default, so that most users will only need to set one switch to obtain sensible chunking. Power users can tune the three switches in tandem to obtain optimal performance.

The user specifies the desired chunking policy with the ‘-P’ switch (or its long option equivalents, ‘--cnk_plc’ and ‘--chunk_policy’) and its cnk_plc argument. Five chunking policies are currently implemented:

Chunk All Variables [default]

Definition: Chunk all variables possible
Alternate invocation: ncchunk
cnk_plc key values: ‘all’, ‘cnk_all’, ‘plc_all’
Mnemonic: All

Chunk Variables with at least Two Dimensions

Definition: Chunk all variables possible with at least two dimensions
Alternate invocation: none
cnk_plc key values: ‘g2d’, ‘cnk_g2d’, ‘plc_g2d’
Mnemonic: Greater than or equal to 2 Dimensions

Chunk Variables with at least Three Dimensions

Definition: Chunk all variables possible with at least three dimensions
Alternate invocation: none
cnk_plc key values: ‘g3d’, ‘cnk_g3d’, ‘plc_g3d’
Mnemonic: Greater than or equal to 3 Dimensions

Chunk Variables Containing Explicitly Chunked Dimensions

Definition: Chunk all variables possible that contain at least one dimension whose chunksize was explicitly set with the ‘--cnk_dmn’ option. Alternate invocation: none
cnk_plc key values: ‘xpl’, ‘cnk_xpl’, ‘plc_xpl’
Mnemonic: EXPLicitly specified dimensions

Unchunking

Definition: Unchunk all variables
Alternate invocation: ncunchunk
cnk_plc key values: ‘uck’, ‘cnk_uck’, ‘plc_uck’, ‘unchunk’
Mnemonic: UnChunK

Equivalent key values are fully interchangeable. Multiple equivalent options are provided to satisfy disparate needs and tastes of NCO users working with scripts and from the command line.

The chunking algorithms must know the chunksizes of each dimension of each variable to be chunked. The correspondence between the input variable shape and the chunksizes is called the chunking map. The user specifies the desired chunking map with the ‘-M’ switch (or its long option equivalents, ‘--cnk_map’ and ‘--chunk_map’) and its cnk_map argument. Four chunking maps are currently implemented:

Chunksize Equals Dimension Size [default]

Definition: Chunksize defaults to dimension size. Explicitly specify chunksizes for particular dimensions with ‘--cnk_dmn’ option.
cnk_map key values: ‘dmn’, ‘cnk_dmn’, ‘map_dmn’
Mnemonic: DiMeNsion

Chunksize Equals Dimension Size except Record Dimension

Definition: Chunksize equals dimension size except record dimension has size one. Explicitly specify chunksizes for particular dimensions with ‘--cnk_dmn’ option.
cnk_map key values: ‘rd1’, ‘cnk_rd1’, ‘map_rd1’
Mnemonic: Record Dimension size 1

Chunksize Equals Scalar Size Specified

Definition: Chunksize for all dimensions is set with the ‘--cnk_scl’ option.
cnk_map key values: ‘xpl’, ‘cnk_xpl’, ‘map_xpl’
Mnemonic: EXPLicitly specified dimensions

Chunksize Product Equals Scalar Size Specified

Definition: The product of the chunksizes for each variable (approximately) equals the size specified with the ‘--cnk_scl’ option. For a variable of rank R (i.e., with R non-degenerate dimensions), the chunksize in each non-degenerate dimension is the Rth root of cnk_scl.
cnk_map key values: ‘prd’, ‘cnk_prd’, ‘map_prd’
Mnemonic: PRoDuct

It is possible to combine the above chunking map algorithms with user-specified per-dimension (but not per-variable) chunksizes that override specific chunksizes determined by the maps above. The user specifies the per-dimension chunksizes with the (equivalent) long options ‘--cnk_dmn’ or ‘--chunk_dimension’). The option takes two comma-separated arguments, dmn_nm,cnk_sz, which are the dimension name and its chunksize, respectively. The ‘--cnk_dmn’ option may be used as many times as necessary.

# Simple chunking and unchunking
ncks -O -4 --cnk_plc=all     in.nc out.nc # Chunk in.nc
ncks -O -4 --cnk_plc=unchunk in.nc out.nc # Unchunk in.nc

# Chunk data then unchunk it, printing informative metadata
ncks -O -4 -D 4 --cnk_plc=all ~/nco/data/in.nc ~/foo.nc
ncks -O -4 -D 4 --cnk_plc=uck ~/foo.nc ~/foo.nc

# More complex chunking procedures, with informative metadata
ncks -O -4 -D 4 --cnk_scl=8 ~/nco/data/in.nc ~/foo.nc
ncks -O -4 -D 4 --cnk_scl=8 /data/zender/dstmch90/dstmch90_clm.nc ~/foo.nc
ncks -O -4 -D 4 --cnk_dmn lat,64 --cnk_dmn lon,128 /data/zender/dstmch90/dstmch90_clm.nc ~/foo.nc 
ncks -O -4 -D 4 --cnk_plc=uck ~/foo.nc ~/foo.nc
ncks -O -4 -D 4 --cnk_plc=g2d --cnk_map=rd1 --cnk_dmn lat,32 --cnk_dmn lon,128 /data/zender/dstmch90/dstmch90_clm_0112.nc ~/foo.nc

# Chunking works with all operators...
ncap2 -O -4 -D 4 --cnk_scl=8 -S ~/nco/data/ncap2_tst.nco ~/nco/data/in.nc ~/foo.nc
ncbo -O -4 -D 4 --cnk_scl=8 -p ~/nco/data in.nc in.nc ~/foo.nc
ncecat -O -4 -D 4 -n 12,2,1 --cnk_dmn lat,32 -p /data/zender/dstmch90 dstmch90_clm01.nc ~/foo.nc
ncflint -O -4 -D 4 --cnk_scl=8 ~/nco/data/in.nc ~/foo.nc
ncpdq -O -4 -D 4 -P all_new --cnk_scl=8 -L 5 ~/nco/data/in.nc ~/foo.nc
ncrcat -O -4 -D 4 -n 12,2,1 --cnk_dmn lat,32 -p /data/zender/dstmch90 dstmch90_clm01.nc ~/foo.nc
ncwa -O -4 -D 4 -a time --cnk_plc=g2d --cnk_map=rd1 --cnk_dmn lat,32 --cnk_dmn lon,128 /data/zender/dstmch90/dstmch90_clm_0112.nc ~/foo.nc

It is appropriate to conclude by informing users about an aspect of chunking that may not be expected: Record dimensions are always chunked with a chunksize of one. Hence all variables that contain the record dimension are also stored as chunked (since data must be stored with chunking either in all dimensions, or in no dimensions). Unless otherwise specified by the user, the other (fixed, non-record) dimensions of such variables are assigned default chunk sizes. The HDF5 layer does all this automatically to optimize the on-disk variable/file storage geometry of record variables. Do not be surprised to learn that files created without any explicit instructions to activate chunking nevertheless contain chunked variables.

3.23 Deflation

All NCO operators that define variables support the netCDF4 feature of storing variables compressed with Lempel-Ziv deflation. The Lempel-Ziv algorithm is a lossless data compression technique. Activate this deflation with the -L dfl_lvl short option (or with the same argument to the ‘--dfl_lvl’ or ‘--deflate’ long options). Specify the deflation level dfl_lvl on a scale from no deflation (dfl_lvl = 0) to maximum deflation (dfl_lvl = 9). Minimal deflation (dfl_lvl = 1) achieves considerable storage compression with little time penalty. Higher deflation levels require more time for compression. File sizes resulting from minimal (dfl_lvl = 1) and maximal (dfl_lvl = 9) deflation levels typically differ by a few percent in size.

To compress an entire file using deflation, use

ncks -4 -L 0 in.nc out.nc # No deflation (fast, no time penalty)
ncks -4 -L 1 in.nc out.nc # Minimal deflation (little time penalty)
ncks -4 -L 9 in.nc out.nc # Maximal deflation (much slower)

Unscientific testing shows that deflation compresses typical climate datasets by 30-60%. Packing, a lossy compression technique available for all netCDF files (see Packed data), can easily compress files by 50%. Packed data may be deflated to squeeze datasets by about 80%:

ncks  -4 -L 1 in.nc out.nc # Minimal deflation (~30-60% compression)
ncks  -4 -L 9 in.nc out.nc # Maximal deflation (~31-63% compression)
ncpdq         in.nc out.nc # Standard packing  (~50% compression)
ncpdq -4 -L 9 in.nc out.nc # Deflated packing  (~80% compression)

ncks prints deflation parameters, if any, to screen (see section ncks netCDF Kitchen Sink).

3.24 Packed data

The phrase packed data refers to data which are stored in the standard netCDF3 packing format which employs a lossy algorithm. See ncks netCDF Kitchen Sink for a description of deflation, a lossless compression technique available with netCDF4 only. Packed data may be deflated to save additional space.

Packing Algorithm

Packing The standard netCDF packing algorithm is lossy, and produces data with the same dynamic range as the original but which requires no more than half the space to store. The packed variable is stored (usually) as type NC_SHORT with the two attributes required to unpack the variable, scale_factor and add_offset, stored at the original (unpacked) precision of the variable (26). Let min and max be the minimum and maximum values of x.
scale_factor = (max-min)/ndrv
add_offset = 0.5*(min+max)
pck = (upk-add_offset)/scale_factor = (upk-0.5*(min+max))*ndrv/(max-min)

where ndrv is the number of discrete representable values for given type of packed variable. The theoretical maximum value for ndrv is two raised to the number of bits used to store the packed variable. Thus if the variable is packed into type NC_SHORT, a two-byte datatype, then there are at most 2^16 = 65536 distinct values representible. In practice, the number of discretely representible values is taken to be two less than the theoretical maximum. This leaves space for a missing value and solves potential problems with rounding that may occur during the unpacking of the variable. Thus for NC_SHORT, ndrv = 65536 - 2 = 65534. Less often, the variable may be packed into type NC_CHAR, where ndrv = 256 - 2 = 254, or type NC_INT where where ndrv = 4294967295 - 2 = 4294967293. One useful feature of (lossy) netCDF packing algorithm is that additional, loss-less packing algorithms perform well on top of it.

Unpacking Algorithm

Unpacking The unpacking algorithm depends on the presence of two attributes, scale_factor and add_offset. If scale_factor is present for a variable, the data are multiplied by the value scale_factor after the data are read. If add_offset is present for a variable, then the add_offset value is added to the data after the data are read. If both scale_factor and add_offset attributes are present, the data are first scaled by scale_factor before the offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)

When scale_factor and add_offset are used for packing, the associated variable (containing the packed data) is typically of type byte or short, whereas the unpacked values are intended to be of type int, float, or double. An attribute’s scale_factor and add_offset and _FillValue, if any, should all be of the type intended for the unpacked data, i.e., int, float or double.

Default Handling of Packed Data

All NCO arithmetic operators understand packed data. The operators automatically unpack any packed variable in the input file which will be arithmetically processed. For example, ncra unpacks all record variables, and ncwa unpacks all variable which contain a dimension to be averaged. These variables are stored unpacked in the output file.

On the other hand, arithmetic operators do not unpack non-processed variables. For example, ncra leaves all non-record variables packed, and ncwa leaves packed all variables lacking an averaged dimension. These variables (called fixed variables) are passed unaltered from the input to the output file. Hence fixed variables which are packed in input files remain packed in output files. Completely packing and unpacking files is easily accomplished with ncpdq (see section ncpdq netCDF Permute Dimensions Quickly). Packing and unpacking individual variables may be done with ncpdq and the ncap2 pack() and unpack() functions (see section Methods and functions).

3.25 Operation Types

The ‘-y op_typ’ switch allows specification of many different types of operations Set op_typ to the abbreviated key for the corresponding operation:

avg

Mean value

sqravg

Square of the mean

avgsqr

Mean of sum of squares

max

Maximium value

min

Minimium value

rms

Root-mean-square (normalized by N)

rmssdn

Root-mean square (normalized by N-1)

sqrt

Square root of the mean

ttl

Sum of values

NCO assumes coordinate variables represent grid axes, e.g., longitude. The only rank-reduction which makes sense for coordinate variables is averaging. Hence NCO implements the operation type requested with ‘-y’ on all non-coordinate variables, but not on coorniate variables. When an operation requires a coordinate variable to be reduced in rank, i.e., from one dimension to a scalar or from one dimension to a degenerate (single value) array, then NCO always averages the coordinate variable regardless of the arithmetic operation type performed on the non-coordinate variables.

The mathematical definition of each arithmetic operation is given below. See section ncwa netCDF Weighted Averager, for additional information on masks and normalization. If an operation type is not specified with ‘-y’ then the operator performs an arithmetic average by default. Averaging is described first so the terminology for the other operations is familiar.

The definitions of some of these operations are not universally useful. Mostly they were chosen to facilitate standard statistical computations within the NCO framework. We are open to redefining and or adding to the above. If you are interested in having other statistical quantities defined in NCO please contact the NCO project (see section Help Requests and Bug Reports).

EXAMPLES

Suppose you wish to examine the variable prs_sfc(time,lat,lon) which contains a time series of the surface pressure as a function of latitude and longitude. Find the minimium value of prs_sfc over all dimensions:

ncwa -y min -v prs_sfc in.nc foo.nc 

Find the maximum value of prs_sfc at each time interval for each latitude:

ncwa -y max -v prs_sfc -a lon in.nc foo.nc

Find the root-mean-square value of the time-series of prs_sfc at every gridpoint:

ncra -y rms -v prs_sfc in.nc foo.nc
ncwa -y rms -v prs_sfc -a time in.nc foo.nc

The previous two commands give the same answer but ncra is preferred because it has a smaller memory footprint. Also, by default, ncra leaves the (degenerate) time dimension in the output file (which is usually useful) whereas ncwa removes the time dimension (unless ‘-b’ is given).

These operations work as expected in multi-file operators. Suppose that prs_sfc is stored in multiple timesteps per file across multiple files, say ‘jan.nc’, ‘feb.nc’, ‘march.nc’. We can now find the three month maximium surface pressure at every point.

ncea -y max -v prs_sfc jan.nc feb.nc march.nc out.nc

It is possible to use a combination of these operations to compute the variance and standard deviation of a field stored in a single file or across multiple files. The procedure to compute the temporal standard deviation of the surface pressure at all points in a single file ‘in.nc’ involves three steps.

ncwa -O -v prs_sfc -a time in.nc out.nc
ncbo -O -v prs_sfc in.nc out.nc out.nc 
ncra -O -y rmssdn out.nc out.nc

First construct the temporal mean of prs_sfc in the file ‘out.nc’. Next overwrite ‘out.nc’ with the anomaly (deviation from the mean). Finally overwrite ‘out.nc’ with the root-mean-square of itself. Note the use of ‘-y rmssdn’ (rather than ‘-y rms’) in the final step. This ensures the standard deviation is correctly normalized by one fewer than the number of time samples. The procedure to compute the variance is identical except for the use of ‘-y var’ instead of ‘-y rmssdn’ in the final step.

ncap2 can also compute statistics like standard deviations. Brute-force implementation of formulae is one option, e.g.,

ncap2 -s 'prs_sfc_sdn=sqrt((prs_sfc-prs_sfc.avg($time)^2).total($time))/($time.size-1)'
      in.nc out.nc

The operation may, of course, be broken into multiple steps in order to archive intermediate quantities, such as the time-anomalies

ncap2 -s 'prs_sfc_anm=prs_sfc-prs_sfc.avg($time)' \
      -s 'prs_sfc_sdn=sqrt((prs_sfc_anm^2).total($time))/($time.size-1)' \
      in.nc out.nc

ncap2 supports intrinsic standard deviation functions (see section Operation Types) which simplify the above expression to

ncap2 -s 'prs_sfc_sdn=(prs_sfc-prs_sfc.avg($time)).rmssdn($time)' in.nc out.nc

These instrinsic functions compute the answer quickly and concisely.

The procedure to compute the spatial standard deviation of a field in a single file ‘in.nc’ involves three steps.

ncwa -O -v prs_sfc,gw -a lat,lon -w gw in.nc out.nc
ncbo -O -v prs_sfc,gw in.nc out.nc out.nc
ncwa -O -y rmssdn -v prs_sfc -a lat,lon -w gw out.nc out.nc

First the appropriately weighted (with ‘-w gw’) spatial mean values are written to the output file. This example includes the use of a weighted variable specified with ‘-w gw’. When using weights to compute standard deviations one must remember to include the weights in the initial output files so that they may be used again in the final step. The initial output file is then overwritten with the gridpoint deviations from the spatial mean. Finally the root-mean-square of the appropriately weighted spatial deviations is taken.

The ncap2 solution to the spatially-weighted standard deviation problem is

ncap2 -s 'prs_sfc_sdn=(prs_sfc*gw-prs_sfc*gw.avg($lat,$lon)).rmssdn($lat,$lon)' \
      in.nc out.nc

Be sure to multiply the variable by the weight prior to computing the the anomalies and the standard deviation.

The procedure to compute the standard deviation of a time-series across multiple files involves one extra step since all the input must first be collected into one file.

ncrcat -O -v tpt in.nc in.nc foo1.nc
ncwa -O -a time foo1.nc foo2.nc
ncbo -O -v tpt foo1.nc foo2.nc foo3.nc
ncra -O -y rmssdn foo3.nc out.nc

The first step assembles all the data into a single file. Though this may consume a lot of temporary disk space, it is more or less required by the ncbo operation in the third step.

3.26 Type Conversion

Type conversion (often called promotion or demotion) refers to the casting of one fundamental data type to another, e.g., converting NC_SHORT (two bytes) to NC_DOUBLE (eight bytes). Type conversion is automatic when the language carries out this promotion according to an internal set of rules without explicit user intervention. In contrast, manual type conversion refers to explicit user commands to change the type of a variable or attribute. Most type conversion happens automatically, yet there are situations in which manual type conversion is advantageous.

3.26.1 Automatic type conversion

As a general rule, automatic type conversions should be avoided for at least two reasons. First, type conversions are expensive since they require creating (temporary) buffers and casting each element of a variable from the type it was stored at to some other type. Second, the dataset’s creator probably had a good reason for storing data as, say, NC_FLOAT rather than NC_DOUBLE. In a scientific framework there is no reason to store data with more precision than the observations were made. Thus NCO tries to avoid performing automatic type conversions when performing arithmetic.

Automatic type conversion during arithmetic in the languages C and Fortran is performed only when necessary. All operands in an operation are converted to the most precise type before the operation takes place. However, following this parsimonious conversion rule dogmatically results in numerous headaches. For example, the average of the two NC_SHORTs 17000s and 17000s results in garbage since the intermediate value which holds their sum is also of type NC_SHORT and thus cannot represent values greater than 32,767 (27). There are valid reasons for expecting this operation to succeed and the NCO philosophy is to make operators do what you want, not what is most pure. Thus, unlike C and Fortran, but like many other higher level interpreted languages, NCO arithmetic operators will perform automatic type conversion when all the following conditions are met (28):

1. The operator is ncea, ncra, or ncwa. ncbo is not yet included in this list because subtraction did not benefit from type conversion. This will change in the future
2. The arithmetic operation could benefit from type conversion. Operations that could benefit (e.g., from larger representable sums) include averaging, summation, or any "hard" arithmetic. Type conversion does not benefit searching for minima and maxima (‘-y min’, or ‘-y max’).
3. The variable on disk is of type NC_BYTE, NC_CHAR, NC_SHORT, or NC_INT. Type NC_DOUBLE is not type converted because there is no type of higher precision to convert to. Type NC_FLOAT is not type converted because, in our judgement, the performance penalty of always doing so would outweigh the (extremely rare) potential benefits.

When these criteria are all met, the operator promotes the variable in question to type NC_DOUBLE, performs all the arithmetic operations, casts the NC_DOUBLE type back to the original type, and finally writes the result to disk. The result written to disk may not be what you expect, because of incommensurate ranges represented by different types, and because of (lack of) rounding. First, continuing the above example, the average (e.g., ‘-y avg’) of 17000s and 17000s is written to disk as 17000s. The type conversion feature of NCO makes this possible since the arithmetic and intermediate values are stored as NC_DOUBLEs, i.e., 34000.0d and only the final result must be represented as an NC_SHORT. Without the type conversion feature of NCO, the average would have been garbage (albeit predictable garbage near -15768s). Similarly, the total (e.g., ‘-y ttl’) of 17000s and 17000s written to disk is garbage (actually -31536s) since the final result (the true total) of 34000 is outside the range of type NC_SHORT.

Type conversions use the floor function to convert floating point number to integers. Type conversions do not attempt to round floating point numbers to the nearest integer. Thus the average of 1s and 2s is computed in double precisions arithmetic as (1.0d + 1.5d)/2) = 1.5d. This result is converted to NC_SHORT and stored on disk as floor(1.5d) = 1s (29). Thus no "rounding up" is performed. The type conversion rules of C can be stated as follows: If n is an integer then any floating point value x satisfying n <= x < n+1 will have the value n when converted to an integer.

3.26.2 Manual type conversion

ncap2 provides intrinsic functions for performing manual type conversions. This, for example, converts variable tpt to external type NC_SHORT (a C-type short), and variable prs to external type NC_DOUBLE (a C-type double).

ncap2 -s 'tpt=short(tpt);prs=double(prs)' in.nc out.nc

See section ncap2 netCDF Arithmetic Processor, for more details.

3.27 Batch Mode

If the output-file specified for a command is a pre-existing file, then the operator will prompt the user whether to overwrite (erase) the existing output-file, attempt to append to it, or abort the operation. However, interactive questions reduce productivity when processing large amounts of data. Therefore NCO also implements two ways to override its own safety features, the ‘-O’ and ‘-A’ switches. Specifying ‘-O’ tells the operator to overwrite any existing output-file without prompting the user interactively. Specifying ‘-A’ tells the operator to attempt to append to any existing output-file without prompting the user interactively. These switches are useful in batch environments because they suppress interactive keyboard input.

3.28 History Attribute

All operators automatically append a history global attribute to any file they create or modify. The history attribute consists of a timestamp and the full string of the invocation command to the operator, e.g., ‘Mon May 26 20:10:24 1997: ncks in.nc foo.nc’. The full contents of an existing history attribute are copied from the first input-file to the output-file. The timestamps appear in reverse chronological order, with the most recent timestamp appearing first in the history attribute. Since NCO and many other netCDF operators adhere to the history convention, the entire data processing path of a given netCDF file may often be deduced from examination of its history attribute. As of May, 2002, NCO is case-insensitive to the spelling of the history attribute name. Thus attributes named History or HISTORY (which are non-standard and not recommended) will be treated as valid history attributes. When more than one global attribute fits the case-insensitive search for "history", the first one found will be used. history attribute To avoid information overkill, all operators have an optional switch (‘-h’, ‘--hst’, or ‘--history’) to override automatically appending the history attribute (see section ncatted netCDF Attribute Editor). Note that the ‘-h’ switch also turns off writing the nco_input_file_list attribute for multi-file operators (see section File List Attributes).

3.29 File List Attributes

Many methods of specifying large numbers of input file names pass these names via pipes, encodings, or argument transfer programs (see section Large Numbers of Files). When these methods are used, the input file list is not explicitly passed on the command line. This results in a loss of information since the history attribute no longer contains the exact command by which the file was created.

NCO solves this dilemma by archiving input file list attributes. When the input file list to a multi-file operator is specified via stdin, the operator, by default, attaches two global attributes to any file they create or modify. The nco_input_file_number global attribute contains the number of input files, and nco_input_file_list contains the file names, specified as standard input to the multi-file operator. This information helps to verify that all input files the user thinks were piped through stdin actually arrived. Without the nco_input_file_list attribute, the information is lost forever and the “chain of evidence” would be broken.

The ‘-H’ switch overrides (turns off) the default behavior of writing the input file list global attributes when input is from stdin. The ‘-h’ switch does this too, and turns off the history attribute as well (see section History Attribute). Hence both switches allows space-conscious users to avoid storing what may amount to many thousands of filenames in a metadata attribute.

3.30 CF Conventions

NCO recognizes the Climate and Forecast (CF) metadata conventions, and applies special rules to such data. NCO also handles older NCAR model datasets, such as CCM and early CCSM datasets, with its CF rules even though the earlier data may not contain an explicit Conventions attribute (e.g., ‘CF-1.0’). We refer to all such data collectively as CF data. Skip this section if you never work with CF data.

The CF netCDF conventions are described here. Most CF netCDF conventions are transparent to NCO (30). There are no known pitfalls associated with using any NCO operator on files adhering to these conventions (31). However, to facilitate maximum user friendliness, NCO applies special rules to certain variables in CF files. The special functions are not required by the CF netCDF conventions, yet experience shows that they simplify data analysis.

Currently, NCO determines whether a datafile is a CF output datafile simply by checking (case-insensitively) whether the value of the global attribute Conventions (if any) equals ‘CF-1.0’ or ‘NCAR-CSM’ Should Conventions equal either of these in the (first) input-file, NCO will apply special rules to certain variables because of their usual meaning in CF files. NCO will not average the following variables often found in CF files: ntrm, ntrn, ntrk, ndbase, nsbase, nbdate, nbsec, mdt, mhisf. These variables contain scalar metadata such as the resolution of the host geophysical model and it makes no sense to change their values.

Furthermore, the size and rank-preserving arithmetic operators try not to operate on certain grid properties. These operators are ncap, ncbo, ncea, ncflint, and ncpdq (when used for packing, not for permutation). These operators do not operate, by default, on (i.e., add, subtract, pack, etc.) the following variables: ORO, area, datesec, date, gw, hyai, hyam, hybi. hybm, lat_bnds, lon_bnds, msk_*. These variables represent the Gaussian weights, the orography field, time fields, hybrid pressure coefficients, and latititude/longitude boundaries. We call these fields non-coordinate grid properties. Coordinate grid properties are easy to identify because they are coordinate variables such as latitude and longitude.

Users usually want all grid properties to remain unaltered in the output file. To be treated as a grid property, the variable name must exactly match a name in the above list, or be a coordinate variable. The handling of msk_* is exceptional in that any variable name beginning with the string msk_ is considered to be a “mask” and is thus preserved (not operated on arithmetically).

You must spoof NCO if you would like any grid properties or other special CF fields processed normally. For example rename the variables first with ncrename, or alter the Conventions attribute.

As of NCO version 4.0.8 (April, 2011), NCO supports the CF bounds convention for cell boundaries described here. This convention allows coordinate variables (including multidimensional coordinates) to describe the boundaries of their cells. This is done by naming the variable which contains the bounds in in the bounds attribute. Note that coordinates of rank N have bounds of rank N+1. NCO-generated subsets of CF-compliant files with bounds attributes will include the coordinates specified by the bounds attribute, if any. Hence the subsets will themselves be CF-compliant.

As of NCO version 3.9.6 (January, 2009), NCO supports the CF coordinates convention described here. This convention allows variables to specify additional coordinates (including multidimensional coordinates) in a space-separated string attribute named coordinates. NCO attaches any such coordinates to the extraction list along with variable and its usual (one-dimensional) coordinates, if any. These auxiliary coordinates are subject to the user-specified overrides described in Subsetting Coordinate Variables.

3.31 ARM Conventions

ncrcat has been programmed to correctly handle data files which utilize the Atmospheric Radiation Measurement (ARM) Program convention for time and time offsets. If you do not work with ARM data then you may skip this section. ARM data files store time information in two variables, a scalar, base_time, and a record variable, time_offset. Subtle but serious problems can arise when these type of files are just blindly concatenated. Therefore ncrcat has been specially programmed to be able to chain together consecutive ARM input-files and produce and an output-file which contains the correct time information. Currently, ncrcat determines whether a datafile is an ARM datafile simply by testing for the existence of the variables base_time, time_offset, and the dimension time. If these are found in the input-file then ncrcat will automatically perform two non-standard, but hopefully useful, procedures. First, ncrcat will ensure that values of time_offset appearing in the output-file are relative to the base_time appearing in the first input-file (and presumably, though not necessarily, also appearing in the output-file). Second, if a coordinate variable named time is not found in the input-files, then ncrcat automatically creates the time coordinate in the output-file. The values of time are defined by the ARM conventions time = base_time + time_offset. Thus, if output-file contains the time_offset variable, it will also contain the time coordinate. A short message is added to the history global attribute whenever these ARM-specific procedures are executed.

3.32 Operator Version

All operators can be told to print their version information, library version, copyright notice, and compile-time configuration with the ‘-r’ switch, or its long-option equivalent ‘revision’. The ‘--version’ or ‘--vrs’ switches print the operator version information only. The internal version number varies between operators, and indicates the most recent change to a particular operator’s source code. This is useful in making sure you are working with the most recent operators. The version of NCO you are using might be, e.g., 3.9.5. Using ‘-r’ on, say, ncks, produces something like ‘NCO netCDF Operators version "3.9.5" last modified 2008/05/11 built May 12 2008 on neige by zender Copyright (C) 1995--2008 Charlie Zender ncks version 20090918’. This tells you that ncks contains all patches up to version 3.9.5, which dates from May 11, 2008.

4. Operator Reference Manual

This chapter presents reference pages for each of the operators individually. The operators are presented in alphabetical order. All valid command line switches are included in the syntax statement. Recall that descriptions of many of these command line switches are provided only in NCO Features, to avoid redundancy. Only options specific to, or most useful with, a particular operator are described in any detail in the sections below.

4.1 ncap2 netCDF Arithmetic Processor

SYNTAX

ncap2 [-3] [-4] [-6] [-A] [-C] [-c] [-D dbg] [-F] [-f] [-L dfl_lvl] 
[-l path] [-O] [-o output-file] [-p path] [-R] [-r] 
[-s algebra] [-S fl.nco] [-t thr_nbr] [-v]
input-file [output-file]  

DESCRIPTION

ncap2 arithmetically processes netCDF files (32). The processing instructions are contained either in the NCO script file ‘fl.nco’ or in a sequence of command line arguments. The options ‘-s’ (or long options ‘--spt’ or ‘--script’) are used for in-line scripts and ‘-S’ (or long options ‘--fl_spt’ or ‘--script-file’) are used to provide the filename where (usually multiple) scripting commands are pre-stored. ncap2 was written to perform arbitrary algebraic transformations of data and archive the results as easily as possible. See section Missing values, for treatment of missing values. The results of the algebraic manipulations are called derived fields.

Unlike the other operators, ncap2 does not accept a list of variables to be operated on as an argument to ‘-v’ (see section Subsetting Variables). Rather, the ‘-v’ switch takes no arguments and indicates that ncap2 should output only user-defined variables. ncap2 neither accepts nor understands the -x switch.

Defining new variables in terms of existing variables is a powerful feature of ncap2. Derived fields inherit the metadata (i.e., attributes) of their ancestors, if any, in the script or input file. When the derived field is completely new (no identically-named ancestors exist), then it inherits the metadata (if any) of the left-most variable on the right hand side of the defining expression. This metadata inheritance is called attribute propagation. Attribute propagation is intended to facilitate well-documented data analysis, and we welcome suggestions to improve this feature.

The only exception to this rule of attribute propagation is in cases of left hand casting (see section Left hand casting). The user must manually define the proper metadata for variables defined using left hand casting.

4.1.1 Syntax of ncap2 statements

Mastering ncap2 is relatively simple. Each valid statement statement consists of standard forward algebraic expression. The ‘fl.nco’, if present, is simply a list of such statements, whitespace, and comments. The syntax of statements is most like the computer language C. The following characteristics of C are preserved:

Array syntax

Arrays elements are placed within [] characters;

Array indexing

Arrays are 0-based;

Array storage

Last dimension is most rapidly varying;

Assignment statements

A semi-colon ‘;’ indicates the end of an assignment statement.

Comments

Multi-line comments are enclosed within /* */ characters. Single line comments are preceded by // characters.

Nesting

Files may be nested in scripts using #include script. Note that the #include command is not followed by a semi-colon because it is a pre-processor directive, not an assignment statement. The filename ‘script’ is interpreted relative to the run directory.

Attribute syntax

The at-sign @ is used to delineate an attribute name from a variable name.

4.1.2 Expressions

Expressions are the fundamental building block of ncap2. Expressions are composed of variables, numbers, literals, and attributes. The following C operators are “overloaded” and work with scalars and multi-dimensional arrays:

Arithmetic Operators: * / % + - ^
Binary Operators:     > >= < <= == != == || && >> <<
Unary Operators:      + - ++ -- !
Conditional Operator: exp1 ? exp2 : exp3
Assign Operators:     = += -= /= *=

In the following section a variable also refers to a number literal which is read in as a scalar variable:

Arithmetic and Binary Operators

Consider var1 ’op’ var2

Precision

• When both operands are variables, the result has the precision of the higher precision operand.
• When one operand is a variable and the other an attribute, the result has the precision of the variable.
• When both operands are attributes, the result has the precision of the more precise attribute.
• The exponentiation operator “^” is an exception to the above rules. When both operands have type less than NC_FLOAT, the result is NC_FLOAT. When either type is NC_DOUBLE, the result is also NC_DOUBLE.

Rank

• The Rank of the result is generally equal to Rank of the operand that has the greatest number of dimensions.
• If the dimensions in var2 are a subset of the dimensions in var1 then its possible to make var2 conform to var1 through broadcasting and or dimension reordering.
• Broadcasting a variable means creating data in non-existing dimensions from the data in existing dimensions.
• More specifically: If the numbers of dimensions in var1 is greater than or equal to the number of dimensions in var2 then an attempt is made to make var2 conform to var1 ,else var1 is made to conform to var2. If conformance is not possible then an error message will be emitted and script execution will cease.
  

Even though the logical operators return True(1) or False(0) they are treated in the same way as the arithmetic operators with regard to precision and rank.
Examples:

dimensions: time=10, lat=2, lon=4
Suppose we have the two variables:

double  P(time,lat,lon);
float   PZ0(lon,lat);  // PZ0=1,2,3,4,5,6,7,8;

Consider now the expression:
 PZ=P-PZ0

PZ0 is made to conform to P and the result is
PZ0 =
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,
   1,3,5,7,2,4,6,8,

Once the expression is evaluated then PZ will be of type double;

Consider now 
 start=four-att_var@double_att;  // start =-69  and is of type intger;
 four_pow=four^3.0f               // four_pow=64 and is of type float  
 three_nw=three_dmn_var_sht*1.0f; // type is now float
 start@n1=att_var@short_att*att_var@int_att; 
                                  // start@n1=5329 and is type int 

Binary Operators
Unlike C the binary operators return an array of values. There is no such thing as short circuiting with the AND/OR operators. Missing values are carried into the result in the same way they are with the arithmetic operators. When an expression is evaluated in an if() the missing values are treated as true.
The binary operators are, in order of precedence:

	
!   Logical Not
----------------------------
<<  Less Than Selection
>>  Greater Than Selection
----------------------------
>   Greater than
>=  Greater than or equal to
<   Less than
<=  Less than or equal to
----------------------------
==  Equal to
!=  Not equal to
----------------------------
&&  Logical AND
----------------------------
||  Logical OR
----------------------------

To see all operators: see section Operators precedence and associativity Examples:

tm1= time>2 && time <7;  // tm1 = 0, 0, 1, 1, 1, 1, 0, 0, 0, 0 ; type double;
tm2= time==3 || time>=6; // tm2 = 0, 0, 1, 0, 0, 1, 1, 1, 1, 1 ; type double
tm3= int(!tm1);          // tm3=  1, 1, 0, 0, 0, 0, 1, 1, 1, 1 ; type int 
tm4= tm1 && tm2;         // tm4=  0, 0, 1, 0, 0, 1, 0, 0, 0, 0 ; type double;
tm5= !tm4;               // tm5=  1, 1, 0, 1, 1, 0, 1, 1, 1, 1 ; type double;

Regular Assign Operator
var1 ’=’ exp1
If var1 doesn’t already exist in Output then var1 is written to Output with the values and dimensions from expr1. If var1 already exists in Output, then the only requirement on expr1 is that the number of elements must match the number already on disk. The type of expr1 is converted if necessary to the disk type.

Other Assign Operators +=,-=,*=./=
var1 ’ass_op’ exp1
if exp1 is a variable and it doesn’t conform to var1 then an attempt is made to make it conform to var1. If exp1 is an attribute it must have unity size or else have the same number of elements as var1. If expr1 has a different type to var1 the it is converted to the var1 type.

example:

z1=four+=one*=10 // z1=14 four=14 one=10;	
time-=2          // time= -1,0,1,2,3,4,5,6,7,8

Increment/ Decrement Operators
These work in a similar fashion to their regular C counterparts. If say the variable "four" is input only then the statement "++four" effectively means -read four from input increment each element by one , then write the new values to Output;

Example:

n2=++four;   n2=5, four=5 
n3=one--+20; n3=21  one=0;	 
n4=--time;   n4=time=0.,1.,2.,3.,4.,5.,6.,7.,8.,9.;

Conditional Operator ?:
exp1 ? exp2 : exp3
The conditional operator (or ternary Operator) is a succinct way of writing an if/then/else. If exp1 evaluates to true then exp2 is returned else exp3 is returned.

Example:

weight_avg= weight.avg();
weight_avg@units= (weight_avg ==1 ? "kilo" : "kilos");  
PS_nw= PS - (PS.min() >100000 ? 100000 : 0 );

Clipping Operators

<< Less-than Clipping

For arrays, the less-than selection operator selects all values in the left operand that are less than the corresponding value in the right operand. If the value of the left side is greater than or equal to the corresponding value of the right side, then the right side value is placed in the result

>> Greater-than Clipping

For arrays, the greater-than selection operator selects all values in the left operand that are greater than the corresponding value in the right operand. If the value of the left side is less than or equal to the corresponding value of the right side, then the right side value is placed in the result.

Example:

RDM2= RDM >>100.0; RDM2=100,100,100,100,126,126,100,100,100, 100; // type double
RDM2= RDM << 90s ; RDM3=  1,  9, 36, 84, 90, 90, 84, 36,  9,   1; // type int

4.1.3 Dimensions

Dimensions are defined in Output using the defdim() function.

defdim("cnt",10);

This dimension name must then be prefixed with a dollar-sign ‘$’ when referred to in method arguments or left-hand-casting, e.g.,

new_var[$cnt]=time;
temperature[$time,$lat,$lon]=35.5;
temp_avg=temperature.avg($time);

The size method allows the dimension size to be used in an arithmetic expression:

time_avg=time.total() / $time.size;

Increase the size of a new variable by one and set new member to zero:

defdim("cnt_new",$cnt.size+1);
new_var[$cnt_new]=0.0;
new_var(0:($cnt_new.size-2))=old_var;

Dimension Abbreviations
It is possible to use dimension abbreviations as method arguments:
$0 is the first dimension of a variable
$1 is the second dimension of a variable
$n is the n+1 dimension of a variable

float four_dmn_rec_var(time,lat,lev,lon);
double three_dmn_var_dbl(time,lat,lon);

four_nw=four_dmn_rev_var.reverse($time,$lon)
four_nw=four_dmn_rec_var.reverse($0,$3);

four_avg=four_dmn_rec_var.avg($lat,$lev);  
four_avg=four_dmn_rec_var.avg($1,$2);  

three_mw=three_dmn_var_dbl.permute($time,$lon,$lat);
three_mw=three_dmn_var_dbl.permute($0,$2,$1);

ID Quoting
If the dimension name contains non-regular characters use ID quoting. See see section ID Quoting

defdim("a--list.A",10);
A1['$a--list.A']=30.0;

GOTCHA
It is not possible to manually define in Output any dimensions that exist in Input. When a variable from Input appears in an expression or statement its dimensions in Input are automagically copied to Output (if they are not already present)

4.1.4 Left hand casting

The following examples demonstrate the utility of the left hand casting ability of ncap2. Consider first this simple, artificial, example. If lat and lon are one dimensional coordinates of dimensions lat and lon, respectively, then addition of these two one-dimensional arrays is intrinsically ill-defined because whether lat_lon should be dimensioned lat by lon or lon by lat is ambiguous (assuming that addition is to remain a commutative procedure, i.e., one that does not depend on the order of its arguments). Differing dimensions are said to be orthogonal to one another, and sets of dimensions which are mutually exclusive are orthogonal as a set and any arithmetic operation between variables in orthogonal dimensional spaces is ambiguous without further information.

The ambiguity may be resolved by enumerating the desired dimension ordering of the output expression inside square brackets on the left hand side (LHS) of the equals sign. This is called left hand casting because the user resolves the dimensional ordering of the RHS of the expression by specifying the desired ordering on the LHS.

ncap2 -s 'lat_lon[lat,lon]=lat+lon' in.nc out.nc
ncap2 -s 'lon_lat[lon,lat]=lat+lon' in.nc out.nc

The explicit list of dimensions on the LHS, [lat,lon] resolves the otherwise ambiguous ordering of dimensions in lat_lon. In effect, the LHS casts its rank properties onto the RHS. Without LHS casting, the dimensional ordering of lat_lon would be undefined and, hopefully, ncap2 would print an error message.

Consider now a slightly more complex example. In geophysical models, a coordinate system based on a blend of terrain-following and density-following surfaces is called a hybrid coordinate system. In this coordinate system, four variables must be manipulated to obtain the pressure of the vertical coordinate: PO is the domain-mean surface pressure offset (a scalar), PS is the local (time-varying) surface pressure (usually two horizontal spatial dimensions, i.e. latitude by longitude), hyam is the weight given to surfaces of constant density (one spatial dimension, pressure, which is orthogonal to the horizontal dimensions), and hybm is the weight given to surfaces of constant elevation (also one spatial dimension). This command constructs a four-dimensional pressure prs_mdp from the four input variables of mixed rank and orthogonality:

ncap2 -s 'prs_mdp[time,lat,lon,lev]=P0*hyam+PS*hybm' in.nc out.nc

Manipulating the four fields which define the pressure in a hybrid coordinate system is easy with left hand casting.

4.1.5 Arrays and hyperslabs

Generating a regularly spaced one-dimensional array with ncap2 is simple with the array() function. The syntax is

output=array(val_srt,val_ncr,$dmn_nm); // One-dimensional output
output=array(val_srt,val_ncr,var_tpl);  // Multi-dimensional output
h

where the arguments are the starting value val_srt, incremental value val_ncr, and, for one-dimensional output, the single dimension $dmn_nm, or, for multi-dimensional output, a template variable var_tpl, i.e., a variable with the same shape as the desired output.

Once the associated dimensions have been defined, the start and increment arguments may be supplied as values, mathmatical expressions, or variables:

var_out=array(1,1,$time); // 1,2,3,4,5,6,7,8,9,10
var_out=array(1+2-2,one,$time); // 1,2,3,4,5,6,7,8,9,10
var_out=array(1,2,three_dmn_rec_var); // 1,3,5,...155,157,159

Hyperslabs in ncap2 are more limited than hyperslabs with the other NCO operators. ncap2 does not understand the shell command-line syntax used to specify multi-slabs, wrapped co-ordinates, negative stride or coordinate value limits. However with a bit of syntactic magic they are all are possible. ncap2 accepts (in fact, it requires) N-hyperslab arguments for a variable of rank N:

var1(arg1,arg2 ... argN);

where each hyperslab argument is of the form

start:end:stride 

and the arguments for different dimensions are separated by commas. If "start" is omitted, it defaults to 0. If "end" is omitted, it defaults to dimension size minus one. If "stride" is omitted, it defaults to 1.
If a single value is present then it is assumed that that dimension collapses to a single value (i.e., a cross-section). The number of hyperslab arguments MUST equal the variable’s rank.

Hyperslabs on the Right Hand Side of an assign

A simple 1D example:

($time.size=10)
od[$time]={20,22,24,26,28,30,32,34,36,38};

od(7);     // 34
od(7:);    // 34,36,38
od(:7);    // 20,22,24,26,28,30,32,34 
od(::4);   // 20,28,36
od(1:6:2)  // 22,26,30
od(:)      // 20,22,24,26,28,30,32,34,36,38 

A more complex three dimensional example:

($lat.size=2,$lon.size=4)
th[$time,$lat,$lon]=      
                          {1, 2, 3, 4, 5, 6, 7, 8,
                          9,10,11,12,13,14,15,16,
                          17,18,19,20,21,22,23,24,
                          -99,-99,-99,-99,-99,-99,-99,-99,
                          33,34,35,36,37,38,39,40,
                          41,42,43,44,45,46,47,48,
                          49,50,51,52,53,54,55,56,
                          -99,58,59,60,61,62,63,64,
                          65,66,67,68,69,70,71,72,
                          -99,74,75,76,77,78,79,-99 };

th(1,1,3);        // 16
th(2,0,:);        // 17, 18, 19, 20
th(:,1,3);        // 8, 16, 24, -99, 40, 48, 56, 64, 72, -99 
th(::5 ,:,0:3:2); // 1, 3, 5, 7, 41, 43, 45, 47

If hyperslab arguments collapse to a single value (a cross-section has been specified), then that dimension is removed from the returned variable. If all the values collapse then a scalar variable is returned. So, for example, the following is valid:

th_nw=th(0,:,:)+th(9,:,:); 
// th_nw has dimensions $lon,$lat 
// NB: the time dimension has become degenerate

The following is invalid:

th_nw=th(0,:,0:1)+th(9,:,0:1);

because the $lon dimension now only has two elements. The above can be calculated by using a LHS cast with $lon_nw as replacement dim for $lon:

defdim("lon_nw",2);
th_nw[$lat,$lon_nw]=th(0,:,0:1) +th(9,:,0:1);

Hyperslabs on the Left Hand Side of an assign
When hyperslabing on the LHS, the expression on the RHS must evaluate to a scalar or a variable/attribute with the same number of elements as the LHS hyperslab. Set all elements of the last record to zero:

th(9,:,:)=0.0;

Set first element of each lon element to 1.0:

th(:,:,0)=1.0;

One may hyperslab on both sides of an assign. For example, this sets the last record to the first record:

th(9,:,:)=th(0,:,:);

Say th0 represents pressure at height=0 and th1 represents pressure at height=1. Then it is possible to insert these hyperslabs into the records

prs[$time,$height,$lat,$lon]=0.0;
prs(:,0,:,:)=th0;
prs(:,1,:,:)=th1

Reverse method
Use the reverse() method to reverse a dimension’s elements in a variable with at least one dimension. This is equivalent to a negative stride, e.g.,

th_rv=th(1 ,:,:).reverse($lon); // {12,11,10,9 }, {16,15,14,13}
od_rv=od.reverse($time);        // {38,36,34,32,30,28,26,24,22,20}

Permute method
Use the permute() method to swap the dimensions of a variable. The number and names of dimension arguments must match the dimensions in the variable. If the first dimension in the variable is of record type then this must remain the first dimension. If you want to change the record dimension then consider using ncpdq.

Consider the variable:

float three_dmn_var(lat,lev,lon);
three_dmn_var_prm=three_dmn_var.permute($lon,$lat,$lev);
// The permuted values are
three_dmn_var_prm= 
  0,4,8,
  12,16,20,
  1,5,9,
  13,17,21,
  2,6,10,
  14,18,22,
  3,7,11,
  15,19,23;

4.1.6 Attributes

Attributes are referred to by var_nm@att_nm
All the following are valid statements:

global@text="Test Attributes"; /* Assign a global variable attribute */
a1[$time]=time*20;
a1@long_name="Kelvin";
a1@min=a1.min();
a1@max=a1.max();
a1@min++;
--a1@max; q
a1(0)=a1@min;
a1($time.size-1)=a1@max;

A value list can be used on the RHS of an assign...

a1@trip1={1,2,3} ;
a1@triplet={a1@min,(a1@min+a1@max)/2,a1@max}; 

The netCDF specification allows all attribute types to have a size greater than one. The maximum is defined by NC_MAX_ATTRS. The following is an ncdump of the metadata for variable a1

double a1(time) ;
  a1:long_name = "Kelvin" ;
  a1:max = 199. ;
  a1:min = 21. ;
  a1:trip1 = 1, 2, 3 ;
  a1:triplet = 21., 110., 199. ;

The size() method can be used with attributes. For example, to save an attribute text string in a variable,

defdim("sng_len", a1@long_name.size());
sng_arr[$sng_len]=a1@long_name; // sng_arr now contains "Kelvin" 

Attributes defined in a script are stored in memory and are written to Output after script completion. To stop the attribute being written use the ram_delete() method or use a bogus variable name.

Attribute Propagation & Inheritance

• Attribute propagation occurs in a regular assign statement. The variable being defined on the LHS gets copies of the attributes from the leftermost variable on the RHS
• Attribute Inheritance: The LHS variable "inherits" attributes from an Input variable with the same name
• It is possible to have a regular assign statement for which both propagation and inheritance occur.

prs_mdp[time,lat,lon,lev]=P0*hyam+hybm*PS;   // prs_mdp gets attributes from PO
th_min=1.0 + 2*three_dmn_var_dbl.min($time); // th_min  gets attributes from three_dmn_var_dbl

If the attribute name contains non-regular characters use ID quoting. See see section ID Quoting

'b..m1@c--lost'=23;

4.1.7 Number literals

The table below lists the postfix character(s) to add to a number literal for type cohesion. To use the new netCDF4 types NCO must be compiled/linked to the netCDF4 library and the output file must be HDF5.

n1[$time]=1UL; // n1 will now by type NC_UINT
n2[$lon]=4b;   // n2 will be of type NC_BYTE
n3[$lat]=5ull; // n3 will be of type NC_UINT64  
n3@a1=6.0d;   // attribute will be type NC_DOUBLE
n3@a2=-666L;  // attribute will be type NC_INT

A floating point number without a postfix will default to NC_DOUBLE. An integer without a postfix will default to type NC_INT. There is no postfix for characters, use a quoted string instead.

n4[$rlev]=0.1      // n4 will be of type NC_DOUBLE
n5[$lon_grd]=2.0   // n5 will be of type NC_DOUBLE
n6[$gds_crd]=2e3;  // n6 will be of type NC_DOUBLE
n7[$gds_crd]=2e3f;  // n7 will be of type NC_FLOAT
n6@a1=41;         // attribute will be type NC_INT
n6@a2=-21;        // attribute will be type NC_INT  
n6@units="kelvin" // attribute will be type NC_CHAR

netCDF3/4 Types

b|B

NC_BYTE, a signed 1-byte integer

none

NC_CHAR, an ISO/ASCII character

s|S

NC_SHORT, a signed 2-byte integer

l|L

NC_INT, a signed 4-byte integer

f|F

NC_FLOAT, a single-precision (4-byte) floating point number

d|D

NC_DOUBLE, a double-precision (8-byte) floating point number

netCDF4 Types

ub|UB

NC_UBYTE, an unsigned 1-byte integer

us|US

NC_USHORT, an unsigned 2-byte integer

u|U|ul|UL

NC_UINT, an unsigned 4-byte integer

ll|LL

NC_INT64, a signed 8-byte integer

ull|ULL

NC_UINT64, an unsigned 8-byte integer

4.1.8 if statement

The syntax of the if statement is similar to its C counterpart. The Conditional Operator (ternary operator) has also been implemented.

if(exp1)
   stmt1;
else if(exp2)     
   stmt2;
else
   stmt3;

# Can use code blocks as well:
if(exp1){
   stmt1;
   stmt1a;
   stmt1b;
} else if(exp2)     
   stmt2; 
else {
   stmt3;
   stmt3a;
   stmt3b;
}     

For a variable or attribute expression to be logically true all its non-missing value elements must be logically true, i.e., non-zero. The expression can be of any type. Unlike C there is no short-circuiting of an expression with the OR (||) AND (&&) operators. The whole expression is evaluated regardless if one of the AND/OR operands are true/false.

# Simple example
if(time>0)
  print("All values of time are greater than zero\n");
else if( time<0)
  print("All values of time are less than zero\n");   
else {
  time_max=time.max();
  time_min=time.min();
  print("min value of time=");print(time_min,"%f");
  print("max value of time=");print(time_max,"%f");
}

# Example from ddra.nco
if(fl_typ==fl_typ_gcm){
  var_nbr_apx=32;
  lmn_nbr=1.0*var_nbr_apx*varsz_gcm_4D; /* [nbr] Variable size */
  if(nco_op_typ==nco_op_typ_avg){
    lmn_nbr_avg=1.0*var_nbr_apx*varsz_gcm_4D; /* [nbr] Averaging block size */
    lmn_nbr_wgt=dmnsz_gcm_lat; /* [nbr] Weight size */
  } // !nco_op_typ_avg
}else if(fl_typ==fl_typ_stl){
  var_nbr_apx=8;
  lmn_nbr=1.0*var_nbr_apx*varsz_stl_2D; /* [nbr] Variable size */
  if(nco_op_typ==nco_op_typ_avg){
    lmn_nbr_avg=1.0*var_nbr_apx*varsz_stl_2D; /* [nbr] Averaging block size */
    lmn_nbr_wgt=dmnsz_stl_lat; /* [nbr] Weight size */
  } // !nco_op_typ_avg
} // !fl_typ

Conditional Operator

// netCDF4 needed for this example
th_nw=(three_dmn_var_sht >= 0 ? three_dmn_var_sht.uint() : three_dmn_var_sht.int() ); 

4.1.9 print statement