NWChem - computational chemistry on parallel computers


Notes on running NWChem on the ANL IBM SP1

Much of the information below has been exracted from information available through the web at ANL or IBM SP man pages by Robert Harrison (rj_harrison@pnl.gov).

For NWChem support mail nwchem-support@emsl.pnl.gov or visit the NWChem homepage.

For support on SP specific issues contact spsupport@mcs.anl.gov.


General SP information

ANL use their own environment (EASY) for scheduling interactive and batch use of the machine. You need to know about just a few facts and commands to get going.

You will log into either bonnie or clyde.mcs.anl.gov which are not really part of the SP2 - they are 'front-ends'. Information on the SP, examples etc. can be found on the MCS SP Web pages.

The command sphelp gives a useful summary of SP specific commands and is a good starting point ... here is the sp command summary

cacReport
Prints report on current state of CAC's allocation.
spfree
Prints the number of nodes that are currently free.
spq
Prints out who's running what on the machine and the queue, how much time they requested or when they'll finish.
sprelease
Is used to remove jobs, from either the queue, or the machine, if the job has already started. This returns node(s) back to the system, and stops the charging against the Charge Allocation Category (CAC) on the ANL system.
spsubmit
Used to submit jobs to the queue.
spstatus
Prints out the current status of the SP2.
spusage
Shows who is using what nodes. The format is:
  node# 0|1 username JID JobTerminationTime (MMDDhhmm)
  0 = node is available, 1 = in use.
xspusage
Same as above except for x windows.

To run either interactively or with batch you have to schedule the resources using spsubmit. By mail you'll be told when the resources become available. If running interactively, you can then login into the nodes you've been assigned (details below) or if running in batch you provided spsubmit a script to run (provided for NWChem). From prior experiences, you may experience a significant delay in receiving mail that you have the nodes. The use of xspusage is best (see below). If you don't have X, use spq.

If a node is not currently assigned to you then you will not be able to log into it.

Each node of the SP is a Power 1 cpu with 128 MB of physical memory and a local scratch disk named /tmp of about 0.25 to 0.4 GB. The O/S and I/O buffers consume about 10 MB (estimate), and the NWChem executable is 5 MB, leaving about 113 MB to be allocated. The local scratch disk is wiped clean at the end of each job. Thus, in order for useful files to be saved by default and to make sure that scratch files with high bandwidth requirements are in /tmp, you should always

Running NWChem in batch

Do this first - it's a lot less confusing than running interactively.

In the directory /sphome/harrison/hpcci are several files

ibm_sp@anl.html
this file
nwchem
the SP executable
library
the basis set library
job
a script for running spsubmit
runnwchem
a script for running nwchem in batch
interactive
a script for running nwchem interactively
examples
a directory with example inputs and outputs for NWChem
user.ps.Z
NWChem documentation (compressed postscript)

Copy one of the example input files into your login directory (e.g., /sphome/harrison/hpcci/examples/scf_h2o.nw - input for a conventional SCF calculation on water) into your login directory. (This input file is also commented and tailored to the SP).

To run the calculation in batch you must use spsubmit to

The standard script /sphome/harrison/hpcci/runnwchem takes 3 arguments
  1. the work directory (made if necessary)
  2. the full pathname of input file
  3. the full pathname of output file which defaults to the name of input file with .nw replaced by .out
e.g.
     runnwchem /tmp /sphome/harrison/test.nw 
or
     runnwchem /tmp /sphome/harrison/test.nw /sphome/harrison/test.out

Spsubmit prompts for the parameters it wants ... here is a sample dialog that runs a five minute nwchem job on eight nodes in batch (when you run this use your own username instead of harrison).

    /sphome/harrison % spsubmit
    *********************************************************************
    112 nodes are available
    16 nodes are allocated for I-Way use

    Please report any problems to spsupport@mcs.anl.gov
    *********************************************************************
    Charge Allocation Category: [default harrison] 
    Maximum Wall-clock Run-Time (minutes): (1-???) 5
    Number of Nodes Required: (1 - 128) 8
    CAC:  "harrison"  *now* has 4397 RUs available, after committing.
    (I)nteractive (B)atch: B
    Job Classification (M)PL, (T)ask Farm, (P)4: M
    IP over the switch [y/n]: Y
    Full path to Shell Script/Program: /sphome/harrison/hpcci/runnwchem
    Command Line arguments for your job: /sphome/harrison \
                                         /sphome/harrison/scf_h2o.nw
    (C)ommit or (A)bort: C

In order to be crystal clear, I typed (comments in parentheses)

     
    spsubmit (the command)
    carriage return (to accept charging to the default account)
    5 (for a 5 minute job)
    8 (for 8 processors)
    B (for batch)
    M (for MPL)
    Y (to let IP also run over the switch)
    /sphome/harrison/hpcci/runnwchem (name of script)
    /sphome/harrison /sphome/harrison/scf_h2o.nw (runnwchem arguments)
    C (to commit)

The script /sphome/harrison/hpcci/job runs spsubmit from input in a file - take a copy and edit it to automate use of spsubmit.

The command line arguments given to the runnwchem script specify the work directory (/sphome/harrison) and the full path to the input file (/sphome/harrison/scf_h2o.nw). Runnwchem will leave the output in /sphome/harrison/scf_h2o.out.

Since it can take a large elapsed time to get a job started five minutes is the practical lower limit that should be used for the job time.

The command spq will indicate the position of the job in the queue and its status. When the job is started you'll receive mail indicating the nodes actually allocated. At the end of the job you'll receive another mail message providing the output of the shell script. Only the NWChem output goes into the output file. The advantage of doing this is that you can examine the output while the job is executing. The unix command 'tail -f filename' is useful to watch a job run.

Note that only one node (the master) executes the script and that only the commands run with POE are actually executed in parallel. This causes seemingly simple things to fail. For instance, if you request to run in a subdirectory of /tmp (e.g., /tmp/rjh) then runnwchem will make this directory and cd into it. Subsequent POE commands will fail since this directory will not be present except on the master node.

If you want to run multiple NWChem calculations in the same job you'll have to modify the runnwchem script. Basically, this just requires replicating the line running poe and specifying the correct input and output file on each line. Make sure that you leave the sprelease at the end of the file since it ensures that you won't be charged for more time than you use.

Running NWChem interactively

Again, you use spsubmit to reserve nodes for your use, but this time specifying interactive access. In this instance it will not prompt you for a script or arguments. Mail will notify you when your nodes are available. Alternatively, you can keep looking with spq to see when your job starts. There is also a graphical program (xspusage) that names nodes according to who 'owns' them.

The mail message, or the file SPnodes.$JOBID in your login directory (you can get the jobid from the output of spsubmit), or xspusage tell you which nodes you have access to. Simply telnet to one of these and login with your usual id and password, or with xspusage you just click on a node you own to open an xterm on that node.

Now that you're logged into the partition you can run NWChem using the /sphome/harrison/hpcci/interactive shell script. This differs from the runnwchem script only in that output comes to the terminal and the sprelease has been removed. Thus, you can run multiple calculations in one interactive session.

Don't forget to do an sprelease with the correct jobid after completing the calculations otherwise you will be charged for all the time that you booked, rather than just the time that you used. Once sprelease is issued you no longer own the nodes and you will be unable to log into them.

E.g.,

  (... log into spnode ...)
  interactive /tmp/ /sphome/harrison/hpcci/examples/scf_h2o.nw
  interactive /tmp/ /sphome/harrison/hpcci/examples/mp2_h2o.nw
  interactive /tmp/ /sphome/harrison/hpcci/examples/dft_h2o.nw
  sprelease 070714025108
  exit

Xspusage is the easiest way to run interactively. Before using xspusage make sure that your DISPLAY environment variable is correctly set (hostname:0.0) and that your local X server permits remote window creation (use the command 'xhost +' in a local shell).

Troubleshooting

Keep the email with the runnwchem shell script output since this can be used to identify nodes with problems.

Job dies with a strange error code and an unknown error message

I've had this happen when I've specified a work directory that is not accessible to all processes (e.g., a subdirectory of /tmp). This often works for a single process, but fails for multiple processes since only the master process has access to this directory. Rerun the job specifying either just /tmp or a subdirectory of your login directory.

Job dies complaining of undefined symbols

This was a problem with incomplete instalation of a system Fortran upgrade. It should be fixed, but if the problem appears contact spsupport@mcs.anl.gov sending them the shell script output and the output in your logfile. They may ask you to rerun it with the MP_INFO variable increased to 3 or 4 in order to generate debug information. To do this take a copy of the runnwchem script and edit it.

Timeout error messages

I haven't seen this problem but apparently it can arise from excessive load from others on the Ethernet. Just resubmit the job.

Running out of disk space

Some of the lower numbered nodes apparently have some system junk sitting in the /tmp filesystem that is not automatically deleted. You can rely on only about 0.25 GB per node, though some will have up to 0.4 GB.

Accounting

Development accounts have just 125 node hours per quarter - I blew this in two days. The command 'cacReport -u userid' reports the time available to you. The accounting is not rigorously enforced and a request to spsupport to increase your time will usually be granted.

Expansion of ~ (tilde) on SP nodes

Only the user that currently has allocated an SP node can log into it. This is done by restricting entries in the node's /etc/passwd file. A side effect of this is that an SP node has no knowledge of other users. Hence, the usual shell expansion mechanism for specifying a login directory by prefacing a user name with a tilde will not work, unless you are referring to yourself. The rule of thumb is always to use fully qualified filenames.

NWChem support

Report NWChem problems, suggestions, feedback, etc., to nwchem-support@emsl.pnl.gov or via the NWChem home page.

There is a mailing list for NWChem users that can be used for announcements and discussion amoung users. To subscribe send email to majordomo@emsl.pnl.gov with the body 'subscribe nwchem-users'. You can do this with the following command

     echo subscribe nwchem-users | mail majordomo@emsl.pnl.gov

To post a message to the mailing list send it to nwchem-users@emsl.pnl.gov.

Acknowledgements

Thanks to Dave Bernholdt and Mike Minkoff for useful input to this document and to Jeff Tilson for an initial version of the execution scripts.


Prepared by RJ Harrison: Email: nwchem-support@emsl.pnl.gov.