How to Automatically Retrieve Results / Postprocess When Job is Done

Ever wish you could set up a script to automatically retrieve and / or process results as soon as your job is finished? Read on to learn more.

Installing the Script

OIT staff have developed a shell script to simplify this task. It works by periodically checking the status of the job, and letting the shell continue when it is done. Download the "sjobwait" script below.

sjobwait

Now transfer the script to ACRES and put it somewhere where you keep utilities for easy access through your PATH environment variable. If you don't have such a directory, or you're not familiar with the PATH environment variable, follow the steps below to establish this directory and set your PATH.

Create appropriately named directory, such as "bin", in your home directory, and add its path to your PATH environment variable:
```
[user@acres-head0 ~]$mkdir bin[user@acres-head0 ~]$export PATH=${HOME}/bin:$PATH
```

The PATH environment variable lists the paths that the computer will search to find commands, separated by colons. In order to make the change to your PATH variable permanent, so it is set every time you log in, edit your ".bashrc" file located in your home directory, and add the following line to it, at the end. (Note the ".bashrc" file is a hidden file so you will not see it with "ls". Use "ls -a" to see it.
```
PATH=${HOME}/bin:$PATH
```

Now copy the sjbowait script into your new ~/bin directory.

Using the sjobwait Script with a New Job

Suppose you have a script that submits your job and post-processes the results. To use sjobwait, "pipe" ( | ) the output from the sbatch command into sjobwait, and then optionally provide the number of seconds to wait () between checking if the job is done. If is not specified, the default sleeptime is 30 seconds.

#!/usr/bin/sh## sutff to do before submitting job#submit jobsbatch myjobscript.sh|sjobwait <sleeptime>#postprocessing operations after job is done

You should see the following output, as your job is running:

When the job is no longer RUNNING, or PENDING, your script will continue.

Using the sjobwait Script with an Existing Job

If you already have a job running, that you'd like to continually check the status of, and wait for, you may use sjobwait in a script in the following way:

sjobwait <sleeptime> <jobid>

sjobwait will wait for the job with the "jobid" to finish, before continuing.

Using sjobwait Reomtely

To use sjobwait remotely, simply use it as you would on ACRES, as part of your submitted command.

From a local Windows computer, using the PuTTY plink.exe command, this would look like:

From a local Linux computer, using ssh:

$ssh -t user@acres.clarkson.edu "cd path/to/rundir/;sbatch test_submit.sh|sjobwait 5"