Chapter 5 Jobs

Jobs are similar to tools. They are independent compute resources but instead of a UI they are focused on running a single script. The use case for jobs is long running process (like web scraping, cross-validation, intense munging or calculations) that you want to run once and receive output from.

IMPORTANT: Jobs require absolute paths, both within the scripts being run and the command itself. Jobs have a set of variables, like tools. However, there is some variation.

  • Application: Use the tool type with the libraries you want to use in your job script
  • Command: Use a terminal command with absolute path of the script to execute
  • R scripts example (Rscript /home/rstudio/myScript.R)
  • Python Scripts example (python /home/matrix/myScript.py)
  • Docker Image: Same as Tools
  • Mount Path: Same as Tools useful for knowing tools absolute path

The following is an example job that runs an R script and writes a csv back to the file system.