Chapter 5 Jobs

Jobs are similar to tools. They are independent compute resources, but instead of a UI, they are focused on running a single script. The use case for jobs is a long-running process (like web scraping, cross-validation, intense munging or calculations) that you want to run once and receive output from.

IMPORTANT: Jobs require absolute paths, both within the scripts being run and the command itself. Jobs have a set of variables, like tools. However, there is some variation.

  • Application: Use the tool type with the libraries you want to use in your job script
  • Command: Use a terminal command with the absolute path of the script to execute
  • R scripts example (Rscript /home/rstudio/myScript.R)
  • Python Scripts example (python /home/matrix/myScript.py)
  • Docker Image: Same as Tools
  • Mount Path: Same as Tools useful for knowing tools absolute path

The following is an example job that runs an R script and writes a CSV back to the file system.