Chapter 5 Jobs
Jobs are similar to tools. They are independent compute resources but instead of a UI they are focused on running a single script. The use case for jobs is long running process (like web scraping, cross-validation, intense munging or calculations) that you want to run once and receive output from.
IMPORTANT: Jobs require absolute paths, both within the scripts being run and the command itself. Jobs have a set of variables, like tools. However, there is some variation.
- Application: Use the tool type with the libraries you want to use in your job script
- Command: Use a terminal command with absolute path of the script to execute
- R scripts example (Rscript /home/rstudio/myScript.R)
- Python Scripts example (python /home/matrix/myScript.py)
- Docker Image: Same as Tools
- Mount Path: Same as Tools useful for knowing tools absolute path
The following is an example job that runs an R script and writes a csv back to the file system.