Domino Jobs Skill

Description

This skill helps users create, run, and manage Domino Jobs - batch executions for running scripts, training models, and processing data.

Activation

Activate this skill when users want to:

• Run a script or notebook as a batch job
• Schedule recurring jobs
• Configure job settings (hardware, environment)
• Monitor job status and results
• Run jobs via API or CLI

What is a Domino Job?

A Job is a batch execution that runs a script or command in Domino. Unlike workspaces, jobs:

• Run to completion without user interaction
• Are fully reproducible with tracked inputs/outputs
• Can be scheduled to run automatically
• Scale to any hardware tier

Creating and Running Jobs

Via Domino UI

1. Navigate to your project
2. Click Jobs in the navigation
3. Click Run
4. Configure:
   • File to Run: Script path (e.g., train.py)
   • Arguments: Command-line arguments (optional)
   • Hardware Tier: Select resources
   • Compute Environment: Select environment
5. Click Start

Via Python (requests)

import requests, os

TOKEN = requests.get("http://localhost:8899/access-token").text.strip()
BASE = os.environ["DOMINO_API_HOST"]
PROJECT_ID = os.environ["DOMINO_PROJECT_ID"]
headers = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}

# Start a job
response = requests.post(
    f"{BASE}/api/jobs/v1/jobs",
    headers=headers,
    json={
        "projectId": PROJECT_ID,
        "runCommand": "python train.py --epochs 100",
        "title": "Training run",
    }
)
job = response.json()
print(f"Job ID: {job['id']}")

Via Domino CLI

# Start a job with script
domino run train.py

# Run with arguments
domino run train.py arg1 arg2 arg3

# Wait for job to complete
domino run --wait train.py arg1 arg2

# Run direct command (not a script)
domino run --direct "pip freeze | grep pandas"

Via REST API

TOKEN=$(curl -s http://localhost:8899/access-token)
curl -X POST "$DOMINO_API_HOST/api/jobs/v1/jobs" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"projectId\": \"$DOMINO_PROJECT_ID\",
    \"runCommand\": \"python train.py\",
    \"hardwareTierId\": \"tier-id\",
    \"environmentId\": \"env-id\"
  }"

Job Commands

Run Python Script

python train.py

Run with Arguments

python train.py --data /mnt/data/train.csv --output /mnt/artifacts/model.pkl

Run Jupyter Notebook

jupyter nbconvert --to notebook --execute notebook.ipynb

Run R Script

Rscript analysis.R

Run Shell Script

bash pipeline.sh

Scheduled Jobs

Creating a Scheduled Job

1. Go to Scheduled Jobs in your project
2. Click New Scheduled Job
3. Configure:
   • Name: Descriptive name
   • Command: Script to execute
   • Schedule: Cron expression or preset
   • Hardware Tier: Resources to use
   • Environment: Compute environment

Schedule Examples

Cron Expression Format

Domino scheduled jobs use Quartz cron (six required fields, optional seventh year field; seconds first). This is not Unix five-field crontab. Use ? in day-of-month or day-of-week when the other field is set. Field rules and special characters (*, -, /, L, W, #): Quartz CronTrigger tutorial.

┌───────────── second (0-59)
│ ┌───────────── minute (0-59)
│ │ ┌───────────── hour (0-23)
│ │ │ ┌───────────── day of month (1-31)
│ │ │ │ ┌───────────── month (1-12)
│ │ │ │ │ ┌───────────── day of week (0-7, SUN-SAT)
│ │ │ │ │ │
* * * * * *

Run Modes

Sequential: Wait for previous job to complete before starting next

# Good for jobs that depend on previous output
# Example: Daily model retrain that uses previous day's data

Concurrent: Allow multiple jobs to run simultaneously

# Good for independent jobs
# Example: Hourly data refresh that doesn't depend on previous runs

Job Notifications

Email Notifications

Configure in job settings:

• Success notifications
• Failure notifications
• Specific email addresses

Model API Updates

Trigger Model API republish after job completes:

1. In scheduled job settings
2. Select Update Model API option
3. Choose the Model API to update

Accessing Job Results

Output Files

Files written to /mnt/ directories are available after job completion:

• /mnt/results/ - Custom outputs
• /mnt/artifacts/ - Model artifacts

Job Logs

View logs in Domino UI or via API:

# Get job logs
logs = domino.runs_get_logs(run_id)
print(logs)

Stdout/Stderr

All print statements and errors are captured in job logs.

Environment Variables in Jobs

import os

# Domino-provided
run_id = os.environ.get('DOMINO_RUN_ID')
project_name = os.environ.get('DOMINO_PROJECT_NAME')
username = os.environ.get('DOMINO_USER_NAME')

# Custom (set in project or job settings)
api_key = os.environ.get('MY_API_KEY')

Job Best Practices

1. Parameterize Scripts

import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--data-path', required=True)
parser.add_argument('--model-output', required=True)
parser.add_argument('--epochs', type=int, default=100)
args = parser.parse_args()

2. Log Progress

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

logger.info("Starting training...")
logger.info(f"Epoch {epoch}/{total_epochs}")
logger.info("Training complete!")

3. Handle Failures Gracefully

try:
    train_model(data)
except Exception as e:
    logger.error(f"Training failed: {e}")
    # Save checkpoint
    save_checkpoint(model, "checkpoint.pt")
    raise

4. Save Artifacts

import joblib

# Save model
joblib.dump(model, "/mnt/artifacts/model.joblib")

# Save metrics
with open("/mnt/artifacts/metrics.json", "w") as f:
    json.dump(metrics, f)

Monitoring Jobs

Job Status

• Pending: Waiting for resources
• Running: Currently executing
• Succeeded: Completed successfully
• Failed: Exited with error
• Stopped: Manually cancelled

Check Status via API

status = domino.runs_status(run_id)
print(f"Status: {status['status']}")
print(f"Started: {status['startedAt']}")

Stop a Running Job

domino.runs_stop(run_id)

Troubleshooting

Job Fails Immediately

• Check script syntax
• Verify file paths exist
• Check environment has required packages

Job Times Out

• Increase hardware tier resources
• Optimize code performance
• Check for infinite loops

Out of Memory

• Use larger hardware tier
• Optimize data loading (chunking, generators)
• Clear variables when no longer needed

API Reference

Before writing or verifying any API call, confirm endpoint paths and field names in API-SPECS.md (use the Public routes section for /api/jobs/…; use Internal routes for legacy /v4/jobs/… and /v4/runs/…). Use public docs for workflow context and field explanations.

Get the cluster base URL: $DOMINO_API_HOST (injected by Domino into every workspace, job, and app).

Public docs (workflow context and field explanations):

• API Guide
• Create and run Jobs
• Scheduled Jobs
• Work with Jobs