--- title: "Schedule a job" summary: "Declare a scheduled command, bound retries, coordinate deployments, and inspect host run records." description: "Run one container command on a host timer with bounded retries, deployment coordination, and useful run records." status: shipped read_when: - "Adding a nightly or recurring task" - "Working out why a scheduled run failed, or whether it ran at all" - "Running a scheduled job now with different parameters" - "Understanding why a cron expression was refused" - "Working out how a scheduled job resolves secrets with no Onebox process running" --- ```yaml workloads: nightly-dump: role: Job image: postgres:17 command: ["sh", "-c", "pg_dump \"$POSTGRES_URL\" | gzip > /backups/$(date -u +%F).sql.gz"] dataEffect: None needs: [postgres] volumes: [{name: backups, path: /backups}] schedule: cron: "0 2 * * *" timezone: Europe/Berlin timeout: 45m catchUp: true ``` ## It becomes a host timer A `schedule` is translated into a timer on the host. It fires **without any Onebox process running** and survives a reboot. There is no daemon, no listening port, and nothing to keep alive. That design decision has a visible consequence elsewhere: encrypted environment entries are decrypted into the release when it is staged and *stay* there, because a timer firing at 02:00 must resolve the values the deploy resolved. Each run is a systemd oneshot with a wall-time limit. `timeout` defaults to `1h`; when it expires, systemd terminates the run and records `timeout` as its result. Set a longer duration for jobs that legitimately need it. `catchUp` defaults to `true`: if the host was off at the scheduled time, the timer runs once after it returns. Set it to `false` for time-sensitive work that should be skipped if it cannot run on time. ## Retry inside one firing By default a failed run is not retried; the next attempt is the next cron elapse, which is the right answer for anything that fires every few minutes. For a job whose failures are usually transient, a rate limit or an upstream that was busy for a moment, declare a bounded retry: ```yaml schedule: cron: "0 * * * *" timeout: 45m retry: {attempts: 3, backoff: 30s, maxBackoff: 10m} ``` `attempts` counts every attempt including the first, so the default of `1` is today's single run. After a non-zero exit the runner sleeps `backoff`, doubles it after each further failure, and never sleeps longer than `maxBackoff`. Every attempt runs inside the same timer firing, under the same locks and the same `timeout`; when the timeout expires the run ends, with no further attempt. Validation refuses a retry whose worst-case backoff is not smaller than the timeout, because the last attempt could never start and the record would say it did. With `deployLock: Exclusive` the deploy lock is held through the sleeps, so a deploy waits for the run to finish; with `deployLock: Pinned` the release lease is held instead and every attempt runs the same release. ## Deployment coordination is exclusive by default An exclusive host-fired job takes an application-wide kernel lock for its whole container run. A pinned job takes that rendezvous as a shared reader until it establishes its immutable release lease. Different pinned jobs may still start together. Every operation that establishes Onebox's fenced application lock takes the rendezvous as an exclusive writer while publishing its ownership. Readers and writers wait briefly for a handoff; the wait is shortened for a job whose own timeout is near the ten-second default. A timer that still collides does not modify Docker beside a deploy: it records a `skipped` run with the reason and exits cleanly. An application lock older than its TTL is treated as expired, exactly as a deploy treats it, so a runner that died mid-operation cannot silence a timer forever. The target must provide `flock` (part of `util-linux` on supported Linux hosts). Onebox refuses to install or run schedules when that serialization primitive is missing. For a long-running job that does not change shared data, opt into a pinned release instead: ```yaml dataEffect: None schedule: cron: "0 2 * * *" timeout: 6h deployLock: Pinned ``` The runner briefly meets deployment under the same scheduling mutex, resolves `current` to one immutable release, and takes a shared lease on that release. It then releases the application-wide mutex and keeps only a per-job lock, so a data-effect-free deployment can proceed but a second instance of the same job cannot overlap. Deployments containing migration, destructive, or unknown-effect jobs, and deployments containing untyped lifecycle hooks, wait for the pinned job to finish. Other application operations also remain exclusive. Release cleanup preserves the leased Compose document, environment files, and release-bound mounts until the job exits. `ob status` reports the policy, the pinned release, start time, and timeout while it runs. `deployLock: Pinned` is accepted only for a Onebox-rendered job declaring `dataEffect: None`. Migration, destructive, unknown-effect, and adopted-Compose jobs remain exclusive because pinning their files cannot prove that a concurrent deployment is safe for the data or external files they use. Destroy and every other non-deploy application operation also refuse while any pinned release is leased. ## Reconcile after upgrading Onebox The `ob` binary is agentless: upgrading it on the operator workstation does not silently connect to or mutate a target. Existing timers therefore keep the unit content written by the previous runner until the next deploy or an explicit schedule apply: ```sh ob schedule apply ``` This rewrites every declared timer, service, runner, and failure notifier from the current contract, and removes units for jobs no longer declared. It runs under the same application lock, schedule mutex, fence, and journal boundary as a deploy, but it does not stage or activate a release. Run it after upgrading Onebox when an application may not be deployed again soon. Run records are written by the unit, so a host whose units predate them has none until this command or a deploy rewrites them. Until then `ob status` falls back to what systemd retained, reports a failure as it always did, and says that no run record exists yet. ## Every run leaves a record When a run ends, for any reason, the unit's `ExecStopPost` writes one record to the host journal with the syslog identifier `onebox-run` and the job's unit in its `ONEBOX_UNIT` field, so `journalctl SYSLOG_IDENTIFIER=onebox-run ONEBOX_UNIT=onebox-job- -o cat` on the host is the raw history: ```json {"run":"a3f9…","job":"nightly-dump","trigger":"timer","operation":"","release":"20260905-140000-ab12cd","started_at":"2026-09-05T02:00:01Z","finished_at":"2026-09-05T02:04:37Z","duration_s":276,"attempts":2,"exit_status":0,"outcome":"success","reason":"","inputs":{}} ``` `run` is systemd's invocation id, so the record and the run's own log share a key. `outcome` is one of `success`, `failure`, `timeout`, or `skipped`. A skip is a firing that met a running instance of the same job, timed out waiting for the application scheduling rendezvous, or found an application operation holding the deploy lock. The runner writes the `reason` and exits before any container starts, so the unit is not failed and the job's own exit status is never mistaken for a skip. Skips are recorded because a job that is silently never running looks exactly like one that works: one skip is timing, and three in a row are reported by `ob status` as an issue. A skip is news about timing, not about the job, so it never clears a failure: `ob status` keeps reporting the newest run that actually happened, and says that nothing has run since. A job that selects `skipped` in `notify` is told its run did not happen, not that it failed. Read the records back from the workstation: ```sh ob schedule list # every job, its timer state and next elapse ob job history nightly-dump # records, newest first; -n 50 for more ob job logs nightly-dump # the journal of the newest run ob job logs nightly-dump --run a3f9… ``` `ob status` reads the same records. Each scheduled job's line carries the next elapse, the last outcome with its duration and attempt count, and, while a run is in progress, the attempt it is on. A last outcome of `failure` or `timeout` is reported as divergence: ```text schedule nightly-dump last run failed: timeout (exit 143) ⚠ ``` Retention is the journal's. A host whose journal lives in memory keeps records only since its last boot; `ob status` says so on the job's line, and the fix is a persistent journal (`/var/log/journal`), which the supported images have by default. ### Notifications per outcome `schedule.notify` selects which outcomes send the configured notifications. The default is `[Failure, Timeout]`, which is what the failure notifier always did. Add `Success` for a job whose completion someone waits for, and `Skipped` for a job whose firings collide often enough that silence would hide it: ```yaml schedule: cron: "0 2 * * *" notify: [Failure, Timeout, Skipped] ``` Each selected outcome sends a bounded, fail-open POST to every webhook whose own `on` list accepts that class of outcome. The payload carries the run id as its `deploy_id`, so `ob job logs --run ` finds the run; it carries nothing else about the run, because notifications cross the host trust boundary and diagnostics stay on the host. Onebox writes the webhook handler root-only beside the unit, so credentials in webhook paths do not appear in `ExecStart` or `ExecStopPost`. A delivery failure is recorded in the unit's journal and never replaces the job's own result. The webhook must be reachable from the target host; timer delivery uses its network path, not the operator workstation's. ## Cron is translated exactly, or refused ``` 0 2 * * * ✓ every day at 02:00 0 2 * * 1 ✓ every Monday at 02:00 0 2 1 * 1 ✗ schedule_untranslatable ``` The last one declares a day-of-month **and** a day-of-week. Cron treats that as "either matches": it fires on the 1st and on every Monday. Onebox refuses it at load because the host scheduler cannot preserve that meaning. `timezone` takes an IANA zone name and defaults to `UTC`. Generated timers use one-second accuracy so systemd does not coalesce distinct cron minutes into its default one-minute wake-up window. ## Jobs still declare a data effect `dataEffect` is required on every job, scheduled or not. A nightly report is `None`; a nightly prune is `Destructive`. The rollback and abort gates read it, and a job that lies about it defeats them. ## Stopping a job without deleting it ```sh ob schedule pause nightly-dump --reason "upstream is returning garbage" ob schedule resume nightly-dump ``` Pausing stops the timer and nothing else. The units stay installed and later deploys keep updating them, so a fix still lands on the host while the job stays stopped. A run already under way is left alone: this stops the next firing, it does not kill work in progress. The pause is recorded on the host, not in the project, which is what lets it survive a deploy. Reconciliation will not start a paused timer, so a pause cannot be undone by someone else's release. `--reason` is required, and is kept with the operator and the time, because a job that is paused and not running is indistinguishable from one that is broken: ```text schedule nightly-dump PAUSED — by ops@example.com; since 2026-09-06T10:00:00Z; upstream is returning garbage ``` `ob status` prints that line whether or not anything else is wrong, and does not count the stopped timer as divergence. It does still report everything else about the job: a unit that will not load, or a run that failed, is printed beside the pause and still fails the command, because a pause explains a stopped timer and nothing more. A stopped timer with no pause behind it is also still a fault. `ob schedule list` shows a paused job's timer as `paused` rather than `inactive`, so the table does not read the same for a job somebody stopped and one that broke. Both `ob schedule pause` and `ob schedule resume` are journaled and appear in `ob audit` with the operator. Neither will record a change that did not happen: pausing an already-paused job is refused rather than overwriting who stopped it and why, and resuming a job nobody paused is refused rather than logged. Both take `--break-lock`, because a running job holds the application lock for its whole run and a crashed deploy leaves one behind. Resuming starts the timer again; the next run is the next scheduled elapse. It does not run the job now, and does not make up firings missed while paused; `ob job run` is how you ask for an immediate run. Deleting a job from the project clears its pause along with its units, so a name reused later does not come back stopped for a reason from a previous life. ## Run one now with inputs A scheduled job may declare inputs: named parameters that reach the container as environment variables. A timer firing uses the defaults; an operator may run the job now and override them. ```yaml workloads: source-sync: role: Job image: ghcr.io/acme/ingest:1.8.2 command: ["./ingest", "sync"] dataEffect: None inputs: SOURCE: enum: [catalog, prices, reviews] default: catalog description: Which upstream to sync. SINCE: pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})?$' default: "" description: Only records changed since this date. Empty means the stored cursor. schedule: cron: "0 * * * *" deployLock: Pinned ``` ```sh ob job run source-sync --input SOURCE=prices --input SINCE=2026-09-01 ``` Names are upper-case identifiers outside Onebox's `ONEBOX_` namespace and may not repeat an `env` key. Each input declares exactly one of `enum` or `pattern`, and a `default` that satisfies it; a pattern matches the whole value. Whatever the pattern allows, a value may not contain a double quote, a backslash, or a control character, and is at most 256 bytes. Those rules are what let the runner hand values to the container without escaping anything. `ob job run` validates every value on the workstation, refuses if the unit is already running or an operator run is still pending, journals the request as `schedule_run` with the operator and the inputs, and starts the unit. It holds the application lock only while writing and journaling, because the runner skips a run that meets that lock. The host record of the run carries the operation id, so `ob audit` and `ob job history` join on it. The command follows the host-supervised unit by default; `--detach` returns once the unit accepts the run. Only a job with `dataEffect: None` accepts inputs. Every operator invocation uses a sealed plan; migration jobs additionally retain their approval and backup-report gates and remain attached so Onebox can capture result evidence. The runner tells an operator activation from a timer firing by the `TRIGGER_UNIT` variable systemd sets on timer activations, which that project introduced in version 252. Ubuntu 24.04 and Debian 12 qualify; Ubuntu 22.04 and Debian 11 do not. Scheduled jobs still run on those older hosts, unchanged. Two things narrow: - A job that declares `inputs` is refused, by `ob preflight` and by `ob deploy`, before anything is staged. Without `TRIGGER_UNIT` the next timer firing would read the file meant for an operator's run. - `ob job run` is refused for the same reason. The timer keeps firing. The records on such a host say `trigger: unknown`, because the runner cannot observe what started it and will not guess. ## Keep application workflows in the application Onebox runs the job's container command. A retry starts that command again; it does not resume from a completed step. A reboot does not resume interrupted work. Make repeated attempts safe in the application and store any progress there. A command can perform several operations internally; the application owns their ordering, checkpoints, and outputs. ## Running one by hand ```sh ob job plan nightly-dump --out ob-job-plan.json ob approve --plan ob-job-plan.json --out ob-job-approval.json ob job run --plan ob-job-plan.json --approval ob-job-approval.json ``` When the operator job also declares `schedule`, Onebox submits this run to the same installed systemd unit used by its timer. The host therefore owns the container, timeout, overlap lock, retries, and run record if the SSH session or operator terminal disappears. The command follows the unit by default and shows elapsed time. Ctrl-C stops following but does not stop the host job; use `--detach` to return immediately after the unit accepts it. Inspect either form with `ob job history ` and `ob job logs `. Jobs without `schedule` keep the direct foreground path. Migration jobs also stay attached because their result evidence is part of the approved operation. `ob job plan` accepts a job when `operatorRun` is `Allowed`. This property is independent of deployment: `deploymentPhase` decides whether the job runs in the deploy graph, while `operatorRun` decides whether an operator may invoke it. By default, phase `None` allows operator runs; release-phase jobs disable them unless explicitly enabled. A declared `deploymentPhase: None` job remains in the digest-pinned release runtime but never joins the deploy graph. Its job plan binds the current serving release, runtime digest, immutable image, data effect, target, and expiry. Automation supplies the saved plan and its separately recorded local confirmation; migration jobs may also need the exact plan-bound backup report. For an undeclared emergency command, the escape hatch remains: ```sh ob exec --reason "incident investigation" -- ``` > **`ob exec` is an audited escape hatch** > > Nothing it changes belongs to a release and Onebox cannot describe, roll back, > or make its effect idempotent. The journal stores the reason, target, operator, > outcome, and command digest, but never the command bytes or passthrough output. > The reason is durable metadata, so do not put a secret in it. For a job that participates automatically in a deploy, choose a release phase: ```yaml deploymentPhase: PreRelease operatorRun: Allowed # optional: also permit ob job plan/run ``` The three properties are independent. `deploymentPhase` controls deploy hooks, `schedule` controls timer activation, and `operatorRun` controls explicit invocation. A job declaring `schedule:` and `deploymentPhase: PreRelease` runs both at its cron time and on every deploy. For a timer-only job, leave `deploymentPhase` at `None`; set `operatorRun: Disabled` too if operators must not start it explicitly. See [`workloads`](/docs/reference/fields/workloads) for every field.