Make an agent a service, not a forgotten terminal.

Package the worker

Use a supported runtime, pinned dependencies and a service manager or container. Configure restart limits and a dedicated service user. Keep private provider credentials in protected configuration outside the application’s public files.

Save progress before actions matter

Use a durable queue or database for jobs. Assign a unique operation identifier, record external requests and reconcile uncertain outcomes. A restart should not send a second email, create a second order or charge twice.

Add useful observability

Monitor job completion, queue age, error rate and cost rather than checking only whether a process exists. Keep logs small enough to review and redact tokens, prompts and documents that do not need to be retained.

Set a clear ceiling

Limit model spending, browser concurrency, runtime and retry count. Keep an easy pause switch and a human review path for consequential actions. A smaller, well-observed worker is easier to trust than an unrestricted one.

Use a supervisor and a deliberate schedule

Systemd and Docker are common ways to make a process survive a terminal disconnect. For a systemd service, Type=exec can report execution failure more directly than a simple background launch, and Restart=on-failure can restart failures with a bounded policy. Scheduled jobs should avoid overlapping runs. Keep the working directory, service user and environment explicit. The example assumes the koala-agent user, virtual environment, worker module and private environment file already exist; adapt them before installing a unit.

• Use a dedicated service account
• Set a restart delay and failure ceiling
• Check the previous job state before starting a scheduled run

[Unit]
Description=Example agent worker
After=network-online.target

[Service]
Type=exec
User=koala-agent
WorkingDirectory=/opt/koala-agent
EnvironmentFile=/etc/koala-agent.env
ExecStart=/opt/koala-agent/.venv/bin/python -m worker
Restart=on-failure
RestartSec=10
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

Official documentation: Systemd service semantics

Set a daily budget and a duplicate boundary

Give the worker a maximum request count, spend limit and execution timeout. Use a durable idempotency record for actions that affect the outside world. A successful browser click can occur before the process receives a response; inspect the external state before trying it again. Docker Compose secret mounts can keep a secret out of the image, but permissions and the consuming application still need review.

Official documentation: Docker Compose secrets

Give a periodic job an explicit catch-up policy

Use a long-running service for a persistent queue worker and a timer-triggered oneshot service for a bounded scheduled task. With a systemd calendar timer, Persistent=true can trigger one missed activation after downtime; it does not recreate an entire business backlog. Decide whether that catch-up is desirable. RandomizedDelaySec can spread starts, while an application-level lease prevents two replicas from processing the same scheduled work.

• Record the intended timezone and inspect the next run times before enabling the schedule.
• Keep the task’s execution deadline shorter than an interval that must not overlap, or use a durable lease with expiry and ownership.
• Make the job select outstanding work from stored state rather than assuming that every previous timer event ran.
• Check the installed systemd version before using a directive from the latest manual.

# Read-only checks after creating your own matching units.
systemctl list-timers --all
systemctl status koala-agent.service
journalctl -u koala-agent.service --since today --no-pager

Official documentation: systemd: timer scheduling

Differentiate a crash loop from a stalled worker

A restart policy can recover a failed process, but repeating an invalid credential or broken migration every few seconds makes a fault noisier. Bound service starts and alert on repeated failure. A live process can also be stuck: measure age of the oldest job and the last completed operation. Test an API outage, an empty disk and a process stop with disposable jobs, then verify that the durable state prevents duplicate external actions.

• Set a provider budget and request timeout in the application as well as a CPU/memory boundary on the runtime.
• Docker containers have no useful workload-specific memory limit just because they are containers; configure and test limits deliberately.
• Store a secret outside the image and public tree, then test that crash logs and diagnostics redact it.
• Keep a manual stop route and require appropriate review for spending, production writes or external messages.

Official documentation: systemd: service supervision · Docker: container resource constraints