Running for days

Run the daemon in the background or under a service manager, and restart it without losing work.

A campaign can run for days. Keep the daemon running in the background or under a service manager, and restart it without stopping the agents.

The daemon

piloop serve is the daemon. It starts the agents, scores the commits and enforces the limits of every project.

piloop serve --detach     # start in the background; output goes to ~/.piloop/serve.log
piloop status             # the first line shows the daemon's pid and uptime, or "not running"
piloop stop               # stop it; waits up to --timeout seconds
piloop restart            # restart it in place on the installed code

Without --detach, the daemon runs in the foreground. Ctrl-C stops it in the same way as piloop stop.

Only one daemon runs for each PILOOP_HOME. The daemon holds a lock on serve.pid while it runs, so a second piloop serve refuses to start. The operating system releases the lock when the process ends, so a crash leaves nothing to clean up.

Stop and restart

piloop stop stops the running agents:

• The runs of the stopped workers fail with daemon stopped. This does not count as an attempt: the ideas go back to ready.
• What each run made is pinned under refs/piloop/, with a note that its next attempt reads.
• Requested agents stay requested, and start when the daemon starts again.
• The projects stay active. The next piloop serve continues them. A campaign ends only at its [stop] limits.
• The sessions of the stopped agents are not attached to Entire.

piloop restart keeps the agents running. The daemon stops its loop, and the same process starts the installed code again, with the same pid and environment. The new daemon connects to every running agent again, and each agent finishes its run without interruption. The timeout of each agent still counts from its start.

A daemon that starts after a crash does the same: it connects again to the agents that are still running.

Under a service manager

piloop service-unit prints a unit file for your service manager. It installs nothing.

Linux (systemd)

piloop service-unit > ~/.config/systemd/user/piloop.service
systemctl --user enable --now piloop

macOS (launchd)

piloop service-unit > ~/Library/LaunchAgents/dev.piloop.serve.plist
launchctl load -w ~/Library/LaunchAgents/dev.piloop.serve.plist

The unit restarts the daemon when it fails. It sets PILOOP_HOME and your current PATH, because the daemon must find the harness CLIs on its PATH. It also carries the PILOOP_* variables that are set in your shell, such as PILOOP_SECRETS_COMMAND and PILOOP_TICK_SECONDS, and OTEL_EXPORTER_OTLP_ENDPOINT, CLAUDE_CONFIG_DIR and CODEX_HOME. It never carries PILOOP_ACTOR, PILOOP_WORKER_TOKEN or PILOOP_UNITS_FILE, because piloop sets them for one agent or one evaluation, not for the daemon. It carries no other variable, so give API keys to the daemon through PILOOP_SECRETS_COMMAND. Set the variables before you run piloop service-unit. See Secrets.

Under a service manager:

• Start and stop the daemon with the service manager.
• piloop stop stops the daemon until the service manager starts it again.
• Do not use piloop serve --detach. After the managed daemon stops, it starts a daemon that the service manager does not control.
• piloop restart and piloop upgrade keep the daemon under the service manager, because the restart happens in the same process.
• After you change the unit file, load it again. On Linux, run systemctl --user daemon-reload, then systemctl --user restart piloop. On macOS, run launchctl unload and then launchctl load -w on ~/Library/LaunchAgents/dev.piloop.serve.plist.

Logs and traces

The daemon writes one line for each span and span event to stderr. With --detach, stderr goes to ~/.piloop/serve.log. Set OTEL_EXPORTER_OTLP_ENDPOINT to also send the spans to an OpenTelemetry collector.

The events table holds what happened to each row, and is the easiest record to query:

piloop sql "SELECT at, actor, kind, payload FROM events ORDER BY id DESC LIMIT 20"