Operations¶

How the orchestrator is installed, upgraded, checked, launched, supervised, and reconfigured: package installation and rollback, continuous integration, documentation publishing, run modes, the systemd user service, and when an edited .env takes effect. The environment-variable reference is in ../configuration.md, which also routes to the observability settings beside this page.

Package installation, upgrades, and rollback¶

Install the published PyPI package with pipx 1.16.0 or newer. This minimum supports pipx pin and pipx unpin, reuse of saved --pip-args during upgrades, and forced installs into existing uv environments (changelog). Check pipx --version and upgrade older pipx before following this guide. Python 3.12, 3.13, and 3.14 are tested. Poetry and a clone of the orchestrator repository are only needed for development. The maintainer guide describes release publication.

Install and configure¶

pipx install chipping-orchestrator
pipx ensurepath

Open a new terminal after pipx ensurepath if the command is not yet on PATH. If pipx's default interpreter is older than Python 3.12, select an installed supported one with pipx install --python python3.12 chipping-orchestrator. pipx keeps the package's dependencies in an isolated environment and exposes the installed chipping-orchestrator command. See the pipx command reference for installer options.

The forced install examples below use --python python3.12; substitute the interpreter used by your installation. Pass it on every forced install: with the uv backend, pipx install --force can reset an existing environment to pipx's default Python when --python is omitted. Ignore pipx's --python is ignored when --force is passed warning for uv-backed environments and keep --python. On pip-backed environments, a forced install never changes the interpreter, even with --python; use pipx reinstall --python to change it on either backend.

Check the installed version with pipx runpip chipping-orchestrator show chipping-orchestrator, then download that version's templates below and configure the live settings. Keep configuration and logs outside the pipx environment.

Download the chosen release's files¶

Replace X.Y.Z with the installed version, including any prerelease suffix; its release tag is vX.Y.Z. Run these blocks in the same shell. Download both templates without cloning the orchestrator repository:

ORCHESTRATOR_VERSION='X.Y.Z'
ORCHESTRATOR_RELEASE_DIR="$HOME/.local/share/chipping-orchestrator/releases/v${ORCHESTRATOR_VERSION:?}"
(
  set -eu
  umask 077
  mkdir -p "${ORCHESTRATOR_RELEASE_DIR:?}"
  for ORCHESTRATOR_TEMPLATE in .env.example .env.example.advanced; do
    curl --fail --location --output "${ORCHESTRATOR_RELEASE_DIR:?}/$ORCHESTRATOR_TEMPLATE" \
      "https://raw.githubusercontent.com/chippingway/chipping-orchestrator/v${ORCHESTRATOR_VERSION:?}/${ORCHESTRATOR_TEMPLATE}"
  done
)

Stop if a download fails. Retain these version-matched files for upgrade comparisons and rollback; their directory does not select targets or become a launch directory.

Configure live settings¶

For first setup only, copy the basic template into the installed configuration location and set an absolute log directory outside the Python environment. This block refuses to overwrite an existing .env:

(
  set -eu
  umask 077
  mkdir -p "$HOME/.config/chipping-orchestrator" "$HOME/.local/state/chipping-orchestrator/logs"
  test ! -e "$HOME/.config/chipping-orchestrator/.env"
  cp "${ORCHESTRATOR_RELEASE_DIR:?}/.env.example" "$HOME/.config/chipping-orchestrator/.env"
  printf '\nLOG_DIR=%s\n' "$HOME/.local/state/chipping-orchestrator/logs" >> "$HOME/.config/chipping-orchestrator/.env"
)

The block appends the expanded absolute LOG_DIR path at the end of .env; the file does not expand $HOME or ~. To change the path, edit that line rather than add another assignment: the first occurrence in .env wins. orchestrator.log and the default analytics.jsonl then live under ~/.local/state/chipping-orchestrator/logs. Without this override, LOG_DIR defaults to <REPO_ROOT>/logs, inside the installed environment's site-packages. Recreating or uninstalling that environment can delete those logs. For an existing installation using the default, stop the poller, copy its logs to the external directory, and set an absolute LOG_DIR before replacing the environment. Preserve explicit ANALYTICS_LOG_PATH and TRAJECTORY_LOG_PATH overrides outside the environment too.

Edit ~/.config/chipping-orchestrator/.env following basic setup: set REPOS to existing local target clones, ALLOWED_ISSUE_AUTHORS, and HITL_HANDLE, and adjust agent routing if needed. Copy only wanted operational overrides from .env.example.advanced into that same file. Keep GitHub tokens in per-repository token files or the launch environment (credentials); .env does not load them. The launch directory's .env is never read by an installed package.

Select an existing environment¶

pipx list shows the installed package and its exposed command. Use pipx runpip chipping-orchestrator for package inspection and dependency checks; it selects the package's isolated environment without activating it. command -v chipping-orchestrator shows the absolute launcher path to use in a service unit.

Check compatibility and start¶

Review the installed release's notes for Python and agent CLI requirements, settings changes, and workflow-state compatibility. Check the package version, dependency consistency, CLI, and local configuration before starting a poller:

(
  set -eu
  cd "$HOME"
  pipx runpip chipping-orchestrator show chipping-orchestrator
  pipx runpip chipping-orchestrator check
  chipping-orchestrator --help
  "$(pipx environment --value PIPX_HOME)/venvs/chipping-orchestrator/bin/python" -c \
    'from orchestrator.runtime.startup import require_issue_authors; require_issue_authors(); print("Configuration OK")'
)

Changing to $HOME keeps the configuration check outside an orchestrator source checkout so it imports the installed package. The check reads ~/.config/chipping-orchestrator/.env, validates local settings and target checkouts, and checks the required author allowlist without connecting to GitHub or polling. Run it as the service user with the same pipx location and process/service overrides as the real launch. It does not prove token permissions, agent authentication, or compatibility with live issue state. --help checks the CLI only; --once performs a real polling tick and can start agents. Stop on a failed check and repair the settings before starting or resuming polling.

Start in the foreground:

chipping-orchestrator

Startup diagnostics identify invalid settings, missing REPOS, invalid target checkouts, or an empty ALLOWED_ISSUE_AUTHORS. Repair the live .env or overriding process/service environment before restarting. Unknown or removed settings may be ignored, so diagnostics do not replace release-note and template comparisons. For persistent supervision, use the systemd user service.

Moving from a dedicated venv¶

For the v0.13.0 installation route at ~/.local/share/chipping-orchestrator/venv, record the installed version with ~/.local/share/chipping-orchestrator/venv/bin/python -m pip show chipping-orchestrator. Keep that exact version and its tested runtime dependency pins during migration; upgrade separately using the upgrade checklist. Before stopping, read the recorded release's notes, set ORCHESTRATOR_VERSION to that version, and download its templates and constraints.txt using the pinned procedure. Retain a configuration backup and the service unit as described under upgrades.

Choose a safe stopping point, run systemctl --user stop orchestrator.service (or Ctrl+C the foreground poller), and wait for shutdown. Deactivate the dedicated venv if it is active, then run the constrained pipx install command from that procedure with the recorded version and its constraints, followed by pipx ensurepath. The procedure explicitly selects a supported Python interpreter. Open a new terminal if needed so the pipx command is on PATH.

Keep ~/.config/chipping-orchestrator/.env and its absolute LOG_DIR, along with any explicit analytics or trajectory paths outside the environment. Skip the first-setup template copy. Compare the downloaded templates with the existing settings and run the compatibility checks. Confirm the installed version matches the recorded version before restarting. Replace the service's ExecStart with the absolute pipx launcher path shown by command -v chipping-orchestrator. Run systemctl --user daemon-reload and systemctl --user start orchestrator.service, then check startup output and polling logs.

Keep the dedicated venv until the new installation passes these checks. If migration fails, stop the new poller, restore the saved unit and compatible configuration, reload systemd, and restart the dedicated-venv service, honoring the release's rollback restrictions. Remove the old venv only after a successful migration; subsequent upgrades and rollback use the pipx procedures below.

Upgrade a package¶

Package changes are operator-initiated. Use pipx upgrade-all --skip chipping-orchestrator for routine updates of other apps so they do not change the package under a running poller. Before upgrading:

1. Read the release notes for the version you intend to install and record that version. Check compatibility with the current Python, agents, targets, and live workflow state. Record the current Version from pipx runpip chipping-orchestrator show chipping-orchestrator and retain its templates; if they are missing, run the template download for that version first.
2. Choose a safe stopping point, preferably with no live agent child. Stop the supervisor with systemctl --user stop orchestrator.service, or Ctrl+C the foreground command, and wait for shutdown. Stop every poller using this pipx environment before changing it.
3. Back up settings and retain the previous release selection:

sh ORCHESTRATOR_PREVIOUS_VERSION='A.B.C' ORCHESTRATOR_PREVIOUS_RELEASE_DIR="$HOME/.local/share/chipping-orchestrator/releases/v${ORCHESTRATOR_PREVIOUS_VERSION:?}" ORCHESTRATOR_BACKUP_DIR="$HOME/.local/share/chipping-orchestrator/backups/$(date -u +%Y%m%dT%H%M%SZ)" ( set -eu umask 077 mkdir -p "${ORCHESTRATOR_BACKUP_DIR:?}" cp -a "$HOME/.config/chipping-orchestrator" "${ORCHESTRATOR_BACKUP_DIR:?}/configuration" printf '%s\n' "${ORCHESTRATOR_PREVIOUS_VERSION:?}" > "${ORCHESTRATOR_BACKUP_DIR:?}/version.txt" )

Replace A.B.C with the recorded version. Also retain custom service units, referenced environment files, and process/service overrides with their existing access restrictions. Keep credentials, clones, worktrees, and external log paths in place. 4. For an unconstrained, unpinned installation, upgrade to the latest available release:

sh pipx upgrade chipping-orchestrator pipx runpip chipping-orchestrator show chipping-orchestrator

An enabled release-age cooldown is saved and reused by upgrades and forced installs unless PIPX_COOLDOWN or explicit --cooldown overrides it; pipx reinstall always uses the saved window and ignores PIPX_COOLDOWN. If it hides a fresh release you have reviewed, add --cooldown 0 (or a shorter window) to the upgrade or recovery install, for example pipx install --force --python python3.12 --cooldown 0 "chipping-orchestrator==X.Y.Z". pipx saves the chosen window, including zero, for later upgrades and forced installs; restore your usual window explicitly on the next package change if needed.

For a constrained installation, repeat the pinned procedure with the new version and its own constraints. pipx upgrade reuses the --pip-args saved at installation, including the previous release's constraint path. To return to unconstrained, unpinned releases, use the forced reinstall command in that section. After a rollback without constraints, run pipx unpin chipping-orchestrator before the upgrade above, with the poller stopped.

Stop if installation fails or the displayed Version differs from the version reviewed in step 1. For an unconstrained installation, recover that reviewed version explicitly (replace X.Y.Z with it):

sh pipx install --force --python python3.12 "chipping-orchestrator==X.Y.Z" pipx runpip chipping-orchestrator show chipping-orchestrator

Leave the poller stopped until the intended version is installed and passes checks. 5. On every upgrade, repeat the template download for the intended version and compare configuration before resuming. 6. Repeat compatibility checks, then start chipping-orchestrator again or run systemctl --user start orchestrator.service. Check startup output and logs for connections and polling. A changed unit needs systemctl --user daemon-reload; editing .env alone does not.

If a release requires a different Python interpreter, follow Change the Python interpreter before installing it.

Change the Python interpreter¶

Provision a supported interpreter and stop every poller using the environment. Record the installed Version with pipx runpip chipping-orchestrator show chipping-orchestrator. For a pinned installation, unpin before reinstalling and restore the pin after a successful reinstall:

pipx unpin chipping-orchestrator
pipx reinstall --python python3.13 chipping-orchestrator
pipx pin chipping-orchestrator
pipx runpip chipping-orchestrator show chipping-orchestrator

Substitute the chosen interpreter; skip the unpin and pin commands for an unpinned installation. Reinstall reuses the saved package requirement and installation options, including constraints and the cooldown window. It ignores PIPX_COOLDOWN and has no --cooldown option.

A saved ==X.Y.Z keeps that version, while a bare package name can install the newest available release. pipx upgrade and pipx upgrade-all replace a saved version requirement with the bare package name, so a later interpreter-only reinstall can also upgrade the orchestrator. The constrained procedure avoids those upgrade commands and retains ==X.Y.Z.

Confirm the displayed Version matches the intended release; for an interpreter-only change, it must match the recorded version. If it differs, use the explicit version install in Upgrade a package or the pinned procedure with the chosen interpreter to install the intended release. Keep the poller stopped if any command fails. Retain the external settings and logs, then repeat compatibility checks before restarting.

Compare configuration across versions¶

Release notes identify added, renamed, or removed settings, changed defaults, stricter validation, and any migration or rollback restrictions. Compare the retained templates and the live settings in the same shell as the upgrade:

diff -u "${ORCHESTRATOR_PREVIOUS_RELEASE_DIR:?}/.env.example" "${ORCHESTRATOR_RELEASE_DIR:?}/.env.example"
diff -u "${ORCHESTRATOR_PREVIOUS_RELEASE_DIR:?}/.env.example.advanced" "${ORCHESTRATOR_RELEASE_DIR:?}/.env.example.advanced"
diff -u "${ORCHESTRATOR_RELEASE_DIR:?}/.env.example" "$HOME/.config/chipping-orchestrator/.env"

diff exits 1 when files differ; review those differences. Merge required changes into the existing .env rather than replacing it. Keep local target paths, author lists, agent specs, budgets, and other deliberate overrides. For a changed default, decide whether to adopt it or explicitly retain the previous value. Check selected advanced settings and service overrides too; commented examples are not active settings.

Keep the backup and record edits. Older versions can ignore new settings or reject values accepted by a newer one, so review startup diagnostics after every repair and after rollback.

Return to the previous package version¶

Read both releases' rollback notes: restoring .env does not undo workflow state written on GitHub. Before returning to a version that cannot handle late split state, complete the workflow drain with the newer package running and DECOMPOSE=off. Honor other release-specific restrictions, then stop the process and supervisor and wait for shutdown.

Select the retained backup explicitly, replacing YYYYMMDDTHHMMSSZ with its timestamp:

ORCHESTRATOR_BACKUP_DIR="$HOME/.local/share/chipping-orchestrator/backups/YYYYMMDDTHHMMSSZ"
ORCHESTRATOR_PREVIOUS_VERSION="$(cat "${ORCHESTRATOR_BACKUP_DIR:?}/version.txt")"

Stop if the backup or version record is missing. For a constrained installation, repeat the pinned procedure with the recorded previous version and that version's own constraints. For an unconstrained installation, reinstall the recorded version:

pipx install --force --python python3.12 "chipping-orchestrator==${ORCHESTRATOR_PREVIOUS_VERSION:?}"

Stop if installation fails. After a successful rollback, run pipx pin chipping-orchestrator to prevent pipx upgrade and pipx upgrade-all from replacing the chosen version (pipx pinning). Unpin only for a planned upgrade or interpreter change while the poller is stopped.

Restore or translate configuration only where the older version requires it. Preserve the current .env before restoring the backup; if .env.pre-rollback already exists, use another filename throughout:

(
  set -eu
  umask 077
  test ! -e "$HOME/.config/chipping-orchestrator/.env.pre-rollback"
  cp -p "$HOME/.config/chipping-orchestrator/.env" "$HOME/.config/chipping-orchestrator/.env.pre-rollback"
  cp -p "${ORCHESTRATOR_BACKUP_DIR:?}/configuration/.env" "$HOME/.config/chipping-orchestrator/.env"
)

Compare the restored settings with the saved current file:

diff -u "$HOME/.config/chipping-orchestrator/.env" "$HOME/.config/chipping-orchestrator/.env.pre-rollback"

diff exits 1 when files differ. Reconcile edits made since the backup with the older version's supported settings, keeping compatible target paths, author lists, log paths, and other deliberate overrides. Review process/service overrides too.

Repeat compatibility checks, confirm the restored version, then restart and inspect startup output and logs. Keep external logs and credentials at their existing paths throughout.

Pin a release and runtime dependencies¶

For reproducible installs, upgrades, or rollback, choose a release whose matching GitHub release includes constraints.txt. Set ORCHESTRATOR_VERSION to that exact version and use the download block for its templates. Download its runtime constraints too:

curl --fail --location --output "${ORCHESTRATOR_RELEASE_DIR:?}/constraints.txt" \
  "https://github.com/chippingway/chipping-orchestrator/releases/download/v${ORCHESTRATOR_VERSION:?}/constraints.txt"

Stop if the download fails. Select an installed Python supported by the chosen release; for an existing environment, pass its interpreter on every forced install (python3.12 below). To use a different interpreter, first follow Change the Python interpreter, then use the chosen interpreter in the forced install below.

For a fresh release you have reviewed, add --cooldown 0 (or a shorter window) to the command below if an enabled release-age cooldown would exclude it or any constrained dependency. Explicit --cooldown overrides both PIPX_COOLDOWN and the saved value, and pipx saves the chosen window, including zero, for later upgrades and forced installs. Restore your usual window explicitly on the next package change if needed.

Use an absolute ORCHESTRATOR_RELEASE_DIR path without whitespace: the uv backend splits constraint paths on whitespace even when quoted. If $HOME contains spaces, set ORCHESTRATOR_RELEASE_DIR to a directory whose absolute path has no whitespace and repeat the template and constraint downloads there; retain it for later reinstalls.

With the poller stopped, install the chosen version using those constraints:

pipx install --force --python python3.12 \
  --pip-args="--constraint ${ORCHESTRATOR_RELEASE_DIR:?}/constraints.txt" \
  "chipping-orchestrator==${ORCHESTRATOR_VERSION:?}"

pipx install --force requests a forced reinstall and keeps an existing environment's recorded installer backend. The --pip-args constraint is applied by pip or uv, whichever that environment uses. Pip constraints restrict runtime dependency versions; the explicit package requirement selects the orchestrator version. Keep --upgrade out of --pip-args: pipx supplies it during upgrades, and uv rejects duplicate flags. Use each release's own constraints, compare settings, and repeat compatibility checks before starting.

After a successful installation, run pipx pin chipping-orchestrator to exclude the environment from routine upgrades. pipx saves the installation's --pip-args; unpinning alone leaves the constraint path in place. Upgrade a constrained installation by repeating this procedure for the new version with its own templates and constraints.

To return to the latest release without saved constraints, stop every poller using the environment and run:

pipx install --force --python python3.12 chipping-orchestrator

The forced install replaces the saved pip arguments and clears the pin, permitting subsequent upgrades. Compare configuration and repeat compatibility checks before restarting.

Continuous integration¶

../../.github/workflows/ci.yml runs Ruff, WPS, pytest with an informational coverage report, poetry build, and separate wheel and sdist smoke installs on every push to main and every pull request. It installs Poetry 2.5.1 with pipx, checks the committed ../../poetry.lock with poetry check --lock, then installs runtime and development dependencies with poetry sync --no-interaction. poetry run executes the checks without changing the environment. The coverage report sets no minimum threshold.

The job runs the whole set on Python 3.12, 3.13, and 3.14. These are the versions CI verifies within the range ../../pyproject.toml admits: requires-python = ">=3.12" names a floor and no ceiling, so a newer interpreter installs without CI coverage. The explicit classifiers name only the three tested versions. All matrix jobs install the same pins from the same lockfile. actions/setup-python provides each matrix interpreter, which poetry env use "$(command -v python)" selects explicitly. fail-fast: false keeps the remaining jobs running when one fails. Branch protection must require ci (3.12), ci (3.13), and ci (3.14) (../security.md#required-checks).

The packaging steps prove the distribution independently of the editable checkout. poetry build uses poetry-core and builds both artifacts from the tree; installing the sdist separately catches files missing from the archive even when the wheel works. Each smoke step creates its own python -m venv, installs the artifact with that environment's python -m pip install, changes directory outside the checkout, and runs chipping-orchestrator --help. The flat package is declared under [tool.poetry].packages. These environments contain only the distribution and its declared runtime dependencies, resolved fresh from PyPI, so an undeclared dependency supplied by the project lock cannot hide a packaging error. The smoke steps configure nothing, and need nothing configured: the command line is read before any owner that resolves the settings is imported, so --help answers even where a REPOS value would abort a real launch.

The job declares timeout-minutes: 20, generous next to the few minutes a green run takes and far under the six-hour default GitHub would otherwise cancel it at; the reasoning that ceiling serves is the same one the scans below are bounded by. The workflow also declares a concurrency group of ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }} with cancel-in-progress: ${{ github.event_name == 'pull_request' }}. A pull request runs in a lane per ref, and a second push cancels the run its own earlier push started rather than leaving two runs racing for a runner and reporting checks against a commit nobody is reviewing. Every other run — a push to main above all — is keyed on its own run id, so it shares a group with nothing. That, rather than the flag, is what makes those runs uncancellable: cancel-in-progress: false protects a run that has started, but GitHub keeps at most one pending run per group and cancels the one each newcomer replaces, so under a shared group a third push would evict a queued main run while the first still held the runner. A CI run on main is the record of what that commit does, and no later push may erase it, waiting or started. ../../tests/repository/test_ci_workflow.py holds that concurrency block, the interpreter matrix and its floor against requires-python, the packaging steps, and this page's versions against the matrix that proves them.

Ruff rules live in ../../pyproject.toml under [tool.ruff.lint]; WPS is selected inline so Flake8 does not duplicate Ruff's checks; dev tools are declared in [dependency-groups]. The only on-disk Flake8 config is ../../.flake8. WPS uses its default rules and complexity limits. Exact-path WPS410 / WPS412 exclusions preserve the deliberate package APIs; no structural rule is waived.

Most initializers are markers. The declared publishers retain explicit __all__ surfaces for distribution metadata, configuration, agent and GitHub operations, scheduling, workflow entry, usage parsing, and analytics recording. tests/repository/test_package_exports.py checks those surfaces and requires every other initializer to bind nothing. These API exclusions remain exact-path unless a maintainer deliberately redesigns the API.

Resolved process values live in orchestrator.config. Importing or reloading that package invokes the environment resolver and binds fresh values, while callers retain the same module object for attribute reads and patches. Repository types live in config.models, and token resolution in config.credentials.

Refactors separate responsibilities and move callers, mocks, and ownership documentation together. Every resulting module must satisfy the default limits; a refactor cannot add an exclusion, an inline suppression, or a raised complexity limit. flake8 orchestrator tests --select=WPS must pass without diagnostics. The isolated run, flake8 --isolated --select=WPS orchestrator tests, must report only the deliberate initializer violations; every remaining file/rule exclusion must correspond to one of those diagnostics.

The rule set is Ruff's own. [tool.ruff.lint] in ../../pyproject.toml declares no select, so what ruff check orchestrator tests enforces is whatever the ruff resolved in ../../poetry.lock ships as its defaults, plus the single E501 opted into through extend-select. That pair is the whole contract. [dependency-groups] names a floor (ruff>=) rather than a version, and poetry sync installs the lockfile's resolution, so what settles the rule set is the lock and what moves it is a regenerated one — which is what the weekly workflow:dependencies PR carries when it bumps the tool. A select of this repo's own would be a copy of those defaults taken on the day it was written, and a copy buys only the drift it cannot follow: every rule the tool adds afterwards stays off, unreported, and unanswered. Nothing is waived tree-wide under the contract either. The only suppressions are the exact-path per-file entries below and the inline # noqa: <CODE> - <reason> directives naming the rule each covers and why, so a rule a new ruff adds reports here on the first run after the bump.

Ruff and the line-length test enforce a repository-wide 120-column target set once as line-length under [tool.ruff] in ../../pyproject.toml. Ruff applies it to Python via the opted-in E501 rule; the first-party ../../tests/repository/test_line_length.py reads the same value and applies it to tracked Markdown/text files, exempting fenced code blocks, single unbreakable tokens (e.g. long URLs), binary assets, the lockfile, and the verbatim LICENSE.

[tool.ruff.lint.per-file-ignores] in that same file carries one waiver, PLC0414, over six exact paths. The rule reports import X as X, which is the spelling F401 is answered with here: a name aliased to itself is a re-export both Ruff and a reader can see, and the import that drops the alias reads as dead. Only one of the six is nothing but a facade: orchestrator/observability/dashboard/theme.py implements nothing and reads the five style owners back under one name, which is the theme object a dashboard page hands every panel. The other five seed and drive a stage's scenarios and carry the re-export beside that work — the types, the mid-run effects, and the publication heads a sibling test module reads a case back through, bound there and read only there. A waiver covers the whole file rather than the import that earned it, so what holds the rest of the file to the rule is the check below rather than the entry. PLC0414 is in the default set, so those entries are read by ruff check orchestrator tests itself rather than by an audit beside it; a package initializer needs no entry, because Ruff never reports the rule there. Each key is an exact path rather than a glob, and ../../tests/repository/test_reexport_aliases.py holds the list against the tree: a self-alias in a module the list does not name, one on a name its own module already reads, or a key that globs fails there.

BLE001 is answered a line at a time instead. The rule reports a blanket except Exception / except BaseException, and it is in the default set beside PLC0414, so those answers are what ruff check orchestrator tests reads and a handler missing one fails CI. No per-file entry would fit: what earns a waiver is one handler rather than the file around it, so each carries an inline # noqa: BLE001 - <reason> naming what its blanket catch protects, and every other handler beside it stays held to the rule. Three families earn one. The GitHub-API calls — the best-effort PR notices, the issue and pull-request reads, and the child-issue create under orchestrator/git/base_sync/ and orchestrator/workflow/ — catch blind because PyGithub raises its own GithubException and the transport errors underneath it alike, and a narrower handler would turn a connection reset into a stranded run. The fail-open guards under orchestrator/observability/analytics/ and in orchestrator/workflow/late_split/telemetry.py catch blind so a parser bug or an unwritable sink costs the record and never the tick that produced it. And two test helpers keep whatever escapes a worker thread so the main thread can assert on it. Ruff exempts a blind handler that reports through a logger it can resolve back to logging itself, which is why a guard logging through an imported log carries a directive where the identical one beside its own getLogger call does not.

lint.external in ../../pyproject.toml is what keeps a directive from reading as dead when it answers a rule the run does not enable. RUF100 — itself in the default set — reports a # noqa naming such a rule and offers to delete it. The blind-handler directives above need no entry, because BLE001 is in the default set and the run reads them itself. N802 is the one code outside it, and listing it is what leaves the single directive naming it in place with the rule that would strip it still on.

That one entry is a narrow claim rather than a clean bill. N802 answers for the one directive there is — on the DataFrame method of the test double standing in for the pd handle a dashboard page is handed — and for nothing wider. ruff check orchestrator tests --select=N802 reports two deliberate names the convention does not reach, a test spelling codex's -C flag and a shouted stub that raises where a call is expected, so the tree is not held to that rule the way the default run holds it to BLE001.

../../tests/repository/test_noqa_directives.py holds three readings, two of them the halves of that declaration. It runs Ruff itself over the two trees under the configured selectors plus RUF100 — which rules a selector enables is Ruff's answer and not one a prefix test could reproduce, since F covers F401 and not FLY002 — so a directive naming a rule that is neither selected nor listed fails there rather than surviving until someone strips it. The other half is the one Ruff has no reading of: a listed code no directive carries suppresses nothing, and is read off the tree instead. The third sits under both and is what the contract above needs beyond them: it tokenizes every module and fails on a comment that opens a suppression and names no rule for it, line-level # noqa or file-wide # ruff: noqa alike. Ruff honours either spelling and reports neither, so a blanket waiver is the one suppression the two halves would let through.

Import order is Ruff's answer too, under [tool.ruff.lint.isort] in ../../pyproject.toml, and I001 is in the default set with the two rules above, so the run CI makes is what holds that order. One setting is declared there. combine-as-imports keeps every member a module is read for on the single statement that names it: Ruff's default hands an aliased member a statement of its own, and the modules here read most of their neighbours under an alias — a stage module and its tests name several of a package's owners in one import ... as ... — so the default would spell one read as a statement per alias and push whole files past the twelve-import ceiling WPS201 holds them to. ../../tests/repository/test_import_sorting.py runs the rule on its own beside that, so an unsorted block is reported next to the tests a change was written for and not only by the lint step beside them.

WPS235 uses its default ceiling of eight names per from ... import statement. Sorted imports merge all reads of one module into one statement, so splitting an oversized read across several statements cannot satisfy that limit. Read values through their actual owner, keep explicit type and fixture imports where needed, and separate responsibilities when a coordinator needs too many sibling owners. Callers import defining owners directly. Neither WPS201 nor WPS235 is relaxed to accommodate a refactor, and no facade, dynamic dependency lookup, or replacement exemption substitutes for a direct owner.

The CI workflow declares permissions: contents: read so the run's GITHUB_TOKEN is read-only and cannot publish artifacts, push tags, or comment on PRs. The job uses no repository secrets, so PRs from forks run safely under the same scope.

../../.github/workflows/codeql.yml runs CodeQL's default queries against the Python source on every push to main, every pull request targeting main, and a weekly schedule. Its one-language matrix fixes the language at python and uses build-mode: none, so the scan analyzes source without installing or executing the project. The workflow keeps contents: read as its top-level token scope; the analysis job restates it and adds only security-events: write, which the CodeQL action needs to upload its results. It reads no secrets.

../../.github/workflows/scorecard.yml runs OpenSSF Scorecard — the supply-chain grader behind the README badge — on a weekly schedule, on every push to main, and on workflow_dispatch, which is how a maintainer can prove a change to it works without waiting a week. It sets publish_results: true, so a run on main publishes the score the badge and the public viewer read, and hands the SARIF it produces to github/codeql-action/upload-sarif, which files each finding as a code-scanning alert on this repo. It declares the same top-level contents: read and reads no secrets; the job adds exactly the two grants those two publications need — id-token: write for the OIDC token the publication is authenticated by, and security-events: write for the SARIF upload — and restates contents: read, because a job-level permissions: block replaces the top-level one rather than adding to it. Nothing gates on the grade: a low score is a set of alerts to triage rather than a failing run, and the workflow is outside the required checks a merge waits on (../security.md#required-checks).

Every uses: in every workflow names a full 40-character commit SHA with the release it belongs to in a trailing comment (uses: owner/action@<sha> # v1.2.3), so a retagged release cannot change what a run executes. ../../tests/repository/test_workflow_action_pins.py holds that shape for every workflow, but nothing checks the comment against the SHA it labels, so a hand edit has to move both together; Dependabot's github-actions updates rewrite the pair.

Each security scan bounds its own job the same way: 30 for the CodeQL analysis, 20 for Scorecard and the vulnerability scan, 15 for the dependency review. A job that hangs otherwise runs until GitHub's six-hour default cancels it. The dependency review is a required check and CodeQL's findings are enforced by a ruleset (../security.md#required-checks), so a hung job holds a merge for those six hours instead of failing in the minutes the scan takes. Scorecard, the vulnerability scan, and CodeQL's scheduled pass have nobody watching, so a hung one holds a runner while reading as a scan that has yet to report rather than as one that failed. The documentation build and deploy jobs each allow 10 minutes: the build is a required check, so a hung one would hold a merge, and a hung deployment from main holds the pages concurrency group, so later deployments wait. ../../tests/repository/test_workflow_job_timeouts.py holds a declared timeout, shorter than that default, on every job in all six workflows, and holds the list it walks against the workflow directory, so a seventh workflow arrives with a timeout rather than outside every check.

../../.github/dependabot.yml opens weekly update PRs for github-actions and pip (Python pyproject.toml + poetry.lock). For routine version updates, github-actions uses the ecosystem-wide cooldown GitHub supports and holds every release for 30 days. The pip entry holds a major release, and anything SemVer does not classify, for 30 days; it holds a minor or a patch for 14 days. Cooldowns do not delay security updates. The pip entry limits update PRs to direct dependencies with dependency-type: direct in its allow: block. Each entry also declares the service labels GitHub stamps on the PRs it opens: workflow:dependencies on every update PR, so the whole dependency queue is one label filter, plus workflow:github_actions or workflow:python:pip naming which ecosystem moved. Those three share the workflow: prefix with the labels the orchestrator writes but are not workflow states — nothing in the tree reads them, so a PR carrying one is not an issue in a stage. github-actions and pip above name the ecosystems Dependabot updates, not labels. The pip entry uses versioning-strategy: increase so dependency updates raise runtime lower bounds to the locked versions tested by CI. ../../tests/repository/test_dependabot_config.py holds the cooldown policies, the direct-dependency scope, the versioning strategy, and the labels against the config.

Create all three custom labels in GitHub's Issues → Labels before Dependabot runs. The orchestrator does not create these service labels, and Dependabot ignores custom labels that do not exist.

workflow:python:uv is retired and remains on historical PRs. When reviewing pip updates, verify that they change the manifest and lock together, cover dev, docs, and dashboard, raise runtime lower bounds to the locked versions, preserve one entry per package and the major caps, honor cooldowns, and carry the service labels above. Use manual group bumps if Dependabot cannot update a group cleanly. ../../.github/workflows/dependency-review.yml runs actions/dependency-review-action on every PR and fails the check when a PR introduces a vulnerable or non-compliant dependency.

../../.github/workflows/vulnerability-scan.yml is the standing scan beside that diff gate: a weekly schedule plus workflow_dispatch, the second so an edit to it is verifiable from the Actions tab instead of a week away. It exports the pins from ../../poetry.lock with poetry export --all-groups --without-hashes -f requirements.txt, then audits them with pip-audit and fails the job when a published advisory names one of them. The job installs poetry-plugin-export at the workflow's exact pin into the Poetry tool environment with pipx inject; exporting does not require installing the project. Why the export and audit are shaped that way:

• --all-groups covers the dashboard and docs groups as well as the runtime and dev ones, so the audit is the whole lockfile rather than the subset a default poetry sync installs.
• Environment markers are stripped from the export before the audit. A pin kept for another platform (colorama under Windows, tzdata) is a version this repository still ships, and an audit reading markers would skip it as inapplicable to the Linux runner — silently, and while reporting success.
• The scanner is CI-only. pipx run runs pip-audit from a throwaway environment, so it audits the pins without becoming one: it appears in neither ../../pyproject.toml nor the lockfile, and the version range there is what keeps a new major from changing the CLI under the job.
• --no-deps --disable-pip and --strict. The exported versions are already the complete pinned set, and the two collection flags are what have pip-audit read them as written. --no-deps on its own still hands the file to pip's resolver and audits whatever that resolves to — a fresh resolution, which is the step the lockfile exists to settle. --strict closes the one hole left in that reading: a pin PyPI has no record of is otherwise skipped and reported under a green run, while a line carrying no exact version fails the job with or without it. Neither flag is a claim about the advisory data — a service that errors out fails the run on its own, and a pin no advisory names is simply a clean result.

The job declares the same permissions: contents: read and uses no secrets. It is not a required check and cannot be one — it never runs on a pull request — so a red run is triaged from the Actions tab. Turning one of its findings into a bump PR is Dependabot's job, which is why enabling Dependabot security updates is on the operator list in ../security.md#dependabot-security-updates. The audit reads the whole lockfile while those updates reach only what the allow: rules above name, so a finding against a transitive pin those rules do not name is cleared by widening them or by hand rather than by waiting for a PR. For manual security fixes younger than 14 days, follow the manual security-fix procedure.

Dependency tooling¶

Poetry is developer tooling for source checkouts; package users install with pipx. It keeps checkout environments in .venv/, and the default sync installs runtime plus dev dependencies. env -u VIRTUAL_ENV -u CONDA_PREFIX poetry sync --with docs and env -u VIRTUAL_ENV -u CONDA_PREFIX poetry sync --with dashboard select optional groups; use --with docs,dashboard for both. Sync removes unselected groups. poetry run executes commands without syncing. The wrapper's install policy, timeouts, and sync requirements are documented under Launcher dependency refresh.

Poetry uses an inherited VIRTUAL_ENV or non-base CONDA_PREFIX even with virtualenvs.in-project = true. Prefix every manual Poetry command with env -u VIRTUAL_ENV -u CONDA_PREFIX so it selects this checkout's .venv/. Agent and verify subprocesses remove VIRTUAL_ENV and its bin directory from PATH for all managed repositories; Conda markers and their PATH entries stay together so Conda-based targets keep a consistent activation. Commands for a Poetry target launched from Conda must explicitly deactivate it or use the manual override above. Without that override, poetry sync can remove unrelated packages from the operator's active Conda environment; the subprocess filter does not protect it. The launcher clears both markers for its dependency install, while its Python process inherits the caller's Conda activation.

The manifest accepts Poetry >=2.4,<3; CI and developer setup pin 2.5.1. Poetry 2.4 is the minimum for the release-age setting in ../../poetry.toml, which holds new releases for 14 days during resolution. Use env -u VIRTUAL_ENV -u CONDA_PREFIX poetry lock for dependency or group changes and env -u VIRTUAL_ENV -u CONDA_PREFIX poetry check --lock to detect a stale lock. A project version bump does not change the lock. Never commit package sources or mirror URLs. If a mirror is required, validate a maintainer-local mirror plugin's output before using it; the committed lock must stay free of [package.source] entries. env -u VIRTUAL_ENV -u CONDA_PREFIX poetry show --outdated --top-level reports outdated direct dependencies.

Keep runtime dependencies' lower bounds and major-version caps in the manifest, and keep each locked version within its declared range. Repository checks accept lock-only updates within those ranges. Raise runtime lower bounds with manual updates; for a lock-only security update, raise the affected lower bound to exclude vulnerable releases. Use the manual security-fix procedure for security fixes younger than 14 days.

Publishing the documentation¶

The documentation is a static MkDocs site hosted by GitHub Pages at https://chippingway.github.io/chipping-orchestrator/. GitHub supplies the address and HTTPS; a custom domain is optional. The Markdown under docs/ is the source for both the website and GitHub's directory view, with ../README.md serving as the site's homepage.

Local preview and build¶

Install the optional docs dependency group from the lockfile, then start the local preview:

env -u VIRTUAL_ENV -u CONDA_PREFIX poetry sync --with docs
env -u VIRTUAL_ENV -u CONDA_PREFIX poetry run mkdocs serve

Open http://127.0.0.1:8000/ to browse the preview. To check the deployable HTML, build it once and point the checks at that output, as the Documentation workflow does:

env -u VIRTUAL_ENV -u CONDA_PREFIX poetry run mkdocs build --strict
env -u VIRTUAL_ENV -u CONDA_PREFIX DOCS_SITE_DIR=site poetry run pytest tests/repository/test_docs_site.py \
  tests/repository/test_docs_output.py tests/repository/test_docs_navigation.py

The build writes to the ignored site/ directory. DOCS_SITE_DIR makes tests/repository/test_docs_output.py check the generated links, heading anchors, search index, sitemap, and page descriptions of that output instead of building a copy of its own. A directory it names that holds no built site fails the check rather than being rebuilt or skipped. Without the variable, the same pytest command builds a fresh strict copy into a temporary directory. Links in the Markdown sources are checked by tests/repository/test_doc_links.py, which needs no documentation builder and runs with the rest of the suite in Continuous integration. The docs group is separate from runtime and development dependencies, so the default poetry sync does not install the documentation builder.

../../mkdocs.yml defines the navigation, site URL, search-enabled theme, the template directory the theme's custom_dir layers over it (Site navigation, Page descriptions), and validation rules. Every page must appear in navigation; missing pages, links, and heading anchors fail the strict build. The hook in ../../.github/scripts/docs_site.py preserves GitHub-style heading anchors for unique headings and turns links to repository files outside docs/ into GitHub links while building. Repeated headings use Python-Markdown suffixes such as _1 instead of GitHub's -1. Relative links between documentation pages keep pointing within the site. Link examples in code blocks are left intact, and the source files retain the relative paths the repository's documentation checks validate.

Site navigation¶

nav: in ../../mkdocs.yml groups the pages by what a reader came to do: Operate (quick start, settings, operations, snapshot capability check, security), How it works (architecture, the state machine, and the agents), Observability, and Releases, beside the homepage the site name links to. ../README.md lists every page under the same grouping for readers browsing on GitHub. A page's URL follows its path under docs/, not its place in nav:, so moving a page between groups changes no address. The templates in ../../.github/docs-theme/ render that list over the bundled mkdocs theme:

• The top bar names each top-level entry once, as a plain link rather than a dropdown. A section links to its first page, which makes that page the section's landing page, so an area's overview goes first. Search, buttons to ask ChatGPT or Claude about the project, a link to the repository, and the light / dark / auto theme menu sit on the right. The AI buttons open a new tab with a prompt pointing to the documentation homepage from site_url. Below a viewport width of 1200px, the top bar collapses into a menu so both AI buttons fit.
• The sidebar lists the pages of the current page's top-level section above that page's own headings. A subsection there is a link to its first page, and opens onto its own pages only while it holds the current one.
• Previous and next links sit under the page, where the theme's n and p keyboard shortcuts find them.

tests/repository/test_docs_navigation.py builds a sample site through the templates to hold each of those rules. It needs the docs group but not DOCS_SITE_DIR, and the Documentation workflow runs it beside the two checks above.

Page descriptions¶

A page summarizes itself for search results and link previews with description front matter, a YAML block that opens the Markdown file, before its title:

---
description: How the orchestrator is checked, launched, supervised, and reconfigured.
---
# Operations

Write the description as one or two sentences of plain text, about 160 characters at most so a search result can show it whole. Markdown in it is not rendered, and HTML is shown as text. Quote the value when it contains a colon followed by a space or a space followed by #, or starts with a YAML indicator such as ', ", &, *, [, or {, so it parses as one string. Front matter is held to the same 120-column line target as the rest of the page (Continuous integration), so a description longer than one line allows is written as a >- folded block indented under the key, as every described page under docs/ does; its lines join on single spaces and need no quoting.

The template ../../.github/docs-theme/main.html overrides the bundled mkdocs theme's site_meta block, and writes the value into the page's single <meta name="description"> tag, HTML-escaped. On the homepage the page's own description takes precedence over site_description in ../../mkdocs.yml, which remains the homepage's fallback; any other page without front matter emits no description. The rest of that block is the theme's own, so every page keeps the canonical URL and the rest of the head metadata the bundled theme gives it; a page without front matter has the bundled head plus the two verification tags described in GitHub Pages setup and the AI-button stylesheet, with the same search entry as under that theme. tests/repository/test_docs_site.py builds sample sites through the template to hold each of those rules, and checks that mkdocs.yml names the template directory. GitHub's file view shows front matter as a table above the page.

tests/repository/test_docs_output.py holds the repository's own pages to those rules in the built site: the homepage and each landing page it lists emit exactly the description their front matter gives, and no two of those descriptions match. It reads the front matter without a YAML parser, so a page it lists writes its description as a plain or >- folded value rather than a quoted one.

GitHub Pages setup¶

In chippingway/chipping-orchestrator, configure these settings once:

1. Open Settings → Pages, and select GitHub Actions under Build and deployment → Source.
2. Create or open Settings → Environments → github-pages, and restrict deployment branches to main.
3. In the repository's About panel, open its settings and set Website to https://chippingway.github.io/chipping-orchestrator/.

The extrahead block in ../../.github/docs-theme/main.html adds the Google Search Console google-site-verification and Bing Webmaster Tools msvalidate.01 meta tags to every page, including the 404 page; update their verification values when moving the site or changing the verification owner.

../../.github/workflows/docs.yml builds and checks every pull request and the pushes to main that change a documentation source, using the committed lockfile. The pull_request trigger carries no path filter, so a pull request touching no documentation still reports the build check that branch protection requires (../security.md#required-checks). Pull requests only build and validate. A push to main runs the workflow when it changes a path its push filter lists — the published pages, the MkDocs configuration and build hook, .github/docs-theme/ for the site template, the workflow, the documentation tests, and the dependency manifests — then uploads the generated site and deploys through the github-pages environment. The deployment job alone has pages: write and id-token: write permissions; no personal access token is needed. The Documentation workflow can also be run manually from Actions, selecting main to publish or another branch to validate its build.

Documentation runs publishing main share a pages concurrency group: a newer run replaces a queued run so the latest documentation wins, while cancel-in-progress: false lets an active deployment finish. CI gives every main run its own group to preserve each merged commit's test record (Continuous integration). Pull-request documentation runs share a group per ref and cancel in progress when superseded.

Each run builds the site once and checks that site/ output before any upload, so the pages a main run publishes are the pages its checks read. Only the generated site/ artifact is uploaded. Python source links, agent instructions, and configuration examples lead back to the repository; the site does not publish the checkout or notes under plans/.

Run modes¶

Launch commands¶

Use the command installed by pipx:

• chipping-orchestrator — continuous polling.
• chipping-orchestrator --once — one real polling tick, including label setup and possible agent launches.
• chipping-orchestrator --cleanup-terminal-artifacts — reclaim the worktrees and branches of finished issues, then exit. Polls no issue, ensures no labels, and writes no workflow state — no label, no pinned state, no comment. It does delete refs, which is what it is for: the orchestrator-owned branches it proved reclaimable go from the local clone and from the remote, after the fail-closed gates cleared each one. It connects read-only clients and asks each repository only about the issues and pull requests its artifacts name.

Safe to run on the same host as the polling process because it refuses to run beside it: every polling run holds a shared flock claim on WORKTREES_DIR/.artifact-maintenance.lock for its whole life, this mode needs that claim exclusively, and a refusal defers the whole pass and exits 0 (a deferral is what the mode was asked for, not a failure a timer unit should report). Two of these runs exclude each other the same way, and the daemon's own recurring pass takes the same claim — from inside its own scheduler barrier, so it only gives its presence up once it has stopped admitting and drained, and takes it back before admitting again. A second daemon on the host therefore cannot be submitting work while either of them deletes. A polling process that starts while a pass is running waits for it (no deadline, because polling through a teardown is never safe — though only for a lock something is holding: one that does not work at all is reported instead, and the pass simply does not act); a pass gives the host back at a candidate boundary once it has held it for 120s, with whatever it did not reach owed to the next pass. Point a nightly service timer at it for hosts whose orchestrator is not always up; a host running the daemon continuously already gets a pass between polling passes — every TERMINAL_ARTIFACT_CLEANUP_INTERVAL_SECONDS, or inside TERMINAL_ARTIFACT_CLEANUP_WINDOW where one is set (When it runs) — and the timer will simply defer. - chipping-orchestrator --log-level DEBUG — verbose logs.

Source-checkout module launches and the restart wrapper are for developers; see the developer guide.

On first start the orchestrator creates the workflow labels and the backlog / paused / workflow:community_contribution control labels on the repo, then begins polling open issues every POLL_INTERVAL seconds. On a repo it drove before the labels were namespaced, each pre-namespace label is renamed in place instead.

Poetry migration for operators¶

Poetry applies to developer source checkouts. The source-tooling migration covers checkout environments, verification commands, and development service units. Published-package users install and upgrade with pipx.

Launcher dependency refresh¶

Dependency refresh by the source wrapper is developer tooling. Its fingerprints, timeout behavior, and optional groups are in the developer guide. Stop all processes using a pipx environment before upgrading or reinstalling the package.

Running more than one poller¶

The supported topology is one host. More than one polling process may run there against the same repositories — a second daemon, or a --once run beside one — provided every one of them resolves the same WORKTREES_DIR, and the in-process scheduler guards (a duplicate active issue, the caps, the family slot) still apply first inside each. Left unset, WORKTREES_DIR defaults beside the first configured target — the first REPOS entry, or TARGET_REPO_ROOT without REPOS (default) — so pollers whose lists start with different repositories resolve different roots unless every one of them sets it explicitly.

The shared issue claims coordinate workflow work. Set the same WORKTREES_DIR in every poller's process or service environment. Installed pollers under one account read the same ~/.config/chipping-orchestrator/.env; use process or service overrides for different REPOS lists and a distinct absolute LOG_DIR for each poller, including concurrent --once runs. Override any explicit ANALYTICS_LOG_PATH and TRAJECTORY_LOG_PATH with distinct absolute paths per poller too; the default analytics path follows LOG_DIR. Runtime log rotation and analytics retention rewrites do not coordinate across processes, so sharing these files can lose records even when issue claims are separate. Stop every process using a shared pipx environment before upgrading it. Developer source wrappers also need the isolation described under Development restart wrapper.

• One writer per issue. Each dispatched issue is taken under a host-local writer claim: an exclusive flock on a file in WORKTREES_DIR/.issue-writer-claims/, named repo-<id>-issue-<n>.lock for the repository's numeric GitHub id — which no rename or transfer changes, unlike the configured slug or the owner/name a poller fetched at startup — and the issue number. A poller that finds the issue held by another skips it for that tick — no refetch, guard, recovery pass, close receipt, or handler, so no label, comment, or pinned write, no agent run, and no usage or evaluation record — and takes the issue up on a later tick once the holder is done. The pre-tick base refresh takes the same claim for each worktree it syncs, before it reads the issue, and holds it until that worktree's rebase, push, park, or crash recovery is done, so a refresh and a dispatch — or two pollers' refreshes — never write one issue at once; a refresh that finds the issue held skips that worktree for the tick with no read, rebase, push, or write, and syncs the rest. A close a refused poll read for the issue costs a pinned read, an issue read behind it, and a pinned read behind that, and is kept in its own memory and nowhere else: a reopen before then cannot take the reading away, and a later tick sweeps it under the claim. It is tied to the late cycle the record names only where the issue still reads closed behind that read and the record read behind the issue still names that cycle — a record that moved is read again behind another issue read, a few times at most, so a close of a cycle the holder restarted between the reads is tied to that cycle — or, where a retirement has just taken the cycle off the record, to the cycle the holder noted on the claim it is retiring, a note read as the claim is refused, so a holder letting go during those reads costs nothing — so a cycle the holder settled and an operator restarted in the meantime is left alone. One the poller cannot tie to a cycle ends one later only if the issue is closed again then. A close the record says ends no cycle is not kept. A poller granted the claim reads the issue again behind it before routing it, so it never resumes a stage another poller advanced the issue past since its poll, and a close its own poll read ends the cycle the record names only where it still reads closed, or where no other poller has held the issue since the poll began (see what a claim file holds, below). Different issues, and one issue number in two repositories, never contend. Inside one process the claim is exclusive between threads too, logged as reason=held_here, with one exception: the receipt a poll posts for a close it observed, and the notice a parent posts telling a consumer its snapshot is gone, are comments built to land beside that process's own worker, so they are let in alongside one. The skip is logged on orchestrator.scheduler as writer claim skip repo=<owner/name> repo_id=<id> issue=#<n> reason=held_elsewhere, where the name is only a label and the id is the key.
• A parent writes a child under the child's claim. A decomposed parent's handler writes its children too — the walk that relabels a workflow:blocked child workflow:ready, the seeds that give a child its parent link and ancestry, the finalize of a child whose pull request merged, and the notice that a reclaimed snapshot is gone — and each of those is made under that child's own claim, off what it reads behind it: the release walk and a merged child's finalize read each child again once claimed, and a seed adds to whatever record the child carries by then. A child another poller holds is left for a later pass, which parks nothing, and one that poller moved since the walk's scan is judged as it reads now: the release walk starts none of them, and the finalize counts it as it stands. An ordinary split creates the rest and leaves the held child's seed and its own finalize to the next tick's recovery. Only a late split's placement of a held child parks the parent, exactly as a seed that could not be written does.
• What the namespace assumes. Every participating poller can read each repository's numeric id from GitHub, which is what makes two configured spellings of one repository, or a poller started before a rename and one started after it, meet on one key. They run as a user that can create and open files in that directory, and see it on a local filesystem whose flock is honored between them — a network filesystem that emulates flock per client coordinates nothing. They read one monotonic clock, as every process on a host does unless it runs in a time namespace of its own. Anything else that can write the directory can hold a claim, so keep it writable by the orchestrator's user alone.
• Failures withhold. A claim that cannot be worked — the namespace cannot be created or opened, or flock fails for any reason other than another holder — skips the issue as a held one is skipped, with a reason=unusable warning, rather than letting it be written uncoordinated. Repair the directory and the next tick proceeds.
• What a claim file holds. The lock, and a few lines from the last hold on it: a token the poller draws at startup, which is how a poller tells another's holds from its own; while a hold retires a late cycle, that cycle's id, which a refused poller reads; and the moment the hold let go, on the host's monotonic clock. Every acquisition empties the file before signing it, a moment after it takes the lock, and a refused poller reads a note only off a file its hold signed and has not stamped as released, so no note outlives a hold that stamped its release. A close a poller's poll read is tied to the record it holds next only where every other poller's hold it has found had let go before the poll began, so a poller restarted over the namespace its predecessor used is not held up by the tokens that predecessor stamped. A hold whose process was killed holding it — SIGKILL, an OOM kill, the forced exit past SHUTDOWN_GRACE_SECONDS — stamps nothing and counts as ending when the next holder finds it; its note, if it left one, is read until that holder empties the file, since it was the last hold to write the issue until then. The same holds for a hold whose release stamp could not be written. Each line is written whole or not at all: one a file-size limit or a full disk lets land only in part is taken back and logged as not written, and a line with no end, or a file nobody signed, is never read as a release time or a retiring cycle — only as a hold that ended when it is found. That includes an empty file, even the one an issue's first claim on the host creates, since another poller can have held and emptied it before that claim locked it. So a close a poll reads on an issue no poller on the host has claimed since its predecessor was killed holding it, or ever, and that is reopened before the poller claims it, ends no cycle by that reading. Pollers must all run a build that signs and stamps its claims: one that does not is a hold nobody sees, and a close read beside it can end a cycle that poller restarted.
• Release. A claim ends when the dispatch or base-refresh route that took it ends, however it ends, and the kernel drops it when a process dies, so a crashed poller leaves nothing held; its descriptor is closed with it and is not inherited by an agent the holder spawns. The files stay: never delete the directory or a file in it while any poller runs, since a recreated file is a new inode another process can lock beside a holder of the old one. They are small, one per issue key ever claimed, and safe to remove once every poller has stopped.
• Not covered. Pollers on different hosts, or on one host with different WORKTREES_DIR values, are not coordinated at all and are not supported against the same repository. And the claim is separate from the artifact presence on WORKTREES_DIR/.artifact-maintenance.lock: a maintenance pass neither takes nor reads a writer claim, and holding one says nothing about the host's artifacts.

Running under systemd (user service)¶

For persistent operation, use a systemd user service supervising the command installed by pipx. A foreground process does not survive a reboot, a tty logout, or the user manager being torn down. Source-checkout service configuration belongs to the developer guide.

A detached screen / tmux session wrapped in a Type=forking unit looks similar but is the wrong shape: systemd ends up supervising screen, not the orchestrator; ExecStop races the screen session's own lifecycle; logs split; and the unit silently does nothing at boot unless linger is enabled. Keep screen / tmux for interactive debugging.

Unit file¶

Drop this at ~/.config/systemd/user/orchestrator.service. Replace <user> with the account that installed the package, and adjust ExecStart to the path shown by command -v chipping-orchestrator:

[Unit]
Description=chipping-orchestrator
After=network-online.target

[Service]
Type=simple
WorkingDirectory=/home/<user>
ExecStart=/home/<user>/.local/bin/chipping-orchestrator
Restart=always
RestartSec=5
Environment=PATH=/home/<user>/.local/bin:/usr/local/bin:/usr/bin:/bin

[Install]
WantedBy=default.target

WorkingDirectory must exist. It selects neither the installed configuration file, ~/.config/chipping-orchestrator/.env, nor the log directory. Set an absolute LOG_DIR outside the pipx environment as described in package setup.

• Type=simple tracks the foreground orchestrator process. SIGTERM from systemctl stop triggers its shutdown.
• Restart=always restarts an exited process; systemctl stop suppresses that restart. Package version changes require an explicit pipx upgrade while the service is stopped. A refused start (empty ALLOWED_ISSUE_AUTHORS or invalid settings or targets) repeats every five seconds (RestartSec=5); check journalctl --user-unit orchestrator.service -f for the diagnostic, then stop the service and repair its configuration.
• A non-interactive service does not inherit your shell's PATH. Include directories containing codex, claude, or agy in Environment=PATH=…, or configure absolute CODEX_BIN, CLAUDE_BIN, and AGY_BIN paths.

Enabling¶

systemctl --user daemon-reload
systemctl --user enable --now orchestrator.service
loginctl enable-linger <user>

enable-linger is required for boot-time start: without it the per-user systemd manager only runs while the user has an active login session.

Operating¶

systemctl --user status orchestrator.service        # current state and last log lines
systemctl --user restart orchestrator.service       # bounce the orchestrator
systemctl --user stop orchestrator.service          # SIGTERM the supervised command; suppress restarts
journalctl --user-unit orchestrator.service -f      # tail the supervised command's stdout/stderr

systemd's journal captures the supervised command's stdout/stderr (process lifecycle, exit codes, restart messages). The orchestrator's own structured log lives at LOG_DIR/orchestrator.log (rotated, ~10 MiB × 5). Package setup above records an absolute LOG_DIR pointing to ~/.local/state/chipping-orchestrator/logs. WorkingDirectory does not select the default log location. Check the journal first for "did it start / did it die", then LOG_DIR/orchestrator.log for per-issue handler detail.

Reclaiming what a split leaves on the remote¶

An issue whose committed implementation is adjudicated as a split leaves two things on the target repository: the branch its superseded candidate was committed on, and an immutable snapshot ref under refs/orchestrator/late-split/issue-<n>/cycle-<c>/gen-<g> holding the exact commit its children were told to reuse. Both are recorded on the issue's own pinned obligation ledger, and the orchestrator reclaims them itself — there is no TTL and no background garbage collection. Three passes do it, and between them every live ledger stays visited:

• the umbrella's terminal, which settles what is owed before it closes the parent (see ../state-machine/delivery-stages.md);
• the umbrella's park, on the way out of the two dispositions that stop the parent for a human — a child rejected, or one closed without reaching a terminal label. Both closed the child, which is the reading the rule takes, so the ledger is settled from the same scan that parked the parent; the park itself is unchanged and no terminal is decided. Nothing else revisits an issue parked this way, which is why it happens here;
• the closed-owner cleanup sweep, for an issue a human closed mid-cycle while it still carried workflow:decomposing or workflow:umbrella — or workflow:ready / workflow:blocked, where a decomposition outcome that landed after the close can leave an ending nothing else would find (see ../state-machine/delivery-stages.md).

The sweep runs on the existing CLOSED_ISSUE_SWEEP_EVERY_N_TICKS cadence and adds no per-tick traffic of its own, so raising that knob (the multi-repo rate-limit advice in ../configuration.md) also stretches how long an unreclaimed branch or ref survives, by the same factor. It is cleanup-only: it never spawns an agent, resumes a workflow, activates a child, or touches one that already exists. It does end the cycle the close interrupted — the cancellation is marked, any pull request that cycle was holding is closed over one notice, and the issue moves to rejected once nothing is owed. That last write is the only label it makes, and it is also what stops the sweep: an owner still owing the remote keeps its label and keeps being visited, so a repository whose closed owners stay on workflow:decomposing or workflow:umbrella is a repository with something the orchestrator could not reclaim.

Reopening such an owner does not get the workflow going again. The same cleanup runs from the dispatcher instead, once per dispatch, the issue reaches no stage handler, and the same rejected is written once the ledger settles — each held dispatch logs a warning naming what is still owed. Clearing the refusal is clearing the obligation; starting a fresh attempt afterwards is removing rejected, which is the handshake below.

Rolling back to an older orchestrator¶

The late generation an oversized candidate is adjudicated under lives in additive late_* fields on the issue's own pinned comment. An orchestrator that predates them ignores those fields entirely, which is safe for an issue that never entered the size gate and not safe for one currently in it: the older binary would read a workflow:decomposing issue carrying a frozen candidate as one waiting to be decomposed, spawn the initial decomposer against it, and decide the size question by not asking it. The same holds for an issue whose ledger still records a snapshot ref or a superseded branch — nothing in the older binary reclaims either.

So a rollback is drained rather than cut over. Set DECOMPOSE=off and restart, which stops new candidates entering the gate while every recorded generation goes on being adjudicated, cancelled, cleaned up, and restarted. Then wait for both halves of the drain, because they end at different times:

• No open issue carries late_cycle_id in its pinned comment. That is the adjudication half — a cycle still deciding, revising, or splitting.
• No CLOSED issue carries an unsettled late_resources ledger, on any of the four labels the cleanup sweep visits. A closed owner is exactly the case the first check cannot see: its cycle was ended by the close, and what is left is the sweep reclaiming the superseded branch and the snapshot ref its children were cut from, at the CLOSED_ISSUE_SWEEP_EVERY_N_TICKS cadence rather than every tick. Deploying the older binary over one abandons precisely the resources the paragraph above says nothing else reclaims. A ledger is settled when every entry reads reconciled; a retained ref (its consumers are still live) is not settled and is the one state that can take arbitrarily long, since it waits on issues that are still being worked.

The section above lists the ref namespace and the git ls-remote that shows what is still out there, which is the direct check when a pinned comment is ambiguous. An issue that has to be parked mid-cycle instead is closed, which ends its cycle irreversibly and settles what it owes the remote — the work stays committed in its worktree and the section below says how to start it again.

Restarting an issue whose cycle was cancelled¶

Reopen the issue, then remove rejected. That is the whole gesture, and both halves are required: the orchestrator restarts an issue that is open and wearing no workflow label at all, over a pinned comment that records a cancelled cycle with nothing left owed and records that this cycle's rejected was actually handed out. Removing the label is what authorizes it — GitHub grants label writes only to a repository's own people — so the restart runs whoever filed the issue, ALLOWED_ISSUE_AUTHORS included.

Three things follow from that last condition. Removing a workflow label from an owner whose cleanup has not finished does not authorize anything: the ending was never in a state it could write rejected from, so it writes it once the cleanup settles, and the handshake starts from there. Neither does a rejected the orchestrator tried and failed to apply — what is recorded is a label a pass could see on the issue, not an attempt, so a refused write is simply retried. And an issue whose cancellation is still owing the remote is refused until the cleanup finishes — including the one obligation no pass can clear for you, a recorded held pull request with no preserved description beside it, which the held-terminal warning names as held PR #<n> (no preserved description). Repair that record first; nothing restarts over it.

Cancellations the orchestrator ended itself need nothing from you, closed owners included: the pass that writes rejected records it in the same breath, so the first removal is the one that counts. Cancellations that ended before this record existed are sitting on rejected with nothing recorded, and the first tick that sees one there writes it down — so removing the label works the first time on those too, as long as a tick has seen the issue open and still labelled. If one of those older issues is closed, then reopened and unlabelled inside a single poll interval, the orchestrator asks GitHub's own label history for that issue before deciding: where the most recent workflow label it applied is rejected, the removal counts and the fresh cycle starts one tick later. Only its own applications count — a rejected a collaborator applied and removed by hand is somebody else's label, not a terminal this orchestrator wrote. A history whose newest one is some other state — which is what an issue that has already been restarted once looks like, since the restart applied a label of its own — and one the API would not serve both re-apply the terminal, and removing it again starts the cycle.

Removing a terminal from an issue that never had a late cycle does not start anything either. An issue that already carries a pinned comment is one the orchestrator has met, and greeting it a second time would write a second pinned comment that every later read shadows, so an unlabeled issue with a pinned comment is left exactly where it is and logged once a tick. To drive such an issue again, apply the workflow label you want it to run from by hand — the same way an outsider's issue is driven past ALLOWED_ISSUE_AUTHORS.

What you will see, on the next tick after the label comes off: one comment naming the new cycle and where the issue is going, and one label — workflow:decomposing, or workflow:implementing when DECOMPOSE=off. The issue then runs as a fresh attempt. Everything the cancelled cycle recorded is dropped from the pinned comment: its agent sessions, its pull request and branch, its child issues and dependency graph, the snapshot it was cut from, any park, the drift baseline, the retry and review counters, and the timestamps. What is kept is the pinned comment itself, the ids of the comments the orchestrator has posted on the thread, and the issue's cumulative agent-run, token, and cost counters — so the receipt a later terminal posts still reports what the whole issue has spent, not just what the newest attempt did. The record of developer conversations retired as failed (dev_session_retirement) is kept too, so the fresh attempt never pins or resumes one of them, even where a backend hands its id back to the first spawn. A child an ordinary split created keeps its seed as well — its parent_number and its late_ancestry_* group, the snapshot pointer included — because the receipt in its body holds every later dispatch to that seed (below); nothing of the cancelled cycle's candidate, exemption, or authorization comes with it. Child issues the cancelled cycle created on GitHub are not touched: they are real issues carrying real work, and what happens to them stays a human's decision.

If you apply that same target label by hand while a restart is mid-transaction, you will see the orchestrator take it off and put it straight back. That is deliberate, not a fight over the label: its own application of the target is what separates the fresh cycle from the previous one's rejected in the label history, and GitHub records no event for a label that is already there. Leave it be — the issue ends up on the label you wanted either way.

If the restart cannot finish — GitHub refuses the comment or the label — the issue keeps the pending marker in its pinned comment, a late_failure carrying restart_failed reaches both sinks, and the next tick resumes at the step that is still owed rather than starting over. backlog or paused on the issue defers the whole restart until the control label comes off; the authorization is not lost meanwhile.

What to look at when something is not going away. Four signals, and they mean different things:

• A late_failure carrying snapshot_delete_failed or branch_cleanup_failed (see ../observability/event-streams.md), together with an umbrella that will not close, or a closed owner that keeps its label. The remote refused the delete. That is a permission or ruleset problem — a protected-ref rule over refs/orchestrator/*, or a token that lost push scope — and only an operator can clear it. Nothing is retried into success meanwhile; the retry itself is every visit, but the record of it is not: the sinks carry the move to failed once and say nothing on the visits that reach the same answer again, so what tells you a refusal is still standing is the terminal that never fires and the warning logged on every visit that holds — not a second event.
• A snapshot ref that is simply still there, with no failure recorded, and an umbrella that will not close. It is retained on purpose: a ref is kept until every recorded direct consumer has ended — which means the consumer's issue is closed, since reaching done, being rejected, and a human closing it all close it, and reopening leaves the label where it was. Those readings are taken fresh on every visit rather than latched, and every obligation that is not reconciled holds the owner's terminal, so the umbrella stays open and logs what it is waiting for on each due dependency poll. A child that stays open forever keeps its ancestor's ref forever, which is the deliberate trade — invalidating a live child's only copy of the work it was told to reuse is worse. Closing (or finishing) the child is what lets both go. That includes a child the umbrella no longer tracks: after an edit re-decomposed the umbrella, the children its drift notice named as ORPHANED are still the ref's recorded consumers, so an orphan left open (or one the orchestrator cannot read) holds the ref and the umbrella after every replacement is done. A replacement that re-decomposition pointed at the ref is a recorded consumer as well, so it holds the ref exactly as an original does until it is closed — reopened after it resolved included. Close the orphans that no longer apply; the ref goes on the next dependency poll that finds the replacements resolved and every recorded consumer, original or replacement, closed. Where the re-decomposition kept work for the parent, the same hold shows as a parent staying on workflow:blocked after every child is done, logging what it holds its return to implementation on; closing the consumers is what lets it go. On its way out it retires the split's late cycle, so its next implementation run starts a cycle of its own rather than looking for the split's superseded candidate.
• An umbrella that will not close with nothing owed at all — every obligation reconciled, no failure on either sink. The issue was split on the far side of publication, and the pull request that split superseded is open again (or was merged, or has been pushed to since). Everything the umbrella still had to do was licensed by that change being closed, so the parent holds rather than handing itself done over a live change carrying superseded work: no child is released, the branch is left wherever it is, and the terminal waits. There is nothing on the ledger to look at because nothing is owed the remote — the reason is only in the log line, said on every tick that holds and naming the pull request. Settling that pull request is what clears it; the label staying put is the retry. Do not expect the orchestrator to re-delete a branch you restored — a branch put back by hand is yours, and refusing to remove it is the point.
• A late_cleanup carrying outcome: reclaiming, and an umbrella that will not close over a ref that is already gone. That state is progress, not failure: the decision is written down before the delete, so an entry left in it is one whose delete landed while a child it owes a receipt could not be reached. Every obligation short of reconciled holds the terminal, so the parent stays open until the next visit finishes the telling — which it does by finding the ref absent, which counts as reclaimed, and then saying so to the children that were missed. A child that stays unreachable is the one case that repeats, and the log line names it on every visit.

To list what a repository is holding:

git ls-remote origin 'refs/orchestrator/late-split/*'

Deleting one by hand is safe once you are sure no child still needs it, and it does not strand the ledger: an absent ref counts as a successful reclamation, so the entry reconciles on the next visit at the cost of one read.

Do not recreate a ref the orchestrator has already reclaimed. Its value was that it provably carried one exact commit, and a ref pushed again from whatever is reachable now proves nothing. The orchestrator does not do it either, and it makes sure nothing resumes against one:

• As the ref goes, each child cut from it gets one comment saying so, carrying a hidden marker naming the owner, cycle, and generation so it is said once. That is all the owner does to a child. It never edits a child's pinned comment: that comment is written whole by whoever writes it, and a handler of the child's own that read it first and wrote it after would silently undo the edit — so the receipt is an appended comment, which nothing can lose. A cycle a human's close cancelled leaves even that unsaid: it is responsible for none of those children, so it reclaims the ref and touches nothing. The child is not left uncovered — the mirror goes before the remote ref, so the guard below reaches the same park a receipt would have earned it, one ls-remote later.
• The child refuses the work itself, at its own dispatch. Before any handler runs, an issue whose recorded ancestry still names a snapshot looks for that receipt on its own thread — marked with the owner, cycle, and generation it was born of, and authored by the orchestrator — and refuses if it finds one. That answer is authoritative: it says the reclamation happened, which no later look at the ref can contradict. A thread the tick could not read is not a thread with no receipt on it, and stops the dispatch there rather than falling through to readings a receipt would have overruled.
• Where the thread answered and carries none, the ref itself decides, and its three answers are three different outcomes. Gone → park. Still there under another commit → park too, under late_snapshot_repointed: the name survived and what it stood for did not, and nothing here re-points or deletes that ref. Unreachable → the dispatch is held for the next dispatch, with nothing written, because an outage is evidence of nothing. A park drops the dangling pointer, marks the issue awaiting_human, and returns before the label's handler is reached — including for a reopened done / rejected child, which is otherwise a dispatch no-op.
• The steady state costs nothing on the wire. The ref reading asks this host's own copy first, and a reclamation takes that copy down before it touches the remote ref — refusing the whole reclamation if it cannot prove the copy gone — so a copy still here means nothing has been reclaimed. It is read for the exact commit it carries, since it lives in the object store the agents' own worktrees share, and it is trusted only for a pointer this binary wrote (children created before that ordering existed pay one ls-remote instead).

A child the split recorded but never managed to seed — an ancestry write that failed after the issue existed — carries the split's marker in its body, and that marker is corroborated rather than believed: the owner's own generation is read fresh and has to name the same cycle and generation and carry this issue among its recorded consumers. Vouched, it hands over the ref and the commit the owner preserved, and the same three answers follow. Unvouched, the guard steps aside — a marker anyone can paste into a body parks nobody. Unreadable, opaque, or naming no candidate, the dispatch is held.

This is also why deleting a snapshot ref by hand is not the same as the orchestrator reclaiming one: no receipt is written, so the children are never told — and on the host that already fetched the ref, its own copy is still there and is what a child reads first, so that child goes on working from the candidate until the copy goes too.

Continuing after such a park means either implementing the issue as an ordinary change or starting an explicit new split cycle on the owner, which preserves a candidate of its own.

Re-decomposing an issue inside a late lineage¶

A genuine edit to an issue a late split made, or to the umbrella of one, sends it back to the ordinary decomposer. The children that answer creates stay inside the same lineage: each is seeded one level below its parent under the same root, and a parent whose own split still holds its snapshot ref points each child at that ref, appends the reuse instructions to its body, and records it on the ref's consumer ledger in the write that records the child — so the replacements hold the ref exactly as the originals do (see the signals above). Nothing of the parent's own size gate is carried over: a child is measured on its own commits, at its own depth. Each child also carries a hidden receipt naming its parent, the split's attempt, and its slice, which is how a crash between creating a child and recording it is recovered without opening a second issue for the same slice.

Every refusal on this road parks the issue it lands on: a notice on its thread mentioning HITL_HANDLE, saying which refusal it is, and awaiting_human: true on its pinned comment. The pinned park_reason is left cleared, as every ordinary park leaves it; the reason is recorded only on the park_awaiting_human record both sinks receive, as reason: replacement_lineage_unproved (see ../observability/event-streams.md), so filter the event stream on that to find every such park. Where it lands tells you what to repair:

• On the parent, before any child exists. Its record cannot prove the lineage, it already sits at the lineage depth bound, its own split's snapshot reads as neither held nor released or is held with no base recorded, or a slice the decomposer proposed names a snapshot ref its child would not be kept. Nothing was created and no split marker was written, so repair the record the notice names, or reply asking the decomposer not to split (or to leave snapshot refs out of its slices); the reply resumes the decomposer.
• On the parent, after children exist. A crash recovery or a dependency poll found a recorded child it cannot vouch for — an unreadable pinned comment, a parent_number that is not the parent's number, a late ancestry or pointer the split did not write, text naming a snapshot ref it cannot keep, a register naming it twice, or a receipt in its body other than the one the split stamped for its slice (another slice's, another split's, or one owing a different lineage). No child of that split is started while it stands, and the parent is parked once rather than on every poll. Repair or close the child the notice names; for a receipt, put back the one the split stamped, since the child's own dispatch holds its seed to it.
• On a child. Its pinned seed is not the one its receipt says it was owed: it never landed, or its late ancestry was taken off, cut down, or rewritten. It is held under every label but done and rejected — a hand relabel to workflow:ready, and the adjudication of an oversized candidate of its own, included — until its seed is whole. Where its parent is still recovering the split, that recovery writes the seed and clears awaiting_human in the same write; otherwise write the parent link and the lineage the receipt names, set awaiting_human back to false, or close the child. A child whose pinned comment will not parse is held without a park or a write, logged on every tick, so the comment is left for you to repair. A restart you authorize on such a child keeps its seed (see the restart), so it is not held afterwards.

A crash recovery that cannot take over the child it finds — one a human closed or relabelled, one whose body ends on a receipt other than this slice's, one the register already records, or a receipt more than one issue carries — parks the parent under decomposition_crash and names it rather than creating the slice again. A drift reset that discards a manifest also drops the split's attempt, so a child the discarded split left behind is never adopted into the next one — but it is written onto late_consumers first, so the ref waits for it as an orphan.

While such a split is still short of its own count, the snapshot ref its replacements are pointed at is held whatever the original split's ledger says: the replacement it created last may be on GitHub and not yet recorded as a consumer. So is a finished one's ref while a replacement it recorded is missing from late_consumers, until it is recorded there again. Either holds the umbrella's terminal, a blocked parent's return to its own work, and the closed-owner sweep — so an owner closed mid-split keeps workflow:decomposing and its ref, visited on every sweep. Two ways out, each in this order:

• Recover, then close. On an open owner, the next workflow:decomposing tick adopts the unrecorded replacement, records it on the ledger, and finalizes; the ref then waits on it like any consumer. Do not close it first: the recovery refuses a closed candidate and parks the owner under decomposition_crash instead.
• Discard by hand where nothing will recover it — a closed owner, or a recovery parked on a candidate it could not adopt: close every issue this orchestrator opened carrying the owner's receipt for that split_attempt, then remove split_attempt from the owner's pinned record. The ledger reads as whole again and the ref goes once every recorded consumer has ended; closing the unrecorded replacements first keeps it from going while one still works from it.

Reclaiming a finished issue's artifacts¶

An issue this orchestrator finished with leaves two things behind on the host that ran it: the per-issue checkout an agent worked in, and the orchestrator/-namespaced branch the work was published on — in the clone and on the remote. Nothing in the state machine deletes them. A terminal-artifact maintenance pass does, once a day by default, and this section is what an operator needs to run it, read what it reports, and settle what it would not touch.

Why maintenance and not a workflow stage. Every stage handler is about one issue in one repository, and several of them run at once. What this pass reclaims is not per-issue: it is one host's disk and one clone's ref store, shared by every configured repository and by every worker on the host. A stage cannot say the host is quiet, because a tick is what makes it busy — and a handler that deleted a directory would be deleting it beside workers it cannot see. So the pass runs between polling passes, over a host that has been proved quiet, and it is deliberately the one deletion path here that no issue transition triggers. It writes no workflow state at any point: no label, no pinned state, no comment, no agent session. An issue that has ended keeps every record of how it ended, and a host tidying its own disk cannot change what GitHub says happened.

When it runs¶

• On the daily interval, when no window is set. With TERMINAL_ARTIFACT_CLEANUP_WINDOW unset or blank, the polling loop fits one pass in at the end of the wait between two polling passes — never inside a tick — at most once every TERMINAL_ARTIFACT_CLEANUP_INTERVAL_SECONDS (default 86400, one day; see ../configuration.md). The due gate is in memory on a monotonic clock, so an NTP step, a suspend, or a timezone change cannot bring a pass forward or push it out, and nothing is persisted: a restart costs at most one extra pass. A turn is spent when it is handed out, so a pass that defers waits for the next interval.
• In a local window, when one is set. TERMINAL_ARTIFACT_CLEANUP_WINDOW takes precedence and the interval is not consulted. The same boundary between polling passes attempts a pass at its first opportunity inside the window, read off the wall clock in TERMINAL_ARTIFACT_CLEANUP_TIMEZONE — the start inclusive, the end exclusive:
• The window decides when a pass may start, and is read twice. Once between polling passes, and again after the pass has drained its own workers and taken the host lock, since both of those wait. A window that closed during that wait starts nothing. A pass that did start keeps its usual bounds — the 120s host-hold budget below — whatever the clock does next.
• A started pass spends the window. It is spent as the last thing before discovery, so a pass that then finds no candidate, keeps or fails every candidate, raises, or runs out of budget has had its night. Only a deferral before that point — workers that would not drain within 30s, or a host lock another process holds — leaves the window owed, and the next attempt comes no sooner than 15 elapsed minutes later, timed on the monotonic clock so a stepped wall clock cannot make attempts more frequent.
• Nothing is caught up. A window the host stayed busy through, or the process was down for, is missed; the next attempt is the next window, never a daytime pass.
• A window is named by the local date it opened on. An overnight 23:00-01:00 is one window on both sides of midnight, and an hour a daylight-saving fall-back repeats is that same window again, so neither owes a second pass. An hour a spring-forward transition skips never occurs, so a window lying wholly inside it is missed that night.
• Scheduling state is in memory only. A restart outside the window waits for the next one; a restart inside it may repeat a pass the previous process already made that night, which is safe — the pass reads the host again and reports whatever is already gone as done. Nothing guarantees exactly one pass per window across restarts.
• On demand. chipping-orchestrator --cleanup-terminal-artifacts runs exactly the same pass once and exits, regardless of the interval or the window — the form to point a nightly service timer at on hosts whose orchestrator is not always up. See Run modes for what that mode connects and why it exits 0 when it defers.
• Only on a quiet host. Before it reads anything, the pass closes its own scheduler's admission (submit refuses with reason=maintenance, track_active hands back claimed=False), waits up to 30s for the work already admitted to drain, and acts only if the host went quiet inside that bound. Nothing running is cancelled or hurried. A bound that expires, a shutdown that started, or a barrier it could not take defers the whole pass — which costs one finished issue's disk until the next scheduled attempt and nothing else: the next interval with no window set, or no sooner than 15 elapsed minutes later inside a window that is still open. Admission always reopens afterwards, whatever the pass did.
• Only one process at a time. The barrier answers for this process's own workers; a second orchestrator on the same host is outside it. So every pass also holds WORKTREES_DIR/.artifact-maintenance.lock exclusively while it acts, and every polling run holds that same lock shared for its whole life. A pass refused the lock defers; a poller that wants it back waits without a deadline. What bounds that wait is the pass itself: it gives the host back at a candidate boundary once it has held it for 120s, with whatever it did not reach owed to the next pass. The whole mechanism is in ../configuration.md#parallel-processing.

Both of those are in-application coordination between orchestrator processes. Neither is an OS boundary: see ../security.md#scheduled-artifact-reclamation for what that does and does not protect against, and for the isolation to use where the boundary has to hold against an arbitrary process.

What it looks for¶

One host-wide scan per pass, then one repository at a time. A candidate is one issue, however many artifacts it is holding and wherever they are:

• Branches, in the exact orchestrator namespace. The clone's refs/heads/orchestrator/ listing and one git ls-remote of the same namespace on the remote. Two names per issue are recognized and no others: the current orchestrator/<sanitized-slug>/issue-<n> this orchestrator publishes now, and the legacy flat orchestrator/issue-<n> an issue already in flight when slug namespacing landed is still on. A name outside that pair is never a candidate — no tag, no pull-request ref, no default branch, and no branch of anyone else's is reachable from this path.
• Checkouts, under the two roots it writes. WORKTREES_DIR/<sanitized-slug>/issue-<n>, the per-repository path it uses now, and WORKTREES_DIR/issue-<n>, the flat path that predates it. A directory has to be a real directory under the exact name (never a symlink) and a worktree of this repository's own clone.
• Remote-only artifacts. A branch the remote still carries under the owned namespace with nothing left on this host — a re-cloned checkout, a host rebuilt, a local ref somebody deleted by hand. The local and remote halves are folded into one candidate per issue, so a branch that exists in both places is one artifact over two hosts rather than two artifacts.

The layout a candidate is reported under says which of those it was found as: current, legacy, mixed (both names at once, which a migration leaves behind), or remote_only (nothing local at all, whatever the remote's copy is called).

Shared clones. Two configured REPOS entries can point at one target_root — the case slug namespacing exists for. Attribution re-derives each spec's own names and refuses anything ambiguous: a name several entries could equally own (every legacy flat branch on a shared clone, a checkout directory two lossily-sanitized slugs both produce) is attributed to none of them and left alone. The flat pre-namespacing checkout is the one artifact whose name says nothing at all, so it is attributed by the clone the directory turns out to be a worktree of; where that still cannot settle it, the whole issue is withheld from the scan — branches included, because reporting a branch whose checkout nobody may remove would hand a teardown a ref to delete out from under a live tree. A repository whose ref store, checkout root, or remote listing could not be read is left out of the pass entirely rather than reported in part, and said out loud on the log: a partial list of what a repository still holds reads exactly like a complete one.

What clears a candidate, and what keeps it¶

Every gate fails closed, and they are asked cheapest first. Nothing is deleted unless all of them pass:

1. Nothing is running for this issue. The live scheduler is asked per candidate, underneath the barrier that already emptied it. An unreadable answer keeps the candidate.
2. The issue has really ended. It is closed, it carries exactly one terminal workflow label, and its pinned state parses. Two workflow labels at once, a pinned comment holding something that is not a state, an issue or a comment that could not be read: each keeps the candidate.
3. No pull request still stands on it. Open pull requests are looked up for both branch spellings and for the recorded number, whatever base they target. A human reviewing a branch is standing on it even though the issue underneath has ended. Every branch lookup is owner-qualified — the query head is <this repository's owner>:<branch>, never the bare branch name — so a fork that happens to carry a branch of the same name is not mistaken for this candidate's pull request, in either direction: a fork's open PR cannot keep an artifact this repository owns, and it cannot be read as the ended PR that accounts for a commit. The recorded number is read straight off this repository for the same reason.
4. Every commit an artifact holds survives its deletion. A branch's tip and the commit a checkout's own HEAD is on each have to be contained in the configured base or in a pull request that has ended. The remote is asked what the branch is at whatever the base says, because the two answer different questions — a merged tip can sit under a branch somebody has since pushed past.
5. The checkout carries nothing of its own. It has to be this issue's own branch, prove it holds nothing modified or untracked, and prove it hides nothing besides: a path the repository's own .gitignore covers — a .env, a virtualenv, a build directory — keeps the whole candidate, because git worktree remove would take all of it and git does not count it as dirty.
6. Nothing has touched the checkout for an hour. Its own directory (for what is created or removed at the top of it) and its index and reflog (for the edit-and-commit that moves neither) are read against a one-hour quiet period. That period is a constant, not a knob: what a shorter one buys is the chance to delete a tree somebody is standing in.

Ambiguous evidence is not permission. Every probe answers "confirmed", "refuted", or "could not read", and the third is never collapsed into either: a reading that failed keeps the artifact exactly as a reading that said no does.

What it deletes, and in what order¶

Order is not a detail here — it is what a failed pass leaves behind:

1. Every checkout first, both layouts, in the order the scan reported them. A branch a worktree still has checked out cannot be deleted, so a pass that took branches first would strand the trees and make their branches undeletable. Removal is never forced: git worktree remove refuses a tree written in since the proof, and that refusal is the answer.
2. Then each branch, on the remote before the clone. The remote is read first so the three answers stay apart — a branch that is not there is a step already done (the ordinary shape of a merged pull request's head), a branch at another commit is a push nobody here cleared, a reading that failed is not permission to delete. The delete itself is a leased push pinned to the proved commit, so a branch somebody pushed to between the proof and the delete is refused by the remote rather than removed. The local delete is the pinned update-ref -d naming that same commit and refusing to dereference, so a branch an agent committed onto survives and a symbolic ref planted under a branch name is deleted as itself rather than followed onto the base.
3. Local last, on purpose. The local ref is the cheapest thing the next discovery finds, so a failed remote delete leaves it standing and the candidate discoverable; taking it first would leave a remote-only artifact that only a remote listing would ever surface again.

Immediately before each mutation the artifact's tip is re-read against the commit the classification proved. An artifact that has moved since (an agent committed, a human pushed) is not a failure — it is a race, and the candidate is kept for the next pass to start over on.

There is no retry, and that is deliberate. The first refusal ends the pass for that candidate; the ones behind it are still evaluated. Nothing is written down — no queue, no ledger, no backoff state — because the discovery that found the candidate once finds whatever is left of it again. That is what makes a repeated pass cheap: it reads the host as it is now and reports whatever is already gone as done. A candidate refused by a branch-protection ruleset therefore reports failed once per pass until the rule or the branch changes, and costs one pass each time.

Reading a cleanup result¶

Every candidate the pass decided about earns one bounded terminal_artifact_cleanup record on the analytics sink — repository, issue, outcome, reason, layout, and a branch where the reason names one — and one line per candidate on the orchestrator.worktree_lifecycle log channel, followed by the pass's own cleaned=/retained=/failed= tally. The log is where the artifact each reason is about is named; the record is deliberately narrower and never carries a path, a command, or git's own output. Both surfaces are observation-only: losing either changes nothing about what was deleted. A sink turned off writes nothing and says nothing, and a sink the filesystem refuses (read-only mount, full disk, a path that is not one) is reported on the orchestrator.analytics channel by the writer itself rather than on the lifecycle channel — so an operator missing records looks there first.

• cleaned — done. Every artifact that candidate was found holding is gone from this host and the remote, absences included. It will not be reported again.
• retained — the pass declined to act, and it will decline again every pass until something changes. This is the one outcome that can repeat forever, so a candidate retained for the same reason day after day is one to settle by hand. Match the reason against the reason-code table:
• active_claim / recent_activity are self-clearing — work was in flight, or the checkout had been touched inside the hour. Nothing to do.
• tip_moved is a race that also clears itself: something moved the artifact between the proof and the mutation.
• unproven is the classification's own answer, and the log line beside it names the artifact it kept the candidate for. Look at that artifact: an open pull request still on the branch (close or merge it), a checkout carrying modified, untracked, or ignored files (take what you want off it — an ignored .env or a stray build directory is the common one), commits the base and every ended pull request do not account for (push them somewhere that outlives the branch, or accept the artifact stays), an issue that is not closed, or a workflow label that is not exactly one terminal.
• The *_unreadable reasons are a read that failed rather than a fact about the artifact: an expired token, a repository this installation cannot see, a remote that was down, a git command that would not run. They clear themselves when the underlying read works, and the log line beside them says which artifact could not be read.
• branch_checked_out means some worktree of the clone is still standing on the branch — often an operator's own git worktree add. Move that tree off the branch (or remove it) and the next pass takes it.
• failed — a step ran and was refused, which is a host or a remote to look at:
• worktree_removal_failed — git would not remove the checkout. It is never forced, so the usual cause is a tree written into after the proof, or a permission problem on the directory. Inspect the path on the log line; remove it by hand once you are sure nothing wants it.
• remote_delete_failed — the remote refused the leased delete. A ruleset forbidding deletions under refs/heads/orchestrator/*, a token without the permission, or a branch that moved. This blocks tidying and nothing else: no issue is parked or relabelled, and the local ref is deliberately kept so the candidate is found again.
• local_delete_failed — the pinned local delete would not run: another process holding the clone's ref store, or a ref that moved between the read and the delete.

A candidate the pass never reached has no record: nothing was decided about it. The one a stop landed on says so on the log — artifacts left alone: this pass may no longer act — and earns no record either, for the same reason. That is not a silent failure; see below.

Recovery after an interrupted pass¶

A pass can stop part way through for four ordinary reasons: SIGINT / SIGTERM, the scheduler closing under it, its 120s host-hold budget running out (which is what a polling process waiting for the host is owed), or the process being killed outright. Whichever it is, recovery is nothing:

• Nothing is persisted, so there is no partial state to repair, no lock file to clear (a flock dies with the process that held it), and no queue to drain. The next pass rediscovers the host as it is.
• A candidate the pass stopped before is exactly as it was, and has no result to interpret.
• A candidate whose teardown was under way cannot be half-deleted in a way that matters. A signal is never answered mid-teardown: the last reading is taken immediately before the first mutation, and past it the candidate is taken as one unit. What a kill can interrupt is therefore a checkout removal and, per branch, a leased remote delete and a pinned local one — each idempotent and each already leased to a proved commit. What ran is done, and what did not is rediscovered; an artifact already gone is simply reported as done next time.
• The pass that ran can leave a checkout removed with its branch still standing, or a branch gone from the remote and still in the clone. Both are discoverable states, and the next pass finishes them.

So an interrupted pass costs the wait for the next scheduled pass — the next interval with no window set, or the next night's window, since a pass that started has spent its own — and a crashed one costs no more: the process that comes back is owed a pass at once on the interval, and inside the window may repeat that night's. If a host was killed mid-pass often enough to matter, the thing to check is not the artifacts but why: the pass gives the host back on its own at a candidate boundary, and never blocks workflow progress either way.

Applying .env changes¶

~/.config/chipping-orchestrator/.env is read once, when chipping-orchestrator starts. Most edits take effect on the next process start; there is no signal to reload configuration. Restart the foreground command or systemd service at a safe stopping point. Developer checkouts use their own root .env.

What survives a restart¶

Per-issue progress lives in the issue's pinned JSON comment on GitHub and in the per-issue worktree on disk. Restarting between ticks loses nothing — the next tick picks each issue back up from its label and pinned state. Two restart-time hazards are worth knowing:

• A live codex / claude child. Stage handlers spawn agent subprocesses that may run for as long as AGENT_TIMEOUT. On a SIGTERM/SIGINT stop the loop terminates in-flight agent process groups up front (and, as a hard backstop, the shutdown watchdog force-terminates them after SHUTDOWN_GRACE_SECONDS and exits), so the process leaves within the grace window instead of holding open for up to AGENT_TIMEOUT and being SIGKILLed at the systemd stop deadline. The interrupted child is still killed mid-session. The runner flags such a run interrupted (distinct from timed_out), and every dev-resume stage handler (implementing, validating, fixing, documenting, in_review, resolving_conflict) short-circuits on it — they ignore the partial result and leave durable GitHub state untouched, so the next process retries the resume from scratch rather than parking on awaiting_human or routing through timeout recovery. A dirty worktree may still remain on disk for the next tick to reconcile.
• In-flight agent spec is pinned. When a codex / claude session starts, the orchestrator writes the full DEV_AGENT / DECOMPOSE_AGENT spec into pinned state and re-parses it (not the current .env) on every resume. Flipping DEV_AGENT or DECOMPOSE_AGENT after a session is locked does nothing for that issue until it reaches done or rejected. The question and discussion stages each seed from DECOMPOSE_AGENT on their own first spawn and pin to question_agent / discussion_agent for the rest of that conversation. REVIEW_AGENT is not pinned — the reviewer spawns fresh each round.

Safe restart guidance¶

• Idle / between ticks — safe. Restart freely; the next tick resumes from GitHub state.
• Issue mid-stage with no agent child — generally safe. Workflow state is on GitHub and in the worktree.
• Live codex / claude child — avoid. Wait for the agent to exit. Forcing a restart can park the issue or leave a dirty worktree behind.

Set the shell's LOG_DIR to the running process's log directory: ~/.local/state/chipping-orchestrator/logs after package setup above, unless overridden. Use an absolute path; settings in .env are not automatically shell variables. Then inspect processes and logs:

pgrep -af 'chipping-orchestrator|codex|claude|agy'
tail -f "${LOG_DIR:?}/orchestrator.log"
journalctl --user -u orchestrator.service -f   # systemd users

Per launch style¶

Foreground terminal.

1. Edit ~/.config/chipping-orchestrator/.env.
2. Wait for a safe stopping point without an agent child; inspect processes with pgrep -af 'codex|claude|agy'.
3. Ctrl+C the command and wait for shutdown.
4. Start chipping-orchestrator again.

systemd user service.

1. Edit ~/.config/chipping-orchestrator/.env.
2. Run systemctl --user daemon-reload only if the .service unit changed; it reloads unit definitions, not .env.
3. When safe, run systemctl --user restart orchestrator.service.
4. Tail logs with journalctl --user -u orchestrator.service -f.

When GITHUB_TOKEN is supplied via the unit's EnvironmentFile=, edit that file and restart. Changing an inline Environment= value requires editing the unit and running daemon-reload before restarting.

Each chipping-orchestrator --once invocation starts a new process and reads current settings. Restarts keep the installed package version; package upgrades are explicit operator actions. Development wrapper restarts are described in the developer guide.

Setting-by-setting expectations¶

When each setting's change takes effect:

• TERMINAL_ARTIFACT_CLEANUP_INTERVAL_SECONDS — next Python start, like the settings below, and the restart also starts a fresh due gate: the cadence is held in memory, so with no window set the run that comes back is owed a pass at once rather than at the end of an interval it cannot remember. A repeated pass costs one discovery and reports whatever is already gone as done.
• TERMINAL_ARTIFACT_CLEANUP_WINDOW, TERMINAL_ARTIFACT_CLEANUP_TIMEZONE — next Python start, where both are validated again: a malformed window, or a blank or unresolvable timezone beside a set one, stops that start. The restart also forgets which window was spent: a run that comes back outside the window waits for the next one, and one that comes back inside it may repeat that night's pass.
• POLL_INTERVAL, AGENT_TIMEOUT, REVIEW_TIMEOUT, SHUTDOWN_GRACE_SECONDS, MAX_REVIEW_ROUNDS, MAX_CONFLICT_ROUNDS, MAX_RETRIES_PER_DAY, MAX_ADDED_LINES, DEV_SESSION_MAX_RESUMES, MAX_AGENT_RUNS_PER_ISSUE, IN_REVIEW_DEBOUNCE_SECONDS, DECOMPOSE, SQUASH_ON_APPROVAL, PR_REF_IN_SUBJECT, EXPOSE_TRACKED_REPOS, VERIFY_COMMANDS, VERIFY_TIMEOUT, LOG_DIR, EVENT_LOG_PATH, ANALYTICS_LOG_PATH, ANALYTICS_RETENTION_DAYS, TRACK_SKILL_TRIGGERS, TRAJECTORY_LOG_PATH, TRAJECTORY_RETENTION_DAYS, REPO / REPOS / TARGET_REPO_ROOT / BASE_BRANCH / REMOTE_NAME, HITL_HANDLE, ALLOWED_ISSUE_AUTHORS — next Python start
• WORKTREES_DIR — next Python start. Left unset, it follows the first REPOS entry, or TARGET_REPO_ROOT without REPOS, so an edit that changes either moves it too: set it explicitly before such an edit on a host with issues in flight (default)
• ANALYTICS_DB_URL — next analytics sync invocation and dashboard launch (operator sync commands, developer dashboard commands). The value is parsed once when the analytics settings holder is first imported, so a browser reload is not enough — relaunch Streamlit. The polling loop does not read this setting.
• DASHBOARD_PARALLEL_READS — next developer analytics dashboard launch. The value is parsed once per process, on the first render's import of the read-mode owner.
• MAX_PARALLEL_ISSUES_PER_REPO, MAX_PARALLEL_ISSUES_GLOBAL — next Python start. Per-REPOS parallel_limit overrides take precedence over MAX_PARALLEL_ISSUES_PER_REPO.
• WORKFLOW_TRANSITION_GUARD — next Python start (parsed at config import).
• DEV_AGENT, DECOMPOSE_AGENT — next Python start, except for issues whose pinned state already names a dev_agent / decomposer_agent / question_agent / discussion_agent — those keep the pinned spec until the issue reaches done or rejected
• REVIEW_AGENT — next reviewer spawn after the next Python start (not pinned per issue)
• GITHUB_TOKEN — not loaded from .env. Update the process environment or rewrite the token file before the next start: the one ORCHESTRATOR_TOKEN_FILE names, or else each configured repository's own ~/.config/<owner>/<repo>/token
• ORCHESTRATOR_BASE_BRANCH — developer source wrappers capture this before the restart loop, so editing it requires restarting the wrapper itself (developer guide). The Python process picks it up on the same next start. An installed package never uses it: the self-restart guard runs only in a source checkout.