Download Pipeline & Recovery

Comicarr keeps a durable pipeline journal so downloads and post-processing can survive restarts without double-processing completed work. Operators do not need to run a special recovery command — recovery runs automatically at startup.

Why it exists

Historically, a restart mid-download or mid-post-process could leave items stuck as “Snatched” with no further progress, or risk re-running work. The journal records each release’s stage so startup can:

1. Reconstruct missing journal rows when a snatch committed but the journal write was lost
2. Re-drive every open obligation through remaining stages exactly once
3. Mark downloads that are truly gone from the client as failed (with a retained payload for later retry)

Stage lattice

Stages are forward-only (monotonic). A later stage never regresses to an earlier one.

Open (still-in-flight) stages: snatched, downloaded, post_processing, moved.
Terminal stages: post_processed, failed.

Release keys

Each journal row is keyed by a release_key derived once in shared code (comicarr.app.downloads.journal). The key ties together issue identity and provider so snatch and post-process seams agree.

Notes:

• Synthetic “one-off” issue IDs (≥ 900000) are not stable across restarts and are handled specially so they are not misclassified as done/gone solely via nzblog presence.
• Provider labels are normalized (whitespace, case, [RSS] suffix stripping) so RSS and active-search snatches share the same key.

Startup recovery (automatic)

On process start — after init and credential decryption, before the web server is fully serving — Comicarr runs pipeline replay once:

1. Anchor reconstruction — rebuild a snatched journal row only when the snatch is durable but the journal write was lost, and the release has not already advanced (avoids re-driving every historical snatch).
2. Snapshot open rows — read all open journal obligations.
3. Classify each row against the download client / DDL state:
   • still — still in flight → re-queue remaining work
   • complete — finished while down → advance / finish without re-snatching
   • gone — client no longer has the item and no “already done” signal → mark failed with reason download_gone
   • unknown — ambiguous (e.g. client unreachable) → do not mark gone; retry next start
4. Throttle enqueue bursts so SQLite is not hammered against live workers.
5. Cap inline full post-processing re-drives per pass so startup does not block the UI indefinitely; remaining open PP rows resume on the next start (idempotent).

You should see [RECOVERY] lines in the application log when reconstruction or re-drives occur.

Interaction with failed download handling

Journal failed / download_gone is about pipeline durability. Separate settings still control automatic re-search after failures:

See Downloading for search/snatch behavior.

Operator tips

• Prefer a clean shutdown (POST /api/system/shutdown or docker stop with grace period) so workers can finish journal writes. Compose sets stop_grace_period: 30s.
• If items look stuck after a crash, restart Comicarr once and check logs for [RECOVERY] / [JOURNAL].
• Do not manually edit pipeline_journal rows unless you know the stage lattice; invalid stages are ignored or rejected by the monotonic guard.
• DDL and NZB/torrent clients are classified differently; client history eviction is treated carefully so a completed-but-pruned history row is not marked gone when the library already shows post-processed.

Related

• Downloading — search loops and post-processing steps
• Direct Downloads — DDL hosts and stuck-download checks
• REST API — downloads and system endpoints