Download Pipeline & Recovery

Comicarr keeps a durable pipeline journal so downloads and post-processing can survive restarts without double-processing completed work. Recovery runs automatically at startup, so there is no special command to run.

Why it exists

Historically, a restart mid-download or mid-post-process could leave items stuck as “Snatched” with no further progress, or risk re-running work. The journal records each release’s stage so startup can:

1. Reconstruct missing journal rows when a snatch committed but the journal write was lost
2. Re-drive every open obligation through remaining stages exactly once
3. Mark downloads that are truly gone from the client as failed (with a retained payload for later retry)

Stage lattice

Stages are forward-only (monotonic). A later stage never regresses to an earlier one.

Open (still-in-flight) stages: snatched, downloaded, post_processing, moved.
Terminal stages: post_processed, failed.

Release keys

Each journal row is keyed by a release_key derived once in shared code (comicarr.app.downloads.journal). The key ties together issue identity and provider so snatch and post-process seams agree.

Notes:

• Synthetic “one-off” issue IDs (≥ 900000) are not stable across restarts and are handled specially so they are not misclassified as done/gone solely via nzblog presence.
• Provider labels are normalized (whitespace, case, [RSS] suffix stripping) so RSS and active-search snatches share the same key.

Startup recovery (automatic)

Comicarr runs pipeline replay once on process start, after init and credential decryption and before the web server is fully serving:

1. Anchor reconstruction: rebuild a snatched journal row only when the snatch is durable but the journal write was lost, and the release has not already advanced. That guard keeps every historical snatch from being re-driven.
2. Snapshot open rows: read all open journal obligations.
3. Classify each row against the download client / DDL state:
   • still: still in flight, so remaining work is re-queued
   • complete: finished while down, so it advances or finishes without re-snatching. Since v0.34.0, "finished" requires library placement evidence, meaning the issue's stored location or Downloaded status. Old download-history state on its own no longer counts. A download that completed but was never imported is re-imported when the completed folder can be resolved; otherwise it lands in Needs attention as "download finished but was never imported into the library", with the files still in the download directory ready for a manual import.
   • gone: the client no longer has the item and there is no "already done" signal, so the row is marked failed with reason download_gone
   • unknown: ambiguous, for example when the client is unreachable. The row is left alone and retried next start.
4. Throttle enqueue bursts so SQLite is not hammered against live workers.
5. Cap inline full post-processing re-drives per pass so startup does not block the UI indefinitely; remaining open PP rows resume on the next start (idempotent).

You should see [RECOVERY] lines in the application log when reconstruction or re-drives occur.

Healing rows that older versions closed out wrongly

Versions before v0.34.0 could mark a download post_processed on download-history state alone, leaving the files stranded in the download directory while Activity claimed a successful import. Those rows were then permanently stuck, because the pipeline treated them as finished and no later startup would look at them again.

Since v0.34.1, startup recovery re-examines finished pipeline records that the library itself contradicts: no stored file location, no Downloaded status, and no operator decision such as Ignored or Archived. Those rows go back through the normal recovery path. When the completed download folder still resolves, the import runs for real; otherwise the item lands in Needs attention with the files ready for a manual import. Genuinely imported items, one-off downloads, and issues from removed series are left alone.

v0.34.1 also stopped completed Usenet downloads from being quarantined with immutable_payload_conflict:provider. The pipeline had recorded the same download's provider two ways, as DrunkenSlug (newznab) at grab time and plain DrunkenSlug at completion, which tripped the identity guard on every finished NZBGet download. Both spellings now resolve to one provider, and records written under the old spelling are reconciled on the next startup.

Interaction with failed download handling

Journal failed / download_gone is about pipeline durability. Separate settings still control automatic re-search after failures:

See Downloading for search/snatch behavior.

Operator tips

• Prefer a clean shutdown (POST /api/system/shutdown or docker stop with grace period) so workers can finish journal writes. Compose sets stop_grace_period: 30s.
• If items look stuck after a crash, restart Comicarr once and check logs for [RECOVERY] / [JOURNAL].
• Do not manually edit pipeline_journal rows unless you know the stage lattice; invalid stages are ignored or rejected by the monotonic guard.
• DDL and NZB/torrent clients are classified differently; client history eviction is treated carefully so a completed-but-pruned history row is not marked gone when the library already shows post-processed.

Related

• Downloading for search loops and post-processing steps
• Direct Downloads for DDL hosts and stuck-download checks
• REST API for downloads and system endpoints