Skip to content
Paquetier

Scanning Existing Repos

The scan feature lets you import packages from an existing S3 bucket that already contains R packages (e.g. a legacy CRAN-style repository or a Posit Package Manager mirror).

How Scanning Works

When you trigger a scan, Paquetier:

  1. Looks for PACKAGES.gz or PACKAGES index files in the S3 bucket.
  2. Probes several standard path conventions:
    • CRAN-style: src/contrib/
    • Binary: bin/{os}/contrib/{rver}/
    • Posit/RSPM-style: {arch}/{os}/latest/src/contrib/
  3. Parses the DCF-format index to discover packages, versions, and metadata.
  4. Imports all discovered packages, versions, and artifacts into the Paquetier database.
  5. Scans the Archive/ subdirectory for older versions.
  6. Auto-detects source vs binary packages by inspecting tarball contents.

Scanning runs in parallel (4 workers) and reports progress via the web UI.

Scans are durable: each scan is persisted to the database with a checkpoint of how many index entries have been processed. A scan interrupted by a process restart or instance failure resumes from its checkpoint instead of starting over, re-probing only the handful of entries that were in flight at crash time. For a full-distro CRAN scan (~100,000 entries, roughly 20 minutes of S3 HEAD probes), this turns a restart from “redo everything” into “pick up exactly where we left off”.

Triggering a Scan

Scans can be triggered by repository owners and admins:

  1. Navigate to the repository you want to scan.
  2. Click Scan (or use the scan option in the repository settings).
  3. The scan runs in the background. Progress is streamed in real-time.

Restart, Failure, and HA

Each scan is owned by exactly one instance at a time via a row-level lease (leased_by + lease_expires_at columns on the scans table). The owning instance extends the lease via a heartbeat every 15 seconds (configurable). If the heartbeat stops — because the process died, was killed, or lost its DB connection — the lease expires and another instance can take over.

Single-instance restart

After a process restart, the boot-time reconciler scans the scans table for rows whose lease has expired, claims them, re-fetches the PACKAGES index, verifies the index hasn’t changed upstream (via stored sha256), and resumes from the checkpoint. This happens automatically before the HTTP server starts accepting traffic, so a UI client reconnecting after the restart sees the scan progressing again rather than appearing stuck.

Multi-instance / HA setups

When multiple Paquetier instances run against the same database, the same primitives provide failover:

  • A reconciler ticker on every instance polls for abandoned scans every 30 seconds (configurable).
  • The atomic claim guarantees exactly one peer takes over each abandoned scan, even when several reconcile passes race simultaneously.
  • The streaming endpoint /api/v1/jobs/{id}/stream falls back to row-polling when the receiving instance does not own the scan, so a browser hitting any instance can follow progress regardless of which peer is doing the work.
  • Each instance respects PAQUETIER_SCAN_MAX_CONCURRENT to prevent one peer from greedily claiming every abandoned scan and starving others.

On graceful shutdown (SIGTERM), the owning instance releases its leases immediately so peers can take over within seconds rather than waiting for the full TTL.

Index drift

If the upstream PACKAGES file changes between the original scan and a resume attempt — for example, a new package version landed mid-scan — the resume detects the checksum mismatch and fails the scan with index changed during scan; please re-trigger. Re-trigger from the UI to pick up the latest index from scratch. The existing duplicate-detection ensures already-imported artifacts are not re-imported.

When to Use Scanning

  • Migrating from an existing CRAN mirror — Import all packages at once instead of re-uploading them individually.
  • Connecting to a shared S3 bucket — If packages are already stored in S3 by another tool, scan to register them in Paquetier.
  • Disaster recovery — Re-populate the database from S3 if the database is lost.

Limitations

  • Scanning only discovers packages that have a valid PACKAGES index file.
  • The scan does not modify or move any files in S3; it only creates database records pointing to existing artifacts.
  • If the upstream PACKAGES file changes mid-scan, resume fails with a clear error rather than silently merging — re-trigger to pick up the latest state.