Orchestration des mises à jour des données de l’Assemblée nationale
  • Shell 91.5%
  • JavaScript 8%
  • Dockerfile 0.5%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-11-11 15:40:24 +01:00
.forgejo/workflows feat(ci): expose the retrieval-only ingest phase for frozen datasets 2026-11-11 15:40:08 +01:00
ci feat: matérialiser les portraits des députés 2026-09-10 15:49:50 +02:00
docs feat(ci): process anticipated documents end to end with targeted metadata import 2026-10-01 18:16:48 +02:00
examples feat(ci): process anticipated documents end to end with targeted metadata import 2026-10-01 18:16:48 +02:00
scripts Add retrieve-annexes phase for PLF dossier annexes 2026-10-05 10:59:49 +02:00
.gitignore feat: add Forgejo Assemblee orchestration 2026-09-01 02:00:09 +02:00
Dockerfile fix: identify non-root runner users in runtime image 2026-09-01 02:15:07 +02:00
README.md feat(ci): expose the retrieval-only ingest phase for frozen datasets 2026-11-11 15:40:08 +01:00

Tricoteuses Assemblée update orchestration

This repository runs the production pipeline that turns Assemblée nationale open-data archives into versioned Git datasets, PostgreSQL projections, and search indexes.

The canonical repository is https://git.tricoteuses.fr/logiciels/tricoteuses-assemblee-update. The former GitLab project is archived and is no longer synchronized.

It contains orchestration and safety controls. The parsers and database/indexing implementations live in separate source repositories:

  • tricoteuses-assemblee: downloads, reorganizes, and cleans Assemblée data;
  • tricoteuses-assemblee-canutes: projects cleaned data into Canutes;
  • tricoteuses-juridique: builds the PostgreSQL-backed and Typesense search projections.

Forgejo Actions on git.tricoteuses.fr is the only CI control plane, and all produced datasets are stored in canonical Forgejo repositories. The former GitLab projects are archived redirects and no longer accept publications.

Production topology

The host names used by the workflows refer to physical infrastructure, not to software components:

Name Role in this pipeline
Rook Linux worker server. Runs up to two production jobs and one watcher job. Its persistent corpus is exposed to jobs as /srv/tricoteuses-assemblee-data.
Macdo Second Linux worker server and failover for Rook. Runs one production job and one watcher job, with its own synchronized corpus.
Duck Third worker with its own canonical corpus. duck-assemblee serves assemblee-prod and retains assemblee-duck-canary; duck-assemblee-watch serves assemblee-watch.
Mole Server hosting the Forgejo service and mesh-only HTTP/SSH proxies used by the runners.
Swan Server hosting the fenced-lease broker that prevents concurrent writers on Rook and Macdo.
Jay Server endpoint used for the Canutes PostgreSQL database and Typesense search service.

Rook and Macdo are persistent workers: the roughly 185–220 GiB corpus is not copied into each CI job. Both servers run repository-scoped, non-privileged Forgejo runners without access to the Docker socket. Duck restored 74 canonical worktrees and validated Scrutins materialization and indexing in run 948; return runs 949 and 950 succeeded on Rook. The isolated full run 957 and return run 958 also succeeded, both with changed=false and no Scrutins ref modified. Duck has since joined the shared production pools. Those no-op canaries do not prove data push authorization; check the affected deploy-key associations before a publishing validation.

All traffic between these servers uses their WireGuard mesh. Detailed host, runner, proxy, secret, and service provisioning belongs to the separate infrastructure repository eraviart/serveurs.

Update flow

A normal update follows these steps:

  1. watch.yml checks the versions advertised by the Assemblée open-data endpoints.
  2. For each changed dataset, it dispatches one ingest.yml run. The run title includes the dataset name, for example Update Assemblée — Agenda_XVII.
  3. The dataset category is mapped to a shard: a declared set of Git repositories and shared resources that must be updated together.
  4. A worker synchronizes the shard's persistent worktrees, downloads and cleans the source data, then commits and pushes the generated repositories.
  5. PostgreSQL materialization runs under the shared database lease when the category has a database projection.
  6. Search indexing runs under the lock associated with the target collection or queue.
  7. The ci-open-data-state branch records independently whether the result was published, applied, and indexed.

A failure after publication is therefore visible: it does not incorrectly mark the database or search projection as complete. The failed downstream phase can be resumed without downloading and publishing the dataset again.

Shards and concurrency

The executable resource map is ci/shards.json, interpreted by scripts/shard-plan.mjs. It declares six shards and rejects unknown categories, undeclared paths, missing dependencies, and parallel shards with overlapping Git write sets.

Notable shared resources:

  • ActeursEtOrganes also owns the deputy and senator photo repositories; photo refreshes run in the same fenced shard instead of a downstream GitLab CI, and deputy portrait bytes are materialized in Canutes with the actor data;
  • Agendas and ComptesRendus share the agenda-reports shard because their cleaners reconcile common outputs;
  • PostgreSQL writers use assemblee-materialize-db;
  • writers to textes_juridiques use the same search lock;
  • Amendements indexing drains its dedicated PostgreSQL-backed Tisseuse queue;
  • Questions and ComptesRendus have no PostgreSQL projection and therefore do not acquire the database lease merely to advance their applied marker.

all.yml publishes independent shards in parallel, then applies the shared PostgreSQL and search phases in dependency order.

Workflows

Workflow Purpose
watch.yml Hourly detection and dispatch of changed datasets.
ingest.yml Update one dataset/shard. Manual runs may select the end-to-end full phase, the retrieval-only ingest phase, documents, retrieve-documents, the urgent end-to-end process-documents, materialize-documents, recover-links, materialize, sync-documents, or index, and may separately request a full PostgreSQL import.
all.yml Run all shards, followed by ordered shared projections; full PostgreSQL imports are opt-in.
runtime-image.yml Build the minimal production image referenced by immutable digest.

Production source checkouts are detached at revisions resolved at run time from the protected default branches (master for tricoteuses-assemblee and tricoteuses-assemblee-canutes, main for tricoteuses-juridique); dispatch inputs accept a full commit SHA as an explicit fallback. The runtime image and third-party actions are also pinned by digest or commit.

Safety model

  • Every mutating phase obtains a fenced lease from the broker on Swan.
  • A pre-push hook checks the lease immediately before Git publication.
  • Losing the broker or lease terminates the complete job process group and prevents a late push.
  • Persistent worktrees are synchronized only after acquiring the relevant shard lease. Open-data-derived repositories discard their uncommitted generated state at synchronization: an interrupted run leaves nothing that the next retrieval cannot re-derive. The documentary checkpoint repositories (Documents, Documents_enrichis) hold acquired content that cannot always be re-fetched and still fail closed on any uncommitted change; unpublished local commits remain proof-gated everywhere.
  • Secrets and runtime .env files are mounted read-only from root-managed host paths; they are never stored in ci-open-data-state.
  • Every runtime job mounts /etc/tricoteuses-assemblee-ci/passwd read-only at /etc/passwd. It must describe the numeric container UID/GID from the runner configuration, not necessarily the runner daemon's account. The dispatcher checks NSS resolution before source resolution, npm, secret copies or leases. Without this mount, OpenSSH fails locally (No user exists for uid) even when Git access rights and the SSH key are correct. Provision and allow the mount on every eligible host before deploying workflow changes.
  • Forgejo and the still-canonical GitLab data remotes are reached through mesh-only proxies on Mole.
  • Shared PostgreSQL and Typesense writes are serialized even when independent Git publication phases run in parallel.

Do not run an ingestion script against production data without the corresponding fenced lease. Use disposable copies for shard-map or transformation tests.

Operational behavior

An exact DATASET is forwarded unchanged to retrieval, raw reorganization, read-only raw validation and cleaning. Only retrieval receives --only-if-new. Shard synchronization and fencing remain shard-wide, but transformations must not touch sibling datasets whose inputs were not downloaded. This requires a source revision supporting reorganize_data.ts --dataset and input-presence preflight; do not pin the source to a revision predating that safety fix.

The watcher normally runs for 55 minutes. Setting repository variable ASSEMBLEE_WATCHER_DISPATCH=false turns a scheduled or manual watcher run into a single read-only observation that neither dispatches work nor publishes state.

Watcher pending entries are dispatch deduplication leases, not the fenced resource leases used by ingestion. They clear when the accepted manifest covers the dispatched artifact or when Forgejo identifies the exact dispatch as failed or cancelled. Missing or unavailable status remains fail-closed until the 24-hour expiry. Correlation uses watch_dispatch_id; timestamp lower bounds must use Forgejo's second precision, including the pagination cutoff. Comparing its second-precision created with millisecond dispatchedAt can hide a same-second failure and retain the lease for 24 hours. The regression fix is Assemblée PR #37. Watcher run 1877 loaded the corrected source and redispatched both Scrutins and Amendements on 2026-10-05 without manually removing controller state. After a correction is merged to the source default branch, dispatch a fresh watch.yml run to load it; a running watcher retains the source revision resolved at startup.

Manual ingest.yml and all.yml runs expose a full boolean, disabled by default. Enabling it passes --full to the PostgreSQL import; otherwise imports only apply changes committed since the last successful materialization. This is independent of ingest.yml's phase=full, which means running the complete publish, materialize, and index pipeline.

Dossiers materialization gives persistent document-content acquisition 75 minutes and PostgreSQL text indexing 30 minutes by default. Both workers consume only queue entries belonging to the requested LEGISLATURE, so older failures from another legislature cannot prevent the current dataset from progressing; LEGISLATURE=0 retains global queue consumption for an explicit all-legislatures run. Remaining queue items stay durable for later cycles. The limits can be overridden with ASSEMBLEE_DOCUMENT_CONTENT_BUDGET_MINUTES and ASSEMBLEE_TEXT_SEARCH_BUDGET_MINUTES for a planned catch-up run. A manual DossiersLegislatifs run with phase sync-documents can instead set document_uids to a comma-separated list: acquisition and PostgreSQL indexing then process exactly those documents, independently of their position in the durable queues. This dedicated phase only checks out tricoteuses-juridique, does not touch the persistent document worktrees, and still acquires the assemblee-materialize-db fenced lease.

Long missing-document catch-ups use the manual documents phase for one legislature at a time. Retrieval, link, and enrichment outputs are checkpointed every 250 changed documents by default on the append-only temporary branch ci/document-catchup/L<legislature>; ASSEMBLEE_DOCUMENT_CHECKPOINT_SIZE can adjust that bound. Once both repositories are complete, each checkpoint branch is squash-committed to master, producing one canonical commit per repository and legislature, then deleted. A stage marker outside the Git worktrees lets a retry discard only the uncommitted generated tail from an interrupted stage and recompute it from the last committed checkpoint. At startup, documentary catch-ups, targeted runs and Dossiers ingestion check all L14–L17 stage markers, not just the requested legislature: an interrupted L16 enrichment must not block a new L17 request. Recovery runs under assemblee-ingest-dossiers, preflights both worktrees before cleanup, and preserves every committed checkpoint (including unpublished local commits). Only known stage-specific generated files on the matching CI checkpoint branch are reset or removed; malformed partial outputs are recomputed, without systematic backups. Unknown changes, missing or invalid markers, Git conflicts and non-CI branches still fail closed. After every requested legislature succeeds, run materialize and then index once with the intended legislature scope.

The retrieve-documents phase is the urgent, UID-targeted variant of that catch-up: a manual DossiersLegislatifs run with document_uids retrieves exactly those documents from the Assemblée site, adds links and enrichment, and publishes them through the same ci/document-catchup/L<legislature> checkpoint protocol, so tail recovery and the assemblee-ingest-dossiers lease are shared with full catch-ups. Documents are read independently of their dossiers: missing references in dossier acts do not prevent processing. A requested UID whose metadata is missing now fails explicitly instead of silently publishing nothing. Targeted runs refetch and reparse the source and force links/enrichment, so a corrected document under the same UID is not hidden by a completion marker.

A document published before its open-data metadata

Use ingest.yml with:

  • categories=DossiersLegislatifs, legislature=17 (or the actual legislature);
  • phase=process-documents, document_uids=<exact institutional document UID>;
  • optional document_metadata: a JSON manifest of verified metadata, following examples/plf-2026-document-metadata.json. This is a historical example, not the UID of the PLF 2027.

process-documents runs retrieval/parsing/links/enrichment and Git publication under the dossiers lease, then targeted metadata import and content/search synchronization under the PostgreSQL lease, then Typesense projection under its shared lease. A failure stops the next phase. Typesense's existing --assemblee-textes projection is global, not UID- or legislature-filtered. Targeted operations do not advance the dataset-wide applied/indexed markers.

When metadata is already in the cleaned dataset, omit the manifest. Otherwise, each entry supplies document (a valid clean document object) and sourceUrl (the corresponding HTTPS dyn/opendata/<UID>.html or dyn/docs/<UID>.raw URL). An optional note records provenance. Fields are validated; the legislature, title and documentary date must be verified. No date defaults to today, and no missing author/classification/adoption information is fabricated. This first version accepts explicit metadata, not automatic extraction of a complete fiche from arbitrary HTML or PDF. Root document UIDs are required for PostgreSQL materialization; retrieval-only runs also accept division UIDs and a complete parent metadata tree.

The scripts stage only the selected metadata in a disposable input dataset, with links to the canonical content repositories. They do not modify the official cleaned dataset or fabricate dossier acts. Acquisition metadata is archived as metadata-anticipation.json in the document's usual UID directory; an explicit targeted retry may reuse that sidecar. Normal imports and catch-ups ignore it. HTML must be published and parseable for bills; an upstream error, unavailable text or parsing failure cannot produce a successful canonical publication.

The targeted import inserts into the normal assemblee.documents table, marking anticipated JSON with _tricoteusesAnticipation. It can refresh an existing anticipation but never overwrite official metadata with an anticipation. The next official import replaces the entire JSON under the same UID, removing that marker. No parallel catalogue or special conservation table is maintained. Accepted risk: an upstream full import that is still late may delete the entry and its downstream projections; the archived metadata permits an explicit retry. If the official metadata changes the title/date used for Git processing, rerun the targeted phase to refresh those outputs. PostgreSQL's normal triggers queue content/search reconciliation on metadata changes.

For separate stages, retrieve-documents publishes Git outputs; materialize-documents imports only the requested metadata and synchronizes PostgreSQL content/search. sync-documents remains available when the rows are already present. Outside Forgejo, DOCUMENT_METADATA_FILE accepts a local manifest file instead of the DOCUMENT_METADATA JSON environment variable. Never set both; all mutations still require the normal fenced leases.

A dirty persistent worktree remains a fail-closed condition. The explicit recover-links phase handles the legacy worktree from run 227: it validates that every dirty path is an expected, complete link output, splits the recovered files by legislature onto the corresponding temporary branches, then restores the canonical worktree without publishing a partial update to master.

The upstream CDN currently fails on high byte ranges of the large Amendements archive. Download retries remain bounded to 30 minutes. Such a run is expected to fail without publishing or advancing downstream markers; the orchestration does not attempt to compensate for this external outage.

Retrieving a frozen legislature dataset

Datasets that no longer change upstream (for example the XIV legislature amendements, frozen since 2018) are not watched. Retrieve them once with a manual ingest.yml dispatch: categories=Amendements, dataset=Amendements_XIV, legislature=14, phase=ingest. That phase retrieves, reorganizes, cleans and publishes both Git repositories without touching the PostgreSQL or search projections.

Preconditions on every assemblee-prod worker: the two persistent worktrees (Amendements_XIV and Amendements_XIV_nettoye) must exist in the shared corpus with origin pointing at the canonical Forgejo repositories (assemblee-opendata/Amendements_XIV_opendata and assemblee/Amendements_XIV), owned by the corpus account, on branch master. ci/shards.json needs no change: the amendments shard already declares Amendements_* Git writes. Because the dataset predates the watchers, the run is manual; nothing schedules it again afterwards.

Portability

The execution layer is reusable, but this production deployment is not turnkey. The Bash/Node.js scripts, shard planner, progress model, lease protocol, and Git publication checks are independent of a particular worker host.

A deployment on other persistent Linux servers can retain that layer if it provides:

  • persistent Git worktrees for the generated datasets;
  • access to the three source repositories and canonical data remotes;
  • PostgreSQL and search endpoints;
  • a compatible fenced-lease service;
  • equivalent secrets, SSH identities, ownership, and pre-push enforcement.

The current Forgejo workflows encode production-specific runner labels, volume paths, secret paths, UID/GID mappings, image references, and Mole/Swan network endpoints. Moving to another Forgejo instance therefore requires replacing those values and provisioning equivalent runners. Ephemeral or SaaS-hosted runners are not a drop-in target: the corpus and stateful services would first need to be externalized without weakening the lease guarantees.

Repository layout

  • .forgejo/workflows/: Forgejo Actions entry points;
  • ci/shards.json: categories, repositories, dependencies, and shared locks;
  • scripts/forgejo-job: common workflow phase dispatcher;
  • scripts/run-ingest: download, transform, commit, and publish one shard;
  • scripts/run-materialize: PostgreSQL and persistent queue projections;
  • scripts/run-index: search projections;
  • scripts/ci-open-data-state: portable state preparation and publication;
  • scripts/ci-state-lock: interface to the fenced-lease broker;
  • scripts/git-hooks/pre-push: final lease check before publication.

Local validation

The local tests do not mutate production data:

find scripts -type f ! -name '*.mjs' -exec bash -n {} +
node --check scripts/shard-plan.mjs
node --check scripts/record-progress.mjs
./scripts/test-runtime-identity.sh
./scripts/test-shard-plan.sh
./scripts/test-ci-open-data-state.sh
./scripts/test-run-index.sh
./scripts/test-run-ingest.sh
./scripts/test-run-materialize.sh
./scripts/test-run-document-catchup.sh
./scripts/test-run-retrieve-documents.sh
./scripts/test-validate-document-link-checkpoint.sh
./scripts/test-recover-document-link-checkpoint.sh
./scripts/test-document-checkpoint-git.sh
./scripts/test-recover-document-checkpoint-tail.sh
./scripts/test-pre-push-hook.sh