- Shell 91.5%
- JavaScript 8%
- Dockerfile 0.5%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| .forgejo/workflows | ||
| ci | ||
| docs | ||
| examples | ||
| scripts | ||
| .gitignore | ||
| Dockerfile | ||
| README.md | ||
Tricoteuses Assemblée update orchestration
This repository runs the production pipeline that turns Assemblée nationale open-data archives into versioned Git datasets, PostgreSQL projections, and search indexes.
The canonical repository is https://git.tricoteuses.fr/logiciels/tricoteuses-assemblee-update. The former GitLab project is archived and is no longer synchronized.
It contains orchestration and safety controls. The parsers and database/indexing implementations live in separate source repositories:
tricoteuses-assemblee: downloads, reorganizes, and cleans Assemblée data;tricoteuses-assemblee-canutes: projects cleaned data into Canutes;tricoteuses-juridique: builds the PostgreSQL-backed and Typesense search projections.
Forgejo Actions on git.tricoteuses.fr is the only CI control plane, and all
produced datasets are stored in canonical Forgejo repositories. The former
GitLab projects are archived redirects and no longer accept publications.
Production topology
The host names used by the workflows refer to physical infrastructure, not to software components:
| Name | Role in this pipeline |
|---|---|
| Rook | Linux worker server. Runs up to two production jobs and one watcher job. Its persistent corpus is exposed to jobs as /srv/tricoteuses-assemblee-data. |
| Macdo | Second Linux worker server and failover for Rook. Runs one production job and one watcher job, with its own synchronized corpus. |
| Duck | Third worker with its own canonical corpus. duck-assemblee serves assemblee-prod and retains assemblee-duck-canary; duck-assemblee-watch serves assemblee-watch. |
| Mole | Server hosting the Forgejo service and mesh-only HTTP/SSH proxies used by the runners. |
| Swan | Server hosting the fenced-lease broker that prevents concurrent writers on Rook and Macdo. |
| Jay | Server endpoint used for the Canutes PostgreSQL database and Typesense search service. |
Rook and Macdo are persistent workers: the roughly 185–220 GiB corpus is not copied
into each CI job. Both servers run repository-scoped, non-privileged Forgejo
runners without access to the Docker socket. Duck restored 74 canonical worktrees
and validated Scrutins materialization and indexing in run 948; return runs 949
and 950 succeeded on Rook. The isolated full run 957 and return run 958 also
succeeded, both with changed=false and no Scrutins ref modified. Duck has since
joined the shared production pools. Those no-op canaries do not prove data push
authorization; check the affected deploy-key associations before a publishing
validation.
All traffic between these servers uses their WireGuard mesh. Detailed host,
runner, proxy, secret, and service provisioning belongs to the separate
infrastructure repository eraviart/serveurs.
Update flow
A normal update follows these steps:
watch.ymlchecks the versions advertised by the Assemblée open-data endpoints.- For each changed dataset, it dispatches one
ingest.ymlrun. The run title includes the dataset name, for exampleUpdate Assemblée — Agenda_XVII. - The dataset category is mapped to a shard: a declared set of Git repositories and shared resources that must be updated together.
- A worker synchronizes the shard's persistent worktrees, downloads and cleans the source data, then commits and pushes the generated repositories.
- PostgreSQL materialization runs under the shared database lease when the category has a database projection.
- Search indexing runs under the lock associated with the target collection or queue.
- The
ci-open-data-statebranch records independently whether the result waspublished,applied, andindexed.
A failure after publication is therefore visible: it does not incorrectly mark the database or search projection as complete. The failed downstream phase can be resumed without downloading and publishing the dataset again.
Shards and concurrency
The executable resource map is ci/shards.json, interpreted
by scripts/shard-plan.mjs. It declares six shards and
rejects unknown categories, undeclared paths, missing dependencies, and
parallel shards with overlapping Git write sets.
Notable shared resources:
ActeursEtOrganesalso owns the deputy and senator photo repositories; photo refreshes run in the same fenced shard instead of a downstream GitLab CI, and deputy portrait bytes are materialized in Canutes with the actor data;AgendasandComptesRendusshare theagenda-reportsshard because their cleaners reconcile common outputs;- PostgreSQL writers use
assemblee-materialize-db; - writers to
textes_juridiquesuse the same search lock; - Amendements indexing drains its dedicated PostgreSQL-backed Tisseuse queue;
QuestionsandComptesRendushave no PostgreSQL projection and therefore do not acquire the database lease merely to advance theirappliedmarker.
all.yml publishes independent shards in parallel, then applies the shared
PostgreSQL and search phases in dependency order.
Workflows
| Workflow | Purpose |
|---|---|
watch.yml |
Hourly detection and dispatch of changed datasets. |
ingest.yml |
Update one dataset/shard. Manual runs may select the end-to-end full phase, the retrieval-only ingest phase, documents, retrieve-documents, the urgent end-to-end process-documents, materialize-documents, recover-links, materialize, sync-documents, or index, and may separately request a full PostgreSQL import. |
all.yml |
Run all shards, followed by ordered shared projections; full PostgreSQL imports are opt-in. |
runtime-image.yml |
Build the minimal production image referenced by immutable digest. |
Production source checkouts are detached at revisions resolved at run time from
the protected default branches (master for tricoteuses-assemblee and
tricoteuses-assemblee-canutes, main for tricoteuses-juridique); dispatch
inputs accept a full commit SHA as an explicit fallback. The runtime image and
third-party actions are also pinned by digest or commit.
Safety model
- Every mutating phase obtains a fenced lease from the broker on Swan.
- A
pre-pushhook checks the lease immediately before Git publication. - Losing the broker or lease terminates the complete job process group and prevents a late push.
- Persistent worktrees are synchronized only after acquiring the relevant
shard lease. Open-data-derived repositories discard their uncommitted
generated state at synchronization: an interrupted run leaves nothing that
the next retrieval cannot re-derive. The documentary checkpoint
repositories (
Documents,Documents_enrichis) hold acquired content that cannot always be re-fetched and still fail closed on any uncommitted change; unpublished local commits remain proof-gated everywhere. - Secrets and runtime
.envfiles are mounted read-only from root-managed host paths; they are never stored inci-open-data-state. - Every runtime job mounts
/etc/tricoteuses-assemblee-ci/passwdread-only at/etc/passwd. It must describe the numeric container UID/GID from the runner configuration, not necessarily the runner daemon's account. The dispatcher checks NSS resolution before source resolution, npm, secret copies or leases. Without this mount, OpenSSH fails locally (No user exists for uid) even when Git access rights and the SSH key are correct. Provision and allow the mount on every eligible host before deploying workflow changes. - Forgejo and the still-canonical GitLab data remotes are reached through mesh-only proxies on Mole.
- Shared PostgreSQL and Typesense writes are serialized even when independent Git publication phases run in parallel.
Do not run an ingestion script against production data without the corresponding fenced lease. Use disposable copies for shard-map or transformation tests.
Operational behavior
An exact DATASET is forwarded unchanged to retrieval, raw reorganization,
read-only raw validation and cleaning. Only retrieval receives --only-if-new.
Shard synchronization and fencing remain shard-wide, but transformations must
not touch sibling datasets whose inputs were not downloaded. This requires a
source revision supporting reorganize_data.ts --dataset and input-presence
preflight; do not pin the source to a revision predating that safety fix.
The watcher normally runs for 55 minutes. Setting repository variable
ASSEMBLEE_WATCHER_DISPATCH=false turns a scheduled or manual watcher run into a
single read-only observation that neither dispatches work nor publishes state.
Watcher pending entries are dispatch deduplication leases, not the fenced
resource leases used by ingestion. They clear when the accepted manifest covers
the dispatched artifact or when Forgejo identifies the exact dispatch as failed
or cancelled. Missing or unavailable status remains fail-closed until the
24-hour expiry. Correlation uses watch_dispatch_id; timestamp lower bounds
must use Forgejo's second precision, including the pagination cutoff. Comparing
its second-precision created with millisecond dispatchedAt can hide a
same-second failure and retain the lease for 24 hours. The regression fix is
Assemblée PR #37.
Watcher run 1877
loaded the corrected source and redispatched both Scrutins and Amendements on
2026-10-05 without manually removing controller state. After a correction is
merged to the source default branch, dispatch a fresh watch.yml run to load it;
a running watcher retains the source revision resolved at startup.
Manual ingest.yml and all.yml runs expose a full boolean, disabled by
default. Enabling it passes --full to the PostgreSQL import; otherwise imports
only apply changes committed since the last successful materialization. This is
independent of ingest.yml's phase=full, which means running the complete
publish, materialize, and index pipeline.
Dossiers materialization gives persistent document-content acquisition 75
minutes and PostgreSQL text indexing 30 minutes by default. Both workers consume
only queue entries belonging to the requested LEGISLATURE, so older failures
from another legislature cannot prevent the current dataset from progressing;
LEGISLATURE=0 retains global queue consumption for an explicit all-legislatures
run. Remaining queue items stay durable for later cycles. The limits can be overridden
with ASSEMBLEE_DOCUMENT_CONTENT_BUDGET_MINUTES and
ASSEMBLEE_TEXT_SEARCH_BUDGET_MINUTES for a planned catch-up run. A manual
DossiersLegislatifs run with phase sync-documents can instead set
document_uids to a comma-separated list: acquisition and PostgreSQL indexing
then process exactly those documents, independently of their position in the
durable queues. This dedicated phase only checks out tricoteuses-juridique,
does not touch the persistent document worktrees, and still acquires the
assemblee-materialize-db fenced lease.
Long missing-document catch-ups use the manual documents phase for one
legislature at a time. Retrieval, link, and enrichment outputs are checkpointed
every 250 changed documents by default on the append-only temporary branch
ci/document-catchup/L<legislature>; ASSEMBLEE_DOCUMENT_CHECKPOINT_SIZE can
adjust that bound. Once both repositories are complete, each checkpoint branch
is squash-committed to master, producing one canonical commit per repository
and legislature, then deleted. A stage marker outside the Git worktrees lets a
retry discard only the uncommitted generated tail from an interrupted stage
and recompute it from the last committed checkpoint. At startup, documentary
catch-ups, targeted runs and Dossiers ingestion check all L14–L17 stage
markers, not just the requested legislature: an interrupted L16 enrichment must
not block a new L17 request. Recovery runs under assemblee-ingest-dossiers,
preflights both worktrees before cleanup, and preserves every committed checkpoint
(including unpublished local commits). Only known stage-specific generated files
on the matching CI checkpoint branch are reset or removed; malformed partial
outputs are recomputed, without systematic backups. Unknown changes, missing or
invalid markers, Git conflicts and non-CI branches still fail closed. After every requested legislature succeeds,
run materialize and then index once with the intended legislature scope.
The retrieve-documents phase is the urgent, UID-targeted variant of that
catch-up: a manual DossiersLegislatifs run with document_uids retrieves
exactly those documents from the Assemblée site, adds links and enrichment, and
publishes them through the same ci/document-catchup/L<legislature> checkpoint
protocol, so tail recovery and the assemblee-ingest-dossiers lease are shared
with full catch-ups. Documents are read independently of their dossiers: missing
references in dossier acts do not prevent processing. A requested UID whose
metadata is missing now fails explicitly instead of silently publishing nothing.
Targeted runs refetch and reparse the source and force links/enrichment, so a
corrected document under the same UID is not hidden by a completion marker.
A document published before its open-data metadata
Use ingest.yml with:
categories=DossiersLegislatifs,legislature=17(or the actual legislature);phase=process-documents,document_uids=<exact institutional document UID>;- optional
document_metadata: a JSON manifest of verified metadata, followingexamples/plf-2026-document-metadata.json. This is a historical example, not the UID of the PLF 2027.
process-documents runs retrieval/parsing/links/enrichment and Git publication
under the dossiers lease, then targeted metadata import and content/search
synchronization under the PostgreSQL lease, then Typesense projection under its
shared lease. A failure stops the next phase. Typesense's existing
--assemblee-textes projection is global, not UID- or legislature-filtered.
Targeted operations do not advance the dataset-wide applied/indexed markers.
When metadata is already in the cleaned dataset, omit the manifest. Otherwise,
each entry supplies document (a valid clean document object) and sourceUrl
(the corresponding HTTPS dyn/opendata/<UID>.html or dyn/docs/<UID>.raw URL).
An optional note records provenance. Fields are validated; the legislature,
title and documentary date must be verified. No date defaults to today, and no
missing author/classification/adoption information is fabricated. This first
version accepts explicit metadata, not automatic extraction of a complete fiche
from arbitrary HTML or PDF. Root document UIDs are required for PostgreSQL
materialization; retrieval-only runs also accept division UIDs and a complete
parent metadata tree.
The scripts stage only the selected metadata in a disposable input dataset, with
links to the canonical content repositories. They do not modify the official
cleaned dataset or fabricate dossier acts. Acquisition metadata is archived as
metadata-anticipation.json in the document's usual UID directory; an explicit
targeted retry may reuse that sidecar. Normal imports and catch-ups ignore it.
HTML must be published and parseable for bills; an upstream error, unavailable
text or parsing failure cannot produce a successful canonical publication.
The targeted import inserts into the normal assemblee.documents table,
marking anticipated JSON with _tricoteusesAnticipation. It can refresh an
existing anticipation but never overwrite official metadata with an anticipation.
The next official import replaces the entire JSON under the same UID, removing
that marker. No parallel catalogue or special conservation table is maintained.
Accepted risk: an upstream full import that is still late may delete the
entry and its downstream projections; the archived metadata permits an explicit
retry. If the official metadata changes the title/date used for Git processing,
rerun the targeted phase to refresh those outputs. PostgreSQL's normal triggers
queue content/search reconciliation on metadata changes.
For separate stages, retrieve-documents publishes Git outputs;
materialize-documents imports only the requested metadata and synchronizes
PostgreSQL content/search. sync-documents remains available when the rows are
already present. Outside Forgejo, DOCUMENT_METADATA_FILE accepts a local
manifest file instead of the DOCUMENT_METADATA JSON environment variable.
Never set both; all mutations still require the normal fenced leases.
A dirty persistent worktree remains a fail-closed condition. The explicit
recover-links phase handles the legacy worktree from run 227: it validates
that every dirty path is an expected, complete link output, splits the recovered
files by legislature onto the corresponding temporary branches, then restores
the canonical worktree without publishing a partial update to master.
The upstream CDN currently fails on high byte ranges of the large Amendements archive. Download retries remain bounded to 30 minutes. Such a run is expected to fail without publishing or advancing downstream markers; the orchestration does not attempt to compensate for this external outage.
Retrieving a frozen legislature dataset
Datasets that no longer change upstream (for example the XIV legislature
amendements, frozen since 2018) are not watched. Retrieve them once with a
manual ingest.yml dispatch: categories=Amendements,
dataset=Amendements_XIV, legislature=14, phase=ingest. That phase
retrieves, reorganizes, cleans and publishes both Git repositories without
touching the PostgreSQL or search projections.
Preconditions on every assemblee-prod worker: the two persistent worktrees
(Amendements_XIV and Amendements_XIV_nettoye) must exist in the shared
corpus with origin pointing at the canonical Forgejo repositories
(assemblee-opendata/Amendements_XIV_opendata and
assemblee/Amendements_XIV), owned by the corpus account, on branch master.
ci/shards.json needs no change: the amendments shard already declares
Amendements_* Git writes. Because the dataset predates the watchers, the run
is manual; nothing schedules it again afterwards.
Portability
The execution layer is reusable, but this production deployment is not turnkey. The Bash/Node.js scripts, shard planner, progress model, lease protocol, and Git publication checks are independent of a particular worker host.
A deployment on other persistent Linux servers can retain that layer if it provides:
- persistent Git worktrees for the generated datasets;
- access to the three source repositories and canonical data remotes;
- PostgreSQL and search endpoints;
- a compatible fenced-lease service;
- equivalent secrets, SSH identities, ownership, and pre-push enforcement.
The current Forgejo workflows encode production-specific runner labels, volume paths, secret paths, UID/GID mappings, image references, and Mole/Swan network endpoints. Moving to another Forgejo instance therefore requires replacing those values and provisioning equivalent runners. Ephemeral or SaaS-hosted runners are not a drop-in target: the corpus and stateful services would first need to be externalized without weakening the lease guarantees.
Repository layout
.forgejo/workflows/: Forgejo Actions entry points;ci/shards.json: categories, repositories, dependencies, and shared locks;scripts/forgejo-job: common workflow phase dispatcher;scripts/run-ingest: download, transform, commit, and publish one shard;scripts/run-materialize: PostgreSQL and persistent queue projections;scripts/run-index: search projections;scripts/ci-open-data-state: portable state preparation and publication;scripts/ci-state-lock: interface to the fenced-lease broker;scripts/git-hooks/pre-push: final lease check before publication.
Local validation
The local tests do not mutate production data:
find scripts -type f ! -name '*.mjs' -exec bash -n {} +
node --check scripts/shard-plan.mjs
node --check scripts/record-progress.mjs
./scripts/test-runtime-identity.sh
./scripts/test-shard-plan.sh
./scripts/test-ci-open-data-state.sh
./scripts/test-run-index.sh
./scripts/test-run-ingest.sh
./scripts/test-run-materialize.sh
./scripts/test-run-document-catchup.sh
./scripts/test-run-retrieve-documents.sh
./scripts/test-validate-document-link-checkpoint.sh
./scripts/test-recover-document-link-checkpoint.sh
./scripts/test-document-checkpoint-git.sh
./scripts/test-recover-document-checkpoint-tail.sh
./scripts/test-pre-push-hook.sh