Récupération et nettoyage des données publiques (open data) du Sénat
Find a file
2025-07-10 16:38:46 +02:00
.forgejo/workflows Fix CI not running 2025-06-30 16:28:19 +02:00
src Update model/texte.ts 2025-07-08 10:26:11 +02:00
.dockerignore ci: add docker & gitlab ci config 2024-12-03 17:01:15 +01:00
.eslintrc.js Clean code base: harmonize with tricoteuses-assemblee 2025-05-30 17:15:22 +02:00
.gitignore Remove package-lock.json from .gitignore 2025-03-11 18:41:43 +01:00
.prettierrc Update prettier config 2025-06-06 17:13:12 +02:00
Dockerfile Update fromSession argument to include 2022 dossiers & documents 2025-07-07 14:44:03 +02:00
example.env First step to senat data conversion into files 2024-10-31 16:30:30 +01:00
LICENSE.md chore: update Docker installation in release workflow and change repository URLs in LICENSE and README files to reflect new hosting location 2025-06-30 10:30:46 +02:00
package-lock.json 2.5.9 2025-07-08 10:26:33 +02:00
package.json 2.5.9 2025-07-08 10:26:33 +02:00
README.md Update README.md 2025-07-10 16:38:46 +02:00
tsconfig.json Update build config 2025-06-02 10:18:48 +02:00

Tricoteuses-Senat

Retrieve, clean up & handle French Sénat's open data

Requirements

  • Node >= 22

Installation

git clone https://git.tricoteuses.fr/logiciels/tricoteuses-senat
cd tricoteuses-senat/

Create a .env file to set PostgreSQL database informations and other configuration variables (you can use example.env as a template). Then

npm install

Database creation (not needed if downloading with Docker image)

Using Docker

docker run --name local-postgres -d -p 5432:5432 -e POSTGRES_PASSWORD=$YOUR_CUSTOM_DB_PASSWORD postgres
# Default Postgres user is postgres
# But scripts require an "opendata" role
docker exec -it local-postgres psql -U postgres -c "CREATE ROLE opendata;"

Download data

Create a folder where the data will be downloaded and run the following command to download the data and convert it into JSON files.

mkdir ../senat-data/

# Available options for optional `categories` parameter : All,  Ameli, Debats, DosLeg, Questions, Sens
npm run data:download ../senat-data -- [--categories All]

Data from other sources is also available :

# Retrieval of textes and rapports from Sénat's website
# Available options for optional `formats` parameter : xml, html, pdf
# Available options for optional `types` parameter : textes, rapports
npm run data:retrieve_documents ../senat-data -- --fromSession 2022 [--formats xml pdf] [--types textes]

# Retrieval & parsing (textes in xml format only for now)
npm run data:retrieve_documents ../senat-data -- --fromSession 2022 --parseDocuments

# Parsing only
npm run data:parse_textes_lois ../senat-data

# Retrieval (& parsing) of agenda from Sénat's website
npm run data:retrieve_agenda ../senat-data -- --fromSession 2022 [--parseAgenda]

# Retrieval of sénateurs' pictures from Sénat's website
npm run data:retrieve_senateurs_photos ../senat-data

Data download using Docker

A Docker image that downloads and converts the data all at once is available. Build it locally or run it from the container registry. Use the environment variables FROM_SESSION and CATEGORIES if needed.

docker run --pull always --name tricoteuses-senat -v ../senat-data:/app/senat-data -d git.tricoteuses.fr/logiciels/tricoteuses-senat:latest

Use the environment variable CATEGORIES and FROM_SESSION if needed.

Using the data

Once the data is downloaded, you can use loaders to retrieve it. To use loaders in your project, you can install the @tricoteuses/senat package, and import the iterator functions that you need.

npm install @tricoteuses/senat
import { iterLoadSenatQuestions } from "@tricoteuses/senat/loaders"

// Pass data directory and legislature as arguments
for (const { item: question } of iterLoadSenatQuestions("../senat-data", 17)) {
  console.log(question.id)
}

Generation of raw types from SQL schema (for contributors only)

npm run data:generate_schemas ../senat-data

Publishing

To publish a new version of this package onto npm, bump the package version and publish.

npm version x.y.z # Bumps version in package.json and creates a new tag x.y.z
npm publish

The Docker image will be automatically built during a CI Workflow if you push the tag to the remote repository.

git push --tags