- TypeScript 99.2%
- Shell 0.6%
- PLpgSQL 0.2%
|
All checks were successful
CI Staging / run-integration-tests (push) Successful in 3m22s
CI Staging / release-image-worker (push) Successful in 35s
CI Staging / deploy-open-data (push) Successful in 38s
CI Staging / release-image (push) Successful in 2m17s
CI Staging / deploy-api (push) Successful in 11s
|
||
|---|---|---|
| .adminjs | ||
| .forgejo/workflows | ||
| images/organes | ||
| kubernetes | ||
| prisma | ||
| sandbox | ||
| src | ||
| tests | ||
| .dockerignore | ||
| .gitignore | ||
| .prettierrc | ||
| Dockerfile | ||
| Dockerfile.worker | ||
| example.env | ||
| LICENSE.md | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
| tsconfig.json | ||
Tricoteuses-API-Parlement
REST API based on open data from the French Parliament (Assemblée & Sénat)
Requirements
- Node >= 20
Installation
git clone https://git.tricoteuses.fr/logiciels/tricoteuses-api-parlement
cd tricoteuses-api-parlement
Create a .env file to store environment variables (you can use example.env as a template).
In particular, edit the database URL as follows:
TRICOTEUSES_ASSEMBLEE_API_DB_URL="postgresql://USER:PASSWORD@HOST:PORT/DATABASE?schema=public"
Then:
npm install
Database configuration
Install PostgreSQL & pgvector (for vector similarity search functionalities), then in database, execute:
CREATE EXTENSION IF NOT EXISTS vector;
Alternatively, you can use this docker image that has everything set up:
docker run --name parlement-db -d -p 5432:5432 -e POSTGRES_PASSWORD=$YOUR_CUSTOM_DB_PASSWORD ankane/pgvector
# Default Postgres user is postgres
Then map the application data model to the database schema:
npm run db:migrate
Loading data
To use the API, you first need to load your database with the data from the Assemblée nationale and the Sénat.
Retrieval & cleaning of open data from Assemblée nationale and Sénat
See README files of Tricoteuses Assemblée and Tricoteuses Sénat and follow instructions to download data.
Import of open data files to database
npm run db:load -- -A ../assemblee-data -S ../senat-data
Available options
-A <path>: Path to the Assemblée data directory-S <path>: Path to the Sénat data directory-l <number>: Specify one or more legislatures to import (e.g.,-l 16 -l 17)--fromSession <number>: Specify one session to start importing from (e.g.,--fromSession 2022)-k <category>: Filter import by one or more categories (e.g.,-k ActeursEtOrganes -k Questions)--skipStats: Skip computing statistics
Available categories are:
- ActeursEtOrganes
- Agendas
- Amendements
- DossiersLegislatifs
- Photos
- Scrutins
- Questions
- ComptesRendus
Example: Import from multiple sources with filters
npm run db:load -- -A ../assemblee-data -S ../senat-data -l 17 --fromSession 2022
Example: Import Acteurs, Organes and Questions from multiple legislatures from Assemblée data
npm run db:load -- -A ../assemblee-data -l 16 -l 17 -k ActeursEtOrganes -k Questions
It's possible to view the database with the following command:
npx prisma studio
Sandboxed import
There is a small application you can use to extract, transform and load the data into a dockerized database.
See sandbox/README.md.
Usage
Launch web server
npm run start
Generate documentation
The documentation is generated in the OpenAPI format. Routes are documented with annotations in controller files.
To generate or update the documentation, simply generate the Prisma client:
npx prisma generate
Using the Prisma client in your project
To use the Prisma client in your project, you can install the package from source:
"devDependencies": {
"@tricoteuses/api-parlement": "git+https://git.tricoteuses.fr/logiciels/tricoteuses-api-parlement.git#master"
}
Running npm install will generate the Prisma client in your project. To then use the client, simply import it:
import { PrismaClient } from '@prisma/client'
export const prisma = new PrismaClient()
await prisma.acteur.findMany()
Updating schema
To update the schema, edit the files in prisma/models/.
Run integration testing
Test the data import pipeline against a dockerized Postgres database:
npm run test
Generate tests files :
You need docker and docker compose for this to work
1/ Set up a test-db if you don't already have one
docker run --name parlement-test-db -d -p 5436:5432 -e POSTGRES_PASSWORD=$YOUR_CUSTOM_DB_PASSWORD ankane/pgvector
Field transformation rules are defined in tests/expected.config.ts
You can declare how each field should appear in the generated expected output.
2/ Then use the following script:
# Sénat only
npm run gen:expected -- --senat-only
# Assemblée nationale only
npm run gen:expected -- --assemblee-only
# Assemblée and sénat for 2 specific models (case sensitive)
npm run gen:expected -- --assemblee-and-senat --models StatistiqueHebdomadaire,Stats
Each run creates/updates its own dedicated output directory
tests/expected/senat-only/
tests/expected/assemblee-only/