Trevas Jupyter

Jupyter notebook providing VTL support through Trevas engine

Usage

Run mvn package to bundle the VTL Kernel.

Without Docker

TODO

With Docker

The image includes the Trevas VTL kernel, jupyterlab-vtl-2-1 (editor: highlighting, lint, autocomplete), Elyra 4.1.1 (pipelines) and Arbalister 0.2.1 (Parquet/CSV viewer). JupyterLab is upgraded to 4.5.x during the image build because Arbalister requires >= 4.5.0 (the Onyxia base image ships 4.4.x).

Build locally (requires Node.js 22, Docker — Maven/Java build runs inside Docker):

cd Trevas-Jupyter
chmod +x docker/build-trevas-jupyter-image.sh
./docker/build-trevas-jupyter-image.sh
docker run --rm -p 8888:8888 trevas-jupyter:local

(build-trevas-jupyter-image.sh builds the VTL wheel on the host, then runs docker buildx. Do not run docker build alone — the wheel must exist in extensions/jupyterlab-vtl-2.1/dist/.)

On Apple Silicon, keep --platform linux/amd64 (the base image is amd64).

After docker run, open the JupyterLab URL printed in the logs. In a new notebook with kernel Trevas VTL, type invalid VTL to verify editor lint in cells.

Optional checks inside a running container:

docker exec -it <container_id> /opt/python/bin/jupyter kernelspec list
docker exec -it <container_id> /opt/python/bin/jupyter labextension list | grep -i vtl

Published images: makingsenseinfo/trevas-jupyter (built and pushed on git tags vX.Y.Z).

Parquet viewer troubleshooting: if opening a .parquet file shows "is not UTF-8 encoded", JupyterLab is using the text editor instead of Arbalister. After writeParquet, open a part-*.parquet file inside the output/ directory (Spark writes a folder, not a single file). On a running container (as root): pip install "jupyterlab>=4.5.0,<5" arbalister==0.2.1 then restart JupyterLab.

Custom functions

Custom functions have been introduced into the Trevas engine.

Name	Arguments	Returned type	Description
loadParquet	String url	Dataset	Load Parquet dataset
loadCSV (1)	String url	Dataset	Load CSV dataset
loadSas	String url	Dataset	Load Sas dataset
loadSDMXEmptySource	(String sdmxMesUrl, String structureId)	Dataset	Load SDMX empty source
loadSDMXSource (2)	(String sdmxMesUrl, String structureId, String dataUrl)	Dataset	Load SDMX source
writeParquet	(String url, Dataset ds)	String	Write given dataset in Parquet
writeCSV (3)	(String url, Dataset ds)	String	Write given dataset in CSV
show	Dataset ds	DisplayData	Display firt rows of a given dataset
showMetadata	Dataset ds	DisplayData	Display metadata of a given dataset
runSDMXPreview	String sdmxMesUrl	DisplayData	Run SDMX VTL transformations, with empty datasets, to obtain Persitent defined datasets
runSDMX (4)	(String sdmxMesUrl, String dataLocations )	DisplayData	Run SDMX VTL transformations, with sources, to obtain Persitent defined datasets
getTransformationsVTL	String sdmxMesUrl	String	Display VTL transformations defined in the SDMX Message file
getRulesetsVTL	String sdmxMesUrl	String	Display VTL rulesets defined in the SDMX Message file
size	Dataset ds	String	Display size of a given dataset

Data locations

loadCSV, writeCSV, loadParquet and writeParquet accept the following location formats:

Format	Example	Notes
Local path (relative)	./data/file.csv	Resolved from the current working directory of the Trevas kernel process
Local path (absolute)	/home/jovyan/work/file.csv	Typical path inside the Jupyter container
HTTPS	https://example.com/data/file.csv	Passed through to Spark
S3	s3://my-bucket/path/file.csv	Mapped internally to s3a:// (Hadoop S3A). Requires AWS/S3 credentials in the kernel environment (see below)

loadCSV, writeCSV, loadParquet, writeParquet and loadSas all resolve remote locations the same way. You can write s3://… in VTL; it is converted to s3a://… before calling Spark/Hadoop.

CSV options (delimiter, quote, header, …) can still be appended as URL query parameters on any of these formats (values must be URL-encoded), for example:

loadCSV("s3://my-bucket/data.csv?delimiter=%7C")

For s3:// and s3a:// URLs, credentials must be available to the Trevas JVM process (not only the Python kernel).

Onyxia: S3 credentials and endpoint are injected automatically into the container; the Trevas kernel maps them to fs.s3a.*.

Local standalone: no AWS_* variables are required. Local paths (./file.csv, /path/to/file.csv) work as before. To access S3/MinIO locally, export the same variables before starting Jupyter:

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=...          # if needed
export AWS_S3_ENDPOINT=http://localhost:9000   # MinIO / non-AWS only
export AWS_DEFAULT_REGION=us-east-1

When AWS_S3_ENDPOINT points to a MinIO-compatible store, path-style access is enabled automatically. Real AWS endpoints (*.amazonaws.com) keep the default virtual-host style.

(1) loadCSV

Default option values:

Name	Value
header	true
delimiter	;
quote	"

Any CSV option can be defined or overridden thanks to url parameters (values have to be encoded).

For instance, to read a CSV content where delimiter is | and quote is ':

loadCSV("s3://my-bucket/data.csv?delimiter=%7C&quote=%27")

Or from a local file in the container:

loadCSV("./data/file.csv")

(2) loadSDMXSource

Sources has to be .csv files for now.

(3) writeCSV

Default option values:

Name	Value
header	true
delimiter	;
quote	"

Any CSV option can be defined or overridden thanks to url parameters (values have to be encoded).

For instance, to write a CSV with a content delimited by | and quoted by ':

• writeCSV(...?delimiter=%7C&quote=%27)
• writeCSV("s3://my-bucket/out.csv?delimiter=%7C&quote=%27", ds)

(4) runSDMX

Sources has to be .csv files for now.

Second argument, dataLocations has to be a string separated field containing SDMX structure id and source location (ex: structId1,dataLocation1,structId2,dataLocation2)

Launch with demo project

INIT_PROJECT_URL docker environment variable enable to load a default project in your Trevas Jupyter instance.

Have a look to this project definition for instance.

Fill the INIT_PROJECT_URL environment variable with your script adress and run:

docker pull makingsenseinfo/trevas-jupyter:latest
docker run -p 8888:8888 -e INIT_PROJECT_URL="https://raw.githubusercontent.com/Making-Sense-Info/Trevas-Jupyter-Training/main/init-notebook.sh" makingsenseinfo/trevas-jupyter:latest