feat: Experimenting with local models

.gitignore

@@ -163,4 +163,7 @@ cython_debug/
 .devenv/
 .direnv/
 .vscode/
-responses.db
+responses.db
+
+# Encoder experiment outputs
+scripts/encoder_experiment/results/

pyproject.toml

@@ -10,7 +10,7 @@ maintainers = [
 ]
 readme = "README.md"
 license = {file = "LICENSE.md"}
-requires-python = ">=3.10,<3.14"
+requires-python = ">=3.12,<3.14"
 dependencies = [
     "google-cloud-core>=2.4.3",
     "pydantic>=2.11.7",
@@ -20,6 +20,7 @@ dependencies = [
     "scipy-stubs>=1.15.3.0",
     "types-requests>=2.32.4.20250809",
     "tenacity>=9.1.2",
+    "python-dotenv>=1.2.2",
 ]
 
 [tool.uv.sources]
@@ -41,6 +42,16 @@ dev = [
     "pytest>=7.4.0",
     "pytest-asyncio>=0.21.1",
 ]
+ml-labeller = [
+    "transformers>=4.40",
+    "datasets>=4.8.4",
+    "torch>=2.10.0",
+    "accelerate>=1.13.0",
+    "scikit-learn>=1.8.0",
+    "sentencepiece>=0.2.1",
+    "tiktoken>=0.12.0",
+]
+
 
 
 [tool.pytest.ini_options]

src/local_models/README.md

@@ -0,0 +1,28 @@
+# Using local models for Pastel
+
+Originally, the Pastel library relied on Gemini to answer a set of questions around each piece of text. While this produced good results, the rising price of Gemini and comparable tools encourages the use of small, local, fine-tuned encoder models.
+
+When a new question is added, we need to create a training set and then use it to fine tune a new model. Each model will only answer one question. We use Gemini to create the training set as a one-off task.
+
+## Create a training set
+
+The `label_sentences.py` module `main()` function processes a single question. It loads a set of sentences that already have labels for some questions. It then passes the sentences and the new question to Gemini and records the results. The end result is an extra set of labels in the `labelled_sentences.jsonl` file.
+
+## Training a model
+
+A new model can then be trained using `finetune_encode.py`. Pass the question to `build_one_question_answerer()` and it will extract the question and labelled sentences from the training file and fine tune a model.
+
+When all new questions have been set up, a new Pastel model can be trained by adding the questions to `demo_beam_search.py`. 
+
+## Inference
+
+After training, `local_answerer.py` uses a fine-tuned model to label new sentences.
+
+# Known issues!
+
+* The function `pastel.py / _get_answers_for_single_sentence()` can either pass the questions to Gemini (needed when creating a training set) or to the new fine-tuned model (for inference). Currently, this switch is done by commenting out bits of code! In the long run, in production, we'll want to only use Gemini during training. However, the library is open source, so it might want to keep both options with a flag set in the environment to indicate 'gemini or local'.
+
+* The trained local models will need to be stored in a bucket and downloaded as required.
+
+* Evaluation of individual local models and the combined Pastel model.
+