Action Search Eval

Build a system that matches natural language queries to the correct connector action.

The Task

You are given:

• A catalog of 31 actions across various connectors (Slack, BambooHR, Salesforce, etc.) in data/actions.json
• Daily evaluation batches of (query, action_id) pairs in data/days/day_*.jsonl
• A server (server.py) that serves daily evaluation batches

Your job: build a model pipeline that takes a natural language query (e.g. "send a message to the team channel") and returns the correct action_id (e.g. slack_send_message).

We are going to run your pipeline on a simulated month but we are only giving you access to the first ten days for training. We will use your version of the evaluate.py script, feel free to make any modification to main for your work but the days.jsonl format will be the same and we want this to be evaluated iteratively to simulate a real-world environment.

Setup

uv sync

Workflow

1. Start the server:

   uv run server.py

2. Explore the data:

   • GET /actions - the full action catalog
   • GET /day/1 - evaluation queries for day 1 (no labels)

3. Build your model. A bare-bones baseline is in baseline.py for reference. You should improve on it.

4. Submit predictions:

   uv run python evaluate.py                       

   Or POST directly:

   curl -X POST http://localhost:5117/day/1/submit \
     -H "Content-Type: application/json" \
     -d '{"predictions": [{"id": 0, "action_id": "slack_send_message"}, ...]}'

   The server returns accuracy and per-category breakdown.

5. Evaluate across all 10 days. Each day is a separate batch. Your aggregate performance across all days is what matters.

Data Format

Action:

{"id": "slack_send_message", "connector": "slack", "label": "Send Message", "description": "Send a message to a Slack channel or user", "category": "messaging"}

Day sample (data/days/day_*.jsonl):

{"query": "post a message on slack", "action_id": "slack_send_message"}

What We Evaluate

• Accuracy across all ten days
• Quality and clarity of your approach
• How you handle the pipeline end to end.

Guidelines

• The baseline uses all-MiniLM-L6-v2 but you're free to use anything open-source.
• You expect this task to not take more than 2 hours. Commit regularly to your fork/repository so we can see the commit history.
• Feel free to use LLMs but you are responsible for everything in the submission.
• Write a markdown document recording experiments / thought process. Detail improvements that you didn't have time to implement and what you would add to a production version of this system.

Have fun.