AgentX OpenProbe

🚀 Getting Started

✨ Features

• Automated Planning: Breaks down complex queries into multiple sub-queries for efficient searching.
• Adaptive Replanning: Revises search strategies when initial plans fall short (up to 2 replans).
• Reflection: Explains why previous plans failed and how they were improved.
• Web Search Integration: Seamlessly integrates with multiple search APIs for information retrieval.

🧭 How It Works

1. Analyzes the user’s question.
2. Generates a search plan with multiple sub-queries.
3. Executes searches based on the plan.
4. If results are insufficient, replans up to 2 times with improved queries.
5. Synthesizes information into a final, comprehensive answer.

---

⚙️ Setup and Usage

1️⃣ Configure API Keys

Create a .env file inside the openprobe directory and add your API keys:

GOOGLE_API_KEY=your_gemini_api_key
LAMBDA_API_KEY=your_lambda_api_key
WEB_SEARCH_API_KEY=your_serper_dev_api_key
JINA_API_KEY=your_jina_api_key
MISTRAL_API_KEY=your_mistral_api_key

2️⃣ Install Dependencies

Run the following commands to set up your environment:

cd openprobe
pip install -e .
crawl4ai-setup
crawl4ai-doctor

3️⃣ Run a Single Query

Test the system with a single question:

python test_deepsearch.py

4️⃣ Run Evaluation on FRAMES

Evaluate on the FRAMES dataset:

python evals/eval_tasks.py \
    --eval-tasks ./evals/datasets/frames_custom_set.csv \
    --parallel-workers 8

After completion, the evaluation results will be saved as a .jsonl file in the output directory.

5️⃣ Run Auto Grading

Grade the evaluation results using LLM auto-grading:

python evals/autograde_df.py \
    PATH_TO_RESULT_JSONL_FILE \
    --provider mistral \
    --num_cpus 2

The grading output will be appended to the input .jsonl file.

6️⃣ Compute Accuracy

Calculate accuracy on an experiment result:

python evals/accuracy.py \
    PATH_TO_GRADED_JSONL_FILE

---

📄 License

This project is licensed under the Apache License Version 2.0. You are free to use, modify, and distribute this code, subject to the terms of the license.

---

🧩 References and Acknowledgements

This project builds upon and integrates ideas and code from various open-source projects, including:

• LangChain
• LangGraph
• LlamaIndex — For data connectors and query engines.
• Serper API — For web search capabilities.
• Jina AI — For computing text embeddings.
• Mistral — For LLM-based grading and evaluation.
• LangGraph ReWOO implementation - Reference implemenation of ReWOO.
• OpenDeepSearch - For implementing the web search tool and FRAMES evaluation.

Many thanks to these projects and their communities for making this work possible!

---

👥 Contributors

We’d like to thank the following people for their contributions to this project:

• Kuo-Hsin Tu — Team Lead, Developer, Researcher
• Suryansh Singh Rawat — Developer
• Jean Yu — Developer
• Ankit Basu — Researcher