AutoCrew

Automatic Generation of Agentic AI Crews for Financial Modeling Tasks

Final Year Project (BSCS) — FAST National University, Karachi
Based on: "Agentic AI Systems for Financial Services" by Okpala et al. (arXiv:2502.05439, 2025)

---

What is AutoCrew?

AutoCrew extends the base paper by introducing a Meta Agent that automatically generates multi-agent crew configurations from natural language. Instead of hardcoding agents, tasks, and tools in Python, you describe the task in English and the system builds everything dynamically.

The project provides two paths for comparison:

• Path A (Manual): Hardcoded agents, tasks, and tools — replicates the base paper
• Path B (AutoCrew): Meta Agent generates a JSON config, Crew Factory builds live agents from it

Both paths share the same tools, Human-in-the-Loop (HITL) gate, and CrewAI execution engine. A built-in comparison framework evaluates both side-by-side on accuracy, F1 score, timing, and MRM verdict.

---

Project Structure

AutoCrew/
├── src/
│   ├── main.py                  # Entry point (manual/auto/compare)
│   ├── config.py                # LLM provider setup (Groq/Gemini/Ollama/Mock)
│   ├── state.py                 # Shared state between crews
│   │
│   ├── meta_agent/              # Path B: Auto crew generation
│   │   ├── orchestrator.py      # Chains the 5-step pipeline
│   │   ├── task_decomposer.py   # NL task → subtask list
│   │   ├── agent_selector.py    # Subtask → agent role + persona
│   │   ├── tool_assigner.py     # Agent → tool names from registry
│   │   ├── instruction_writer.py# Generate detailed task instructions
│   │   └── json_generator.py    # Assemble crew_config.json
│   │
│   ├── crew_factory/            # JSON → live CrewAI objects
│   │   ├── crew_builder.py      # Parses JSON, builds Agent/Task/Crew
│   │   └── auto_runner.py       # Runs the auto pipeline end-to-end
│   │
│   ├── manual/                  # Path A: Base paper replication
│   │   ├── modeling_crew.py     # Hardcoded modeling agents
│   │   ├── mrm_crew.py          # Hardcoded MRM agents
│   │   └── manual_runner.py     # Runs the manual pipeline
│   │
│   ├── tools/                   # Shared tools (both paths use these)
│   │   ├── code_execution.py    # Executes Python scripts
│   │   ├── eda_tool.py          # Automated EDA on CSV files
│   │   ├── cag_tool.py          # Compliance checking tool
│   │   └── tool_registry.py     # String → tool instance mapping
│   │
│   ├── evaluation/              # Comparison & reporting
│   │   ├── comparison.py        # Run both paths, compare metrics
│   │   ├── metrics_collector.py # Extract accuracy/F1/timing from output
│   │   ├── generate_charts.py   # Generate all figures (PNG)
│   │   ├── report_generator.py  # Generate PDF report (ReportLab)
│   │   └── terminal_logger.py  # Generate Terminal output 
│   │
│   └── hitl/                    # Human-in-the-loop
│       └── human_gate.py        # Y/N approval gate between phases
│
├── data/                        # Place your datasets here
│   └── .gitkeep
│
├── configs/                     # Auto-generated crew configs
│   └── generated/
│       └── .gitkeep
│
├── prompts/                     # LLM prompt templates
│   ├── decompose_task.txt
│   ├── select_agents.txt
│   ├── assign_tools.txt
│   └── write_instructions.txt
│
├── registries/                  # Agent and tool catalogs
│   ├── agent_templates.json
│   └── tool_catalog.json
│
├── tests/
│   └── .gitkeep
│
├── .env                         # API keys (create this yourself)
├── .gitignore
├── .gitattributes
├── requirements.txt
└── README.md

---

Setup

1. Clone the repository

git clone https://github.com/ssarimm/AutoCrew-Meta-Agent-Research

2. Create virtual environment

python -m venv venv

# Windows
venv\Scripts\activate

# macOS/Linux
source venv/bin/activate

3. Install dependencies

pip install -r requirements.txt

4. Add your datasets

Place your CSV datasets in the data/ folder:

#	Dataset	Source	Task
1	credit_card_approval.csv	UCI ML Repository	Binary classification (approve/reject)
2	creditcard_2023.csv	Kaggle	Fraud detection (imbalanced)
3	cs-training.csv	Kaggle - Give Me Some Credit	Credit scoring (financial distress)

5. Set up API keys

Create a .env file in the project root:

# Pick one (or more) LLM provider
#Add Api Keys in `.env` file at root directory 
OPENROUTER_API_KEY=                     

---

Usage

Run the system

python -m src.main

You'll be prompted to select:

1. LLM Provider — Groq / Gemini / Ollama / Mock / Open Router / Others
2. Dataset — Credit Approval / Fraud Detection / Credit Scoring
3. Mode:
   • 1 Manual Pipeline (Path A - base paper)
   • 2 AutoCrew Pipeline (Path B - meta agent)
   • 3 Comparison Mode (runs both, generates comparison report)

Outputs

After a run, you'll find:

Location	Contents
results/AutoCrew_Report.pdf	Full PDF report with charts
results/auto_results_*.json	Raw metrics from auto path
results/comparison_*.md	Markdown comparison report
results/comparison_*.json	Raw comparison data
figures/*.png	Generated charts (5 figures)
configs/generated/*.json	Auto-generated crew configs

---

How It Works

Meta Agent Pipeline (Path B)

The Meta Agent converts a natural language task into a complete crew configuration in 5 steps:

"Predict credit approval" 
    → Task Decomposer (subtasks)
    → Agent Selector (roles + personas)
    → Tool Assigner (tools from registry)
    → Instruction Writer (detailed prompts)
    → JSON Generator (crew_config.json)

Each step is one LLM call with structured JSON output. Total time: ~6 seconds on Groq.

Execution Flow

User Input → Meta Agent → crew_config.json → Crew Factory → Modeling Crew → HITL Gate → MRM Crew → Verdict

---

Tips for Best Results

• Use credit_card_approval.csv for demos — it's small (689 rows) and won't exhaust free this
• If rate-limited, wait 24 hours or switch to Gemini (higher free tier)
• Run modes 1 and 2 separately if you want to avoid rate limits during comparison mode

---

Key Contributions

#	Contribution	Description
1	Meta Agent	LLM-powered 5-step orchestrator for automatic crew creation
2	JSON Config	Structured crew_config.json generated from natural language
3	Crew Factory	JSON parser that builds live CrewAI Agent/Task/Crew objects
4	Dual-Path Architecture	Manual vs Auto for fair, controlled comparison
5	Tool Registry	Extensible string-to-tool mapping with validation
6	Evaluation Framework	Side-by-side metrics (accuracy, F1, timing, verdict)

---

Tech Stack

• Framework: CrewAI + LangChain
• LLM Routing: LiteLLM
• ML: scikit-learn, pandas, numpy
• Reporting: ReportLab (PDF), matplotlib (charts)
• Language: Python 3.10+

---

License

This project is part of an academic Final Year Project (FYP) at FAST National University, Karachi.