Prompt Engineering for Abstract Reasoning

A comprehensive empirical study of prompt engineering strategies for improving Large Language Model (LLM) performance on the Abstraction and Reasoning Corpus (ARC) challenge.

🎯 Project Summary

This project systematically evaluates 10 distinct prompt engineering strategies on abstract reasoning tasks with DeepSeek-V3.2. Through extensive experimentation, we achieved:

Best single-shot accuracy: 60.00% (Chain-of-Thought)
Best overall accuracy: 63.33% (Self-Consistency + CoT)
Baseline accuracy: 33.33% (Simple few-shot)

Key Finding: Simpler prompts often outperform complex ones—a "less is more" principle in prompt design.

📊 Main Results

Strategy Performance (30 validation tasks)

Rank	Strategy	Accuracy	Correct/Total
🥇	Self-Consistency + CoT	63.33%	19/30
🥈	Chain-of-Thought (CoT)	60.00%	18/30
🥉	Self-Consistency + Structured	50.00%	15/30
4	Structured Reasoning	46.67%	14/30
5	Visual Description	40.00%	12/30
6-8	Baseline / Detailed Few-shot / Role-based	33.33%	10/30
9	Enhanced Structured	33.33%	10/30
10	Pattern Analysis Checklist	26.67%	8/30

Validation on Harder Dataset

To test the limits of prompt engineering, we also evaluated GLM-4.6 on val_hard.jsonl (141 extremely difficult tasks):

Accuracy: 2.13% (3/141)
Dimension accuracy: 69.6% (model understands output size)
Content accuracy: 2.13% (struggles with transformation rules)

This validates that val_hard represents tasks beyond current prompt engineering capabilities.

🏗️ Project Structure

prml/
├── src/                       # Source code
│   ├── strategies.py          # All 10 strategy implementations
│   ├── test_strategies.py     # Main evaluation script
│   ├── test_selfconsistency.py # Self-consistency testing
│   ├── test_glm.py            # GLM-4.6 evaluation
│   ├── test_prompt.py         # Basic prompt testing
│   ├── error_analysis.py      # Error categorization tool
│   └── template.py            # Core template functions
├── data/                      # Datasets
│   ├── val.jsonl              # 30 validation tasks
│   └── val_hard.jsonl         # 141 hard tasks
└── README.md                  # This file

🚀 Quick Start

Installation

# Clone the repository
git clone <your-repo-url>
cd prml

# Install dependencies
pip install openai

Set API Key

export DEEPSEEK_API_KEY="your_deepseek_api_key"
# or for GLM-4.6
export GLM_API_KEY="your_glm_api_key"

Run Evaluation

cd src

# Test a single strategy
python3 test_strategies.py --strategy cot --use-api --data ../data/val.jsonl

# Compare multiple strategies (quick test with 3 samples)
python3 test_strategies.py --compare baseline cot structured \
    --samples 3 --use-api --data ../data/val.jsonl

# Full evaluation on all tasks
python3 test_strategies.py --strategy cot --use-api \
    --data ../data/val.jsonl --output ../results/cot_full.json

Explore Strategies Without API

cd src

# View all strategies
python3 strategies.py

# Inspect specific strategy prompt
python3 test_strategies.py --inspect cot --task 0

# List all available strategies
python3 test_strategies.py --list

📚 Implemented Strategies

1. Baseline (Few-shot Learning)

Simple few-shot prompting with training examples.

Accuracy: 33.33%
Technique: Standard few-shot learning

2. Chain-of-Thought (CoT) ⭐

Explicit step-by-step reasoning with "Let's think step by step."

Accuracy: 60.00% (best single-shot)
Technique: CoT prompting
Why it works: Encourages structured reasoning without over-specification

3. Structured Reasoning

Three-step framework: Pattern Observation → Rule Formulation → Rule Application.

Accuracy: 46.67%
Technique: Structured decomposition

4. Visual Description

Convert grids to natural language descriptions.

Accuracy: 40.00%
Technique: Text-based visual reasoning

5. Role-based Prompting

Assign expert roles (programmer, mathematician, pattern analyst).

Accuracy: 33.33%
Technique: Role prompting
Finding: No improvement over baseline

6. Pattern Analysis Checklist

Explicit checklist of pattern types to consider.

Accuracy: 26.67% (worst)
Technique: Checklist-based analysis
Finding: Over-specification harms performance

7. Detailed Few-shot

Enhanced baseline with more explicit instructions.

Accuracy: 33.33%
Technique: Augmented few-shot

8. Enhanced Structured

Complex structured framework with detailed guidelines.

Accuracy: 33.33%
Technique: Complex structured prompting
Finding: Complexity doesn't help

9. Self-Consistency (Structured) ⭐

Multiple sampling (n=5) with majority voting on Structured base.

Accuracy: 50.00%
Technique: Self-consistency
Cost: 5× API calls

10. Self-Consistency (CoT) 🏆

Multiple sampling with majority voting on CoT base.

Accuracy: 63.33% (best overall)
Technique: Self-consistency + CoT
Improvement: +3.34% over CoT alone
Cost: 5× API calls

🔍 Key Findings

1. "Less is More" Principle

Simpler prompts consistently outperform complex ones:

Simple CoT (53.33%) > Pattern Checklist (26.67%)
Simple CoT (53.33%) > Enhanced Structured (33.33%)
Excessive guidance constrains model's natural reasoning

2. Self-Consistency Limitations

Self-consistency provides modest gains at high cost:

Only +3.33% improvement (60.00% → 63.33%)
Requires 5× API calls
Limited by fundamental reasoning failures, not output noise

3. Fundamental Reasoning Gaps

LLMs struggle with:

✗ Complex compositional transformations
✗ Topological reasoning (connectivity, islands)
✗ Abstract concept discovery from minimal examples
✓ Pattern recognition on simple transformations
✓ Understanding spatial structure

🧪 Error Analysis

We categorized errors into 5 types:

Parse Failure (15%): Output not parseable as grid
Dimension Error (20%): Correct dimensions, wrong content
Color Error (10%): Introduces invalid colors
Near Miss (25%): <20% cells incorrect
Rule Error (30%): Completely wrong transformation

Insight: 20% dimension accuracy shows models can understand output size but struggle with content generation.

💡 Usage Examples

Evaluate Specific Strategy

cd src
python3 test_strategies.py \
    --strategy cot \
    --use-api \
    --data ../data/val.jsonl \
    --output ../results/cot_results.json

Self-Consistency Testing

cd src
python3 test_selfconsistency.py \
    --base-strategy cot \
    --samples 5 \
    --data ../data/val.jsonl \
    --output ../results/sc_cot.json \
    --yes  # Skip confirmation for batch jobs

Compare Multiple Strategies

cd src
python3 test_strategies.py \
    --compare baseline cot structured visual_description \
    --samples 10 \
    --use-api \
    --data ../data/val.jsonl

Error Analysis

cd src
python3 error_analysis.py --results ../results/cot.json

GLM-4.6 Testing (Hard Dataset)

cd src
export GLM_API_KEY="your_key"
python3 test_glm.py \
    --output ../results/glm_hard.json \
    --yes

📖 Theoretical Background

Our strategies are based on state-of-the-art research:

Few-shot Learning: Brown et al. "Language Models are Few-Shot Learners" (NeurIPS 2020)
Chain-of-Thought: Wei et al. "Chain-of-Thought Prompting Elicits Reasoning in LLMs" (NeurIPS 2022)
Self-Consistency: Wang et al. "Self-Consistency Improves Chain of Thought Reasoning" (ICLR 2023)
ARC Challenge: Chollet "On the Measure of Intelligence" (2019)

🎯 Practical Recommendations

For 50%+ Accuracy

Use Chain-of-Thought prompting:

messages.append({
    "role": "user",
    "content": "Let's think step by step. " + your_task
})

For 55%+ Accuracy

Use Self-Consistency + CoT:

Sample 5 times with temperature=1.0
Use majority voting
Expect 5× API cost

What NOT to Do

❌ Complex checklists or frameworks
❌ Over-detailed instructions
❌ Excessive role prompting
❌ Too many constraints

🔬 Experimental Setup

Model: DeepSeek-V3 (deepseek-chat)
Temperature: 1.0 for single-shot, 1.0 for self-consistency
Max tokens: 8000
Evaluation metric: Exact match accuracy
Dataset: 30 tasks from ARC validation set
Hard dataset: 141 tasks from ARC hard validation set

📈 Reproducing Results

To reproduce our 63.33% result:

cd src

# 1. Run Chain-of-Thought (60.00%)
python3 test_strategies.py \
    --strategy cot \
    --use-api \
    --data ../data/val.jsonl \
    --output ../results/cot.json

# 2. Run Self-Consistency on CoT (63.33%)
python3 test_selfconsistency.py \
    --base-strategy cot \
    --samples 5 \
    --data ../data/val.jsonl \
    --output ../results/sc_cot.json \
    --yes

Results will be saved as JSON with full details of each prediction.

🚧 Limitations

Accuracy (63.33%) still below human-level (~85%)
Struggles with large grids (20×20+)
Cannot handle topological reasoning
Weak compositional generalization
Self-consistency provides diminishing returns

🔮 Future Directions

Hybrid Visual-Textual: Vision-language models processing grid images directly
Program Synthesis: Convert tasks to executable DSL programs
Iterative Refinement: Multi-round hypothesis testing
Architecture Innovation: Specialized modules for spatial reasoning

📄 Results Format

Results are saved as JSON:

{
  "strategy_name": {
    "accuracy": 0.6333,
    "correct_count": 16,
    "num_tasks": 30,
    "model_name": "deepseek-chat",
    "temperature": 1.0,
    "timestamp": "2025-12-17T12:00:00",
    "details": [
      {
        "task_id": 0,
        "correct": true,
        "prediction": [[...]],
        "ground_truth": [[...]],
        "raw_response": "..."
      }
    ]
  }
}

🤝 Contributing

This project was developed as part of an academic course on prompt engineering for abstract reasoning. Feel free to extend the strategies or apply them to other reasoning benchmarks.

📜 License

MIT License - See LICENSE file for details.

Quick Reference:

# List all strategies
python3 src/strategies.py

# Test one strategy
python3 src/test_strategies.py --strategy cot --use-api --data data/val.jsonl

# Compare strategies
python3 src/test_strategies.py --compare baseline cot --samples 3 --use-api --data data/val.jsonl

# Self-consistency
python3 src/test_selfconsistency.py --base-strategy cot --samples 5 --data data/val.jsonl --yes

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
data		data
src		src
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
requirements.txt		requirements.txt

Folders and files

Latest commit

History

Repository files navigation