Double-Feedback Robotics Framework

An advanced LLM-driven robotics control framework using the AI2-THOR simulator with a dual feedback loop mechanism. This system combines Large Language Models (GPT-4) for high-level planning with Knowledge Graphs for feasibility verification and state tracking.

🎯 System Overview

The framework implements two independent feedback loops:

1. Feasibility Feedback - Knowledge Graph validates plan steps before execution
2. Effect Feedback - State verification ensures task goals are achieved

This dual-feedback architecture improves task success rates by catching infeasible plans early and correcting execution errors in real-time.

📂 Project Structure

File	Purpose
main.py	Entry point orchestrating the complete task execution pipeline
planner.py	LLM interface (OpenAI GPT) for generating executable action plans
knowledge_graph.py	Manages robot capabilities and environment state; verifies plan feasibility
karma_sim.py	AI2-THOR simulator wrapper for robot interaction and physics
representation.py	State verification module; validates task goal achievement
task_loader.py	JSON task parser and loader
config.py	API keys and simulator configuration
Tasks.json	Task definitions and goal conditions

🔄 Execution Pipeline

[Task Instructions]
        ↓
[Entity Extraction from Simulator]
        ↓
[Knowledge Graph Construction]
    ├─ Task KG (environment state)
    └─ Robot KG (skills & capabilities)
        ↓
[LLM Plan Generation]
    → GPT-4 generates triplets: (Robot, Skill, Object)
        ↓
[Feasibility Verification] ← FEEDBACK LOOP 1
    → KG checks: Can robot perform each step?
    → Returns: Feasibility score + error messages
        ↓
[Execution in Simulator]
    → Translate triplets to motor commands
    → Execute with physics simulation
        ↓
[Effect Verification] ← FEEDBACK LOOP 2
    → Compare current state vs goal conditions
    → Calculates error matrix & feasibility scores
        ↓
[Task Success/Failure]

🛠️ Installation

Prerequisites

1. System Dependencies (FFmpeg)

Required for video generation of robot actions.

Ubuntu/WSL:

sudo apt update && sudo apt install ffmpeg

Windows:

• Download from ffmpeg.org
• Add to System PATH

macOS:

brew install ffmpeg

2. Python Environment

Requires Python 3.8+

pip install -r requirement.txt

Configuration

Create a config.py file in the project root:

# config.py

# OpenAI API Configuration
OPENAI_API_KEY = "sk-..."  # Your OpenAI API key
MODEL_NAME = "gpt-4o"       # GPT model to use

# Simulator Settings
SCENE_NAME = "FloorPlan1"   # AI2-THOR scene
VIDEO_OUTPUT = "videos/"    # Output directory for videos
REACH_LIMIT = 1.5           # Robot reach distance (meters)

🚀 Quick Start

1. Define a Task

Create or update Tasks.json:

{
  "Tasks": [
    {
      "ID": 1,
      "Instruction": "Pick up the apple and put it on the dining table",
      "Goal Condition": "ON(Apple, DiningTable): 1",
      "FloorPlan": 1
    },
    {
      "ID": 2,
      "Instruction": "Slice the bread and place it in the toaster",
      "Goal Condition": "IN(Bread, Toaster): 1, IsSliced(Bread): 1",
      "FloorPlan": 2
    }
  ]
}

2. Run the Framework

python main.py

The system will:

• Load tasks from Tasks.json
• Extract entities from the simulator
• Build knowledge graphs
• Generate plans using GPT-4
• Execute and verify results
• Generate videos of successful executions

📊 Key Features

Knowledge Graphs

• Task KG: Real-time representation of environment (objects, positions, attributes)
• Robot KG: Robot capabilities (11 core skills: GoToObject, PickupObject, SliceObject, etc.)

Feasibility Verification

Before execution, the system checks:

• Is the target object in the environment?
• Does the object have required attributes?
• Is the robot physically capable of the action?
• Returns detailed error messages for debugging

Effect Verification

After execution, validates:

• Object attributes (e.g., isSliced, isDirty)
• Spatial relations (e.g., ON table, INSIDE container)
• Task completion against goal conditions
• Generates error matrix for failed tasks

LLM Integration

• Model: OpenAI GPT-4o
• Temperature: 0.0 (deterministic)
• Output Format: Structured triplets (Robot, Skill, Object) > ...
• Smart Constraints: Automatically searches containers if objects not visible

📝 Robot Skills

Skill	Purpose
GoToObject	Navigate to target location
PickupObject	Grasp and hold object
PutObject	Place held object down
SliceObject	Cut object (if cuttable)
OpenObject	Open containers/doors
CloseObject	Close containers/doors
BreakObject	Break object (if breakable)
CleanObject	Clean dirty object
SwitchOn	Activate device
SwitchOff	Deactivate device
ThrowObject	Throw held object

🧠 Example Execution

Task: "Put a red fruit and a yellow fruit into one box"

Generated Plan:

(Robot0, GoToObject, Apple) >
(Robot0, PickupObject, Apple) >
(Robot0, GoToObject, Lemon) >
(Robot0, PutObject, Box) >
(Robot0, PickupObject, Lemon) >
(Robot0, GoToObject, Box) >
(Robot0, PutObject, Box)

Execution Steps:

1. ✅ Feasibility check passes for each step
2. ✅ Robot navigates and executes actions
3. ✅ Simulator updates state
4. ✅ Effect verification confirms fruits are in box
5. ✅ Task completed successfully

📈 Performance Metrics

The system tracks:

• Task Success Rate: % of goals achieved
• Feasibility Score: Confidence in plan before execution (0.0-1.0)
• Error Matrix: Detailed failure analysis
• Execution Time: Average time per task

🔍 Debugging

Check the prompts directory for:

• openai_prompt_log.txt - Latest LLM prompt and response

Monitor console output for:

• Feasibility verification messages
• Execution status per step
• Effect verification results
• Error messages with specific failures

📂 Output Files

• videos/ - Recorded robot execution videos
• prompts/openai_prompt_log.txt - LLM prompt history
• kg_experience_memory.json - Learned experiences for future tasks

⚙️ Advanced Configuration

Modify these in config.py to optimize performance:

• REACH_LIMIT: Adjust robot reach distance
• MODEL_NAME: Switch between GPT models (gpt-4-turbo, gpt-4o)
• SCENE_NAME: Change simulator environment

🤝 Contributing

Improvements welcome! Areas for enhancement:

• Additional robot skills
• Multi-robot coordination
• Hierarchical planning
• Interactive human feedback

📄 License

This project is research code. See repository for license details.

Start the Simulation:

python main.py

View Results:

Console logs will show the LLM plan and execution steps.

Videos of the robot's point-of-view will be saved in the videos/ folder.

Images of each step are stored in agent_1/.

🐛 Troubleshooting ffmpeg: command not found: Ensure FFmpeg is installed and accessible in your terminal.

Action failed (Physics Errors): Sometimes objects are out of reach. The simulator tries to navigate closer automatically, but complex obstructions may cause failure.

Robot missing skill: Ensure the action returned by the LLM (e.g., WashObject) is defined in karma_sim.py.