ImaginationWithLLM

newly added imagine() in src/imagine_env.py can be the input context of LLM.

Try with LLM

1. agent generate: input affordance, imagine()code, object size/picture/.obj file, output agent.urdf and agent number.
2. action generate: input affordance, imagine code, agent and object information, output object and agents path (each path is a list of keypoint poses) in each environment. One environment means one trial.
3. constraints generate: input affordance, imagine code, and imagination result (object and agents configuration) output a constraint function to classify whether each trial is successful from the resultant configuration of imagination.
4. evaluation function generate: output a function to evaluate how well this object could afforde such functionality.
5. if the object is not good enough, input the results to step 2, ask LLM to generate a better group of action, and repeat step 2-5 until certain iterations.

When checking all stable poses of the object, you might try to find the bbox that can contains the object in all stable poses, and ask LLM to generate plan only once. But when eval, you need to check on each stable pose, and then analyze all results.

Executable programs:

1. Generation mode: Generate simulation profile for affordance verification.

source ~/Director2Workspace/devel/setup.bash
python main_imagine.py

2. Testing mode: Verify affordance of objects using saved configuration.

source ~/Director2Workspace/devel/setup.bash
python test_imagine.py