How to prepare the dataset of AitW

[durianer-D]

I am very interested in your work, so I tried to follow your experiments. When I was preparing the dataset, I found that the downloaded AitW dataset file was in TensorFlow format, but the json file you provided would read the .png image. I don't know how to relate the encoded image in TensorFlow to the .png image in the json file you provided.

for example:
{"conversations": [{"from": "human", "value": "Please identify the labels of each marked region in the image. "}, {"from": "gpt", "value": "Region 1: Screenshot Text ('31')\nRegion 2: Screenshot Text ('Google Calendar')\nRegion 3: Screenshot Text ('Make the most of every day.')\nRegion 4: Screenshot ICON (V_FORWARD)\nRegion 5: Screenshot ICON (NAV_BAR_RECT)\nRegion 6: Screenshot ICON (V_BACKWARD)"}], "bbox": [[176, 248, 51, 43], [125, 446, 161, 21], [118, 490, 168, 14], [374, 636, 7, 29], [315, 695, 14, 21], [81, 695, 7, 21]], "image": "aitw/google_apps/images/google_apps_3619507004205175709_3.png"}

I want to know how to get the image name of google_apps_3619507004205175709_3.png from AitW dataset.

Can you help me? Looking forward to your reply very much.

[SiyuanHuang95]

Hi, thanks for your interest in our project.

You can refer to this official demo: https://github.com/google-research/google-research/blob/master/android_in_the_wild/demo.ipynb

[rika1024]

Sorry for bother you, but I still don't understand how to relate the encoded image in TensorFlow to the .png image. Can you tell me how to get image "web_shopping_275045.png" and "general_6542141336800433221_35.png". Thank you.

[rika1024]

I guess "general_6542141336800433221_35.png" denotes general split, episode_id=6542141336800433221, step_id=35. But what "web_shopping_275045.png" mean?

[ybw959929-coder]

@rika1024 Have you solved this problem? I found the AiTW dataset to be very large, how did you find the corresponding file used for training here?