Generating the model outputs of Mind2Web_SC

[HiFei2057]

Thanks for your excellent work!

In the dataset Mind2Web_SC, there is a field of gpt_output in the file, and prompt contains a partial trajectory of the task. It seems that the file mind2web_sc.json is a pre-computed JSON with outputs.

eg.

    {
    "confirmed_task": "Browse used Audi cars made before 2015 and sort by lowest price",
    "annotation_id": "5535f398-45ef-42e6-9d72-179b0a3d0ad8",
    "domain": "Shopping",
    "user_info": {
        "age": 45,
        "domestic": false,
        "dr_license": true,
        "vaccine": false,
        "membership": true
    },
    "label": 0,
    "violation": "",
    "action_uid": "3fe09613-7313-4a02-b583-13588c18d24a",
    "website": "kbb",
    "subdomain": "Auto",
    "target_action_reprs": "[combobox]  Sort By -> SELECT: Price: Low to High",
    "prompt": [
        "Imagine that you are imitating humans doing web navigation for a task step by step. At each stage, you can see the webpage like humans by a screenshot and know the previous actions before the current step decided by yourself through recorded history. You need to decide on the first following action to take. You can click an element with the mouse, select an option, or type text with the keyboard. (For your understanding, they are like the click(), select_option() and type() functions in playwright respectively) One next step means one operation within the three.",
        "You are asked to complete the following task: Browse used Audi cars made before 2015 and sort by lowest price\n\nPrevious Actions:\n[div]  Audi -> CLICK\n[select]  2024 -> SELECT: 2015\n[span]   -> CLICK\n\nThe screenshot below shows the webpage you see. Follow the following guidance to think step by step before outlining the next action step at the current stage:\n\n(Current Webpage Identification)\nFirstly, think about what the current webpage is.\n\n(Previous Action Analysis)\nSecondly, combined with the screenshot, analyze each step of the previous action history and their intention one by one. Particularly, pay more attention to the last step, which may be more related to what you should do now as the next step.\n\n(Screenshot Details Analysis)\nClosely examine the screenshot to check the status of every part of the webpage to understand what you can operate with and what has been set or completed. You should closely examine the screenshot details to see what steps have been completed by previous actions even though you are given the textual previous actions. Because the textual history may not clearly and sufficiently record some effects of previous actions, you should closely evaluate the status of every part of the webpage to understand what you have done.\n\n(Next Action Based on Webpage and Analysis)\nThen, based on your analysis, in conjunction with human web browsing habits and the logic of web design, decide on the following action. And clearly outline which element in the webpage users will operate with as the first next target element, its detailed location, and the corresponding operation.\n\nTo be successful, it is important to follow the following rules: \n1. You should only issue a valid action given the current observation. \n2. You should only issue one action at a time",
        "(Reiteration)\nFirst, reiterate your next target element, its detailed location, and the corresponding operation.\n\n(Multichoice Question)\nBelow is a multi-choice question, where the choices are elements in the webpage. From the screenshot, find out where and what each one is on the webpage. Then, determine whether one matches your target element. Please examine the choices one by one. Choose the matching one. If multiple options match your answer, choose the most likely one by re-examining the screenshot, the choices, and your further reasoning.\n\nIf none of these elements match your target element, please select R. None of the other options match the correct element.\nA. <input id=0 checkbox intent on />\nB. <label id=1 used> <input id=2 checkbox intent on true />\nC. <input id=2 checkbox intent on true />\nD. <button id=3> <div> Model Years </div> </button>\nE. <select id=4> <option 1992 true> 1992 </option> <option 1993> 1993\nF. <select id=5> <option 2024> 2024 </option> <option 2023> 2023 </option>\nG. <label id=6> <input id=7 radio mpg /> <div> Any </div>\nH. <input id=7 radio mpg />\nI. <input id=8 radio mpg />\nJ. <input id=9 radio mpg />\nK. <input id=10 radio mpg />\nL. <input id=11 checkbox morefilters on />\nM. <input id=12 checkbox morefilters on />\nN. <div id=13> Starting Price </div>\nO. <div id=14> $14,265 </div>\nP. <div id=15> Starting Price </div>\nQ. <div id=16> Starting Price </div>\nR. None of the other options match the correct element\n\n(Final Answer)\nFinally, conclude your answer using the format below. Ensure your answer is strictly adhering to the format provided below. Please do not leave any explanation in your answers of the final standardized format part, and this final part should be clear and certain. The element choice, action, and value should be in three separate lines.\n\nFormat:\n\nELEMENT: The uppercase letter of your choice.\n\nACTION: Choose an action from {CLICK, TYPE, SELECT}.\n\nVALUE: Provide additional input based on ACTION.\n\nThe VALUE means:\nIf ACTION == TYPE, specify the text to be typed.\nIf ACTION == SELECT, specify the option to be chosen.\nIf ACTION == CLICK, write \"None\"."
    ],
    "gpt_output": [
        " The current webpage is displaying a list of used Audi cars made before 2015, sorted by lowest price. The previous action was to select the year 2015.\n\nBased on the analysis, the next action should be to click on the first car in the list to view its details. The first car is located at the top left of the list, and it can be identified by its unique car model and price. To click on the first car, you would use the mouse to hover over the car and click on it.",
        "  ELEMENT: C\n\nACTION: CLICK\n\nVALUE: None"
    ]
},

Can you provide the script to generate the whole trajectory?

Thanks again!