By enabling users to connect to tools and services, systems that can follow instructions from a graphical user interface (GUI) can automate tedious tasks, increase accessibility, and increase the usefulness of digital assistants. can.
Many GUI-based digital agent implementations rely on HTML-derived textual representations, which are not always readily available. People recognize visual input and use standard mouse and keyboard shortcuts to interact with his GUI. You don’t have to look at the application’s source code to understand how the program works. Quickly learn new programs with an intuitive graphical user interface, regardless of the underlying technology.
The Atari game system is just one example of how a system that learns from pixel-only inputs can perform well. However, when attempting his GUI-based instruction following tasks, learning from pixel-only inputs in combination with common low-level actions presents a number of obstacles. Visually interpreting a GUI requires becoming familiar with the structure of the interface, recognizing and interpreting the natural language in which it is laid out visually, recognizing and identifying visual elements, and understanding what those elements do and how they interact. It should be predictable.
Google DeepMind and Google introduced PIX2ACT, a model that takes pixel-based screenshots as input and selects actions that match basic mouse and keyboard controls. The research group found that agents with only pixel inputs and a general action space outperformed human crowdworkers, and performed on par with state-of-the-art agents using a similar amount of DOM information and human demonstrations. It was demonstrated for the first time that the .
For this, the researchers extended PIX2STRUCT. This Transformer-based image-to-text model has already been trained on large-scale online data to convert screenshots into structured representations based on HTML. PIX2ACT applies a tree search to iteratively build a new expert trajectory for training, combining human demonstrations and interactions with the environment.
The team’s efforts here include creating a framework for a universal browser-based environment and using standard cross-domain observation and action formats, two benchmark datasets (MiniWob++ and WebShop) ready for use in the environment. includes adapting the Using their proposed option (CC-Net without DOM), PIX2ACT performs about 4x better than human crowd workers on MiniWob++. Ablation shows that pixel-based pre-training of PIX2STRUCT is essential for his PIX2ACT performance.
For pixel-based inputs followed by GUI-based instructions, our findings demonstrate the effectiveness of pre-training PIX2STRUCT with screenshot analysis. Pre-training in a behavioral cloning environment increased task scores for MiniWob++ and WebShop by 17.1 and 46.7, respectively. Although there are still performance drawbacks compared to large language models that use HTML-based inputs and task-specific actions, this work sets a first baseline in this environment.
please check out paper. don’t forget to join 23,000+ ML SubReddit, Discord channeland email newsletterShare the latest AI research news, cool AI projects, and more. If you have any questions regarding the article above or missed something, feel free to email us. Asif@marktechpost.com
🚀 Check out 100’s of AI Tools at the AI Tools Club
Tanushree Shenwai is a consulting intern at MarktechPost. She is currently pursuing her bachelor’s degree at the Indian Institute of Technology (IIT), Bhubaneswar. She is a data her science enthusiast and has a keen interest in the range of applications of artificial intelligence in various fields. She is passionate about exploring new advances in technology and its practical applications.
