Multimodal Large Language Models (MLLMs) have evolved significantly in recent months. These direct people’s attention to the Large Language Model (LLM), where they may discuss the input image. These models can understand visual content, but cannot communicate to the user about the exact location of materials. Both the user and the model cannot provide the specific location of the mentioned material within the photograph. In contrast, as shown in Figure 1, everyday human conversation often addresses discrete areas or items within a scene, with individuals conversing and pointing to specific areas for effective information sharing. can do.
They call this kind of communication Reference Dialogue (RD). Once MLLM is implemented in this area, many attractive applications will emerge. A user can specify something to communicate with an AI assistant, for example when using a Mixed Reality (XR) headset such as the Apple Vision Pro. If necessary, the AI assistant can show the immediate area in the field of view. It also helps visual robots interact with people by understanding their unique reference points. Helping consumers learn more about objects of interest in photos encourages online purchases. For this study, they developed MLLM to open the curtain on referential conversations.
Researchers from SenseTime Research, SKLSDE, Beihang University, and Shanghai Jiao Tong University have developed and created Shikra, an integrated model that can handle spatial coordinate input and output. All coordinates, both input and output, are provided in natural-language numeric form, without the use of additional vocabularies or position encoders. The Alignment Layer, LLM, and Vision Encoder are all part of the Shikra architecture. It makes Shikra uniform and easy by not introducing pre/post detection modules or other plugin models. These offer a multitude of user interactions that users can use to compare differences between different areas on her website, find out what thumbnails mean, or talk about specific items. Shikla can answer any question with verbal or geographical justification.
The Vision Language (VL) job of referential discourse surpasses some other jobs. Skilled in RD, Shikra is naturally able to perform tasks such as visual question answering (VQA), image captioning, and location-related tasks such as reference formula comprehension (REC) and pointing, with promising results. increase. Additionally, this essay discusses interesting issues such as how to represent a place in a photograph. Can old MLLMs understand absolute position? Would using geographic information for reasoning lead to more accurate answers to the question? They hope that these analytical experiments will stimulate more MLLM research in the future.
The main contributions of this essay are:
• This essay describes the activity of Reference Dialogue (RD), which is an important part of normal human communication and has many practical applications.
• Shikra, a generalist MLLM, is offered as RD. Shikra is easy and integrated, no need to add new vocabularies, pre/post detection modules or other plugin models.
• Shikra can easily manage hidden settings to meet different application situations. Even common visual language tasks such as REC, PointQA, VQA, and image captions perform well without fine-tuning. Code is available on GitHub.
Please check paper and Github link.don’t forget to join 25,000+ ML SubReddit, Discord channeland email newsletterShare the latest AI research news, cool AI projects, and more. If you have any questions regarding the article above or missed something, feel free to email us. Asif@marktechpost.com
🚀 Check out 100’s of AI Tools at the AI Tools Club
Aneesh Tickoo is a consulting intern at MarktechPost. He is currently pursuing his Bachelor of Science in Data Science and Artificial Intelligence from the Indian Institute of Technology (IIT), Bhilai. He spends most of his time working on projects aimed at harnessing the power of machine learning. His research interest is in image processing and he is passionate about building solutions around it. He loves connecting with people and collaborating on interesting projects.
