Powering the meteoric rise of AI chatbots, LLMs are the talk of the town. They are showing mind-blowing capabilities in user-tailored natural language processing functions but seem to be lacking the ability to understand the visual world. To bridge the gap between the vision and language world, researchers have presented the All-Seeing (AS) project. 

The AS Project is for open-world panoptic visual recognition and understanding, driven by the goal of creating a vision system that mimics human cognition. The term “panoptic” refers to including everything visible in one view.

The AS Project consists of:

The All-Seeing 1B (AS-1B) dataset covers a wide range of 3.5 million common and rare concepts in the real world and has 132.2 billion tokens that describe the concepts and their attributes.

The All-Seeing model (ASM) is a unified location-aware image-text foundation model. The model consists of two key components: a location-aware image tokenizer and an LLM-based decoder.

The dataset comprises over 1 billion region annotations in various formats, such as semantic tags, locations, question-answering pairs, and captions. Compared with the previous visual recognition datasets like ImageNet and COCO, visual understanding datasets like Visual Genome and Laion-5B, the AS-1B dataset stands out due to its rich and diverse instance-level location annotation and corresponding detailed object concepts and descriptions. 

The architecture of the AS model comprises a unified framework of varying levels. It supports contrastive and generative image-text tasks at both the image level and region levels. By leveraging pre-trained LLMs and powerful vision foundation models (VFMs), the model demonstrates promising performance in discriminative tasks like image-text retrieval and zero classification, as well as generative tasks such as visual question answering (VQA), visual reasoning, image captioning, region captioning/VQA, etc. Additionally, researchers claim to see potential in grounding tasks like phrase grounding and referring expression comprehension with the assistance of a class-agnostic detector.

The All-Seeing Model (ASM) comprises of three key designs:

A location-aware image tokenizer extracts feature from the image and region levels based on the input image and bounding box, respectively.

A trainable task prompt is incorporated at the beginning of the vision and text tokens to guide the model in distinguishing between discriminative and generative tasks.

An LLM-based decoder is utilized to extract vision and text features for discriminative tasks and auto-regressively generate response tokens in generative tasks. 

Extensive data analysis in terms of quality, scaling, diversity, and experiments was conducted by analyzing and comparing the proposed ASM with a CLIP-based baseline model(displays zero-shot capabilities of GPT-2 and 3) and leading Multimodality Large Language models (VLLMs) on representative vision tasks including zero-shot region recognition, image-level caption, and region-level caption. The findings highlighted the strong region-level text generation capabilities of our model while also showcasing its ability to comprehend the entire image. The human evaluation results indicated that captions generated by our ASM are preferred over those from MiniGPT4 and LLaVA.

The model is trained with open-ended language prompts and locations, which allows it to generalize to various vision and language tasks with remarkable zero-shot performance, including region-text retrieval, region recognition, captioning, and question-answering. This, according to researchers, has given LLMs a “all-seeing eye” and has revolutionized the intersection of vision and language.

Check out the Paper and GitHub. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 28k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

The post Breakthrough in the Intersection of Vision-Language: Presenting the All-Seeing Project appeared first on MarkTechPost.