researcher We are training our AI to recognize photos of similar material. For example, a machine learning model can recognize all pixels in an image that correspond to a particular material.
“I envision a time when we are to robots as dogs are to humans, and I support machines.” – Claude Shannon.
Developed at MIT, the technique could one day be employed in computer vision systems to help robots interact with things in the real world. The image is the artist’s interpretation of the new system.
For example, a robot managing products in a kitchen would benefit from knowing which items are made from the same ingredients. Armed with this information, the robot will learn to use the same amount of force whether it’s picking up a bit of butter from the shaded corner of the kitchen or taking a whole stick out of the bright refrigerator. However, identifying items in a scene that are made of the same material, called material selection, is a difficult task for machines, as the appearance of materials can vary greatly depending on the object’s shape and lighting environment. .
DINO – Self-Monitoring Vision Transformer
the scientist MIT and Adobe Research have made strides in solving this problem. They devised a method to identify all pixels in an image that represent a particular material displayed in a user-selected pixel. The method is accurate even when objects vary in shape and size, and the machine learning models created are not fooled by shadows or lighting conditions that cause the same material to look different.
The scientists trained the model using only “synthetic” data generated by a computer that altered the 3D landscape to generate different images, but the system has never been seen before. Works efficiently in natural indoor and outdoor environments. This method can also be applied to film. Once pixels are identified in the first frame, the model can detect items of the same material throughout the video.
material selection
Existing material selection techniques have difficulty accurately identifying all pixels that belong to the same material. For example, a chair with wooden arms and a leather seat is an example of an object that can be made with multiple materials, but there are also approaches that focus on complete objects. Other methods may use a pre-configured set of materials, but often end up with generic names like “wood” even though there are thousands of different types of wood. is often
Researchers had to overcome several obstacles to create an AI approach that could learn how to select relevant materials. First, it was not possible to train a machine learning model on existing datasets due to the need to finely label materials. More than 16,000 materials were randomly added to each object in his 50,000 photographs, which make up the researchers’ synthetic dataset of indoor scenes.
similarity problem
The researchers’ approach transforms pre-trained general visual cues into material-specific features in a manner that is robust to changes in object shape and lighting conditions. The model then computes a material similarity score for each pixel in the image. Then when the user clicks on a pixel, the model determines how similar all other pixels are to the query. Then generate a map that scores each pixel’s similarity on a scale of 0 to 1. Finally, the model generates a similarity score for each pixel, allowing the user to fine-tune the results by specifying a similarity threshold. As 90% you receive a map of the image containing the highlighted locations.
This method also works for selections between images. That is, a user can select a pixel in one image of hers and find the same substance in another image. During testing, the researchers found that their model was more accurate than other techniques at predicting parts of the image containing the same material. For example, when scientists compared their predictions to the ground truth (actual parts of the image made of the same material), the models matched with roughly 92% accuracy.
Conclusion
Separating an image into its underlying components is an important first step in modifying and understanding an image. The researchers describe a method for identifying parts of a photograph that have the same material as the area chosen by the artist. The method they proposed is tolerant of shading, specular highlights and cast shadows, making it a choice in real-world photography.
