Multimodal large-scale language models (MLLMs) are increasingly being introduced into real-world agent settings where the output must not only be correct but also conform to a predefined data schema. Despite recent advances in structured generation in the text domain, there are still no benchmarks that systematically evaluate schema-based information extraction and inference on visual input. In this study, we conduct a comprehensive study of the visual structure output capabilities of MLLM using the carefully designed SO-Bench benchmark. Covering four visual domains including UI screens, natural images, documents, and charts, SO-Bench is built from over 6.5K diverse JSON schemas and 1.8K hand-picked image schema pairs with human-verified quality. Benchmark experiments of open source Frontier proprietary models reveal that there are still gaps in predicting accurate schema-compliant outputs, highlighting the need for better multimodal structural inference. Beyond the benchmarks, we conduct further training experiments to significantly improve the structured output capabilities of our model. We plan to make benchmarks available to the community.
