In the realm of large multi-modal models (LMMs), efficient modality alignment is crucial yet often constrained by the scarcity of high-quality image-text data. To address this bottleneck, we introduce the ShareGPT4V dataset, a pioneering large-scale resource featuring 1.2 million highly descriptive captions, which surpasses existing datasets in diversity and information content, covering world knowledge, object properties, spatial relationships, and aesthetic evaluations. Specifically, ShareGPT4V originates from a curated 100K high-quality captions collected from advanced GPT4-Vision and has been expanded to 1.2 million with a superb caption model trained on this subset. ShareGPT4V first demonstrates its effectiveness for the Supervised Fine-Tuning (SFT) phase, by substituting an equivalent quantity of detailed captions in existing SFT datasets with a subset of our high-quality captions, significantly enhancing the LMMs like LLaVA-7B, LLaVA-1.5-13B, and Qwen-VL-Chat-7B on the MME and MMBench benchmarks, with respective gains of 222.8/22.0/22.3 and 2.7/1.3/1.5. We further incorporate ShareGPT4V data into both the pre-training and SFT phases, obtaining ShareGPT4V-7B, a superior LMM based on a simple architecture that has remarkable performance across a majority of the multi-modal benchmarks. We will release this project to serve as a pivotal resource for advancing the LMMs community.
Comparison of widely-used caption datasets and ShareGPT4V. 'LCS' abbreviates the LAION, CC, and SBU datasets. The 'Visible' column denotes the image visibility during captioning, and the 'Avg.' column shows the average character number of the caption.
We illustrate the procedure for collecting highly descriptive captions from GPT4-Vision via various image sources and data-specific prompts, resulting in 100K high-quality captions that encapsulate a wide array of information conveyed by the images. A comparison between the caption in our proposed ShareGPT4V dataset and those utilized by recent large multi-modal models (LMMs):
We delineate the process of utilizing the seed captions to train a general captioner (Share-Captioner) and then employing this captioner to generate 1.2M high-quality captions for pre-training usage. A qualitative comparison of caption quality from various sources:
We compare the performance of various large multi-modal models before and after replacing a corresponding portion of their SFT captions with those generated by GPT4-Vision:
The remarkable performance of the proposed LMM, ShareGPT4V-7B, developed with the assistance of the ShareGPT4V dataset:
If you find our work helpful for your research, please consider giving a citation 📃
@article{chen2023sharegpt4v, title={ShareGPT4V: Improving Large Multi-Modal Models with Better Captions}, author={Chen, Lin and Li, Jisong and Dong, Xiaoyi and Zhang, Pan and He, Conghui and Wang, Jiaqi and Zhao, Feng and Lin, Dahua}, journal={arXiv preprint arXiv:2311.12793}, year={2023} }
This website is adapted from Nerfies and LLaVA, licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. We thank the LLaMA team for giving us access to their models, and open-source projects, including Alpaca and Vicuna.
Usage and License Notices: The data, code and checkpoint is intended and licensed for research use only. They are also restricted to uses that follow the license agreement of CLIP, LLaMA, Vicuna and GPT-4. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used outside of research purposes.