Home

Awesome

OBELICS

OBELICS is an open, massive and curated collection of interleaved image-text web documents, containing 141M documents, 115B text tokens and 353M images.

Dataset page: https://huggingface.co/datasets/HuggingFaceM4/OBELICS

Visualization of OBELICS web documents: https://huggingface.co/spaces/HuggingFaceM4/obelics_visualization

Paper: https://arxiv.org/abs/2306.16527

Goal and organization of obelics

The folder obelics is aimed for:

The primary techniques are defined in the sub-folder processors, while their invocation is found in callers. The configs used for the extraction and the filtering of the documents are in configs.

We refer to our paper for details about these steps.

In visualization, there are different streamlit visualizations:

Goal and organization of build_obelics

In the folder build_obelics, we are giving all the scripts that were used for the creation of OBELICS, with numbers indicating the chronology.

These scripts often call methods defined in processors but not only, and also define other useful methods.

Citation

If you are using this dataset or this code, please cite

@misc{laurencon2023obelics,
      title={OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents},
      author={Hugo Laurençon and Lucile Saulnier and Léo Tronchon and Stas Bekman and Amanpreet Singh and Anton Lozhkov and Thomas Wang and Siddharth Karamcheti and Alexander M. Rush and Douwe Kiela and Matthieu Cord and Victor Sanh},
      year={2023},
      eprint={2306.16527},
      archivePrefix={arXiv},
      primaryClass={cs.IR}
}