Home

Awesome

Smol

Smol Vision 🐣

Recipes for shrinking, optimizing, customizing cutting edge vision and multimodal AI models.

Latest examples 👇🏻

Note: The script and notebook are updated to fix few issues related to QLoRA!

NotebookDescription
Quantization/ONNXFaster and Smaller Zero-shot Object Detection with OptimumQuantize the state-of-the-art zero-shot object detection model OWLv2 using Optimum ONNXRuntime tools.
VLM Fine-tuningFine-tune PaliGemmaFine-tune state-of-the-art vision language backbone PaliGemma using transformers.
Intro to Optimum/ORTOptimizing DETR with 🤗 OptimumA soft introduction to exporting vision models to ONNX and quantizing them.
Model ShrinkingKnowledge Distillation for Computer VisionKnowledge distillation for image classification.
QuantizationFit in vision models using QuantoFit in vision models to smaller hardware using quanto
Speed-upFaster foundation models with torch.compileImproving latency for foundation models using torch.compile
VLM Fine-tuningFine-tune Florence-2Fine-tune Florence-2 on DocVQA dataset
VLM Fine-tuningQLoRA/Fine-tune IDEFICS3 or SmolVLM on VQAv2QLoRA/Full Fine-tune IDEFICS3 or SmolVLM on VQAv2 dataset
VLM Fine-tuning (Script)QLoRA Fine-tune IDEFICS3 on VQAv2QLoRA/Full Fine-tune IDEFICS3 or SmolVLM on VQAv2 dataset
Multimodal RAGMultimodal RAG using ColPali and Qwen2-VLLearn to retrieve documents and pipeline to RAG without hefty document processing using ColPali through Byaldi and do the generation with Qwen2-VL
Multimodal Retriever Fine-tuningFine-tune ColPali for Multimodal RAGLearn to apply contrastive fine-tuning on ColPali to customize it for your own multimodal document RAG use case
Speed-up/Memory OptimizationVision language model serving using TGI (SOON)Explore speed-ups and memory improvements for vision-language model serving with text-generation inference
Quantization/Optimum/ORTAll levels of quantization and graph optimizations for Image Segmentation using Optimum (SOON)End-to-end model optimization using Optimum