Awesome
safetensors-merge-supermario
Combine any two models using a Super Mario merge(DARE) as described in the linked whitepaper.
About
Combine capabilities from multiple models. Works with:
- Stable Diffusion (1.5, XL/XL Turbo)
- LLMs(Mistral, Llama, etc)
- LoRas(must be same size)
- Any two homologous models
Example
| Model | Description | Image(same seed) |
|---|---|---|
| sd_xl_turbo | Attempting 1024 | <img src="assets/before_xl_turbo.png" alt="SDXL turbo attempting to render at 1024" width="256" height="256"> |
| sdxl base 1.0 | Attempting to use SDTurboScheduler | <img src="assets/before_xl.png" alt="SDXL attempting to use SDTurboScheduler" width="256" height="256"> |
| merged | Mario merged(DARE) | <img src="assets/after.png" alt="Merged model successfully rendering 1024" width="256" height="256"> |
Usage
python3 merge.py -p [weight drop probability] -lambda [scaling factor] [base_safetensors_model_file_or_folder] [model_to_merge] [output_path]
Example
python3 merge.py -p 0.13 -lambda 3.0 sdxl_base.safetensors sd_xl_turbo_1.0_fp16.safetensors sdxl_merged.safetensors
Note: This also works with arguments reversed.
Models
- https://huggingface.co/martyn/sdxl-turbo-mario-merge - SD Turbo XL merged with SDXL Base
- https://civitai.com/models/215796 - Top Rated - TurboXL+LCM - Super Mario Merge
- https://huggingface.co/martyn/llama-megamerge-dare-13b - A llama 13b mega merge created using
hf_merge.py - https://huggingface.co/collections/martyn/dare-llm-merges-6581f52c0fb25b9aa26fb180 - A collection of LLM merges
- https://huggingface.co/collections/martyn/dare-diffusion-merges-6581f5fab1c4ab777ad43cf7 - A collection of text-to-image diffusion model merges
ComfyUI workflow
Changelog
- Dec 27 2023: Add support for using files, folders, or hf repos in the hf_merge.py merge list.
- Dec 27 2023: Added mergekit-compatible yaml support for
hf_merge.py. Always runs dare and ignores options outside model specification. weight is p and density is 1/λ. - Dec 12 2023: Added
hf_merge.pyfor merging hf repos. - Dec 12 2023: Added support for folders. You can now merge LLMs(mistral, llama, etc) and other large models. Folders with
.binfiles are supported - the first specified model in the cli must be in.safetensorsformat. - Nov 28 2023: Initial release supporting stable diffusion.
References
- https://arxiv.org/pdf/2311.03099.pdf - Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
- https://github.com/yule-BUAA/MergeLM - GitHub for the linked whitepaper
- https://stability.ai/research/adversarial-diffusion-distillation - SXDL Turbo - Adversarial Diffusion Distillation