Home

Awesome

<div align="center"> <a href="https://github.com/argmaxinc/WhisperKit#gh-light-mode-only"> <img src="https://github.com/user-attachments/assets/f0699c07-c29f-45b6-a9c6-f6d491b8f791" alt="WhisperKit" width="20%" /> </a> <a href="https://github.com/argmaxinc/WhisperKit#gh-dark-mode-only"> <img src="https://github.com/user-attachments/assets/1be5e31c-de42-40ab-9b85-790cb911ed47" alt="WhisperKit" width="20%" /> </a>

whisperkittools

</div>

+Unit and Functional Tests

Python tools for WhisperKit and WhisperKitAndroid

[!IMPORTANT] If you are looking for more features such as speaker diarization and upgraded performance, check out WhisperKit Pro!

Table of Contents

Installation

conda create -n whisperkit python=3.11 -y && conda activate whisperkit
cd WHISPERKIT_ROOT_DIR && pip install -e .

<a name="model-generation-apple"></a> Model Generation (Apple)

Convert Hugging Face Whisper Models (PyTorch) to WhisperKit (Core ML) format:

whisperkit-generate-model --model-version <model-version> --output-dir <output-dir>

For optional arguments related to model optimizations, please see the help menu with -h

<a name="publish-custom-model"></a> Publishing Models

We host several popular Whisper model versions here. These hosted models are automatically over-the-air deployable to apps integrating WhisperKit such as our example app WhisperAX on TestFlight. If you would like to publish custom Whisper versions that are not already published, you can do so as follows:

huggingface-cli whoami

If you don't have a write token yet, you can generate it here.

MODEL_REPO_ID=my-org/my-whisper-repo-name whisperkit-generate-model --model-version distil-whisper/distil-small.en --output-dir <output-dir>

If the above command is successfuly executed, your model will have been published to hf.co/my-org/my-whisper-repo-name/distil-whisper_distil-small.en!

<a name="model-generation-android"></a> Model Generation (Android)

WhisperKit currently only supports Qualcomm AI Hub Whisper models on Hugging Face:

whisperkittools generates 3 more support models for input preprocessing and output postprocessing used in the WhisperKitAndroid pipeline. These are all published on Hugging Face here. Nonetheless, you may regenerate these models if you wish by following these steps:

Stay tuned for more options for generating models without creating an account and more model version coverage!

<a name="evaluate-apple"></a> Model Evaluation (Apple)

Evaluate (Argmax- or developer-published) models on speech recognition datasets:

whisperkit-evaluate-model --model-version <model-version> --output-dir <output-dir> --evaluation-dataset {librispeech-debug,librispeech,earnings22}

Install additional dependencies via:

pip install -e '.[evals,pipelines]'

By default, this command uses the latest main branch commits from WhisperKit and searches within Argmax-published model repositories. For optional arguments related to code and model versioning, please see the help menu with -h

We continually publish the evaluation results of Argmax-hosted models here as part of our continuous integration tests.

<a name="evaluate-on-custom-dataset"></a> Model Evaluation on Custom Dataset

If you would like to evaluate WhisperKit models on your own dataset:

export CUSTOM_EVAL_DATASET="my-dataset-name-on-hub"
export DATASET_REPO_OWNER="my-user-or-org-name-on-hub"
export MODEL_REPO_ID="my-org/my-whisper-repo-name" # if evaluating self-published models
whisperkit-evaluate-model --model-version <model-version> --output-dir <output-dir> --evaluation-dataset my-dataset-name-on-hub

Python Inference

Use the unified Python wrapper for several on-device Whisper frameworks:

Install additional dependencies via:

pip install -e '.[pipelines]'
from whisperkit.pipelines import WhisperKit, WhisperCpp, WhisperMLX, WhisperOpenAIAPI

pipe = WhisperKit(whisper_version="openai/whisper-large-v3", out_dir="/path/to/out/dir")
print(pipe("audio.{wav,flac,mp3}"))

Note: WhisperCpp requires ffmpeg to be installed. Recommended installation is with brew install ffmpeg Note: WhisperOpenAIAPI requires setting OPENAI_API_KEY as an environment variable

Example SwiftUI App

TestFlight

Source Code (MIT License)

This app serves two purposes:

Note that the app is in beta and we are actively seeking feedback to improve it before widely distributing it.

<a name="qoi"></a> WhisperKit Quality and Performance Benchmarks

Please visit the WhisperKit Benchmarks Hugging Face Space for detailed benchmark results. Here is a brief explanation to help with navigation of the results. This benchmark is updated for every non-patch release on virtually all supported devices.

FAQ

Q1: xcrun: error: unable to find utility "coremlcompiler", not a developer tool or in PATH A1: Ensure Xcode is installed on your Mac and run sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer.

Citation

If you use WhisperKit for something cool or just find it useful, please drop us a note at info@argmaxinc.com!

If you use WhisperKit for academic work, here is the BibTeX:

@misc{whisperkit-argmax,
title = {WhisperKit},
author = {Argmax, Inc.},
year = {2024},
URL = {https://github.com/argmaxinc/WhisperKit}
}