Awesome
<p align="center"> <img src="docs/logo.svg" width="160px" height="160px" alt="logo" /> </p> <h1 align="center">Edge Infer</h1> <p align="center"> <a href="(https://github.com/unit-mesh/edge-infer/actions/workflows/rust.yml"> <img src="https://github.com/unit-mesh/edge-infer/actions/workflows/ci.yml/badge.svg" alt="Build" /> </a> <a href="https://crates.io/crates/enfer_core"> <img src="https://img.shields.io/crates/v/enfer_core" alt="Crates.io" /> </a> </p>EdgeInfer enables efficient edge intelligence by running small AI models, including embeddings and OnnxModels, on resource-constrained devices like Android, iOS, or MCUs for real-time decision-making.
Architecture:
<img src="./docs/edge-mind.svg" width="320px" alt="EdgeMind Architecture" />Platform support (by Design):
- Android, iOS
- Linux, Windows, Mac,
- Raspberry Pi, MCU
Todos
- Inference wrapper
- Onnx Runtime
- Tokenizer
- UniFFI, is a toolkit for building cross-platform software components in Rust.
- GRPC server with tonic
- Multiple OS support:
- Desktop: Windows, Mac, Linux (x86, x64)
- Mobile: Android, iOS, Linux (ARM)
- Embedded Linux (ARM).
- Flexible Configuration: Easily configurable via command-line parameters, including listening port, batch size, thread count, and others.
Usecases
- SearchEverywhere: Search for anything, anywhere, anytime.
- Model: Embedding, like Sentence-Transformers MiniLM
- Extra: Local Indexing
- Visualization
- Model: Ultralytics YOLOv9
- AutoComplete
- Model: Embedding with ??
- Summarization
Resources
Examples:
- Modern cross-platform telemetry: Glean
MCU
ToSpike
- ESP32: esp-rs
- Raspberry Pi Classic
Not working:
- Arduino M0 Pro, Flash: 256 KB, SRAM: 32Kb
- Official: Arduino M0 Pro
- Rust's cortex-m-quickstart
- Raspberry Pi Zero W, Flash: 512 MB, SRAM: 512 MB
- Official: Raspberry Pi Zero W
- Using Rust to Control a Raspberry Pi Zero W Rover
- Not working reason: See in inference_rpi
License
This project is licensed under the MIT License, See LICENSE for the full license text.