Home

Awesome

ESP32 Mouse/Keyboard for BLE HID

ESP32 implementation for HID over GATT Keyboard and Mouse (Bluetooth Low Energy). Including serial API for external modules (compatible to Adafruit EZKey HID).

Compatibility

Due to the wide variety of hardware & software combinations, it is hard to guarantee compatibility. Nevertheless, we try to test this project with as much devices as possible.

Tested devices by the AsTeRICS team:

DeviceMouseKeyboard
PC with CSR 4.0 BT dongle, Debian Bullseyeyesyes
Lenovo T440s, Debian Busteryesyes
MacBook Pro (Mid 2012) - macOS Mojave 10.14yesyes
PC with CSR 4.0 BT dongle, Win10 Build 2004yesyes
Lenovo T440s, Win10 Build 2004 in VirtualBoxnono
Xiaomi Mi A1 - LineageOS 16.0 (Android 9)yesyes
Asus Nexus 7 (2013) - Android 6.0.1yesyes
iPad Air 2 (MGL12FD/A9 - iOS 13.3.1yes (Assistive Touch)yes (Text input & accessible switch interface)
iPad Air 2 (MGL12FD/A9 - iOS 13.5.1yes (Assistive Touch)yes (Text input & accessible switch interface)
iPad Pro 10.5" (MPHG2TY/A) - iOS 13.5.1yes (Assistive Touch)yes (Text input & accessible switch interface)
iPad 6th Gen (MR7J2FD/A) - iOS 13.5.1yes (Assistive Touch)yes (Text input & accessible switch interface)

Note: On some iPad devices, there might show up a notification in the Bluetooth menu, that this device may affect WiFi or Bluetooth performance. You could ignore this error, according to StackOverflow there is no clear reason for this message, maybe it is related to the additional WiFi interface of the ESP32 (other user report this error if a normal mouse has an additional USB-dongle).

Note: If you can report the compatibility status for a device, we would be happy to add a list of community tested devices, just open an issue with the label device-testreport.

Building the ESP32 firmware

Please follow the step-by-step instructions of Espressif's ESP-IDF manual to setup the build infrastructure: Setup Toolchain

With idf.py build or make you can build this project.

With idf.py -p (PORT) flash or make flash you can upload this build to an ESP32

With idf.py -p (PORT) monitor or make monitor you can see the debug output (please use this output if you open an issue) or trigger basic test commands (mouse movement or a keyboard key press) on a connected target.

Note: Currently tested IDF version: release/v5.0 (other ones won't be supported!)

esp32miniBT vs. Arduino Nano Connect

This firmware is used on 2 different devices in context of our assistive devices:

Note: Please select the correct type of board in idf.py menuconfig!

Note: If you want to use this firmware on a custom board, you can select the external UART interface settings in menuconfig; for our boards the pinning, baudrate and UART config is pre-defined.

Usage via Console or second UART

Control via stdin (make monitor)

For basic mouse and keyboard testing, some Bluetooth HID reports can be triggered via the keyboard when the make monitor console is running (see Espressif IDF docs: https://docs.espressif.com/projects/esp-idf/en/latest/api-guides/tools/idf-monitor.html ).

KeyFunctionDescription
aMouse leftMove mouse left by 30px (MOUSE_SPEED define)
sMouse downMove mouse down by 30px (MOUSE_SPEED define)
dMouse rightMove mouse right by 30px (MOUSE_SPEED define)
wMouse upMove mouse up by 30px (MOUSE_SPEED define)
lClick leftMouse click right
rClick rightMouse click left
qType 'y' (US layout)just for testing keyboard reports

Control via 2nd UART

This interface is primarily used to control mouse / keyboard activities via an external microcontroller.

Commands

Each command is started with a '$' character, followed by a command name (uppercase letters) and a variable number of parameters. A command must be terminated by a '\n' character (LF, ASCII value 10)!

Note: We do not test these on a regular basis!

CommandFunctionParametersDescription
$IDGet ID--Prints out the ID of this module (firmware version number)
$GPGet BLE pairings--Prints out all paired devices' MAC adress. The order is used for DP as well, starting with 0
$GCGet active BLE connections--Prints out connected paired devices' MAC adress. Ordered by connection occurance (first connected device is listed first)
$SWSwitch between devicesBT addr (001122334455)Switch between connected devices, the given BT addr will receive the HID packets
$DPDelete one pairing (or all)number of pairing, given as ASCII-characer '0'-'9'Deletes one pairing. The pairing number is determined by the command GP. If no parameter is given, all pairings are removed!
$PMSet pairing mode'0' / '1'Enables (1) or disables (0) discovery/advertising and terminates an exisiting connection if enabled
$NAMESet BLE device namename as ASCII stringSet the device name to the given name. Restart required.
$UGInitiating firmware update--Boot partition is set to 'factory', if available. Device is restarted and expects firmware (.bin) via UART2
$SVSet a key/value pairkey valueSet a value to ESP32 NVS storage, e.g. "$SV testkey This is a testvalue". Note: no spaces in the key! Returns "OK xx/yy used/free" on success, NVS:"error code" otherwise.
$GVGet a key/value pairkeyGet a value from ESP32 NVS storage, e.g. "$GV testkey". Note: no spaces in the key!
$CVClear all key/value pairs--Delete all stored key/value pairs from $SV.

HID input

We use a format which is compatible to the Adafruit EZKey module, and added a few backwards compatible features.

Mouse:

Byte 0Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
0xFDdon't care0x03button maskX-axisY-axiswheeldon't caredon't care

Keyboard:

Note: currently we only support the US keyboard layout.

Byte 0Byte 1Byte 2Byte 3Byte 4Byte 5Byte 6Byte 7Byte 8
0xFDmodifier mask0x00keycode 1keycode 2keycode 3keycode 4keycode 5keycode 6

Joystick:

Byte 0Byte 1Byte 2Byte 3-8Byte 9Byte 10Byte 11Byte 12Byte 13
0xFDdon't care0x01X,Y,Z,Rz,Rx,Ry axis (each int8_t)hat switch (0 is rest position; 1-8 are directions)buttons 0-7buttons 8-15buttons 16-23buttons 24-31

RAW HID input from sourcecode

If you want to change anything within the sourcecode, it is possible to send an HID report directly via:

hid_dev_send_report(hidd_le_env.gatt_if, hid_conn_id,
                  HID_RPT_ID_MOUSE_IN, HID_REPORT_TYPE_INPUT, HID_MOUSE_IN_RPT_LEN, mouse_report);

Use according HID_RPT_ID_*/HID_*_IN_RPT_LEN, depending on input type (mouse, keyboard). The HID report data is located in array, passed as last parameter of hid_dev_send_report.

Mouse & Keyboard input from sourcecode

Please use the functions provided by esp_hidd_prf_api.c.

Credits and many thanks to:

and to Espressif for providing the HID implementation within the esp-idf.