Awesome
The Google Assistant SDK
A version in node to play around with! I've abstracted it from needing to use the mic and speakers on the device running the code (but it still can!) so that you can pass audio in and play audio back however you want to.
Installation
You need to create a JSON file for OAuth2 permissions! Follow the instructions and then:
$ npm install google-assistant
You may also pass in your own OAuth2 client via the
oauth2Client
param (This will be used instead ofkeyFilePath
andsavedTokensPath
). See the usage below for an example.
If you want the device that you are running this code on to respond to commands (eg. "Turn off"), you'll need to go through the Device Registration instructions.
Usage
const path = require('path');
const GoogleAssistant = require('google-assistant');
const config = {
auth: {
keyFilePath: path.resolve(__dirname, 'YOUR_API_KEY_FILE_PATH.json'),
// where you want the tokens to be saved
// will create the directory if not already there
savedTokensPath: path.resolve(__dirname, 'tokens.json'),
// you can also pass an oauth2 client instead if you've handled
// auth in a different workflow. This trumps the other params.
oauth2Client: YOUR_CLIENT,
},
// this param is optional, but all options will be shown
conversation: {
audio: {
encodingIn: 'LINEAR16', // supported are LINEAR16 / FLAC (defaults to LINEAR16)
sampleRateIn: 16000, // supported rates are between 16000-24000 (defaults to 16000)
encodingOut: 'LINEAR16', // supported are LINEAR16 / MP3 / OPUS_IN_OGG (defaults to LINEAR16)
sampleRateOut: 24000, // supported are 16000 / 24000 (defaults to 24000)
},
lang: 'en-US', // language code for input/output (defaults to en-US)
deviceModelId: 'xxxxxxxx', // use if you've gone through the Device Registration process
deviceId: 'xxxxxx', // use if you've gone through the Device Registration process
deviceLocation: {
coordinates: { // set the latitude and longitude of the device
latitude: xxxxxx,
longitude: xxxxx,
},
},
textQuery: 'What time is it?', // if this is set, audio input is ignored
isNew: true, // set this to true if you want to force a new conversation and ignore the old state
screen: {
isOn: true, // set this to true if you want to output results to a screen
},
},
};
const assistant = new GoogleAssistant(config.auth);
// starts a new conversation with the assistant
const startConversation = (conversation) => {
// setup the conversation and send data to it
// for a full example, see `examples/mic-speaker.js`
conversation
.on('audio-data', (data) => {
// do stuff with the audio data from the server
// usually send it to some audio output / file
})
.on('end-of-utterance', () => {
// do stuff when done speaking to the assistant
// usually just stop your audio input
})
.on('transcription', (data) => {
// do stuff with the words you are saying to the assistant
})
.on('response', (text) => {
// do stuff with the text that the assistant said back
})
.on('volume-percent', (percent) => {
// do stuff with a volume percent change (range from 1-100)
})
.on('device-action', (action) => {
// if you've set this device up to handle actions, you'll get that here
})
.on('screen-data', (screen) => {
// if the screen.isOn flag was set to true, you'll get the format and data of the output
})
.on('ended', (error, continueConversation) => {
// once the conversation is ended, see if we need to follow up
if (error) console.log('Conversation Ended Error:', error);
else if (continueConversation) assistant.start();
else console.log('Conversation Complete');
})
.on('data', (data) => {
// raw data from the google assistant conversation
// useful for debugging or if something is not covered above
})
.on('error', (error) => {
// handle error messages
})
};
// will start a conversation and wait for audio data
// as soon as it's ready
assistant
.on('ready', () => assistant.start(config.conversation))
.on('started', startConversation);
TypeScript
import GoogleAssistant = require("google-assistant");
const googleAssistant: GoogleAssistant = new GoogleAssistant();
Examples
- mic-speaker - If you want to test input and output using your machine’s built-in hardware.
- console-input - Use the console to have a text-based conversation with the assistant.
Pre-reqs for the mic-speaker example
If you are on macOS and are seeing Illegal instruction: 4
when you complete your conversation, just use this command to re-install the speaker:
$ npm install speaker --mpg123-backend=openal
If you are on a Raspberry Pi and having some issues with getting the mic-speaker
example working, try changing the following code around line 54:
const mic = record.start({ threshold: 0, recordProgram: 'arecord', device: 'plughw:1,0' });
This is assuming you have a capture device on 1,0 (hint: type arecord -l
to see what capture devices you have and what card/device they are on)
Assistant Instance {Object}
Expects an object with the following params:
- keyFilePath {String}: Path of the JSON file that you have after following the OAuth flow.
- savedTokensPath {String}: Path where you would like your tokens saved after you give your app permission to access your account.
Events
ready {Assistant}
Emitted once your OAuth2 credentials have been saved. It's safe to start a conversation now. Returns an instance of the assitant that you can start conversations with (after the ready event is fired though)
started {Conversation}
You'll get this right after a call to start
and it returns a conversation
instance (see below).
Methods
start([callback]) {Conversation}
This is called anytime after you've got a ready
event. Optional callback will return Error
or a Conversation
instance.
Conversation Instance [{Object}]
After you call start
on your Assistant instance, you will get this back. It takes an optional config object with the follow params:
- audio {Object}: How audio in/out is handled. Has the folliowing params:
- encodingIn {String}: How the audio coming in is encoded. Supported are
LINEAR16
andFLAC
(defaults toLINEAR16
) - sampleRateIn {Number}: Sample rate of the input audio. Supported rates are between
16000
-24000
(defaults to16000
) - encodingOut {String}: How you would like out output audio to be encoded. Supported are
LINEAR16
,MP3
, andOPUS_IN_OGG
(defaults toLINEAR16
) - sampleRateOut {Number}: Sample rate of output audio. Supported are
16000
and24000
(defaults to24000
)
- encodingIn {String}: How the audio coming in is encoded. Supported are
- lang {String}: Language code for the input / output. (defaults to
en-US
, but here are some more options!) - deviceModelId {String}: Device model id when using custom devices.
- deviceId {String}: Device id when using custom devices.
- deviceLocation {Object}: Set the device's location.
- coordinates {Object}: Set the latitude and longitude of the devic.
- latitude {Number}: The latitude in degrees. It must be in the range [-90.0, +90.0].
- longitude {Number}: The longitude in degrees. It must be in the range [-180.0, +180.0].
- coordinates {Object}: Set the latitude and longitude of the devic.
- textQuery {String}: Text that will be passed to the assistant. Audio input is disabled if this is set!
- isNew {Boolean}: Set this to true if you want to force a new conversation and ignore the old state.
- screen {Object}: Configures output to return visual results.
- isOn {Boolean}: Set this to true to be able to trigger the
screen-data
event.
- isOn {Boolean}: Set this to true to be able to trigger the
Events
error
If things go funky, this will be called.
audio-out {Buffer}
Contains an audio buffer to use to pipe to a file or speaker.
end-of-utterance
Emitted once the server detects you are done speaking.
transcription {Object}
While you are speaking, you will get many of these messages. They contain the following params:
- transcription {String}: What your current speech to text value is.
- done {Boolean}: If
true
the assistant has determined you are done speaking and is processing it.
response {String}
The response text from the assistant.
volume-percent {Number}
There was a request to change the volume. The range is from 1-100.
device-action {Object}
There was a request to complete an action. Check out the Device Registration page for more info on creating a device instance.
ended {Error, Boolean}
After a call to end()
this will be emitted with an error and a boolean that will be true
if you need to continue the conversation. This is basically your cue to call start()
again.
screen-data {Object}
Contains information to render a visual version of the assistant's response.
- format {String}: What format the data is in (eg.
HTML
). - data {Buffer}: Raw data to be used to display the formatted result.
Methods
write(UInt8Array)
When using audio input, this is what you use to send your audio chunks to the assistant. (see the mic-speaker example)
end()
Send this when you are finsished playing back the assistant's response.