Home

Awesome

Assistant Conversation Testing Library

:warning: Warning: Conversational Actions will be deprecated on June 13, 2023. For more information, see Conversational Actions Sunset.

This library provides an easy way to write automated tests for your Action. The library wraps the Actions API, enabling you to define a test suite, send queries to your Action, and make assertions on the output to verify information specific to your Action's conversational state.

Install

Node

The latest version of this library requires Node v10.13.0 or later. You can install the library with

npm install @assistant/conversation-testing --save

Setup

  1. Enable the Actions API for your project (The Actions API is enabled by default for newly created projects):
    1. Visit the Google API console and select your project from the Select a project dropdown.
    2. If the Action API is not enabled, search for "Actions API" and click Enable.
  2. Create a Service Account key:
    1. Visit the Google Cloud console credentials page and select your project from the Select a project dropdown.
    2. In the "Create credentials" click "Service account".
    3. Enter a service account name and click Create.
    4. From the Select a role dropdown, select Actions > Actions Admin.
    5. Click Continue.
    6. Click ADD KEY, then select Create new key, then press CREATE to download the service account JSON file.
    7. Set the service account key file to the GOOGLE_APPLICATION_CREDENTIALS environment variable: export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service_account.json

Usage

Note: The examples below use Mocha as a testing framework. You can change the report style by overriding the Mocha report style. See more information in Mocha's reporter docs

  1. Create file for your tests and define a test suite.

    import 'mocha';

import {ActionsOnGoogleTestManager} from '@assistant/conversation-testing';

const PROJECT_ID = '<ACTION_PROJECT_ID>';
const TRIGGER_PHRASE = 'Talk to <ACTION_DISPLAY_NAME>';

describe('Test Suite', function() {
  // Set the timeout for each test run to 60s.
  this.timeout(60000);
  let testManager;

  before('Before all setup', async function() {
    testManager = new ActionsOnGoogleTestManager({ projectId: PROJECT_ID });
    await testManager.writePreviewFromDraft();
    testManager.setSuiteLocale(DEFAULT_LOCALE);
    testManager.setSuiteSurface(DEFAULT_SURFACE);
  });

  afterEach(function() {
    testManager.cleanUpAfterTest();
  });
});

  2. Update the test suite to include various tests with queries and assertions related to your Action. Examples of updates include:

    • Test your main invocation - The following test is run to verify your invocation points to the intended actions.intent.MAIN intent, the correct scene is initialized, and the prompt for the scene is sent.
    ...
  it('trigger only test', async function() {
    await testManager.sendQuery(TRIGGER_PHRASE);
    testManager.assertIntent('actions.intent.MAIN');
    testManager.assertScene('Welcome');
    testManager.assertSpeech('Welcome to Facts about Google!');
  });
...
    • Test your conversation - The example below shows how you might test multiple conversational turns. This test sets the locale to use for the conversation, checks the main invocation, and responds to the Action, while checking matched intents, session parameters and expected responses.
    ...
  it('main functionality', async function() {
    testManager.setTestLocale('en-GB');
    await testManager.sendQuery(TRIGGER_PHRASE);
    testManager.assertIntent('actions.intent.MAIN');
    testManager.assertScene('Welcome');
    testManager.assertSpeech('Welcome to Facts about Google!');
    testManager.assertText(
      'Welcome to .* about Google!', {isRegexp: true, isExact: true});
    await testManager.sendQuery('Cats');
    testManager.assertSpeech(['Oh great, cats!', 'Good choice cats!']);
    testManager.assertIntent('cats');
    testManager.assertSessionParam('categoryType', 'Cats');
    await testManager.sendQuery('stop');
    testManager.assertConversationEnded();
  });
...
    • Test intent matching - This test asserts the expected intent is in the top number of matched intents to a given query, in the given language. In the example below, the test returns successful if the yes intent is in the top three intent matches, when the query "yes, I do" is sent.
    ...
  it('intent match testing', async function() {
    await testManager.assertTopMatchedIntent('yes, I do', 'yes', 3, 'en');
  });
...

Running tests

  1. Ensure you have set your service account key file to the GOOGLE_APPLICATION_CREDENTIALS environment variable: export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service_account.json

  2. Run: npm install && npm run build && npm test

Note: You can use "npm test -- --bail" if you want to stop after the first failure.

Supported features

This library provides functions to control your conversation, easily assert aspects of the response, setting Suite and Test level defaults, and more.

Send functions

Assertions

Note: All assertXXX and getXXX functions get optional response as a last argument. If the response is not passed, the last turn response is used.

Note: You can use your own custom assertions on values using Chai, or any other node.js package, but the builtin assertions are likely to be more convenient for you. You can also access the full last turn response by calling getLatestResponse(), and run any custom checks on it.

Getters

Match Intents

This are assertions that are run as standalone NLU versification (not in the context of conversation):

Note: Make sure to set the language code and NOT locale code as the queryLanguage.

Troubleshooting

Known issues

References & Issues

Make Contributions

Please read and follow the steps in the CONTRIBUTING.md.

License

See LICENSE.