Home

Awesome

WARNING: This repository is no longer maintained :warning:

This repository will not be updated. The repository will be kept available in read-only mode.

Create voice commands for VR experiences with Watson services

Read this in other languages: 한국어, 中国.

In this Code Pattern we will create a Virtual Reality game based on Watson's Speech-to-Text and Watson's Assistant services.

In Virtual Reality, where you truly "inhabit" the space, speech can feel like a more natural interface than other methods. Providing speech controls allows developers to create more immersive experiences. Google Cardboard is by far the most popular VR headset platform, with 84 million sold in 2016 (http://www.hypergridbusiness.com/2016/11/report-98-of-vr-headsets-sold-this-year-are-for-mobile-phones).

When the reader has completed this Code Pattern, they will understand how to:

architecture

Flow

  1. User interacts in virtual reality and gives voice commands such as "Create a large black box".
  2. The Android phone microphone picks up the voice command and the running application sends it to Watson Speech-to-Text.
  3. Watson Speech-to-Text converts the audio to text and returns it to the running Application on the Android phone.
  4. The application sends the text to Watson Assistant. Watson assistant returns the recognized intent "Create" and the entities "large", "black", and "box". The virtual reality application then displays the large black box (which falls from the sky).

Watch the Video

video

Included components

Featured technologies

Steps

  1. Before you begin
  2. Create IBM Cloud services
  3. Building and Running

1. Before You Begin

2. Create IBM Cloud services

On your local machine:

git clone https://github.com/IBM/vr-speech-sandbox-cardboard.git
cd vr-speech-sandbox-cardboard

In IBM Cloud:

Import the Skill workspace.json:

Create the Assistant and get the Assistant ID

add Skill to Assistant

Choose Settings

3. Building and Running

Note: This has been compiled and tested using Unity 2018.3.0f2 and Unity 2019.1.0f2 and Watson Unity SDK version v3.1.0.

Note: If you are in any IBM Cloud region other than US-South you must use Unity 2018.2 or higher. This is because Unity 2018.2 or higher is needed for TLS 1.2, which is the only TLS version available in all regions other than US-South.

cd SpeechSandbox/Assets
git clone https://github.com/watson-developer-cloud/unity-sdk.git
git checkout v3.1.0
  1. Open Unity and inside the project launcher select the Open button.
  2. Navigate to where you cloned this repository and open the SpeechSandbox directory.
  3. If prompted to upgrade the project to a newer Unity version, do so.
  4. Follow these instructions to add the Watson Unity SDK downloaded in step 1 to the project.
  5. Follow these instructions to create your Speech To Text and Watson Assistant services and find their credentials (using IBM Cloud
  6. In the Unity Hierarchy view, click on Assets -> Scenes -> Playground and double-click to load the scene.
  7. In the Unity Hierarchy view, click on Player and then the Streaming Speech object.
  8. In the Inspector you will see Variables for Speech To Text and Watson Assistant
  9. Add the Speech To Text IAM APIKey and the Assistant IAM APIKey
  10. Fill out the Speech To Text Service Url and the Assistant Service Url, or except the defaults. Also fill out the Assistant Id, and the Assistant Version Date. There are tool tips which will show help and any defaults.

Speech Sandbox Creds

  1. Press Play

  2. To Build an android .apk file and deploy to your phone, you can File -> Build Settings (Ctrl + Shift +B) and click Build.

  3. When prompted you can name your build and then move it to your phone.

  4. Alternately, connect the phone via USB and File-> Build and Run (or Ctrl+B).

    Make sure you have enabled USB Debugging:

Links

Troubleshooting

Assets/Scripts/CreatableObject.cs(162,43): error CS0117: `GvrController' does not contain a definition for `Gyro'

This is because you have your Build Settings -> Platform set to PC, Mac, or Linux Standalone. Solve this problem by changing the Build Settings -> Platform to Android and clicking Switch Platform.

Assets/unity-sdk/Scripts/Utilities/Credentials.cs(399,51): error CS1061: Type `System.DateTimeOffset' does not contain a definition for `ToUnixTimeSeconds' and no extension method `ToUnixTimeSeconds' of type `System.DateTimeOffset' could be found. Are you missing an assembly reference?

The solution is to go to Build Settings->Player Settings-> Other Settings->Scripting Runtime Version and change to: Experimental(.NET 4.6 Equivalent).

Learn more

License

This code pattern is licensed under the Apache Software License, Version 2. Separate third party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 (DCO) and the Apache Software License, Version 2.

Apache Software License (ASL) FAQ