IBM/vr-speech-sandbox-rift

Name: vr-speech-sandbox-rift

Owner: International Business Machines

Description: A sample application that demonstrates how to integrate voice commands and speech recognition into a virtual reality experience. https://developer.ibm.com/code/journey/create-a-virtual-reality-speech-sandbox/

Created: 2017-10-17 16:16:38.0

Updated: 2018-05-14 19:28:12.0

Pushed: 2018-05-14 19:28:11.0

Homepage:

Size: 119922

Language: C#

GitHub Committers

UserMost Recent Commit# Commits

Other Committers

UserEmailMost Recent Commit# Commits

README

Create voice commands for VR experiences with Watson services

Read this in other languages: ??.

Watson Conversation is now Watson Assistant. Although some images in this code pattern may show the service as Watson Conversation, the steps and processes will still work.

In this Code Pattern we will create a Virtual Reality game based on Watson's Speech-to-Text and Watson's Assistant services.

In Virtual Reality, where you truly ?inhabit? the space, speech can feel like a more natural interface than other methods. Providing speech controls allows developers to create more immersive experiences. The Oculus Rift is the 4th most popular head-mounted VR devices (not including Google Cardboard) and an ideal candidate for Speech interaction, selling roughly 355 thousand units in 2016.

When the reader has completed this Code Pattern, they will understand how to:

Flow
  1. User interacts in virtual reality and gives voice commands such as “Create a large black box”.
  2. The Oculus Rift Headset microphone picks up the voice command and the running application sends it to Watson Speech-to-Text.
  3. Watson Speech-to-Text converts the audio to text and returns it to the running Application that powers the Oculus Rift.
  4. The application sends the text to Watson Assistant. Watson conversation returns the recognized intent “Create” and the entities “large”, “black”, and “box”. The virtual reality application then displays the large black box (which falls from the sky).
Included components
Featured technologies

Watch the Video

Steps

  1. Before you begin
  2. Create IBM Cloud services
  3. Building and Running
1. Before You Begin
2. Create IBM Cloud services

On your local machine:

  1. git clone https://github.com/IBM/vr-speech-sandbox-vive.git
  2. cd vr-speech-sandbox-vive

In IBM Cloud:

  1. Create a Speech-To-Text service instance.
  2. Create a Assistant service instance.
  3. Once you see the services in the Dashboard, select the Assistant service you created and click the "Launch Tool" button.
  4. After logging into the Assistant Tool, click the "Import" button.
  5. Import the Assistant workspace.json file located in your clone of this repository.
3. Building and Running

If you followed the previous steps you should already be inside your local clone and ready to get started running the app from Unity.

  1. git clone https://github.com/watson-developer-cloud/unity-sdk.git

  2. Open Unity and inside the project launcher select the Open button.

  3. Navigate to where you cloned this repository and open the “Creation Sandbox” directory.

  4. If prompted to upgrade the project to a newer Unity version, do so.

  5. Follow these instructions to add the Watson Unity SDK downloaded in step 1 to the project.

  6. Follow these instructions to create your Speech To Text and Assistant services and find their credentials (using IBM Cloud).

  7. Open the script vr-speech-sandbox-vive/Creation Sandbox/Assets/Scripts/SpeechSandboxStreaming.cs

  8. Fill in the credentials for Speech to Text and Assistant, and the Assistant workspace id:

    ate string stt_username = "blahblah";
ate string stt_password = "abc123youandme";
ate string stt_url = "https://stream.watsonplatform.net/speech-to-text/api";

ate string convo_username = "dlkfjasdkfdkfjdkjf";
ate string convo_password = "kdfjlsdjf";
ate string convo_url = "https://gateway.watsonplatform.net/conversation/api";
.<snip>.....
ate string convo_workspaceId = "kdjfls2323";

  9. Install Blender

  10. In the Unity editor project tab, select Assets->Scenes->MainGame->MainMenu and double click to load the scene.

  11. Press Play

Links

Learn more

License

Apache 2.0

This work is supported by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number U24TR002306. This work is solely the responsibility of the creators and does not necessarily represent the official views of the National Institutes of Health.