Awesome
Reimagining Customer Experience 360
Omnichannel Customer Service on Google Cloud
Customers ask us all the time. What's the future of customer care? We think your contact center shouldn't be a cost center but a revenue center. It should meet your customers, where they are, 24/7 and be proactive, ubiquitous, and scalable. It should become a trusted partner in your day to day life.
What you will see in the video below, is an early demo, built by DevRel on how we can reimagine customer experience. A true omnichannel approach!
The big keypiece of my architecture is <a href="https://cloud.google.com/conversational-ai">Google Cloud Conversational AI</a>. I am making use of the following components:
<img src="https://github.com/GoogleCloudPlatform/dialogflow-ccai-omnichannel/blob/main/images/experience.png" width="800" /><a href="https://cloud.google.com/conversational-ai/docs">Learn more, checkout the Conversational AI learning hub</a>
Used Technology
Here's an architecture diagram of the used technology:
<img src="https://github.com/GoogleCloudPlatform/dialogflow-ccai-omnichannel/blob/main/images/architecturenew.png" width="800" />Setup
This is a large guide on setting up all the seperate pieces. When possible things are automated through bash scripts. Please read carefully, through all the steps.
Setup guides to the various parts:
- Google Cloud
- Dialogflow
- Twilio Phone & SMS
- Firebase
- Business Messages
- Business Messages Adlingo
- Business Messages Verified Calls
To deploy this environment on GKE use these steps:
To run this environment locally use these setup guides:
Once everything is installed you can run the Demo Script:
Google Cloud Setup
The steps below, will help you to setup your Google Cloud Platform project and enable all the tools you will need for this demo.
- Create GCP Project and assign a billing account to it.
https://console.cloud.google.com
- Either clone this Git repo on your local HD with the following command:
git clone https://github.com/GoogleCloudPlatform/dialogflow-ccai-omnichannel
OR
Click the blue button, to clone in Cloud Shell:
- Copy, modify and rename the env.txt file to .env:
nano env.txt
cp env.txt .env
- Run the following install script to start the installation:
. setup.sh
Choose 2: None, when prompted for bindings.
-
(Optional) In case you want to run this project locally on your machine, the
setup.sh
script will create a service-account with access rights, and download the key to your machine:ccai-360-key.json
.This file can be used to authenticate to Google Cloud Platform services from any environment. To use the file, set the
GOOGLE_APPLICATION_CREDENTIALS
environment variable to the path to the key file, for example:export GOOGLE_APPLICATION_CREDENTIALS=/path/to/ccai-360-key.json
You can run this command each time you run your local environment, or add it to your bash profile scripts which loads when starting the bash command line. On Mac OSX this is the
.bash_profile
file, a hidden file that can be found in your Mac OSX home directory. -
Initialize the Cloud SDK:
Install Cloud SDK on your local machine in case you want to use the project locally. https://cloud.google.com/sdk/docs/install
$ gcloud init
2 (Create a new configuration)
yourname-examples
(login)
list
#number-of-choice
y
gcloud auth login
Dialogflow
Dialogflow Essentials
- Create a new agent: https://dialogflow.cloud.google.com/#/newAgent
- Name: rce-360
- Select Google Project: [project-id]
- Language: English
-
Upgrade to Pay as you Go - plan
-
Go into Settings, and make sure Speech > Speech adaptation is enabled.
-
Import the ES agent into Dialogflow: https://github.com/GoogleCloudPlatform/dialogflow-ccai-omnichannel/tree/main/dialogflow/es-agent.zip
-
To link your Dialogflow instance with a local environment make sure the
GC_PROJECT_ID
variable is correctly set inbackend/.env
-
To make use of Dialogflow Essentials, make sure you have set this config in
backend/config.ts
. By default it's set tov2beta1
.
dialogflow: {
version: 'v2beta1',
},
Dialogflow CX
- Create a new agent: https://dialogflow.cloud.google.com/cx/projects
- Select Google Project: [project-id]
- Name: rce-360
- Choose a region
- Choose a timezone
- Language: English
-
Import the CX agent into Dialogflow CX: https://github.com/GoogleCloudPlatform/dialogflow-ccai-omnichannel/tree/main/dialogflow/cx-agent.blob
-
To link your Dialogflow instance with a local environment make sure the
AGENT_ID
andCX_LOCATION
variables are correctly set inbackend/.env
-
To make use of Dialogflow CX, make sure you have set this config in
backend/config.ts
. By default it's set tocx
.
Twilio Phone/SMS
-
Install Twilio CLI
npm install twilio-cli -g
-
Create a Twilio account: https://www.twilio.com/console/project/
-
And a CLI profile:
twilio profiles:create
-
Buy a phone number:
twilio api:core:available-phone-numbers:local:list --area-code="650" --country-code=US --voice-enabled
or https://www.twilio.com/console/phone-numbers/search -
Wire up your Twilio number with your endpoint on incoming calls. This will automatically start an ngrok tunnel to your machine.
twilio phone-numbers:update +3197010254316 --voice-url=http://localhost:8080/twiml
or manually (to only do this once) (this assumes you have ngrok installed: https://ngrok.com/download)
ngrok http -subdomain=ccai 8080
and modify the URL in the Twilio console to:
https://ccai.ngrok.io/twiml
NOTE: For outbound calls you will need to whitelist the geo permissions you can call: https://console.twilio.com/us1/develop/voice/settings/geo-permissions
-
To link your Twilio instance with a local environment make sure the
TWILIO_ACCOUNT_SID
andTWILIO_ACCOUNT_TOKEN
variables are correctly set inbackend/.env
(env.txt for first time users). Also make sure you have set the correct phone numbers in this file. -
Optionally, you can also buy a seperate test number. Set in
backend/.env
TEST_USER_DEV
that points to a Firebase UID of one of your own test phone numbers. You can execute the script by calling /api/callme/dev/ in the browser.
Firebase
You will need an account on Firebase as we will store authentication data in Firebase auth.
To install the Firebase command-line tools use:
npm install firebase-admin --save
Business Messages
With Business Messages, you can place messaging buttons for brands within organic Google search results. When a user clicks on a messaging button, they start a conversation with an entity representing the brand—the brand's agent. See the Business Messages documentation for more information.
Before you begin
Configure your webhook
You must specify your webhook URL in order to start receiving messages for the Business Messages agent you create.
- Open the Business Communications Developer Console and sign in with your Business Messages Google account.
- Click Account settings.
- Make sure the correct partner account is selected.
- Enter your Business Messages webhook URL as https://YOUR_WEB_SEVER/api/business-messages where
YOUR_WEB_SEVER
is the domain where you are hosting the backend of thedialogflow-ccai-omnichannel
project. - Click Save.
Create your Business Messages agent
Once you've registered, it's time to create your agent.
- Open the Business Communications Developer Console and sign in with your Business Messages Google account.
- Click Create agent.
- If you're prompted for Agent type, choose Business Messages.
- Enter values for Brand name and Agent name.
- Click Create agent.
- When your agent is available, click your agent.
Test your Business Messages agent
Each agent has test URLs that let you see how a conversation with that agent appears to users. Use a test URL to verify that messages sent by users interacting with your agent are being received by your webhook and automatically responded to by the Dialogflow agent you've set up.
To test an agent,
- Open the Business Communications Developer Console and sign in with your Business Messages Google account.
- Choose your agent.
- Under Agent test URLs on the Overview page, click the Android button or iOS button to copy the test URL to your device's clipboard and send to your mobile device or use the Send to email feature.
- Open the test URL on your mobile device and send a message to the Business Messages agent.
Business Messages AdLingo
- Login into your AdLingo environment: https://adsbuilder.adlingo.com/
- Create a conversational agent:
- Specify the GCS bucket name and path to service account (it will be gs://[project-name]-bucket/ccai-360-key.json)
- Event name:
WELCOME-ADLINGO
- Create a creative
- Upload an background image 600x1200 hi-res PNG
- Upload a logo
- Upload an avatar
- For this demo we are integrating an iframe with the preview of the add in the external website interface. In a real world application, this will be provided through an ads platform.
Business Messages Verified Calls
With Verified Calls, you can upgrade a regular phone call from a business to a consumer. When a user receives a call from a business, the user sees who is calling, the logo of the business, that the business has been verified, and the reason for the call. See the Verified Calls documentation for more information.
Before you begin
Configure your webhook
You must specify your webhook URL in order to start receiving messages for the Business Messages agent you create.
- Open the Business Communications Developer Console and sign in with your Business Messages Google account.
- Click Account settings.
- Make sure the correct partner account is selected.
- Enter your Business Messages webhook URL as https://YOUR_WEB_SEVER/api/business-messages where
YOUR_WEB_SEVER
is the domain where you are hosting the backend of thedialogflow-ccai-omnichannel
project. - Click Save.
Create your Verified Calls agent
Once you've registered, it's time to create your agent.
- Open the Business Communications Developer Console and sign in with your Verified Calls Google account.
- Click Create agent.
- If you're prompted for Agent type, choose Verified Calls.
- Enter values for Brand name, Agent name, and Agent logo.
- Click Next.
- Enter the phone number of the business.
- Click Next.
- Enter the call reasons.
- Click Next.
- Enter values for the brand contact, brand website, and yourself.
- Click Finish setup.
Once your agent has been verified by Google, when your business makes a call to a user, the user will see the business information. Note that this only works for Android users.
Deploy to GKE
This script will create a GKE cluster with the backend container and the client website. This will allow you to run a chatbot on a website as a channel. The other channels such as the mobile Flutter app, Twilio phone & SMS and Google Assistant, require additional configuration as desribed below.
-
Rename enx.txt to .env and modify the values. This requires a GCP project id and Twilio configuration in case you want to run the demo via a phone.
-
Modify the properties in the .properties file. You will need a domain to get an SSL certificate. We will need a valid TLS certificate, for secure websockets (WSS), which is what you need in order to make use of Twilio.
-
You will need to open your domain register console and add the following details: (Read more on domain config for GKE)[https://cloud.google.com/kubernetes-engine/docs/tutorials/configuring-domain-name-static-ip]
_Make sure you have executed . setup.sh
before, to enable all services. Also you will need to have envsubst and kubectl on your machine, in case you run it locally. See the tools.sh script.
-
gcloud components install kubectl
-
. tools.sh
-
Execute
. deploy-first-time.sh
Optional
In case you already have a cluster and a static IP. You can rebuild and deploy the containers with:
. deploy.sh
In case you want to re-deploy individual containers, run the following build scripts:
gcloud builds submit --config _cloudbuilder/chatserver.yaml
gcloud builds submit --config _cloudbuilder/web.yaml
To delete deployments use:
kubectl delete deployment web
To deploy another deployment:
cat _cloudbuilder/chatserver-deployment.yaml | envsubst | kubectl apply -f -
cat _cloudbuilder/web-deployment.yaml | envsubst | kubectl apply -f -
Local Components and Channels
The following steps will guide you to run all the various channels. These steps are optionally, in case you want to run it from your own machine.
Node JS Backend Server
As seen in the below architecture, all channels talk to the Node JS backend server. index.ts is the entry point. It sets all the API POST/GET routes. Each individual middleware script takes care of the implementation per channel Business Messages, Twilio, Web, AoG.
They are all handled the same way, as they talk to the Dialogflow ES/Dialogflow CX environment depending on the version that was set in config.ts. No matter which Dialogflow version you are using, the responses are flattent and normalized, so you can switch between versions. See dialogflow-bot-responses.ts.
<img src="https://github.com/GoogleCloudPlatform/dialogflow-ccai-omnichannel/blob/main/images/architecture.png" width="800" />- In order to run this demo on your local machine. You will need to have Node installed.
sudo n latest
sudo npm install -g npm@latestsudo
Here's a brief explanation of what each setting does:
ENV
- development or production, it contains logging and dev settingsVERTICAL
- this demo has fsi assetsGC_PROJECT_ID
- you need to enter your google cloud project IDAGENT_ID
- in case you are making use of an CX agent, you will need to enter the agent id, you can find this in https://dialogflow.cloud.google.com/cx/projects/ccai-360/locations (options button right from the display name > copy agent name), the agent id, is the end string of numbers and characters (the part that's after /agents/)CX_LOCATION
- the cx agent locationTWILIO_ACCOUNT_SID
- when you have followed the twilio steps in this guide, you should get the user SID from the CLITWILIO_ACCOUNT_TOKEN
- - when you have followed the twilio steps in this guide, you should get the password key secret from the CLIDOMAIN1
- in case you plan to deploy this demo to a new bought domainPUBSUB_TOPIC
- the pubsub topic for analyticsBQ_LOCATION
- the location of BigQueryDATASET
- the BigQuery Dataset nameTABLE
- the BigQuery Table nameBOT_PHONE_NUMBER_US
- in case you make use of an USA Twilio number, get the phonenumber from your console.twilio.comBOT_PHONE_NUMBER
- in case you make use of a non USA Twilio number, get the phonenumber from your console.twilio.comBOT_PHONE_NUMBER_DEV
- in case you bought an additional phone number for testing purposes, get the phonenumber from your console.twilio.comTEST_USER_DEV
- in case you want to test the Twilio phone gateway with a test user, without making use of the deployed website. Set this to the firebase Auth user ID that holds a phonenumber.
- Please review the following configurations that are unique for your environment:
backend/.env
backend/config.ts
- You can start the back-end server with the below command. It will start your app in development mode at http://localhost:8080
cd backend && npm run watch
Angular Website
In order to run this demo on your local machine. You will need to have Angular CLI installed
sudo npm install -g @angular/cli
You can start the Angular site with the below command. It will start your website at http://localhost:4200
cd web && ng serve
Flutter Mobile App
-
In order to run this demo on your local machine. You will need to have Flutter installed. Follow the install steps from: https://flutter.dev/docs/get-started/install
-
You will need have a service account key copied into the mobile/assets/ folder, and it should be named credentials.json. Download the service account key:
gcloud iam service-accounts keys create credentials.json \
--iam-account=$SA_EMAIL
- Compile and Run the Flutter app
cd mobile && flutter run
AoG Google Assistant Action
- In order to run this demo on your local machine. You will need to have gActions installed. Follow the install steps from: https://developers.google.com/assistant/conversational/df-asdk/actions-sdk/gactions-cli
Store gActions somewhere on your hard drive, and assign it to your PATH
environment variable.
- On Mac and Linux, run
chmod +x gactions
to make the binary executable. - On Windows, you must have Administrator rights.
Run gactions
from the CLI to update to the latest version.
https://developers.google.com/assistant/actionssdk/gactions
- Create a project in the Actions on Google console: https://actions-console.google.com/u/0/
- Choose project: [project-id]
- Language: English, USA
- pick Custom > Blank Project
Run fulfillment locally
- Install ngrok
npm install ngrok -g
ngrok http 8080
-
Modify actions.json to point to your local ngrok https URL
-
Once your Node backend server is deployed to the Google Cloud, you can take over those URLs
-
You can test your fulfillment by making an example post call
curl --header "Content-Type: application/json" \
--request POST \
--data '{"user":{"locale":"en-US","userVerificationStatus":"VERIFIED"},"conversation":{"conversationId":"ABwppHEjwDSpkSNHiGX76mlQAdMSUcsT9bVTrpPqs2xRTHSAz-UtlLY8_LS-bCgQWbraSrZLtFk","type":"NEW"},"inputs":[{"intent":"actions.intent.MAIN","rawInputs":[{"inputType":"VOICE","query":"Talk to CCAIDemo"}]}],"surface":{"capabilities":[{"name":"actions.capability.SCREEN_OUTPUT"},{"name":"actions.capability.AUDIO_OUTPUT"},{"name":"actions.capability.ACCOUNT_LINKING"},{"name":"actions.capability.MEDIA_RESPONSE_AUDIO"}]},"isInSandbox":true,"requestType":"SIMULATOR"}' \
https://www.conv.dev/api/aog/
Deploy your action
Run the following command:
cd assistant
gactions update --action_package action.json --project $GCP_PROJECT
It will ask you to visit a URL for authentication, and you will have to copy paste a generated key in the terminal.
Prepare for Demo
Create a RCE360 "G-Mortgages" Account:
- Either go to your local environment or the live environment and sign up for an account. This will store your info in Firebase. It's important that you are setting up a Gmail address and a phone number. In case you will demo the Google Assistant, make sure you use the same gmail address.
http://localhost:4200/profile / https://www.conv.dev/profile
-
Login on the G-Mortgages website http://localhost:4200 / https://www.conv.dev
-
Follow this flow in the web:
Welcome to G-Mortgages. I see that you have prequalified for a fixed rate mortgage. Would you like me to book an appointment?
> Yes
I can schedule a mortgage consultation with Mr. Gerrit Oogle. Monday the 8th at 2 pm. Would this work for you?
> Yeah that sounds good
Perfect, you'll have a mortgage appointment for Monday the 8th at 2 pm.
You will receive a text confirmation for the virtual appointment via text.
Is there anything else I can help you with?
> No that's it. Thanks.
- Get your cell phone, you should have received a text message:
Hi [Name]. Thank you for scheduling an appointment with G-Mortgages.
Your virtual consultation with a licensed broker has been set for tomorrow,
Wednesday the 8th at 2:00pm. Feel free to reply, if you want to re-schedule your appointment.
> Can we make it a half hour later?
We have that time available, I will update the appointment to 2:30pm. If you have any questions, you can use the Google Assistant to talk to G-Mortgages.
> Can you give me a call?
Sure. I will contact you on the number that's set in your profile.
- You will receive a phone call:
> Hello
Hi, I am the virtual assistant for G-Mortgages. You've asked me to give you a ring. How can I help?
> I am thinking of changing my job, will this affect my mortgage?
That’s exciting! Unfortunately, taking a job at a new company will affect your length of employment and may impact your application.
> Ok, now what?
I know it’s a tough situation, but if you could defer the start date until after the mortgage is approved that would help. Is there anything else I can help you with?
> What's an adjustable rate mortgage?
Adjustable rate mortgages typically have low initial interest rates. After the initial period, the rate adjusts to the current market conditions.
> Is it better than a fixed rate mortgage?
Well, it depends. How long do you plan to own the house?
> Maybe 30 years.
In that case I would suggest an adjustable rate mortgage. No sense in locking a rate for 30 years if you’re not going to live there that long. Is there anything else I can help you with?
> No thank you, good bye.
Thank you for contacting G-Mortgages. Have a nice day.
- To view the Looker Dashboard go to /dashboard
Insights
To demonstrate the Insights UI you will need to upload over 15K of conversations in the console. To learn more about the configuration see this blog article. I've create a simple data generation script, that generates example mortgage conversations (questions), which can be used to upload in a GCS bucket.
It might be the quickest (upload) solution when you run these commands/scripts from the Cloud Console terminal.
cd _insights/data-generation/
mkdir _temp
Modify the max
conversation count variable in generate-questions.js.
-
Generate example conversation data:
node generate-questions.js
-
Create a new GCS bucket, for example: insights-data
-
Copy all conversations to a bucket
gsutil cp *.json gs://insights-data
-
You can use the batch import tool to import all the chat conversations: https://cloud.google.com/contact-center/insights/docs/import
gsutil cp -r gs://dialogflow-docs-downloads/contact-center-insights-preview .
pip install virtualenv
python3 -m venv .
source bin/activate
Comment out (#
) protobuf in requirements.txt: nano requirements.txt
pip install protobuf
pip install -r requirements.txt
python3 import_conversations.py --source_chat_transcript_gcs_bucket insights-data my-project-id
Agent Assist
To demonstrate the Smart Reply feature in the Agent Assist UI you will need to upload over 30K of conversations in the console. You will use this to train a conversation model from a dataset, and it will auto complete / provide suggestions for the agent.
It might be the quickest (upload) solution when you run these commands/scripts from the Cloud Console terminal.
cd _agentassist/data-generation/
mkdir _temp
Modify the max
conversation count variable in generate-answers.js.
-
Generate example conversation data:
node generate-answers.js
-
Create a new GCS bucket, for example: agentassist-data
-
Copy all conversations to a bucket
cd _temp && gsutil cp *.json gs://agentassist-data
-
In the Agent Assist Console navigate to Create new, and point to your GCS bucket
gs://agentassist-data/*
, to import the dataset. It will make use of the Data Labeling API (which needs to be enabled).
Note, due to a bug in the Agent Assist UI, I am unable to submit the form. I had to manually edit the DOM to enable the Create button. This should be fixed in the future.
- Next, go to *Models, and create a Smart Reply model, by selecting your new data set. Training the model will take some time.