GSP311

Overview

In a challenge lab you’re given a scenario and a set of tasks. Instead of following step-by-step instructions, you will use the skills learned from the labs in the quest to figure out how to complete the tasks on your own! An automated scoring system (shown on this page) will provide feedback on whether you have completed your tasks correctly.

When you take a challenge lab, you will not be taught new Google Cloud concepts. You are expected to extend your learned skills, like changing default values and reading and researching error messages to fix your own mistakes.

To score 100% you must successfully complete all tasks within the time period!

This lab is recommended for students who have enrolled in the Automate Interactions with Contact Center AI quest. Are you ready for the challenge?

Topics tested

• Create a Cloud Storage bucket
• Create a Cloud Function
• Create a BigQuery dataset and table from an existing schema
• Create a Pub/Sub topic
• Deploy Dataflow pipeline
• Write SQL queries

Prerequisites

• Familiarity with AI/ML APIs on Google Cloud, source repositories, Pub/Sub, BigQuery and Dataflow.
• Familiarity to use text editors such as vim, emacs, or nano.

Setup

Before you click the Start Lab button

Read these instructions. Labs are timed and you cannot pause them. The timer, which starts when you click Start Lab, shows how long Google Cloud resources will be made available to you.

This hands-on lab lets you do the lab activities yourself in a real cloud environment, not in a simulation or demo environment. It does so by giving you new, temporary credentials that you use to sign in and access Google Cloud for the duration of the lab.

To complete this lab, you need:

• Access to a standard internet browser (Chrome browser recommended).

Note: Use an Incognito or private browser window to run this lab. This prevents any conflicts between your personal account and the Student account, which may cause extra charges incurred to your personal account.

• Time to complete the lab---remember, once you start, you cannot pause a lab.

Note: If you already have your own personal Google Cloud account or project, do not use it for this lab to avoid extra charges to your account.

How to start your lab and sign in to the Google Cloud Console

1. Click the Start Lab button. If you need to pay for the lab, a pop-up opens for you to select your payment method. On the left is the Lab Details panel with the following:

   • The Open Google Console button
   • Time remaining
   • The temporary credentials that you must use for this lab
   • Other information, if needed, to step through this lab

2. Click Open Google Console. The lab spins up resources, and then opens another tab that shows the Sign in page.

   Tip: Arrange the tabs in separate windows, side-by-side.

   Note: If you see the Choose an account dialog, click Use Another Account.

3. If necessary, copy the Username from the Lab Details panel and paste it into the Sign in dialog. Click Next.

4. Copy the Password from the Lab Details panel and paste it into the Welcome dialog. Click Next.

   Important: You must use the credentials from the left panel. Do not use your Google Cloud Skills Boost credentials. Note: Using your own Google Cloud account for this lab may incur extra charges.

5. Click through the subsequent pages:

   • Accept the terms and conditions.
   • Do not add recovery options or two-factor authentication (because this is a temporary account).
   • Do not sign up for free trials.

After a few moments, the Cloud Console opens in this tab.

Note: You can view the menu with a list of Google Cloud Products and Services by clicking the Navigation menu at the top-left.

Challenge scenario

Your company is ready to launch a new marketing campaign. For that, they would like to know what customers have been calling customer service about and what is the sentiment around those topics. They will be sending the data to a third-party for further analysis, so sensitive data such as customer's name, phone number, address, email, SSN, should all be redacted from the data. Right now all the data that they have is available as call recordings and they have not been processed/analyzed yet.

Your challenge

Your task, should you choose to accept it, is to augment an existing pipeline for speech processing by adding the redaction step, which will remove sensitive data from the audio transcriptions. The current pipeline processes audio, sending it to the STT (Speech-to-Text) API. It then calls the NLP (Natural Language Processing) API for the extraction of entities and the sentiment associated with them. It does all of that with the help of Pub/Sub and Dataflow. BigQuery is then used for analyzing it. You are tasked with creating a new BigQuery table by adding the DLP (Data Loss Prevention) API for data redaction, once all the data has been properly redacted and stored in BigQuery you have completed the challenge. Good luck!

Setting up the environment

• Clone the Speech Analysis Framework source repository from the Cloud Shell:

git clone https://github.com/GoogleCloudPlatform/dataflow-contact-center-speech-analysis.git

Task 1. Create a Cloud Storage bucket

• Create a Regional Cloud Storage bucket in with a Standard storage class for the Audio Uploads.

Create a Regional Cloud Storage bucket

Task 2. Create a Cloud function

Create a Cloud Function to automatically process audio files when files are uploaded to the storage bucket.

• In the cloned repo, within Cloud Shell, go to the saf-longrun-job-func directory and deploy the Cloud Function with following requirements:

• Function to execute: safLongRunJobFunc
• Region:
• Runtime: nodejs12
• Trigger Resource:
• Trigger Event: google.storage.object.finalize

Note: If the command fails because of an authentication error, try running it again. Create a Cloud Function

Task 3. Create a BigQuery dataset

• Create a BigQuery Dataset store the enriched raw audio data processed by the Speech-To-Text and Natural Language APIs. Cloud Dataflow will create the table for you.

Create a BigQuery dataset

Task 4. Create Cloud Pub/Sub topic

• Create Cloud Pub/Sub Topic for Cloud Functions to send a payload with the Speech-To-Text long-running job name and associated metadata.

Create a Pub/Sub topic

Task 5. Create a Cloud Storage bucket for staging contents

• Create a Regional Cloud Storage bucket in with a Standard storage class and create folder in the bucket for Dataflow staging.

Create a Regional Cloud Storage bucket with folder

Task 6. Deploy a Cloud dataflow pipeline

Deploy the Cloud Dataflow Pipeline that processes messages from Pub/Sub, extracts the Speech-To-Text transcript and enriches the data with the Natural Language API.

• In the cloned repo, within Cloud Shell, go to the saf-longrun-job-dataflow directory and deploy the Cloud Dataflow Pipeline with saflongrunjobdataflow.py Python script.

Hints for setting up the Python development environment:

docker run -it python:3.7 /bin/bash pip install apache-beam[gcp] pip install dateparser Note: If you get an error while deploying a dataflow pipeline, please disable dataflow API and enable it again. Deploy Dataflow pipeline

Task 7. Upload sample audio files for processing

• Upload the sample audio files into your Audio Uploads Bucket to be processed by Cloud Functions and Dataflow.

Hints for using gsutil:

Note: Your Python development environment container does not have gsutil. Type and enter exit it. # mono flac audio sample gsutil -h x-goog-meta-dlp:false -h x-goog-meta-callid:1234567 -h x-goog-meta-stereo:false -h x-goog-meta-pubsubtopicname:[TOPIC_NAME] -h x-goog-meta-year:2019 -h x-goog-meta-month:11 -h x-goog-meta-day:06 -h x-goog-meta-starttime:1116 cp gs://qwiklabs-bucket-gsp311/speech_commercial_mono.flac gs://[YOUR_UPLOADED_AUDIO_FILES_BUCKET_NAME] # stereo wav audio sample gsutil -h x-goog-meta-dlp:false -h x-goog-meta-callid:1234567 -h x-goog-meta-stereo:true -h x-goog-meta-pubsubtopicname:[TOPIC_NAME] -h x-goog-meta-year:2019 -h x-goog-meta-month:11 -h x-goog-meta-day:06 -h x-goog-meta-starttime:1116 cp gs://qwiklabs-bucket-gsp311/speech_commercial_stereo.wav gs://[YOUR_UPLOADED_AUDIO_FILES_BUCKET_NAME]

Test your understanding

Once the audio uploaded data has been processed by Cloud Dataflow and Cloud Functions, you can see its output in your provided BigQuery dataset table.

• Please run query against generated data to answer the following question:

Hint: You will need to use UNNEST to count the entities in BigQuery. The result set might take 5 to 10 minutes to populate before you will be able to answer the following question.

Process the sample audio files

Task 8. Run a data loss prevention job

Our data will be in the Streaming buffer. Due to this, to test DLP during the lab, it is best to query all data and set a destination table for query results.

• On the copied table run Scan with DLP.

Hint: To get started, you can select Scan with DLP in the BigQuery console.

Check out the Using Cloud DLP to scan BigQuery data guide to get some hints on how to use Cloud DLP to scan BigQuery data.

Run a Data Loss Prevention Job

Congratulations!

Awesome job completing this challenge lab!

In this lab, you built a solution that allows you to analyze audio files via Cloud Speech-to-Text and Cloud Natural Language. The solution also employed Cloud Pub/Sub, Cloud Dataflow for data streaming and transformation and ran a Data Loss Prevention job to identify sensitive data. The resulting application is completely serverless and your company is now ready to launch a new marketing campaign.

Earn Your next skill badge

This self-paced lab is part of the Automate Interactions with Contact Center AI skill badge quest. Completing this skill badge quest earns you the badge above, to recognize your achievement. Share your badge achievement with your network and on your resume and social platforms, and announce your accomplishment using #GoogleCloudBadge.

This skill badge quest is part of Google Cloud’s Data Scientist/ML Engineer and Contact Center Engineer learning paths.

Google Cloud training and certification

...helps you make the most of Google Cloud technologies. Our classes include technical skills and best practices to help you get up to speed quickly and continue your learning journey. We offer fundamental to advanced level training, with on-demand, live, and virtual options to suit your busy schedule. Certifications help you validate and prove your skill and expertise in Google Cloud technologies.

Manual Last Updated March 14, 2023

Manual Last Tested March 14, 2023

Copyright 2023 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.