Overview

Google App Engine lets you manage resources from the command line, debug source code in production and run API backends. This lab concentrates on the backend service, putting together Pub/Sub, Natural Language, and Spanner services and APIs to collect and analyze feedback and scores from an online Quiz application.

Objectives

In this lab, you perform the following tasks:

• Create and publish messages to a Cloud Pub/Sub topic.
• Subscribe to the topic to receive messages in a separate worker application.
• Use the Cloud Natural Language Machine Learning API.
• Create and configure a Cloud Spanner database instance, then insert data into the database.

Setup and requirements

For each lab, you get a new Google Cloud project and set of resources for a fixed time at no cost.

1. Sign in to Qwiklabs using an incognito window.

2. Note the lab's access time (for example, 1:15:00), and make sure you can finish within that time.
   There is no pause feature. You can restart if needed, but you have to start at the beginning.

3. When ready, click Start lab.

4. Note your lab credentials (Username and Password). You will use them to sign in to the Google Cloud Console.

5. Click Open Google Console.

6. Click Use another account and copy/paste credentials for this lab into the prompts.
   If you use other credentials, you'll receive errors or incur charges.

7. Accept the terms and skip the recovery resource page.

Activate Google Cloud Shell

Google Cloud Shell is a virtual machine that is loaded with development tools. It offers a persistent 5GB home directory and runs on the Google Cloud.

Google Cloud Shell provides command-line access to your Google Cloud resources.

1. In Cloud console, on the top right toolbar, click the Open Cloud Shell button.

   

2. Click Continue.

It takes a few moments to provision and connect to the environment. When you are connected, you are already authenticated, and the project is set to your PROJECT_ID. For example:

gcloud is the command-line tool for Google Cloud. It comes pre-installed on Cloud Shell and supports tab-completion.

• You can list the active account name with this command:

gcloud auth list

Output:

Credentialed accounts: - @.com (active)

Example output:

Credentialed accounts: - google1623327_student@qwiklabs.net

• You can list the project ID with this command:

gcloud config list project

Output:

[core] project =

Example output:

[core] project = qwiklabs-gcp-44776a13dea667a6 Note: Full documentation of gcloud is available in the gcloud CLI overview guide .

Task 1. Prepare the Quiz application

In this section, you access Cloud Shell and enter commands to:

• Clone the git repository containing the Quiz application
• Configure environment variables
• Run the application

Clone source code in Cloud Shell

1. Clone the repository for the lab:

   git clone https://github.com/GoogleCloudPlatform/training-data-analyst

2. Create a soft link as a shortcut to your working directory:

   ln -s ~/training-data-analyst/courses/developingapps/v1.3/python/pubsub-languageapi-spanner ~/pubsub-languageapi-spanner

Configure and run the Quiz application

In this section you'll open two Cloud Shell windows, one for the web part of the Quiz application, the other for the worker part of the application that handles the console.

1. Change the working directory:

   cd ~/pubsub-languageapi-spanner/start

2. Update the App Engine region using the sed command.

   export REGION={{{project_0.startup_script.app_region|REGION}}} sed -i "s/us-central/$REGION/g" prepare_web_environment.sh

3. Run a script to configure the web application:

   . prepare_web_environment.sh

   This script file:

   • Creates and configures a GCP Service Account to provide access to GCP services for your application.
   • Creates an App Engine application.
   • Exports environment variables GCLOUD_PROJECT, GCLOUD_BUCKET, and GOOGLE_APPLICATION_CREDENTIALS.
   • Creates a virtualenv isolated Python environment for Python 3 and activates it.
   • Updates pip and runs pip install -r requirements.txt.
   • Creates entities in Cloud Datastore.
   • Prints out the Google Cloud Platform Project ID.

4. Run the web application:

   python run_server.py

   The application is running when you see a message similar to the example output:

   Example output:

   * Running on http://127.0.0.1:8080/ (Press CTRL+C to quit) * Restarting with stat * Debugger is active! * Debugger PIN: 110-448-781 Note: If you get an error indicating that the service account does not have storage.buckets.get access to the Google Cloud Storage bucket, permissions may not yet have propagated to the service account. Run the python command until it indicates that the script is running.

5. Click Open a new tab (+) on the right of the Cloud Shell tab to add a second Cloud Shell window. This window runs the Worker (console) application.

6. In the second window, change the working directory:

   cd ~/pubsub-languageapi-spanner/start

7. Run the worker application in the second Cloud Shell window:

   . run_worker.sh

   This script file:

   • Exports environment variables GCLOUD_PROJECT, GCLOUD_BUCKET, and GOOGLE_APPLICATION_CREDENTIALS.
   • Activates a virtualenv environment for Python 3.
   • Runs the worker application python -m quiz.console.worker.

Check out the Quiz application

1. In Cloud Shell, click Web preview > Preview on port 8080 to preview the Quiz application.

   

2. In the navigation bar, click Take Test.

3. Click Places.

4. Answer the question.

   After you answer the question, you should see a final screen inviting you to submit feedback.

   You can put information in the form, but the Send Feedback button does not yet work.

5. Return to the first Cloud Shell window, and press Ctrl+C to stop the web application.

Task 2. Examine the Quiz application code

In this lab you'll view and edit files. You can use the shell editors that are installed on Cloud Shell, such as nano or vim or the Cloud Shell code editor.

This lab uses the Cloud Shell code editor to review the Quiz application code.

Launch the Cloud Shell code editor

1. From Cloud Shell, click Open Editor to launch the code editor.

2. Then click Open in new window.

Review the GCP application code structure

1. Navigate to the /pubsub-languageapi-spanner/start folder using the file browser panel on the left side of the editor.

2. Select the pubsub.py file in the .../quiz/gcp folder.

   This file contains a module that allows applications to publish feedback messages to a Cloud Pub/Sub topic and register a callback to receive messages from a Cloud Pub/Sub subscription.

3. Select the languageapi.py file in the .../quiz/gcp folder.

   This file contains a module that allows users to send text to the Cloud Natural Language ML API and to receive the sentiment score from the API.

4. Select the spanner.py file.

   This file contains a module that allows users to save the feedback and Natural Language API response data in a Cloud Spanner database instance.

Review the web application code

1. Select the api.py file in the .../quiz/api folder.

   The handler for POST messages sent to the /api/quizzes/feedback/:quiz route publishes the feedback data received from the client to Pub/Sub.

2. Select the worker.py file in the .../quiz/console folder.

   This file runs as a separate console application to consume the messages delivered to a Pub/Sub subscription.

Task 3. Work with Cloud Pub/Sub

In this section, you create a Cloud Pub/Sub topic and subscription in your Google Cloud Platform project, publish a message, and retrieve it.

Create a Cloud Pub/Sub topic

1. In the Cloud Platform Console, click Navigation menu > Pub/Sub > Topics.

   

2. Click +Create Topic.

3. For Topic ID, type feedback, and then click Create.

Create a Cloud Pub/Sub subscription

1. Return to the second Cloud Shell window and press Ctrl+C to stop the application.

2. Create a Cloud Pub/Sub subscription named worker-subscription against the feedback topic:

   gcloud pubsub subscriptions create worker-subscription --topic feedback

Note: If you receive an error about the active account not having valid credentials, wait for a minute and try the command again.

Publish a message to a Cloud Pub/Sub topic

• Publish a "Hello World" message into the feedback topic:

gcloud pubsub topics publish feedback --message "Hello World"

Retrieve a message from a Cloud Pub/Sub subscription

• Now pull the message from the feedback topic with automatic acknowledgement of the message:

gcloud pubsub subscriptions pull worker-subscription --auto-ack

Output:

DATA: Hello World MESSAGE_ID: 3944934635874333 ORDERING_KEY: ATTRIBUTES: DELIVERY_ATTEMPTS:

Task 4. Publish messages to Cloud Pub/Sub programmatically

Write code to publish messages to Cloud Pub/Sub

Note: Update code within the sections marked as follows:

# TODO

# END TODO

To maximize your learning, review the code, inline comments, and related API documentation.

Import and use the Python Cloud Pub/Sub module

In this section, you'll update ...quiz/gcp/pubsub.py to do the following:

1. Open the ...quiz/gcp/pubsub.py file in the editor.

2. Load the pubsub_v1 module from the google.cloud package.

quiz/gcp/pubsub.py

# TODO: Load the Cloud Pub/Sub module from google.cloud import pubsub_v1 # END TODO

3. Construct a Cloud Pub/Sub Publisher client:

# TODO: Create a Pub/Sub Publisher Client publisher = pubsub_v1.PublisherClient() # END TODO

4. Get the fully qualified path referencing the feedback Pub/Sub topic you created earlier:

# TODO: Create Topic Object to reference feedback topic topic_path = publisher.topic_path(project_id, 'feedback') # END TODO Note: The Cloud Shell Editor is not configured to use the same version of python that is being used for the application, so lint may report some false errors.

Write code to publish a message to Cloud Pub/Sub

1. In the publish_feedback(feedback) function, publish a message to the feedback topic.

quiz/gcp/pubsub.py, publish_feedback(feedback) function

# TODO: Publish the feedback object to the feedback topic payload = json.dumps(feedback, indent=2, sort_keys=True) data = payload.encode('utf-8') future = publisher.publish(topic_path, data=data) return future.result() # END TODO

2. Save the file.

Write code to use the Pub/Sub publish functionality

1. In the .../quiz/api/api.py file, load the pubsub module from the quiz.gcp package.

quiz/api/api.py

# TODO: Add pubsub to import list from quiz.gcp import datastore, pubsub # END TODO

2. In the publish_feedback(feedback) function, remove the placeholder pass statement, invoke the pubsub.publish_feedback(feedback) function, and return a response to the client indicating that feedback was received.

quiz/api/api.py, publish_feedback(feedback) function

# TODO: Publish the feedback using your pubsub module, # return the result result = pubsub.publish_feedback(feedback) response = Response(json.dumps(result, indent=2, sort_keys=True)) response.headers['Content-Type'] = 'application/json' return response # END TODO

3. Save the file.

Run the application and create a Pub/Sub message

1. In the first Cloud Shell tab, restart the web application (if it is running, stop and start it).

2. Preview the web application.

3. Click Take Test.

4. Click Places.

5. Answer the question, select the rating, enter some feedback text, and click Send Feedback.

6. In the second Cloud Shell window, to pull a message from the worker-subscription, execute the following command:

   gcloud pubsub subscriptions pull worker-subscription --auto-ack

   Output:

   DATA: { "email": "app.dev.student"example.org", "feedback": "too easy!", "quiz": "places", "rating": "3", "timestamp": 1645579593499 } MESSAGE_ID: 3945028363760545 ORDERING_KEY ATTRIBUTES: DELIVERY_ATTEMPT

7. Stop the web application in the first Cloud Shell tab.

Task 5. Subscribe to Cloud Pub/Sub topics programmatically

In this section you write code to create a subscription to a Cloud Pub/Sub topic and receive message notifications in the worker console application.

Write code to create a Cloud Pub/Sub subscription and receive messages

1. Return to the ...quiz/gcp/pubsub.py file.

2. Declare a Cloud Pub/Sub Subscriber Client.

/quiz/gcp/pubsub.py

# TODO: Create a Pub/Sub Subscriber Client sub_client = pubsub_v1.SubscriberClient() # END TODO

3. Get the fully qualified path referencing the 'worker-subscription':

# TODO: Create a Subscription object named # worker-subscription sub_path = sub_client.subscription_path(project_id, 'worker-subscription') # END TODO

4. Move to the pull_feedback(callback) function, and remove the placeholder pass statement.

5. Use the subscriber client to subscribe to the worker subscription, invoking the callback when a message is received.

/quiz/gcp/pubsub.py, pull_feedback(callback) function

# TODO: Subscribe to the worker-subscription, # invoking the callback sub_client.subscribe(sub_path, callback=callback) # END TODO

6. Save the file.

Write code to use the Pub/Sub subscribe functionality

1. In the ...quiz/console/worker.py file, load the pubsub module from the quiz.gcp package.

console/worker.py

# TODO: Load the pubsub, languageapi and spanner modules from # the quiz.gcp package from quiz.gcp import pubsub # END TODO

2. In the pubsub_callback(message) function, acknowledge the message.

console/worker.py, pubsub_callback(message) function

# TODO: Acknowledge the message message.ack() # END TODO

3. Log the message to the console:

# TODO: Log the message log.info('Message received') log.info(message) # END TODO

4. In the main() function, register the handler function as the Pub/Sub subscription callback.

console/worker.py, main() function

# TODO: Register the callback pubsub.pull_feedback(pubsub_callback) # END TODO

5. Save the file.

Run the web and worker applications and create a Pub/Sub message

1. In the first Cloud Shell window, start the web application if it's not already running:

   python run_server.py

2. In the second Cloud Shell window, start the worker application:

   . run_worker.sh

3. In Cloud Shell, click Web preview > Preview on port 8080 to preview the quiz application.

4. Click Take Test.

5. Click Places.

6. Answer the question, select the rating, enter some feedback text, and then click Send Feedback.

7. Return to the second Cloud Shell window.

   The worker application should show that it has received the feedback message via its handler and displayed details in the window. The message itself is truncated.

   INFO:root:Worker starting... INFO:root:Message received INFO:root:Message { data: b'{\n "email": "app.dev.student@example.org",\n "fee...' ordering_key: '' attributes: {} }

8. Stop the web and console applications.

Task 6. Use the Cloud Natural Language API

In this section you write the code to perform sentiment analysis on the feedback text submitted by the user.

Note: For more information, refer to the Google Cloud Natural Language API reference.

Write code to invoke the Cloud Natural Language API

1. In the editor, move to the top of the ...quiz/gcp/languageapi.py file.

2. Load the language module from the google.cloud package.

quiz/gcp/languageapi.py

# TODO: Import the language module from google.cloud import language_v1 # END TODO

3. Create a Cloud Natural Language client object:

# TODO: Create the Language API client lang_client = language_v1.LanguageServiceClient() # END TODO

4. Move to the analyze(text) function, and create a Document object to pass to the Natural Language client, assigning the feedback text to the content parameter.

quiz/gcp/languageapi.py, analyze(text) function

# TODO: Create a Document object doc = language_v1.types.Document(content=text, type_='PLAIN_TEXT') # END TODO

5. Use the Natural Language client object to analyze the sentiment of the document:

# TODO: Analyze the sentiment sentiment = lang_client.analyze_sentiment( document=doc).document_sentiment # END TODO

6. Return the sentiment score from the Natural Language API:

# TODO: Return the sentiment score return sentiment.score # END TODO

7. Save the file.

Write code to use the Natural Language API functionality

1. In the .../quiz/console/worker.py file, add languageapi to the import statement.

console/worker.py

# TODO: Load the pubsub, languageapi and spanner modules from # from the quiz.gcp package from quiz.gcp import pubsub, languageapi # END TODO

2. In the pubsub_callback(message) function, and after the existing code, perform sentiment detection on the feedback.

console/worker.py, pubsub_callback(message) function

# TODO: Use the languageapi module to # analyze the sentiment score = languageapi.analyze(str(data['feedback'])) # END TODO

3. Log the score to the console:

# TODO: Log the sentiment score log.info('Score: {}'.format(score)) # END TODO

4. Assign a new score property to the feedback object:

# TODO: Assign the sentiment score to # a new score property data['score'] = score # END TODO

5. Save the file.

Run the web and worker application and test the Natural Language API

1. Return to the first Cloud Shell window.
2. Run the web application.
3. Switch to the second Cloud Shell window.
4. Restart the worker application.
5. Preview the web application.
6. Click Take Test.
7. Click Places.
8. Answer the questions, select the rating, enter some feedback text, and then click Send Feedback.
9. Return to the second Cloud Shell window.

You should see that the worker application has invoked the Cloud Natural Language API and displayed the sentiment score in the console.

10. Stop the web and console applications.

Task 7. Persist data to Cloud Spanner

In this section you create a Cloud Spanner instance, database, and table. Then you write the code to persist the feedback data into the database.

Create a Cloud Spanner instance

1. Return to the Cloud Console and open the Navigation menu > View All Products. In the Databases section, click > Spanner.
2. Click Create a Provisioned Instance.
3. For Instance name, type quiz-instance.
4. In the Configuration section, select as the region.
5. Click Create.

Create a Cloud Spanner database and table

1. On the Instance Details page for quiz-instance, click Create database.
2. For Database name, type quiz-database.
3. Under Define your schema, type the following SQL statement:

CREATE TABLE Feedback ( feedbackId STRING(100) NOT NULL, email STRING(100), quiz STRING(20), feedback STRING(MAX), rating INT64, score FLOAT64, timestamp INT64 ) PRIMARY KEY (feedbackId);

4. Click Create.

Write code to persist data into Cloud Spanner

1. Return to the code editor, and move to the top of the .../quiz/gcp/spanner.py file.

2. Load the spanner module from the google.cloud package.

quiz/gcp/spanner.py

# TODO: Import the spanner module from google.cloud import spanner # END TODO

3. Construct a Cloud Spanner client:

# TODO: Create a spanner Client spanner_client = spanner.Client() # END TODO

4. Get a reference to the Spanner instance:

# TODO: Get a reference to the Cloud Spanner quiz-instance instance = spanner_client.instance('quiz-instance') # END TODO

5. Get a reference to the Spanner database:

# TODO: Get a reference to the Cloud Spanner quiz-database database = instance.database('quiz-database') # END TODO

6. Move to the saveFeedback(data) function.

7. Create a database.batch object using a with block. This can be used to perform multiple operations against a Spanner database.

quiz/gcp/spanner.py, save_feedback(data) function

# TODO: Create a batch object for database operations with database.batch() as batch: # END TODO

8. Create a key for the feedback record from the email, quiz, and timestamp properties from the data:

# TODO: Create a key for the record # from the email, quiz and timestamp feedback_id = '{}_{}_{}'.format( reverse_email(data['email']), data['quiz'], data['timestamp']) # END TODO

The reverse_email(...) function takes the input email and creates a reversed string that is used as a primary key. For example, app.dev.student@example.com is converted to com_example_student_dev_app.

9. Use the batch object to insert a record, using a set of columns and values:

# TODO: Use the batch to insert a record # into the feedback table # This needs the columns and values batch.insert( table='feedback', columns=( 'feedbackId', 'email', 'quiz', 'timestamp', 'rating', 'score', 'feedback' ), values=[ ( feedback_id, data['email'], data['quiz'], data['timestamp'], data['rating'], data['score'], data['feedback'] ) ] ) # END TODO

10. Save the file.

Write code to use the Cloud Spanner functionality

1. In the .../quiz/console/worker.py file, load the spanner module.

quiz/console/worker.py

# TODO: Load the pubsub, languageapi and spanner modules # from the quiz.gcp package from quiz.gcp import pubsub, languageapi, spanner # END TODO

2. After the existing code in the pubsub_callback(message) function, save the feedback into Spanner.

quiz/console/worker.py, pubsub_callback(message) function

# TODO: Use the spanner module to save the feedback spanner.save_feedback(data) # END TODO

3. Log a message to the console to say that the feedback has been saved:

# TODO: Log a message to say the feedback # has been saved log.info('Feedback saved') # END TODO

4. Save the file.

Run the web and worker application and test Cloud Spanner

1. Save all the files, and then restart the web application in the first Cloud Shell window.
2. Restart the worker application in the second Cloud Shell window.
3. Preview the web application.
4. Click Take Test > Places.
5. Answer the questions, select the rating, enter some feedback text, and then click Send Feedback.
6. Return to the second Cloud Shell window.

You should see that the worker application has invoked the Cloud Spanner API and displayed the message in the console window.

7. Return to the Cloud Platform Console and click Navigation menu > View All Products > Spanner.
8. Select quiz-instance > quiz-database > Spanner Studio.
9. Click the blue + icon to open the Query page.
10. Click Clear Query, paste the query below, and then click Run:

SELECT * FROM Feedback

You should see the new feedback record in the Cloud Spanner database, including the message data from Cloud Pub/Sub and the score from the Cloud Natural Language API.

End your lab

When you have completed your lab, click End Lab. Google Cloud Skills Boost removes the resources you’ve used and cleans the account for you.

You will be given an opportunity to rate the lab experience. Select the applicable number of stars, type a comment, and then click Submit.

The number of stars indicates the following:

• 1 star = Very dissatisfied
• 2 stars = Dissatisfied
• 3 stars = Neutral
• 4 stars = Satisfied
• 5 stars = Very satisfied

You can close the dialog box if you don't want to provide feedback.

For feedback, suggestions, or corrections, please use the Support tab.

Copyright 2022 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.