GSP976

Introduction

Site Reliability Engineers (SRE) have a broad set of responsibilities, of which, managing incidents is a critical part of their role. In this lab, you will learn how to take advantage of the integrated capabilities of Google Cloud's operations suite to observe latency of applications deployed and set Service Level Indicators and Objectives to resolve such latencies.

Objectives

In this lab, you will learn how to:

1. Define a Latency SLI and SLO for a microservices based application

2. Select appropriate SLIs/SLOs for an application

3. Implement SLIs using Cloud Monitoring features

4. Troubleshoot and resolve application issues

5. Use Cloud Trace to identify software issues

Setup

Before you click the Start Lab button

Read these instructions. Labs are timed and you cannot pause them. The timer, which starts when you click Start Lab, shows how long Google Cloud resources will be made available to you.

This hands-on lab lets you do the lab activities yourself in a real cloud environment, not in a simulation or demo environment. It does so by giving you new, temporary credentials that you use to sign in and access Google Cloud for the duration of the lab.

To complete this lab, you need:

• Access to a standard internet browser (Chrome browser recommended).

Note: Use an Incognito or private browser window to run this lab. This prevents any conflicts between your personal account and the Student account, which may cause extra charges incurred to your personal account.

• Time to complete the lab---remember, once you start, you cannot pause a lab.

Note: If you already have your own personal Google Cloud account or project, do not use it for this lab to avoid extra charges to your account.

How to start your lab and sign in to the Google Cloud Console

1. Click the Start Lab button. If you need to pay for the lab, a pop-up opens for you to select your payment method. On the left is the Lab Details panel with the following:

   • The Open Google Console button
   • Time remaining
   • The temporary credentials that you must use for this lab
   • Other information, if needed, to step through this lab

2. Click Open Google Console. The lab spins up resources, and then opens another tab that shows the Sign in page.

   Tip: Arrange the tabs in separate windows, side-by-side.

   Note: If you see the Choose an account dialog, click Use Another Account.

3. If necessary, copy the Username from the Lab Details panel and paste it into the Sign in dialog. Click Next.

4. Copy the Password from the Lab Details panel and paste it into the Welcome dialog. Click Next.

   Important: You must use the credentials from the left panel. Do not use your Google Cloud Skills Boost credentials. Note: Using your own Google Cloud account for this lab may incur extra charges.

5. Click through the subsequent pages:

   • Accept the terms and conditions.
   • Do not add recovery options or two-factor authentication (because this is a temporary account).
   • Do not sign up for free trials.

After a few moments, the Cloud Console opens in this tab.

Note: You can view the menu with a list of Google Cloud Products and Services by clicking the Navigation menu at the top-left.

Activate Cloud Shell

Cloud Shell is a virtual machine that is loaded with development tools. It offers a persistent 5GB home directory and runs on the Google Cloud. Cloud Shell provides command-line access to your Google Cloud resources.

1. Click Activate Cloud Shell at the top of the Google Cloud console.

When you are connected, you are already authenticated, and the project is set to your PROJECT_ID. The output contains a line that declares the PROJECT_ID for this session:

Your Cloud Platform project in this session is set to YOUR_PROJECT_ID

gcloud is the command-line tool for Google Cloud. It comes pre-installed on Cloud Shell and supports tab-completion.

2. (Optional) You can list the active account name with this command:

gcloud auth list

3. Click Authorize.

4. Your output should now look like this:

Output:

ACTIVE: * ACCOUNT: student-01-xxxxxxxxxxxx@qwiklabs.net To set the active account, run: $ gcloud config set account `ACCOUNT`

5. (Optional) You can list the project ID with this command:

gcloud config list project

Output:

[core] project = <project_ID>

Example output:

[core] project = qwiklabs-gcp-44776a13dea667a6 Note: For full documentation of gcloud, in Google Cloud, refer to the gcloud CLI overview guide.

Scenario

Your organization has deployed a multi-tier microservices application. It is a web-based e-commerce application called "Hipster Shop", where users can browse for vintage items, add them to their cart and purchase them. Hipster Shop is composed of many microservices, written in different languages, that communicate via gRPC and REST APIs. The architecture of the deployment is optimized for learning purposes and includes modern technologies as part of the stack: Kubernetes, Istio, Cloud Operations, App Engine, gRPC, OpenTelemetry, and similar cloud-native technologies.

As a member of the Site Reliability Engineering (SRE) team, you are contacted when end users report severe latency experienced when browsing the website. You will explore the various services deployed to determine the root cause of the issue and set up a Service Level Objective (SLO) to prevent similar incidents from occurring in the future.

Task 1. Develop sample SLOs and SLIs

The below resources will provide you with a good understanding of Service Level Objectives and can be used to help you define the SLIs and SLOs of your applications. In this lab, given that the application itself serves end user ecommerce traffic, it's going to be very important that user experience remains constant and that performance is good. Monitor SLIs for request latency, error rate, throughput, and availability.

Review the introduction to the chapter on Service Level Objectives from the Site Reliability Engineering book:

"It's impossible to manage a service correctly, let alone well, without understanding which behaviors really matter for that service and how to measure and evaluate those behaviors. To this end, we would like to define and deliver a given level of service to our users, whether they use an internal API or a public product.

We use intuition, experience, and an understanding of what users want to define service level indicators (SLIs), objectives (SLOs), and agreements (SLAs). These measurements describe basic properties of metrics that matter, what values we want those metrics to have, and how we'll react if we can't provide the expected service. Ultimately, choosing appropriate metrics helps to drive the right action if something goes wrong, and also gives an SRE team confidence that a service is healthy.

An SLI is a service level indicator—a carefully defined quantitative measure of some aspect of the level of service that is provided.

Most services consider request latency — how long it takes to return a response to a request — as a key SLI. Other common SLIs include the error rate, often expressed as a fraction of all requests received, and system throughput, typically measured in requests per second. Another kind of SLI important to SREs is availability, or the fraction of the time that a service is usable. It is often defined in terms of the fraction of well-formed requests that succeed. Durability — the likelihood that data is retained over a long period of time — is equally important for data storage systems. The measurements are often aggregated: i.e., raw data is collected over a measurement window and then turned into a rate, average, or percentile."

Application architecture

It's impossible to develop SLIs without understanding how the application is built. Details are in the original repository, but for this lab, it suffices to understand that:

• Users access the application through the Frontend.

• Purchases are handled by CheckoutService.

• The CheckoutService depends on CurrencyService to handle conversions.

• Other services such as RecommendationService, ProductCatalogService, and Adservice are used to provide the frontend with content needed to render the page.

Service Level Indicators (SLIs) and Objectives (SLOs)

The following SLIs and SLOs are selected based on the end-user experience and the theoretical impact to users and business objectives:

SLI	Metric	Description	SLO
Request latency	Front end latency	Measures how long a user is waiting for the page to load. A high latency typically correlates to a negative user experience	99% of requests from the previous 60 minute period are serviced in under 3 seconds
Error rate	Front end error rate	Measures the error rate experienced by users. A high error rate likely indicates an issue.	0 Errors in the previous 60 minute period
	Checkout error rate	Measures the error rate experienced by other services calling the checkout service. A high error rate likely indicates an issue.	0 Errors in the previous 60 minute period
	Currency Service error rate	Measures the error rate experienced by other services calling the currency service. A high error rate likely indicates an issue.	0 Errors in the previous 60 minute period
Availability	Front end success rate	Measures the rate of successful requests as a way to determine the availability of the service. A low success rate likely indicates that users are having a poor experience.	99% of requests are successful over the previous 60 minute period

Task 2. Configure Latency SLI

Create an alerting policy for the Latency SLO. The metrics required to create a Latency SLI are already being collected by default.

Front End Latency

1. From the GCP console, navigate to Navigation Menu > Monitoring > Alerting.

2. Click Create Policy.

3. Click Select a Metric. Next, you will specify the metric and condition to use to trigger the Alerting Policy.

Note: The condition lets you know when you're experiencing performance issues that are impacting user experience.

As described in the Service Level Indicators and Objectives table above, you will use the 99th percentile front end latency as the SLI.

4. Add the following into the Find resource type and metric field then select the following from the dropdown menu:

custom.googleapis.com/opencensus/grpc.io/client/roundtrip_latency

If the metric is not found, wait a minute and then try again.

5. Click Global > Custom and select the Metric, then click Apply

6. Next, set the Time series aggregation to 99th percentile.

7. Next, in the Configure alert trigger area, set the options as follows:

   • Alert triggers if Any time series violates
   • Threshold position: Above threshold
   • Threshold value: 200

8. Click Next and disable the Use notification channel.

9. Name the alerting policy Latency Policy in the Alert policy name field.

10. Click Next and Create Policy.

Click Check my progress to verify the objective. Configure Latency SLI

Task 3. Browse the web site

Now that the Latency SLI has been configured and an alerting policy is set, you will go to the website to trigger a Latency SLI violation.

1. Navigate to Navigation Menu > Kubernetes Engine > Services & Ingress.

2. Find the service frontend-external and click on the external IP provided under the Endpoints column.

This will open a browser window that leads you to the web site.

3. Refresh the page at least ten times to generate data with latency issue present.

Note: As reported by end users, you will notice extreme latency loading the landing page of the website. Wait until the website loads before continuing to the next steps in this lab.

Similarly, you can configure an SLI, SLO and Burn rate alerts for the services availability. See Step By Step SLO creation guide using Cloud Operation Sandbox. In the next lab, we will learn to configure SLIs and SLOs for additional user journeys.

Task 4. Latency SLO violation - diagnose the problem

To diagnose the latency issue observed by the SRE team and reported by end users of the website, you will leverage Cloud Monitoring Application Performance Management (APM) tools to identify and resolve the issue causing poor application latency.

1. Navigate to Cloud Monitoring. Navigation Menu > Monitoring > Alerting.

Note: A Latency Policy incident should appear under the Incidents section once in this view. If not present, wait a few minutes for it to appear before proceeding.

2. Select the alert in the Incidents section by clicking on the text under the Incident summary column.

On the Incident details page, you will be presented with specific information about the incident and what caused it to occur.

To further diagnose the issue, the SRE team decides to utilize the Trace tool to understand why there is latency when entering the website on its landing page.

3. Navigate to Navigation Menu > Trace in the Google Cloud Console. This will take you to the Trace List view in the console.

4. In the Trace List view, enable the Auto Reload feature in the top right corner of the screen and ensure that the window is set to 1 hour.

5. Under the Select a trace section, select the most recent trace with the URI of Span.server-frontend. This will let you see the trace details on the bottom of the page.

Note: If you have a hard time finding the trace with the URI of Span.server-frontend you can also use the right hand table of traces and Sort Descending. You may also need to go to the second page of the table to see the Span.server-frontend trace. Select it in the table once you can see it.

6. Down at the bottom of the page in the Selected trace details section, you will see the call stack of the Span.server-frontend.

Notice that the CurrencyService's Convert method is called multiple times and is taking a long time to complete. You can conclude that the issue is caused by multiple calls to the Convert method of the CurrencyService upon loading the products on the website's landing page.

Task 5. Identify opportunities for application improvement

In order to improve the application, the SRE team decides to identify opportunities to improve the application and make it run faster while using less compute resources. To do so, you will use Cloud Profiler to investigate the currencyservice and determine what can be done to improve its CPU consumption.

1. Navigate to Navigation Menu > Profiler.

2. In the Profiler view, set the following values:

• Service: productcatalogservice

• Profile type: Heap

3. Hover over the top of the sync.(*Once).doSlow call stack and select it by clicking on it.

Note: If you cannot see the call stack try increasing the value set for the Timespan towards the top right hand side of the screen.

From the length of the bar and its percentage of call time it can be deduced that the doSlow method may be causing the application to respond slowly.

Task 6. Remediate the issue

After utilizing Cloud Trace and Profiler the SRE team was able to pinpoint exactly what application code was causing latency on the website. The development team issues a fix for the code which you will deploy below:

1. Open Cloud Shell and clone the following git repository:

git clone --depth 1 --branch csb_1220 https://github.com/GoogleCloudPlatform/cloud-ops-sandbox.git

2. Change directories to the following:

cd ~/cloud-ops-sandbox/sre-recipes

3. Navigate to Navigation Menu > Kubernetes Engine > Clusters. Select the three dots ( ) to the right of the cloud-ops-sandbox cluster and select the option to Connect.

4. On the Connect to the cluster modal dialog, click the RUN IN CLOUD SHELL button.

5. Press Enter to run the command once populated in Cloud Shell.

6. Run the following command to deploy the fix that the development team made:

./sandboxctl sre-recipes restore "recipe0"

7. Once deployed, validate that the landing page of the website loads quickly by opening the website once more. To do so, go to Navigation Menu > Kubernetes Engine > Services & Ingress and select the IP address of the frontend-external service as you did earlier in this lab. The website will now load without the latency noticed prior to re-deploying.

(Optional) Remove your alerting policy

If you set up an email alert as part of your alerting policy, there is a chance that you will receive a few emails about your resources even after the lab is completed.

To avoid this, remove the alerting policy before you complete your lab.

Congratulations!

In this lab, you explored the Cloud Operations suite, which allows Site Reliability Engineers (SRE) to investigate and diagnose issues experienced with workloads deployed. In order to increase the reliability of workloads, you explored how to navigate resource pages or GKE, view operational data from GKE dashboards, create logs-based metrics to capture specific issues and proactively respond to incidents by setting service level objectives and alerts to proactively notify the SRE team about issues experienced before they cause outages.

Finish your quest

This self-paced lab is part of the Google Cloud's Operations Suite on GKE, Cloud Architecture, and DevOps Essentials quests. A quest is a series of related labs that form a learning path. Completing a quest earns you a badge to recognize your achievement. You can make your badge or badges public and link to them in your online resume or social media account. Enroll in any quest that contains this lab and get immediate completion credit. Refer to the Google Cloud Skills Boost catalog for all available quests.

Take your next lab

Continue your quest with Cloud Logging on Kubernetes Engine, or check out these suggestions:

• Monitoring and Logging for Cloud Functions

Next steps

• Learn more about the Google Cloud Operations Suite.
• Practice and learn Cloud Operations on Google Cloud with the Cloud Operations Sandbox.

Google Cloud training and certification

...helps you make the most of Google Cloud technologies. Our classes include technical skills and best practices to help you get up to speed quickly and continue your learning journey. We offer fundamental to advanced level training, with on-demand, live, and virtual options to suit your busy schedule. Certifications help you validate and prove your skill and expertise in Google Cloud technologies.

Manual Last Updated: January 12, 2023

Lab Last Tested: January 12, 2023

End your lab

When you have completed your lab, click End Lab. Your account and the resources you've used are removed from the lab platform.

You will be given an opportunity to rate the lab experience. Select the applicable number of stars, type a comment, and then click Submit.

The number of stars indicates the following:

• 1 star = Very dissatisfied
• 2 stars = Dissatisfied
• 3 stars = Neutral
• 4 stars = Satisfied
• 5 stars = Very satisfied

You can close the dialog box if you don't want to provide feedback.

For feedback, suggestions, or corrections, please use the Support tab.

Copyright 2023 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.