GSP649

Overview

In this lab, you use Cloud Functions and Cloud Scheduler to identify and clean up wasted cloud resources. You trigger a Cloud Function to migrate a storage bucket from a Cloud Monitoring alerting policy to a less expensive storage class.

Google Cloud provides storage object lifecycle rules that automatically moves objects to different storage classes based on a set of attributes, such as their creation date or live state. However, these rules can’t take into account whether the objects have been accessed. Sometimes, you might want to move newer objects to Nearline storage if they haven’t been accessed for a certain period of time.

What you'll do

• Create two storage buckets, add a file to the serving-bucket, and generate traffic against it.
• Create a Cloud Monitoring dashboard to visualize bucket utilization.
• Deploy a Cloud Function to migrate the idle bucket to a less expensive storage class, and trigger the function by using a payload intended to mock a notification received from a Cloud alerting policy.

Setup and requirements

In this section, you configure the infrastructure and identities required to complete the lab.

Before you click the Start Lab button

Read these instructions. Labs are timed and you cannot pause them. The timer, which starts when you click Start Lab, shows how long Google Cloud resources will be made available to you.

This hands-on lab lets you do the lab activities yourself in a real cloud environment, not in a simulation or demo environment. It does so by giving you new, temporary credentials that you use to sign in and access Google Cloud for the duration of the lab.

To complete this lab, you need:

• Access to a standard internet browser (Chrome browser recommended).

Note: Use an Incognito or private browser window to run this lab. This prevents any conflicts between your personal account and the Student account, which may cause extra charges incurred to your personal account.

• Time to complete the lab---remember, once you start, you cannot pause a lab.

Note: If you already have your own personal Google Cloud account or project, do not use it for this lab to avoid extra charges to your account.

How to start your lab and sign in to the Google Cloud console

1. Click the Start Lab button. If you need to pay for the lab, a pop-up opens for you to select your payment method. On the left is the Lab Details panel with the following:

   • The Open Google Cloud console button
   • Time remaining
   • The temporary credentials that you must use for this lab
   • Other information, if needed, to step through this lab

2. Click Open Google Cloud console (or right-click and select Open Link in Incognito Window if you are running the Chrome browser).

   The lab spins up resources, and then opens another tab that shows the Sign in page.

   Tip: Arrange the tabs in separate windows, side-by-side.

   Note: If you see the Choose an account dialog, click Use Another Account.

3. If necessary, copy the Username below and paste it into the Sign in dialog.

   {{{user_0.username | "Username"}}}

   You can also find the Username in the Lab Details panel.

4. Click Next.

5. Copy the Password below and paste it into the Welcome dialog.

   {{{user_0.password | "Password"}}}

   You can also find the Password in the Lab Details panel.

6. Click Next.

   Important: You must use the credentials the lab provides you. Do not use your Google Cloud account credentials. Note: Using your own Google Cloud account for this lab may incur extra charges.

7. Click through the subsequent pages:

   • Accept the terms and conditions.
   • Do not add recovery options or two-factor authentication (because this is a temporary account).
   • Do not sign up for free trials.

After a few moments, the Google Cloud console opens in this tab.

Note: To view a menu with a list of Google Cloud products and services, click the Navigation menu at the top-left.

Activate Cloud Shell

Cloud Shell is a virtual machine that is loaded with development tools. It offers a persistent 5GB home directory and runs on the Google Cloud. Cloud Shell provides command-line access to your Google Cloud resources.

1. Click Activate Cloud Shell at the top of the Google Cloud console.

When you are connected, you are already authenticated, and the project is set to your Project_ID, . The output contains a line that declares the Project_ID for this session:

Your Cloud Platform project in this session is set to {{{project_0.project_id | "PROJECT_ID"}}}

gcloud is the command-line tool for Google Cloud. It comes pre-installed on Cloud Shell and supports tab-completion.

2. (Optional) You can list the active account name with this command:

gcloud auth list

3. Click Authorize.

Output:

ACTIVE: * ACCOUNT: {{{user_0.username | "ACCOUNT"}}} To set the active account, run: $ gcloud config set account `ACCOUNT`

4. (Optional) You can list the project ID with this command:

gcloud config list project

Output:

[core] project = {{{project_0.project_id | "PROJECT_ID"}}} Note: For full documentation of gcloud, in Google Cloud, refer to the gcloud CLI overview guide.

Architecture

In the following diagram, you trigger a Cloud Function to migrate a storage bucket to a less expensive storage class from a Cloud Monitoring alerting policy.

Task 1. Enable APIs and clone repository

1. In Cloud Shell, enable the Cloud Scheduler API:

   gcloud services enable cloudscheduler.googleapis.com

   Click Check my progress to verify the objective.

Enable the Cloud Scheduler API

2. Clone the repository:

   git clone https://github.com/GoogleCloudPlatform/gcf-automated-resource-cleanup.git && cd gcf-automated-resource-cleanup/

3. Set environment variables and make the repository folder your $WORKDIR where you run all commands related to this lab:

   export PROJECT_ID=$(gcloud config list --format 'value(core.project)' 2>/dev/null) WORKDIR=$(pwd)

4. Install Apache Bench, an open source load-generation tool:

   sudo apt-get install apache2-utils -y

Task 2. Create cloud storage buckets and add a file

1. In Cloud Shell, navigate to the migrate-storage directory:

   cd $WORKDIR/migrate-storage

2. Create serving-bucket, the Cloud Storage bucket. You use this later to change storage classes:

   export PROJECT_ID=$(gcloud config list --format 'value(core.project)' 2>/dev/null) gcloud storage buckets create gs://${PROJECT_ID}-serving-bucket -l {{{project_0.default_region|REGION}}}

Click Check my progress to verify the objective.

Create a Cloud Storage bucket

3. Make the bucket public:

   gsutil acl ch -u allUsers:R gs://${PROJECT_ID}-serving-bucket

4. Add a text file to the bucket:

   gcloud storage cp $WORKDIR/migrate-storage/testfile.txt gs://${PROJECT_ID}-serving-bucket

5. Make the file public:

   gsutil acl ch -u allUsers:R gs://${PROJECT_ID}-serving-bucket/testfile.txt

6. Confirm that you’re able to access the file:

   curl http://storage.googleapis.com/${PROJECT_ID}-serving-bucket/testfile.txt

   Your output will be:

   this is a test

Click Check my progress to verify the objective.

Make cloud Storage bucket public

7. Create a second bucket called idle-bucket that won’t serve any data:

   gcloud storage buckets create gs://${PROJECT_ID}-idle-bucket -l {{{project_0.default_region|REGION}}}

Click Check my progress to verify the objective.

Create another Cloud Storage bucket

Task 3. Create a monitoring dashboard

Create a Monitoring Metrics Scope

Set up a Monitoring Metrics Scope that's tied to your Google Cloud Project. The following steps create a new account that has a free trial of Monitoring.

• In the Cloud Console, click Navigation menu () > Monitoring.

When the Monitoring Overview page opens, your metrics scope project is ready.

1. In the left panel, click Dashboards > +Create Dashboard.

2. Name the Dashboard Bucket Usage.

3. Click +ADD WIDGET.

4. Click Line Chart.

5. For Widget Title, type Bucket Access.

6. For select a metric > Metric type gcs_bucket for the resource, select GCS Bucket > Api > Request count metric and click Apply.

Note: If you cannot find the Request count metric, Turn off the toggle for Show only active resources & metrics.

6. Click + Add Filter.

To filter by the method name:

• For Label, select method.
• For Comparison, select =(equals).
• For Value, select ReadObject.
• Click Apply.

7. To group the metrics by bucket name, in the Group By drop-down list, select bucket_name and click ok.

You’ve configured Cloud Monitoring to observe object access in your buckets. There's no data in the chart because there's no traffic to the Cloud Storage buckets.

Task 4. Generate load on the serving bucket

Now that you configured monitoring, use Apache Bench to send traffic to serving-bucket.

1. In Cloud Shell, send requests to the object in the serving bucket:

ab -n 10000 http://storage.googleapis.com/$PROJECT_ID-serving-bucket/testfile.txt

2. In the left panel, click Dashboards and then click the name of your Dashboard that is Bucket Usage to see the Bucket Access chart.

Note: If you closed this tab earlier, select Navigation menu > Monitoring, and then in the left panel, click Dashboards > Bucket Usage. Note: Need to wait for at least 1 minute for the graph to appear on the Bucket Usage dashboard.

3. View traffic details.

Task 5. Review and deploy the Cloud Function

1. In Cloud Shell, enter the following command to view the Cloud Function code that migrates a storage bucket to the Nearline storage class:

   cat $WORKDIR/migrate-storage/main.py | grep "migrate_storage(" -A 15

   The output is:

   def migrate_storage(request): # process incoming request to get the bucket to be migrated: request_json = request.get_json(force=True) # bucket names are globally unique bucket_name = request_json['incident']['resource_name'] # create storage client storage_client = storage.Client() # get bucket bucket = storage_client.get_bucket(bucket_name) # update storage class bucket.storage_class = "NEARLINE" bucket.patch()

   Notice that the Cloud Function uses the bucket name passed in the request to change it's storage class to Nearline.

2. Deploy the Cloud Function:

   gcloud functions deploy migrate_storage --trigger-http --runtime=python37 --region {{{project_0.default_region | Region}}}

   When prompted, enter Y to allow unauthenticated invocations.

3. Capture the trigger URL into an environment variable that you use in the next section:

   export FUNCTION_URL=$(gcloud functions describe migrate_storage --format=json --region {{{project_0.default_region | Region}}} | jq -r '.httpsTrigger.url')

Click Check my progress to verify the objective.

Deploy cloud function

Task 6. Test and validate alerting automation

1. Set the idle bucket name:

   export IDLE_BUCKET_NAME=$PROJECT_ID-idle-bucket

2. Send a test notification to the Cloud Function you deployed using the incident.json file:

   envsubst < $WORKDIR/migrate-storage/incident.json | curl -X POST -H "Content-Type: application/json" $FUNCTION_URL -d @-

   The output is:

   OK

   The output isn’t terminated with a newline and therefore is immediately followed by the command prompt.

3. Confirm that the idle bucket was migrated to Nearline:

   gsutil defstorageclass get gs://$PROJECT_ID-idle-bucket

   The output is:

   gs://-idle-bucket: NEARLINE

Click Check my progress to verify the objective.

Confirm the migration of bucket to Nearline

Congratulations!

You completed the following tasks:

• Created two Cloud Storage buckets.
• Added an object to one of the buckets.
• Configured Cloud Monitoring to observe bucket object access.
• Reviewed the Cloud Function code that migrates objects from a Regional bucket to a Nearline bucket.
• Deployed the Cloud Function.
• Tested the Cloud Function by using a Cloud Monitoring alert.

Finish your quest

This self-paced lab is part of the Optimizing your Google Cloud Costs quest. A quest is a series of related labs that form a learning path. Completing this quest earns you a badge to recognize your achievement. You can make your badge or badges public and link to them in your online resume or social media account. Enroll in this Quest and get immediate completion credit.
See the Google Cloud Skills Boost catalog to see all available quests.

Take your next lab

Continue your Quest with Network Tiers - Optimizing Network Spend or check out these suggestions:

• Setting Up Cost Control with Quota
• Clean Up Unused IP Addresses

Google Cloud training and certification

...helps you make the most of Google Cloud technologies. Our classes include technical skills and best practices to help you get up to speed quickly and continue your learning journey. We offer fundamental to advanced level training, with on-demand, live, and virtual options to suit your busy schedule. Certifications help you validate and prove your skill and expertise in Google Cloud technologies.

Manual Last Updated December 19, 2023

Lab Last Tested December 19, 2023

Copyright 2024 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.