Overview

In this lab, you will identify Map and Reduce operations, execute the pipeline, and use command line parameters.

Objective

• Identify Map and Reduce operations
• Execute the pipeline
• Use command line parameters

Setup

For each lab, you get a new Google Cloud project and set of resources for a fixed time at no cost.

1. Sign in to Qwiklabs using an incognito window.

2. Note the lab's access time (for example, 1:15:00), and make sure you can finish within that time.
   There is no pause feature. You can restart if needed, but you have to start at the beginning.

3. When ready, click Start lab.

4. Note your lab credentials (Username and Password). You will use them to sign in to the Google Cloud Console.

5. Click Open Google Console.

6. Click Use another account and copy/paste credentials for this lab into the prompts.
   If you use other credentials, you'll receive errors or incur charges.

7. Accept the terms and skip the recovery resource page.

Check project permissions

Before you begin your work on Google Cloud, you need to ensure that your project has the correct permissions within Identity and Access Management (IAM).

1. In the Google Cloud console, on the Navigation menu (), select IAM & Admin > IAM.

2. Confirm that the default compute Service Account {project-number}-compute@developer.gserviceaccount.com is present and has the editor role assigned. The account prefix is the project number, which you can find on Navigation menu > Cloud Overview > Dashboard.

Note: If the account is not present in IAM or does not have the editor role, follow the steps below to assign the required role.

1. In the Google Cloud console, on the Navigation menu, click Cloud Overview > Dashboard.
2. Copy the project number (e.g. 729328892908).
3. On the Navigation menu, select IAM & Admin > IAM.
4. At the top of the roles table, below View by Principals, click Grant Access.
5. For New principals, type:

{project-number}-compute@developer.gserviceaccount.com

6. Replace {project-number} with your project number.
7. For Role, select Project (or Basic) > Editor.
8. Click Save.

Task 1. Lab preparations

Specific steps must be completed to successfully execute this lab.

Open the SSH terminal and connect to the training VM

You will be running all code from a curated training VM.

1. In the Console, on the Navigation menu (), click Compute Engine > VM instances.

2. Locate the line with the instance called training-vm.

3. On the far right, under Connect, click on SSH to open a terminal window.

4. In this lab, you will enter CLI commands on the training-vm.

Clone the training github repository

• In the training-vm SSH terminal enter the following command:

git clone https://github.com/GoogleCloudPlatform/training-data-analyst

Task 2. Identify map and reduce operations

• Return to the training-vm SSH terminal and navigate to the directory /training-data-analyst/courses/data_analysis/lab2/python and view the file is_popular.py with Nano. Do not make any changes to the code. Press Ctrl+X to exit Nano.

cd ~/training-data-analyst/courses/data_analysis/lab2/python nano is_popular.py

Can you answer these questions about the file is_popular.py?

• What custom arguments are defined?
• What is the default output prefix?
• How is the variable output_prefix in main() set?
• How are the pipeline arguments such as --runner set?
• What are the key steps in the pipeline?
• Which of these steps happen in parallel?
• Which of these steps are aggregations?

Task 3. Execute the pipeline

1. In the training-vm SSH terminal, run the pipeline locally:

python3 ./is_popular.py

2. Identify the output file. It should be output<suffix> and could be a sharded file:

ls -al /tmp

3. Examine the output file, replacing '-*' with the appropriate suffix:

cat /tmp/output-*

Task 4. Use command line parameters

1. In the training-vm SSH terminal, change the output prefix from the default value:

python3 ./is_popular.py --output_prefix=/tmp/myoutput

2. What will be the name of the new file that is written out?
3. Note that we now have a new file in the /tmp directory:

ls -lrt /tmp/myoutput*

End your lab

When you have completed your lab, click End Lab. Google Cloud Skills Boost removes the resources you’ve used and cleans the account for you.

You will be given an opportunity to rate the lab experience. Select the applicable number of stars, type a comment, and then click Submit.

The number of stars indicates the following:

• 1 star = Very dissatisfied
• 2 stars = Dissatisfied
• 3 stars = Neutral
• 4 stars = Satisfied
• 5 stars = Very satisfied

You can close the dialog box if you don't want to provide feedback.

For feedback, suggestions, or corrections, please use the Support tab.

Copyright 2022 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.