GSP762

Overview

In this lab you will build a PDF converter web app on Cloud Run, which is a serverless service, that automatically converts files stored in Google Drive into PDFs stored in segregated Google Drive folders.

What you'll learn

• Convert a Go application to a container

• Learn how to build containers with Google Cloud Build

• Create a Cloud Run service that converts files to PDF files in the cloud.

• Understand how to create Service Accounts and add permissions

• Use event processing with Cloud Storage

Prerequisites

This is a fundamental level lab. This assumes familiarity with the Cloud Console and Cloud Shell.

You should also be comfortable editing files. You can use your favorite text editor (like nano, vi, etc.) or you can launch the code editor from Cloud Shell, which can be found from the top ribbon:

Setup and requirements

Before you click the Start Lab button

Read these instructions. Labs are timed and you cannot pause them. The timer, which starts when you click Start Lab, shows how long Google Cloud resources will be made available to you.

This hands-on lab lets you do the lab activities yourself in a real cloud environment, not in a simulation or demo environment. It does so by giving you new, temporary credentials that you use to sign in and access Google Cloud for the duration of the lab.

To complete this lab, you need:

• Access to a standard internet browser (Chrome browser recommended).

Note: Use an Incognito or private browser window to run this lab. This prevents any conflicts between your personal account and the Student account, which may cause extra charges incurred to your personal account.

• Time to complete the lab---remember, once you start, you cannot pause a lab.

Note: If you already have your own personal Google Cloud account or project, do not use it for this lab to avoid extra charges to your account.

How to start your lab and sign in to the Google Cloud Console

1. Click the Start Lab button. If you need to pay for the lab, a pop-up opens for you to select your payment method. On the left is the Lab Details panel with the following:

   • The Open Google Console button
   • Time remaining
   • The temporary credentials that you must use for this lab
   • Other information, if needed, to step through this lab

2. Click Open Google Console. The lab spins up resources, and then opens another tab that shows the Sign in page.

   Tip: Arrange the tabs in separate windows, side-by-side.

   Note: If you see the Choose an account dialog, click Use Another Account.

3. If necessary, copy the Username from the Lab Details panel and paste it into the Sign in dialog. Click Next.

4. Copy the Password from the Lab Details panel and paste it into the Welcome dialog. Click Next.

   Important: You must use the credentials from the left panel. Do not use your Google Cloud Skills Boost credentials. Note: Using your own Google Cloud account for this lab may incur extra charges.

5. Click through the subsequent pages:

   • Accept the terms and conditions.
   • Do not add recovery options or two-factor authentication (because this is a temporary account).
   • Do not sign up for free trials.

After a few moments, the Cloud Console opens in this tab.

Note: You can view the menu with a list of Google Cloud Products and Services by clicking the Navigation menu at the top-left.

Activate Cloud Shell

Cloud Shell is a virtual machine that is loaded with development tools. It offers a persistent 5GB home directory and runs on the Google Cloud. Cloud Shell provides command-line access to your Google Cloud resources.

1. Click Activate Cloud Shell at the top of the Google Cloud console.

When you are connected, you are already authenticated, and the project is set to your PROJECT_ID. The output contains a line that declares the PROJECT_ID for this session:

Your Cloud Platform project in this session is set to YOUR_PROJECT_ID

gcloud is the command-line tool for Google Cloud. It comes pre-installed on Cloud Shell and supports tab-completion.

2. (Optional) You can list the active account name with this command:

gcloud auth list

3. Click Authorize.

4. Your output should now look like this:

Output:

ACTIVE: * ACCOUNT: student-01-xxxxxxxxxxxx@qwiklabs.net To set the active account, run: $ gcloud config set account `ACCOUNT`

5. (Optional) You can list the project ID with this command:

gcloud config list project

Output:

[core] project = <project_ID>

Example output:

[core] project = qwiklabs-gcp-44776a13dea667a6 Note: For full documentation of gcloud, in Google Cloud, refer to the gcloud CLI overview guide.

Architecture

In this lab you will assist the Pet Theory Veterinary practice to automatically convert their invoices into PDFs so that customers can open them reliably.

Using Googleapis

During this lab you will use Google APIs. The following APIs have been enabled for you:

Name	API
Cloud Build	cloudbuild.googleapis.com
Cloud Storage	storage-component.googleapis.com
Cloud Run	run.googleapis.com

Task 1. Get the source code

Get started by downloading the code necessary for this lab.

1. Activate your lab account:

gcloud auth list --filter=status:ACTIVE --format="value(account)"

2. Run the following to clone the Pet Theory repository:

git clone https://github.com/Deleplace/pet-theory.git

3. Move to the correct directory:

cd pet-theory/lab03

Task 2. Creating an invoice microservice

In this section you will create a Go application to process requests. As outlined in the architecture diagram, you will integrate Cloud Storage as part of the solution.

1. Click the Open Editor icon and then click Open in a new window.

2. Navigate to pet-theory > lab03 > server.go

3. Open the server.go source code and edit it to match the text below:

package main import ( "fmt" "io/ioutil" "log" "net/http" "os" "os/exec" "regexp" "strings" ) func main() { http.HandleFunc("/", process) port := os.Getenv("PORT") if port == "" { port = "8080" log.Printf("Defaulting to port %s", port) } log.Printf("Listening on port %s", port) err := http.ListenAndServe(fmt.Sprintf(":%s", port), nil) log.Fatal(err) } func process(w http.ResponseWriter, r *http.Request) { log.Println("Serving request") if r.Method == "GET" { fmt.Fprintln(w, "Ready to process POST requests from Cloud Storage trigger") return } // // Read request body containing Cloud Storage object metadata // gcsInputFile, err1 := readBody(r) if err1 != nil { log.Printf("Error reading POST data: %v", err1) w.WriteHeader(http.StatusBadRequest) fmt.Fprintf(w, "Problem with POST data: %v \n", err1) return } // // Working directory (concurrency-safe) // localDir, errDir := ioutil.TempDir("", "") if errDir != nil { log.Printf("Error creating local temp dir: %v", errDir) w.WriteHeader(http.StatusInternalServerError) fmt.Fprintf(w, "Could not create a temp directory on server. \n") return } defer os.RemoveAll(localDir) // // Download input file from Cloud Storage // localInputFile, err2 := download(gcsInputFile, localDir) if err2 != nil { log.Printf("Error downloading Cloud Storage file [%s] from bucket [%s]: %v", gcsInputFile.Name, gcsInputFile.Bucket, err2) w.WriteHeader(http.StatusInternalServerError) fmt.Fprintf(w, "Error downloading Cloud Storage file [%s] from bucket [%s]", gcsInputFile.Name, gcsInputFile.Bucket) return } // // Use LibreOffice to convert local input file to local PDF file. // localPDFFilePath, err3 := convertToPDF(localInputFile.Name(), localDir) if err3 != nil { log.Printf("Error converting to PDF: %v", err3) w.WriteHeader(http.StatusInternalServerError) fmt.Fprintf(w, "Error converting to PDF.") return } // // Upload the freshly generated PDF to Cloud Storage // targetBucket := os.Getenv("PDF_BUCKET") err4 := upload(localPDFFilePath, targetBucket) if err4 != nil { log.Printf("Error uploading PDF file to bucket [%s]: %v", targetBucket, err4) w.WriteHeader(http.StatusInternalServerError) fmt.Fprintf(w, "Error downloading Cloud Storage file [%s] from bucket [%s]", gcsInputFile.Name, gcsInputFile.Bucket) return } // // Delete the original input file from Cloud Storage. // err5 := deleteGCSFile(gcsInputFile.Bucket, gcsInputFile.Name) if err5 != nil { log.Printf("Error deleting file [%s] from bucket [%s]: %v", gcsInputFile.Name, gcsInputFile.Bucket, err5) // This is not a blocking error. // The PDF was successfully generated and uploaded. } log.Println("Successfully produced PDF") fmt.Fprintln(w, "Successfully produced PDF") } func convertToPDF(localFilePath string, localDir string) (resultFilePath string, err error) { log.Printf("Converting [%s] to PDF", localFilePath) cmd := exec.Command("libreoffice", "--headless", "--convert-to", "pdf", "--outdir", localDir, localFilePath) cmd.Stdout, cmd.Stderr = os.Stdout, os.Stderr log.Println(cmd) err = cmd.Run() if err != nil { return "", err } pdfFilePath := regexp.MustCompile(`\.\w+$`).ReplaceAllString(localFilePath, ".pdf") if !strings.HasSuffix(pdfFilePath, ".pdf") { pdfFilePath += ".pdf" } log.Printf("Converted %s to %s", localFilePath, pdfFilePath) return pdfFilePath, nil }

4. Now run the following to build the application:

go build -o server

The functions called by this top-level code are in source files:

• server.go
• notification.go
• gcs.go

With the application has been successfully built, you can create the pdf-conversion service.

Task 3. Create a pdf-conversion service

The PDF service will use Cloud Run and Cloud Storage to initiate a process each time a file is uploaded to the designated storage.

To achieve this you will use a common pattern of event notifications together with Cloud Pub/Sub. Doing this enables the application to concentrate only on processing information. Transporting and passing information is performed by other services, which allows you to keep the application simple.

Building the invoice module requires the integration of two components:

Adding the LibreOffice package means it can be used in your application.

1. In the Open editor, Open the existing Dockerfile manifest and update the file as shown below:

FROM debian:buster RUN apt-get update -y \ && apt-get install -y libreoffice \ && apt-get clean WORKDIR /usr/src/app COPY server . CMD [ "./server" ]

2. Save the updated Dockerfile.

3. Initiate a rebuild of the pdf-converter image using Cloud Build:

gcloud builds submit \ --tag gcr.io/$GOOGLE_CLOUD_PROJECT/pdf-converter

Click Check my progress to verify that you've performed the above task.

Build an image with Cloud Build

4. Deploy the updated pdf-converter service.

Note: It's a good idea to give LibreOffice 2GB of RAM to work with, see the line with the `--memory` option.

5. Run these commands to build the container and to deploy it:

gcloud run deploy pdf-converter \ --image gcr.io/$GOOGLE_CLOUD_PROJECT/pdf-converter \ --platform managed \ --region us-east1 \ --memory=2Gi \ --no-allow-unauthenticated \ --set-env-vars PDF_BUCKET=$GOOGLE_CLOUD_PROJECT-processed \ --max-instances=3

Click Check my progress to verify that you've performed the above task.

PDF Converter service deployed

The Cloud Run service has now been successfully deployed. However we deployed an application that requires the correct permissions to access it.

Task 4. Create a Service Account

A Service Account is a special type of account with access to Google APIs.

In this lab uses a Service Account to access Cloud Run when a Cloud Storage event is processed. Cloud Storage supports a rich set of notifications that can be used to trigger events.

Next, update the code to notify the application when a file has been uploaded.

1. Click the Navigation menu > Cloud Storage, and verify that two buckets have been created. You should see:

• PROJECT_ID-processed

• PROJECT_ID-upload

2. Create a Pub/Sub notification to indicate a new file has been uploaded to the docs bucket ("uploaded"). The notifications will be labeled with the topic "new-doc".

gsutil notification create -t new-doc -f json -e OBJECT_FINALIZE gs://$GOOGLE_CLOUD_PROJECT-upload

3. Create a new service account to trigger the Cloud Run services:

gcloud iam service-accounts create pubsub-cloud-run-invoker --display-name "PubSub Cloud Run Invoker"

4. Give the service account permission to invoke the PDF converter service:

gcloud run services add-iam-policy-binding pdf-converter \ --member=serviceAccount:pubsub-cloud-run-invoker@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com \ --role=roles/run.invoker \ --region us-east1 \ --platform managed

5. Find your project number by running this command:

PROJECT_NUMBER=$(gcloud projects list \ --format="value(PROJECT_NUMBER)" \ --filter="$GOOGLE_CLOUD_PROJECT")

6. Enable your project to create Cloud Pub/Sub authentication tokens:

gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \ --member=serviceAccount:service-$PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com \ --role=roles/iam.serviceAccountTokenCreator

Click Check my progress to verify that you've performed the above task.

Service Account created

With the Service Account created it can be used to invoke the Cloud Run Service.

Task 5. Testing the Cloud Run service

Before progressing further, test the deployed service. Remember the service requires authentication, so test that to ensure it is actually private.

1. Save the URL of your service in the environment variable $SERVICE_URL:

SERVICE_URL=$(gcloud run services describe pdf-converter \ --platform managed \ --region us-east1 \ --format "value(status.url)")

2. Display the SERVICE URL:

echo $SERVICE_URL

3. Make an anonymous GET request to your new service:

curl -X GET $SERVICE_URL

The anonymous GET request will result in an error message
"Your client does not have permission to get URL". This is good; you don't want the service to be callable by anonymous users.

4. Now try invoking the service as an authorized user:

curl -X GET -H "Authorization: Bearer $(gcloud auth print-identity-token)" $SERVICE_URL

5. You will get the response "Ready to process POST requests from Cloud Storage trigger"

Great work, you have successfully deployed an authenticated Cloud Run service.

Task 6. Cloud Storage trigger

To initiate a notification when new content is uploaded to Cloud Storage, add a subscription to your existing Pub/Sub Topic.

Note: Cloud Storage notifications will automatically push a message to your Topic queue when new content is uploaded. Using notifications allows you to create powerful applications that respond to events without needing to write additional code.

• Create a Pub/Sub subscription so that the PDF converter will be run whenever a message is published to the topic new-doc:

  gcloud pubsub subscriptions create pdf-conv-sub \ --topic new-doc \ --push-endpoint=$SERVICE_URL \ --push-auth-service-account=pubsub-cloud-run-invoker@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com

Click Check my progress to verify that you've performed the above task.

Confirm Pub/Sub Subscription

Now whenever a file is uploaded the Pub/Sub subscription will interact with your Service Account. The Service Account will then initiate your PDF Converter Cloud Run service.

Task 7. Testing Cloud Storage notification

To test the Cloud Run service, use the example files available.

1. Copy the test files into your upload bucket:

gsutil -m cp -r gs://spls/gsp762/* gs://$GOOGLE_CLOUD_PROJECT-upload

2. In the Cloud Console, click Cloud Storage > Buckets followed by the bucket name whose name ends in "-upload" and click the Refresh button a few times and see how the files are deleted, one by one, as they are converted to PDFs.

3. Then click Buckets, followed by the bucket whose name ends in "-processed". It should contain PDF versions of all files. Feel free to open the PDF files to make sure they were properly converted.

4. Once the upload is done, click Navigation menu > Cloud Run and click on the pdf-converter service.

5. Select the LOGS tab and add a filter of "Converting" to see the converted files.

6. Navigate to Navigation menu > Cloud Storage and open the bucket name ending in "-upload" to confirm all files uploaded have been processed.

Excellent work, you have successfully built a new service to create a PDF using files uploaded to Cloud Storage.

Congratulations!

The pdf-converter service converts documents and writes them as PDFs to the "processed" bucket. In this lab you have learned how to:

• Convert a Go application to a container

• Build containers with Google Cloud Build

• Create a Cloud Run service

• Enable permissions using a Service Account

• Use Cloud Storage event processing

Finish your quest

This self-paced lab is part of the Google Cloud Run Serverless Workshop quest. A quest is a series of related labs that form a learning path. Completing this quest earns you a badge to recognize your achievement. You can make your badge or badges public and link to them in your online resume or social media account. Enroll in this quest and get immediate completion credit. Refer to the Google Cloud Skills Boost catalog for all available quests.

Learn more / Next steps

• Serverless Containers: Next '19 video on YouTube

Google Cloud training and certification

...helps you make the most of Google Cloud technologies. Our classes include technical skills and best practices to help you get up to speed quickly and continue your learning journey. We offer fundamental to advanced level training, with on-demand, live, and virtual options to suit your busy schedule. Certifications help you validate and prove your skill and expertise in Google Cloud technologies.

Manual Last Updated: January 18, 2023

Lab Last Tested: January 18, 2023

Copyright 2023 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.