arrow_back

Transcoding HTTP/JSON API calls to gRPC through API Gateway

Join Sign in

Transcoding HTTP/JSON API calls to gRPC through API Gateway

1 hour 30 minutes 5 Credits

GSP881

Google Cloud self-paced labs logo

Overview

gRPC Transcoding is a feature for mapping between a gRPC method and one or more HTTP REST endpoints. It allows developers to build a single API service that supports both gRPC APIs and REST APIs. API Gateway enables you to provide secure access to gRPC services through a well-defined API configuration. If securing gRPC APIs through API Gateway, you may also want to support protocol transcoding so that clients can access your gRPC API by using HTTP/JSON.

Transcoding involves mapping HTTP/JSON requests and their parameters to gRPC methods, their parameters and return types. Because of this, while it's possible to map an HTTP/JSON request to any arbitrary API method, it helps to do so if the gRPC API is structured in a resource-oriented way, similar to a traditional HTTP REST API.

In other words, you design the API service so that it uses a small number of standard methods, corresponding to HTTP verbs such as GET and PUT, that operate on the service's resources. These standard methods are List, Get, Create, Update, and Delete. You can find out much more about resource-oriented design and standard transcoding mappings in Google Cloud's API design guide.

In this lab you will learn how to:

  • Secure a gRPC API via API Gateway and provide the ability to call the API using HTTP/JSON.
  • Use annotations in your gRPC .proto file to specify data conversion from HTTP/JSON to gRPC.
  • How to deploy your service on Cloud Run and leverage transcoding to mediate between HTTP/JSON and gRPC.
  • Where to find more reference information about designing and implementing transcoding for gRPC services.

This lab assumes that you have already completed the Managing Cloud Run gRPC Services with API Gateway lab.

Setup and requirements

Before you click the Start Lab button

Read these instructions. Labs are timed and you cannot pause them. The timer, which starts when you click Start Lab, shows how long Google Cloud resources will be made available to you.

This hands-on lab lets you do the lab activities yourself in a real cloud environment, not in a simulation or demo environment. It does so by giving you new, temporary credentials that you use to sign in and access Google Cloud for the duration of the lab.

To complete this lab, you need:

  • Access to a standard internet browser (Chrome browser recommended).
Note: Use an Incognito or private browser window to run this lab. This prevents any conflicts between your personal account and the Student account, which may cause extra charges incurred to your personal account.
  • Time to complete the lab---remember, once you start, you cannot pause a lab.
Note: If you already have your own personal Google Cloud account or project, do not use it for this lab to avoid extra charges to your account.

How to start your lab and sign in to the Google Cloud Console

  1. Click the Start Lab button. If you need to pay for the lab, a pop-up opens for you to select your payment method. On the left is the Lab Details panel with the following:

    • The Open Google Console button
    • Time remaining
    • The temporary credentials that you must use for this lab
    • Other information, if needed, to step through this lab
  2. Click Open Google Console. The lab spins up resources, and then opens another tab that shows the Sign in page.

    Tip: Arrange the tabs in separate windows, side-by-side.

    Note: If you see the Choose an account dialog, click Use Another Account.
  3. If necessary, copy the Username from the Lab Details panel and paste it into the Sign in dialog. Click Next.

  4. Copy the Password from the Lab Details panel and paste it into the Welcome dialog. Click Next.

    Important: You must use the credentials from the left panel. Do not use your Google Cloud Skills Boost credentials. Note: Using your own Google Cloud account for this lab may incur extra charges.
  5. Click through the subsequent pages:

    • Accept the terms and conditions.
    • Do not add recovery options or two-factor authentication (because this is a temporary account).
    • Do not sign up for free trials.

After a few moments, the Cloud Console opens in this tab.

Note: You can view the menu with a list of Google Cloud Products and Services by clicking the Navigation menu at the top-left. Navigation menu icon

Activate Cloud Shell

Cloud Shell is a virtual machine that is loaded with development tools. It offers a persistent 5GB home directory and runs on the Google Cloud. Cloud Shell provides command-line access to your Google Cloud resources.

  1. Click Activate Cloud Shell Activate Cloud Shell icon at the top of the Google Cloud console.

  2. Click Continue.

It takes a few moments to provision and connect to the environment. When you are connected, you are already authenticated, and the project is set to your PROJECT_ID. The output contains a line that declares the PROJECT_ID for this session:

Your Cloud Platform project in this session is set to YOUR_PROJECT_ID

gcloud is the command-line tool for Google Cloud. It comes pre-installed on Cloud Shell and supports tab-completion.

  1. (Optional) You can list the active account name with this command:

gcloud auth list

Output:

ACTIVE: * ACCOUNT: student-01-xxxxxxxxxxxx@qwiklabs.net To set the active account, run: $ gcloud config set account `ACCOUNT`
  1. (Optional) You can list the project ID with this command:

gcloud config list project

Output:

[core] project = <project_ID>

Example output:

[core] project = qwiklabs-gcp-44776a13dea667a6 Note: For full documentation of gcloud, in Google Cloud, refer to the gcloud CLI overview guide. Note: You may be prompted to Authorize the use of Cloud Shell - click the Authorize button.

Task 1. Enable the required APIs

  • In Cloud Shell, run the following commands to enable the required services for this lab:

gcloud services enable apigateway.googleapis.com gcloud services enable run.googleapis.com gcloud services enable servicemanagement.googleapis.com gcloud services enable servicecontrol.googleapis.com

Task 2. Install gRPC tools

  • Run the following commands to download and install gRPC Tools. You will use this tool in a later step to create the descriptor file for the gRPC service.

python3 -m pip install --upgrade pip python3 -m pip install grpcio-tools --quiet &

Task 3. Deploy the gRPC service on Cloud Run

In this section, deploy a service called bookstore-server on Cloud Run using a pre-built container image that has already been uploaded to Google Container Registry.

  1. In Cloud Shell run the following command:

gcloud run deploy bookstore-service --image gcr.io/endpointsv2/python-grpc-bookstore-server:2 --platform managed --region us-central1 --max-instances 3 --min-instances 1 --memory 512Mi --ingress all --allow-unauthenticated
  1. After the command completes navigate to Cloud Run in the Cloud Console by navigating to Navigation Menu > Cloud Run.

  2. Confirm that the bookstore-service was created in the UI. You may need to wait a few minutes after the deployment for the check below to pass.

bookstore_service created

Click Check my progress to verify the objective. Deploy the gRPC service on Cloud Run

Task 4. Creating an API Config with HTTP to gRPC Transcoding

There are two methods of transcoding HTTP/JSON to gRPC. The first involves adding direct annotations to the .proto file while the second method involves adding http rules to the gRPC service configuration defined in one or more YAML files.

After the bookstore-server is successfully deployed you now need to store a local copy of the gRPC service's .proto file so that you can create an API Config to use with API Gateway. You will also need to create an API Gateway once the config is deployed.

  1. Run the following command to save a copy of the http_bookstore.proto from the example repository to your Cloud Shell instance. This file defines the Bookstore service's API:

curl -O https://raw.githubusercontent.com/GoogleCloudPlatform/python-docs-samples/master/endpoints/bookstore-grpc/http_bookstore.proto
  1. In Cloud Shell, run the following command to create a folder which will be used to create the descriptor file for the gRPC service:

mkdir generated_pb2
  1. Clone the googleapis repo from Github and set an environment variable to reference the folder it clones to:

git clone https://github.com/googleapis/googleapis export GOOGLEAPIS_DIR=$PWD/googleapis
  1. Run the following command to create the descriptor file for the gRPC service:

python3 -m grpc_tools.protoc \ --include_imports \ --include_source_info \ --proto_path=. \ --proto_path=$GOOGLEAPIS_DIR \ --descriptor_set_out=api_descriptor.pb \ --python_out=generated_pb2 \ --grpc_python_out=generated_pb2 \ http_bookstore.proto
  1. You're ready to create the API Config which will be used to deploy the gRPC API via API Gateway. Run the following command to create a text file called api_config.yaml in the current working directory of Cloud Shell. The http_bookstore.proto file should be in the same directory as well:

cat << APICONFIG > api_config.yaml # The configuration schema is defined by the service.proto file. # https://github.com/googleapis/googleapis/blob/master/google/api/service.proto type: google.api.Service config_version: 3 name: "*.apigateway.PROJECT_ID.cloud.goog" title: Bookstore API apis: - name: endpoints.examples.bookstore.Bookstore usage: rules: # ListShelves methods can be called without an API Key. - selector: endpoints.examples.bookstore.Bookstore.ListShelves allow_unregistered_calls: true backend: rules: - selector: "*" address: grpcs://ADDRESS APICONFIG

You will run commands against this file in subsequent steps to update the placeholder values.

Note: In the YAML file, note the usage section. This section explicitly allows any calls to the List method to be invoked without an API Key. A call to any other service that the Bookstore Service exposes will require an API Key. You will explore this later in the lab. To understand API access configuration for API Gateway further refer to Restricting API access with API keys.
  1. Set the following environment variables which will replace the PROJECT_ID and ADDRESS of the api_config.yaml file:

export PROJECT_ID=$(gcloud config get-value project) export ADDRESS=$(gcloud run services list --platform managed --format json | jq -r .[].status.address.url) ADDRESS=${ADDRESS:8}
  1. Run the following sed command to update the contents of the api_config.yaml file:

sed -i "s/PROJECT_ID/${PROJECT_ID}/g" api_config.yaml sed -i "s/ADDRESS/${ADDRESS}/g" api_config.yaml

The api_config.yaml file should now look similar to the following:

# The configuration schema is defined by the service.proto file. # https://github.com/googleapis/googleapis/blob/master/google/api/service.proto type: google.api.Service config_version: 3 name: "*.apigateway.qwiklabs-gcp-02-d07314166027.cloud.goog" title: Bookstore API apis: - name: endpoints.examples.bookstore.Bookstore usage: rules: # ListShelves methods can be called without an API Key. - selector: endpoints.examples.bookstore.Bookstore.ListShelves allow_unregistered_calls: true backend: rules: - selector: "*" address: grpcs://bookstore-service-2mpuzpwz6q-uc.a.run.app
  1. To configure HTTP to gRPC transcoding via mappings in the gRPC API Configuration, run the following command which will download an additional configuration file containing HTTP rules to apply to the service:

curl -O https://raw.githubusercontent.com/GoogleCloudPlatform/python-docs-samples/master/endpoints/bookstore-grpc/api_config_http.yaml
  1. Run the following command to update the api_config_http.yaml file's project ID reference:

sed -i "s/<MY_PROJECT_ID>/${PROJECT_ID}/g" api_config_http.yaml
  1. Now create an API Config for the API and use it to create an API Gateway.

  • Run the following command to create the API config. If this command returns an error you may need to wait a few minutes before trying to run the command again to proceed:

gcloud api-gateway api-configs create http-grpc-config \ --api=http-grpc-bookstore \ --project=$PROJECT_ID \ --grpc-files=api_descriptor.pb,api_config_http.yaml,api_config.yaml
  1. Next, run the following command to deploy the API config for the API and create the API Gateway:

gcloud api-gateway gateways create bookstore-transcoding-example \ --api=http-grpc-bookstore \ --api-config=http-grpc-config \ --location=us-central1 \ --project=${PROJECT_ID} Note: The deployment will take a few minutes to complete.
  1. On successful completion, run the following command to store the gateway hostname to use in the next step:

export defaultHostname=$(gcloud api-gateway gateways describe bookstore-transcoding-example --location=us-central1 --project=${PROJECT_ID} --format=json | jq -r .defaultHostname)

Click Check my progress to verify the objective. Creating an API Config with HTTP to gRPC Transcoding

Task 5. Call the API using Transcoding

Now that the API Gateway has been deployed you can test the API via http calls.

To do this, enable the API in Cloud Console.

  1. From the Navigation Menu go to APIs & Services > Library.
  2. Enter the text bookstore into the top search box and select the Bookstore API that results from the search.
  3. Click the Enable button on the next page to enable the API.
  4. Press OK on the resulting dialog.

You will not need a gRPC client to interact with the API and can simply use a tool such as curl from the command line.

  1. Make HTTP calls to the service using the following command:

curl https://${defaultHostname}/v1/shelves

The output should look similar to the following:

{ "shelves": [ { "id": "1", "theme": "Fiction" }, { "id": "2", "theme": "Fantasy" } ] }
  1. Once verified, run the following command to create a new Bookshelf:

curl -d '{"theme":"Music"}' https://${defaultHostname}/v1/shelves

You should receive a 401 Unauthorized error upon invocation. Recall from earlier in this lab, we created an API config with usage rules requiring an API key for all calls except the ListShelves method. You can inspect the api_config.yaml file in Cloud Shell using cat api_config.yaml.

401 error in output

  1. In order to call the API we must create an API key in the Google Cloud project and provide it when invoking the Create Shelf method we used in the last step. Navigate to Navigation Menu > APIs & Services > Credentials.

  2. Click on the + CREATE CREDENTIALS button on the top of the Credentials landing page and select API Key. This will surface a dialog that will display the newly created API Key.

  3. Press the copy button next to the newly created API Key in the resulting dialog box. Run the following command to store the API Key in an environment variable, replacing (place copied API Key value here) with the copied API Key:

export APIKEY=(place copied API Key value here)
  1. Call the API once more and verify you can create a Bookshelf:

curl -d '{"theme":"Music"}' https://${defaultHostname}/v1/shelves?key=$APIKEY
  1. Call the List shelves method once more to verify the new Bookshelf was added:

curl https://${defaultHostname}/v1/shelves

The output should look similar to the following:

{ "shelves": [ { "id": "1", "theme": "Fiction" }, { "id": "2", "theme": "Fantasy" }, { "id": "3", "theme": "Music" } ] }

Click Check my progress to verify the objective. Call the API using Transcoding

Congratulations!

You have successfully called a gRPC service deployed to Cloud Run through API Gateway using HTTP transcoding. You also explored how to secure API calls through API Gateway by restricting which API methods can be invoked with and without an API Key present on the query string of the inbound request from API clients.

Finish your quest

This self-paced lab is part of the Secure and Rate Limit API calls with API Gateway quest. A quest is a series of related labs that form a learning path. Completing this quest earns you a badge to recognize your achievement. You can make your badge or badges public and link to them in your online resume or social media account. Enroll in this quest and get immediate completion credit. Refer to the Google Cloud Skills Boost catalog for all available quests.

Google Cloud training and certification

...helps you make the most of Google Cloud technologies. Our classes include technical skills and best practices to help you get up to speed quickly and continue your learning journey. We offer fundamental to advanced level training, with on-demand, live, and virtual options to suit your busy schedule. Certifications help you validate and prove your skill and expertise in Google Cloud technologies.

Manual Last Updated: September 23, 2022

Lab Last Tested: September 23, 2022

Copyright 2022 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.