GSP208

Overview

In Terraform, a Provider is the logical abstraction of an upstream API. In this lab, you learn how to build a custom provider for Terraform. Terraform supports a plugin model, and all providers are actually plugins. Plugins are distributed as Go binaries. Although technically possible to write a plugin in another language, almost all Terraform plugins are written in Go.

Objectives

In this lab, you build a custom provider for Terraform by:

• Building the plugin.
• Defining resources.
• Invoking the provider.
• Learning about error handling and partial state.
• Implementing the destroy and read functions.

Prerequisites

For this lab, you should have experience with the following:

• Familarity with Linux editor like nano, vi etc.

• Familarity with Go programming language.

Setup and requirements

Before you click the Start Lab button

Read these instructions. Labs are timed and you cannot pause them. The timer, which starts when you click Start Lab, shows how long Google Cloud resources will be made available to you.

This hands-on lab lets you do the lab activities yourself in a real cloud environment, not in a simulation or demo environment. It does so by giving you new, temporary credentials that you use to sign in and access Google Cloud for the duration of the lab.

To complete this lab, you need:

• Access to a standard internet browser (Chrome browser recommended).

Note: Use an Incognito or private browser window to run this lab. This prevents any conflicts between your personal account and the Student account, which may cause extra charges incurred to your personal account.

• Time to complete the lab---remember, once you start, you cannot pause a lab.

Note: If you already have your own personal Google Cloud account or project, do not use it for this lab to avoid extra charges to your account.

How to start your lab and sign in to the Google Cloud Console

1. Click the Start Lab button. If you need to pay for the lab, a pop-up opens for you to select your payment method. On the left is the Lab Details panel with the following:

   • The Open Google Console button
   • Time remaining
   • The temporary credentials that you must use for this lab
   • Other information, if needed, to step through this lab

2. Click Open Google Console. The lab spins up resources, and then opens another tab that shows the Sign in page.

   Tip: Arrange the tabs in separate windows, side-by-side.

   Note: If you see the Choose an account dialog, click Use Another Account.

3. If necessary, copy the Username from the Lab Details panel and paste it into the Sign in dialog. Click Next.

4. Copy the Password from the Lab Details panel and paste it into the Welcome dialog. Click Next.

   Important: You must use the credentials from the left panel. Do not use your Google Cloud Skills Boost credentials. Note: Using your own Google Cloud account for this lab may incur extra charges.

5. Click through the subsequent pages:

   • Accept the terms and conditions.
   • Do not add recovery options or two-factor authentication (because this is a temporary account).
   • Do not sign up for free trials.

After a few moments, the Cloud Console opens in this tab.

Note: You can view the menu with a list of Google Cloud Products and Services by clicking the Navigation menu at the top-left.

Activate Cloud Shell

Cloud Shell is a virtual machine that is loaded with development tools. It offers a persistent 5GB home directory and runs on the Google Cloud. Cloud Shell provides command-line access to your Google Cloud resources.

1. Click Activate Cloud Shell at the top of the Google Cloud console.

When you are connected, you are already authenticated, and the project is set to your PROJECT_ID. The output contains a line that declares the PROJECT_ID for this session:

Your Cloud Platform project in this session is set to YOUR_PROJECT_ID

gcloud is the command-line tool for Google Cloud. It comes pre-installed on Cloud Shell and supports tab-completion.

2. (Optional) You can list the active account name with this command:

gcloud auth list

3. Click Authorize.

4. Your output should now look like this:

Output:

ACTIVE: * ACCOUNT: student-01-xxxxxxxxxxxx@qwiklabs.net To set the active account, run: $ gcloud config set account `ACCOUNT`

5. (Optional) You can list the project ID with this command:

gcloud config list project

Output:

[core] project = <project_ID>

Example output:

[core] project = qwiklabs-gcp-44776a13dea667a6 Note: For full documentation of gcloud, in Google Cloud, refer to the gcloud CLI overview guide.

Task 1. Requirement of custom providers

Some examples of when to author a custom Terraform provider:

• An internal private cloud whose functionality is either proprietary or would not benefit the open source community.

• A "work in progress" provider being tested locally before contributing back.

• Extensions of an existing provider.

Task 2. The provider schema

1. In Cloud shell, create a file named provider.go. This is the root of the provider.

touch provider.go

2. Click the Open Editor button on the toolbar of Cloud Shell. (You can switch between Cloud Shell and the code editor by using the Open Editor and Open Terminal icons as required, or click the Open in new window button to leave the Editor open in a separate tab).

3. In the Editor, add the following content to the provider.go file:

package main import ( "github.com/hashicorp/terraform-plugin-sdk/helper/schema" ) func Provider() *schema.Provider { return &schema.Provider{ ResourcesMap: map[string]*schema.Resource{}, } }

The helper/schema library is part of Terraform Core. It abstracts many of the complexities and ensures consistency between providers. The example above defines an empty provider (there are no resources).

The *schema.Provider type describes the provider's properties including:

• The configuration keys it accepts

• The resources it supports

• Any callbacks to configure

Task 3. Building the plugin

Go requires a main.go file, which is the default executable when the binary is built.

1. Create a file named main.go:

touch main.go

2. Since Terraform plugins are distributed as Go binaries, it is important to define this entry-point, which you'll do by adding the following code in the editor window to the main.go file :

package main import ( "github.com/hashicorp/terraform-plugin-sdk/plugin" "github.com/hashicorp/terraform-plugin-sdk/terraform" ) func main() { plugin.Serve(&plugin.ServeOpts{ ProviderFunc: func() terraform.ResourceProvider { return Provider() }, }) }

This establishes the main function to produce a valid, executable Go binary. The contents of the main function consume Terraform's plugin library. This library deals with all the communication between Terraform core and the plugin.

3. Create a module for your Terraform Go packages:

go mod init gopath/src/github.com/hashicorp/terraform-plugin-sdk

4. Download and install Terraform packages and dependencies:

go get github.com/hashicorp/terraform-plugin-sdk/helper/schema go get github.com/hashicorp/terraform-plugin-sdk/plugin go get github.com/hashicorp/terraform-plugin-sdk/terraform

5. Next, build the plugin using the Go toolchain:

go build -o terraform-provider-example

The output name (-o) is very important. Terraform searches for plugins in the format of:

terraform--

In the case above, the plugin is of type provider and of name example.

6. List out the contents of the directory:

ls

7. To verify things are working correctly, execute the binary you just created:

./terraform-provider-example

Example output:

This binary is a plugin. These are not meant to be executed directly. Please execute the program that consumes these plugins, which will load any plugins automatically

Your file tree should look like this:

Task 4. Defining resources

Terraform providers manage resources. A provider is an abstraction of an upstream API, and a resource is a component of that provider. As an example, the Google provider supports google_compute_instance and google_compute_address.

As a general convention, Terraform providers put each resource in their own file, named after the resource, prefixed with resource_.

1. Create an example_server and name it resource_server.go by convention:

touch resource_server.go

2. Add the following code to the resource_server.go file:

package main import ( "github.com/hashicorp/terraform-plugin-sdk/helper/schema" ) func resourceServer() *schema.Resource { return &schema.Resource{ Create: resourceServerCreate, Read: resourceServerRead, Update: resourceServerUpdate, Delete: resourceServerDelete, Schema: map[string]*schema.Schema{ "address": &schema.Schema{ Type: schema.TypeString, Required: true, }, }, } }

This uses the schema.Resource type. This structure defines the data schema and CRUD operations for the resource. Defining these properties are the only required thing to create a resource.

The schema above defines one element, address, which is a required string. Terraform's schema automatically enforces validation and type casting.

Next there are four fields defined - Create, Read, Update, and Delete. The Create, Read, and Delete functions are required for a resource to be functional. There are other functions, but these are the only required ones. Terraform itself handles which function to call and with what data. Based on the schema and current state of the resource, Terraform can determine whether it needs to create a new resource, update an existing one, or destroy.

Each of the four struct fields point to a function. While it is technically possible to inline all functions in the resource schema, best practice dictates pulling each function into its own method. This optimizes for both testing and readability. You fill in those stubs now, paying close attention to method signatures.

3. Update the resource_server.go file and add the following contents at the end of the file:

func resourceServerCreate(d *schema.ResourceData, m interface{}) error { return nil } func resourceServerRead(d *schema.ResourceData, m interface{}) error { return nil } func resourceServerUpdate(d *schema.ResourceData, m interface{}) error { return nil } func resourceServerDelete(d *schema.ResourceData, m interface{}) error { return nil }

4. Lastly, update the provider schema in provider.go to register the new example_server resource:

func Provider() *schema.Provider { return &schema.Provider{ ResourcesMap: map[string]*schema.Resource{ "example_server": resourceServer(), }, } }

5. Build and test the plugin.

Everything should compile as-is, although all operations are a no-op.

go build -o terraform-provider-example ./terraform-provider-example

Example output:

This binary is a plugin. These are not meant to be executed directly. Please execute the program that consumes these plugins, which will load any plugins automatically

The layout now looks like this:

Task 5. Move the provider to plugins directory

With Terraform 0.13+, you must specify all required providers and their respective source in your Terraform configuration. A provider source string is comprised of hostname/namespace/name.

When you run terraform init, Terraform will attempt to download the provider from the Terraform Registry.

If Terraform can't download the provider from the Terraform Registry (for example if the provider is local, or because of firewall restrictions), you can specify the installation method configuration explicitly. Otherwise, Terraform will implicitly attempt to find the provider locally in the appropriate subdirectory within the user plugins directory, ~/.terraform.d/plugins/${host_name}/${namespace}/${type}/${version}/${target} or %APPDATA%\terraform.d\plugins\${host_name}/${namespace}/${type}/${version}/${target}.

In order to use the local example provider you built, you'll move it into the proper subdirectory and then, later in the lab, point to that location in a main.tf file.

1. First, create the directory:

mkdir -p ~/.terraform.d/plugins/example.com/qwiklabs/example/1.0.0/linux_amd64

2. Then, copy the terraform-provider-example binary into that location:

cp terraform-provider-example ~/.terraform.d/plugins/example.com/qwiklabs/example/1.0.0/linux_amd64

Task 6. Invoking the provider

Previous sections showed running the provider directly via the shell, which outputs a warning message like:

This binary is a plugin. These are not meant to be executed directly. Please execute the program that consumes these plugins, which will load any plugins automatically

Terraform plugins should be executed by Terraform directly.

1. To test this, create a main.tf in the working directory (the same place where the plugin exists):

touch main.tf

2. Next, add the following content to the main.tf file:

# This is required for Terraform 0.13+ terraform { required_providers { example = { version = "~> 1.0.0" source = "example.com/qwiklabs/example" } } } resource "example_server" "my-server" {}

Take note of the source and version included in the required_providers block. When the init command is executed, terraform will search for the example provider in the plugins folder with these specifications.

3. Run terraform init to discover the newly compiled Provider:

terraform init

Example output:

Initializing provider plugins... Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.

4. Now execute terraform plan:

terraform plan

Example output:

Error: Missing required argument on main.tf line 10, in resource "example_server" "my-server": 10: resource "example_server" "my-server" {} The argument "address" is required, but no definition was found.

This validates Terraform is correctly delegating work to your plugin and that your validation is working as intended.

5. Fix the validation error by adding an address field to the main.tf resource:

resource "example_server" "my-server" { address = "1.2.3.4" }

6. Execute terraform plan to verify the validation is passing:

terraform plan

Example output:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # example_server.my-server will be created + resource "example_server" "my-server" { + address = "1.2.3.4" + id = (known after apply) } Plan: 1 to add, 0 to change, 0 to destroy. ------------------------------------------------------------------------

You can optionally run terraform apply, but it will be a no-op because all of the resource options currently take no action.

Task 7. Implement Create

1. Navigate to the resource_server.go file and implement the Create functionality by making the following update:

func resourceServerCreate(d *schema.ResourceData, m interface{}) error { address := d.Get("address").(string) d.SetId(address) return nil }

This uses the schema.ResourceData API to get the value of address provided by the user in the Terraform configuration. Due to the way Go works, we have to typecast it to string. This is a safe operation, however, since your schema guarantees it will be a string type.

Next, it uses SetId, a built-in function, to set the ID of the resource to the address. The existence of a non-blank ID is what tells Terraform that a resource was created. This ID can be any string value, but should be a value that can be used to read the resource again.

Finally, you must recompile the binary and instruct Terraform to reinitialize it by rerunning terraform init. This is only necessary because you have modified the code and recompiled the binary and it no longer matches an internal hash Terraform uses to ensure the same binaries are used for each operation.

2. Recompile and reinitialize the Provider:

go build -o terraform-provider-example

3. Move your provider into the plugins subdirectory with a new version:

mkdir -p ~/.terraform.d/plugins/example.com/qwiklabs/example/1.0.1/linux_amd64 cp terraform-provider-example ~/.terraform.d/plugins/example.com/qwiklabs/example/1.0.1/linux_amd64

4. Update the required_providers block in main.tf to use new 1.0.1 version:

# This is required for Terraform 0.13+ terraform { required_providers { example = { version = "~> 1.0.1" source = "example.com/qwiklabs/example" } } }

5. Reinitialize Terraform:

terraform init -upgrade

6. Run terraform plan:

terraform plan

Example output:

+ example_server.my-server address: "1.2.3.4" Plan: 1 to add, 0 to change, 0 to destroy.

7. Terraform will ask for confirmation when you run terraform apply. Enter yes to create your example server and commit it to state:

terraform apply

Example output:

An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: + example_server.my-server id: address: "1.2.3.4" Plan: 1 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes example_server.my-server: Creating... address: "" => "1.2.3.4" example_server.my-server: Creation complete after 0s (ID: 1.2.3.4)

8. Since the Create operation used SetId, Terraform believes the resource created successfully. Verify this by running terraform plan:

terraform plan

Example output:

example_server.my-server: Refreshing state... [id=1.2.3.4] No changes. Your infrastructure matches the configuration. Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Again, because of the call to SetId, Terraform believes the resource was created. When running plan, Terraform properly determines there are no changes to apply.

9. To verify this behavior, first change the value of the address field then run terraform plan again.

terraform plan

You should see output like this:

Example output:

example_server.my-server: Refreshing state... [id=1.2.3.4] Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: ~ update in-place Terraform will perform the following actions: # example_server.my-server will be updated in-place ~ resource "example_server" "my-server" { ~ address = "1.2.3.4" -> "1.2.3.5" id = "1.2.3.4" } Plan: 0 to add, 1 to change, 0 to destroy.

Terraform detects the change and displays a diff with a ~ prefix, noting the resource will be modified in place, rather than created new.

10. Run terraform apply to apply the changes.

Terraform will again prompt for confirmation.

11. Type yes.

terraform apply

Example output:

example_server.my-server: Refreshing state... (ID: 1.2.3.4) An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: ~ update in-place Terraform will perform the following actions: ~ example_server.my-server address: "1.2.3.4" => "5.6.7.8" Plan: 0 to add, 1 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes example_server.my-server: Modifying... (ID: 1.2.3.4) address: "1.2.3.4" => "5.6.7.8" example_server.my-server: Modifications complete after 0s (ID: 1.2.3.4) Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Since you didn't implement the Update function, you would expect the terraform plan operation to report changes, but it does not! How were your changes persisted without the Update implementation?

Task 8. Error handling and partial state

Previously your Update operation succeeded and persisted the new state with an empty function definition. Navigate to the resource_server.go file and recall the current update function:

func resourceServerUpdate(d *schema.ResourceData, m interface{}) error { return nil }

The return nil tells Terraform that the update operation succeeded without error. Terraform assumes this means any changes requested applied without error. Because of this, your state updated and Terraform believes there are no further changes.

To say it another way: if a callback returns no error, Terraform automatically assumes the entire diff successfully applied, merges the diff into the final state, and persists it.

Functions should never intentionally panic or call os.Exit - always return an error.

In reality, it is a bit more complicated than this. Imagine the scenario where your update function has to update two separate fields which require two separate API calls. What do you do if the first API call succeeds but the second fails? How do you properly tell Terraform to only persist half the diff? This is known as a partial state scenario, and implementing these properly is critical to a well-behaving provider.

Here are the rules for state updating in Terraform:

• If the Create callback returns with or without an error without an ID set using SetId, the resource is assumed to not be created, and no state is saved.
• If the Create callback returns with or without an error and an ID has been set, the resource is assumed created and all state is saved with it. Repeating because it is important: if there is an error, but the ID is set, the state is fully saved.
• If the Update callback returns with or without an error, the full state is saved. If the ID becomes blank, the resource is destroyed (even within an update, though this shouldn't happen except in error scenarios).
• If the Destroy callback returns without an error, the resource is assumed to be destroyed, and all state is removed.
• If the Destroy callback returns with an error, the resource is assumed to still exist, and all prior state is preserved.
• If partial mode (covered next) is enabled when a create or update returns, only the explicitly enabled configuration keys are persisted, resulting in a partial state.

Here is an example of a partial mode with an update function:

func resourceServerUpdate(d *schema.ResourceData, m interface{}) error { // Enable partial state mode d.Partial(true) if d.HasChange("address") { // Try updating the address if err := updateAddress(d, m); err != nil { return err } d.SetPartial("address") } // If we were to return here, before disabling partial mode below, // then only the "address" field would be saved. // We succeeded, disable partial mode. This causes Terraform to save // all fields again. d.Partial(false) return nil } Note: This code will not compile since there is no updateAddress function. You can implement a dummy version of this function to play around with partial state. For this example, partial state does not mean much. If updateAddress were to fail, then the address field would not be updated.

Task 9. Implementing Destroy

The Destroy callback is exactly what it sounds like - it is called to destroy the resource. This operation should never update any state on the resource. It is not necessary to call d.SetId(""), since any non-error return value assumes the resource was deleted successfully.

1. Add the Destroy callback function in resource_server.go:

func resourceServerDelete(d *schema.ResourceData, m interface{}) error { // d.SetId("") is automatically called assuming delete returns no errors, but // it is added here for explicitness. d.SetId("") return nil }

The Destroy function should always handle the case where the resource might already be destroyed (manually, for example). If the resource is already destroyed, this should not return an error. This allows Terraform users to manually delete resources without breaking Terraform.

2. Recompile the Provider:

go build -o terraform-provider-example

3. Place it in the proper plugins subdirectory with a new version numbers:

mkdir -p ~/.terraform.d/plugins/example.com/qwiklabs/example/1.0.2/linux_amd64 cp terraform-provider-example ~/.terraform.d/plugins/example.com/qwiklabs/example/1.0.2/linux_amd64

4. Update the required_providers block in main.tf to use new 1.0.2 version:

# This is required for Terraform 0.13+ terraform { required_providers { example = { version = "~> 1.0.2" source = "example.com/qwiklabs/example" } } }

5. Reinitialize Terraform:

terraform init -upgrade

6. Run terraform destroy to destroy the resource. When prompted for confirmation, type yes.

terraform destroy

Example output:

example_server.my-server: Refreshing state... (ID: 5.6.7.8) An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: - destroy Terraform will perform the following actions: - example_server.my-server Plan: 0 to add, 0 to change, 1 to destroy. Do you really want to destroy? Terraform will destroy all your managed infrastructure, as shown above. There is no undo. Only 'yes' will be accepted to confirm. Enter a value: yes example_server.my-server: Destroying... (ID: 5.6.7.8) example_server.my-server: Destruction complete after 0s Destroy complete! Resources: 1 destroyed.

Task 10. Implementing Read

The Read callback is used to sync the local state with the actual state (upstream). This is called at various points by Terraform and should be a read-only operation. This callback should never modify the real resource.

If the ID is updated to blank, this tells Terraform the resource no longer exists (maybe it was destroyed out of band). Just like the destroy callback, the Read function should gracefully handle this case.

• Update the Read callback function in resource_server.go file:

func resourceServerRead(d *schema.ResourceData, m interface{}) error { client := m.(*MyClient) // Attempt to read from an upstream API obj, ok := client.Get(d.Id()) // If the resource does not exist, inform Terraform. We want to immediately // return here to prevent further processing. if !ok { d.SetId("") return nil } d.Set("address", obj.Address) return nil }

Congratulations!

In this lab, you built a custom provider for Terraform. You accomplished this by building a plugin, defining resources, learning about error handling and partial state, and implementing a few Terraform functions.

Finish your quest

This self-paced lab is part of the Managing Cloud Infrastructure with Terraform quest. A quest is a series of related labs that form a learning path. Completing this quest earns you a badge to recognize your achievement. You can make your badge or badges public and link to them in your online resume or social media account. Enroll in this quest or any quest that contains this lab and get immediate completion credit. See the Google Cloud Skills Boost catalog to see all available quests.

Take your next lab

Continue your quest with Cloud SQL with Terraform, or check out these suggestions:

• Modular Load Balancing with Terraform - Regional Load Balancer

Google Cloud training and certification

...helps you make the most of Google Cloud technologies. Our classes include technical skills and best practices to help you get up to speed quickly and continue your learning journey. We offer fundamental to advanced level training, with on-demand, live, and virtual options to suit your busy schedule. Certifications help you validate and prove your skill and expertise in Google Cloud technologies.

Manual Last Updated: January 20, 2023

Lab Last Tested: January 20, 2023

Copyright 2023 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.