Videos
For larger systems it is common to split infrastructure across multiple separate configurations and apply each of them separately. This is a separate idea from (and complimentary to) using shared modules: modules allow a number of different configurations to have their own separate "copy" of a particular set of infrastructure, while the patterns described below allow an object managed by one configuration to be passed by reference to another.
If some configurations will depend on the results of other configurations, it's necessary to store these results in some data store that can be written to by its producer and read from by its consumer. In an environment where the Terraform state is stored remotely and readable broadly, the terraform_remote_state data source is a common way to get started:
data "terraform_remote_state" "resource_group" {
# The settings here should match the "backend" settings in the
# configuration that manages the network resources.
backend = "s3"
config {
bucket = "mycompany-terraform-states"
region = "us-east-1"
key = "azure-resource-group/terraform.tfstate"
}
}
resource "azurerm_virtual_machine" "example" {
resource_group_name = "${data.terraform_remote_state.resource_group.resource_group_name}"
# ... etc ...
}
The resource_group_name attribute exported by the terraform_remote_state data source in this example assumes that a value of that name was exposed by the configuration that manages the resource group using an output.
This decouples the two configurations so that they have an entirely separate lifecycle. You first terraform apply in the configuration that creates the resource group, and then terraform apply in the configuration that contains the terraform_remote_state data resource shown above. You can then apply that latter configuration as many times as you like without risk to the shared resource group or key vault.
While the terraform_remote_state data source is quick to get started with for any organization already using remote state (which is recommended), some organizations prefer to decouple configurations further by introducing an intermediate data store like Consul, which then allows data to be passed between configurations more explicitly.
To do this, the "producing" configuration (the one that manages your resource group) publishes the necessary information about what it created into Consul at a well-known location, using the consul_key_prefix resource:
resource "consul_key_prefix" "example" {
path_prefix = "shared/resource_group/"
subkeys = {
name = "${azurerm_resource_group.example.name}"
id = "${azurerm_resource_group.example.id}"
}
resource "consul_key_prefix" "example" {
path_prefix = "shared/key_vault/"
subkeys = {
name = "${azurerm_key_vault.example.name}"
id = "${azurerm_key_vault.example.id}"
uri = "${azurerm_key_vault.example.uri}"
}
}
The separate configuration(s) that use the centrally-managed resource group and key vault would then read it using the consul_keys data source:
data "consul_keys" "example" {
key {
name = "resource_group_name"
path = "shared/resource_group/name"
}
key {
name = "key_vault_name"
path = "shared/key_vault/name"
}
key {
name = "key_vault_uri"
path = "shared/key_vault/uri"
}
}
resource "azurerm_virtual_machine" "example" {
resource_group_name = "${data.consul_keys.example.var.resource_group_name}"
# ... etc ...
}
In return for the additional complexity of running another service to store these intermediate values, the two configurations now know nothing about each other apart from the agreed-upon naming scheme for keys within Consul, which gives flexibility if, for example, in future you decide to refactor these Terraform configurations so that the key vault has its own separate configuration too. Using a generic data store like Consul also potentially makes this data available to the applications themselves, e.g. via consul-template.
Consul is just one example of a data store that happens to already be well-supported in Terraform. It's also possible to achieve similar results using any other data store that Terraform can both read and write. For example, you could even store values in TXT records in a DNS zone and use the DNS provider to read, as an "outside the box" solution that avoids running an additional service.
As usual, there is a tradeoff to be made here between simplicity (with "everything in one configuration" being the simplest possible) and flexibility (with a separate configuration store), so you'll need to evaluate which of these approaches is the best fit for your situation.
As some additional context: I've documented a pattern I used successfully for a moderate-complexity system. In that case we used a mixture of Consul and DNS to create an "environment" abstraction that allowed us to deploy the same applications separately for a staging environment, production, etc. The exact technologies used are less important than the pattern, though. That approach won't apply exactly to all other situations, but hopefully there are some ideas in there to help others think about how to best make use of Terraform in their environment.
You can destroy specific resources using terraform destroy -target path.to.resource. Docs
Different parts of a large solution can be split up into modules, these modules do not even have to be part of the same codebase and can be referenced remotely. Depending on your solution you may want to break up your deployments into modules and reference them from a "master" state file that contains everything.
In my current project I'm following a layering pattern for deploying infrastructure to Azure, like the one explained in this article: https://www.padok.fr/en/blog/terraform-iac-multi-layering
So I currently have 4 layers:
-
A bootstrap layer, creating the resource group and state storage (intended to be fire-and-forget)
-
The network layer, setting up the VNET, subnets and firewall/network security group rules
-
Data layer, setting up storage accounts & SQL databases
-
The apps layer, setting up the app services etc. There could be multiple layers here on the same "level", but for different services.
Each of the layers has its own state. Now, this works for the most part great for us, and gives us a nice and tidy organization of our resources. BUT, in the various layers we often need to refer to resources that has been created earlier.
A good example is the subnets, in which we put the SQL databases and app services as well as storage accounts. They are created in the network layer, with a certain naming convention. In e.g. the apps layer, I am using data sources to resolve references, but then I need to know the name of the subnet I want to use. This of course works, but I have to duplicate the subnet name. If I change e.g. the subnet's name, I would have to remember to update ALL subsequent layers.
What is the best way to refer to resources created in earlier stages of deployments? Since all layers are distinct configurations, I can't directly refer to them. Here's the options I've thought of so far:
-
Simply do as I do now - just refer to the subnet (which also requires me to pass in the VNET name as well). For me, this smells - but I'm open to be convinced otherwise.
-
Define all these names for resources that will be referred to in multiple layers as variables that are passed in. I fear this will blow up the number of variables I need to pass in, but somehow feels better than the first option as I then can define the subnet names and VNET name in a common .tfvars file.
-
I have considered simply creating a module in e.g. the network layer that exposes the subnet IDs as outputs. That way at least I can have one place where I define the names. I don't know if this considered an anti-pattern or not, and if so for what reason.
Does anyone have any experience with this design pattern for Terraform, and how to best resolve resources in subsequent layers?
I am trying to make sense of the mechanics and best practices around splitting Terraform code into independent root modules that build on top of each other.
An example of this is Google Cloud's Enterprise foundations blueprint which consists of multiple layers of terraform root modules:
0 bootstrap
1 org
2 environments
3 networks-dual-svpc / networks-hub-and-spoke
4 projects
5 app-infra
I would be interested to hear how much layer separation of this kind is happening in real world projects and how the Terraform root modules are connected (e.g. using terraform_remote_state or something else).
One particular thing I am wondering about is how to set up the Terraform deployment itself. I can see how the automated deployment of a Terraform root module requires infrastructure that already has to be set up. Things like:
A Terraform backend to persist the state
A service account to access the infrastructure
A deployment pipeline that runs
terraform applyand has access to the service account and the Terraform backend
This cannot be managed by the Terraform root module that is supposed to be deployed by it, right? It either has to be set up by a lower layer or by hand. How do you deal with this?
Thanks.