11 Using deployment stacks for grouping resources

This chapter covers

You have learned all about creating and updating Azure resources through IaC. In a template, you describe how you want your resources to be configured, and the Azure Resource Manager makes it so by creating or updating resources. Up to now, we haven’t paid any attention to removing resources you do not need anymore, except for mentioning the Complete deployment mode in chapter 4.

In chapter 4 you learned that Complete deployments are a powerful but risky method for removing resources you do not need anymore. The risk lies in the fact that they remove all resources in a resource group that are not in the current deployment. If someone else deployed something to the same resource group from another context, those resources would be removed as well. In practice, this risky side of Complete deployments is why they are not frequently used in the real world. Some teams use them in non-production environments, but Complete deployments in production environments are rare. We will revisit this in more detail later in this chapter.

Because of the drawbacks of Complete deployments, the call for a reliable mechanism for removing resources through IaC has remained. And as we write this in the spring of 2022, a solution has been developed by the Azure Resource Manager team: deployment stacks. In this chapter, you’ll learn why you might need another grouping mechanism for resources, in addition to resource groups and subscriptions, to remove resources in a less risky way. You will also learn how deployment stacks can be used to provision resources for others without granting control over those resources.

But first a word of warning: deployment stacks are still under heavy development by the Azure Product Group. While we wrote this chapter, the feature was still in private preview and available to only a handful of users. By the time you read this, deployment stacks might be in public preview or will enter public preview soon. Consequently, the code examples in this chapter might require some tweaking by you, as the details of deployment stacks might have changed. If you ever run into such problems, you can always find our updated code examples in the GitHub repository accompanying this book: https://github.com/AzureIaCBook/book-templates. With that out of the way, let’s take a look at why you might want to start using deployment stacks.

11.1 Grouping resources by their lifetime

In Azure, your resources are organized into a multilevel hierarchy: resources are grouped into resource groups, resource groups into subscriptions, and subscriptions into management groups. This hierarchy works really well for building an extensive Azure infrastructure, where you want to delegate the responsibility for some level in the hierarchy to a business unit, department, or team. This hierarchy is often used for grouping resources by who can access them.

However, this way of grouping does not match the official Microsoft guideline for grouping resources by their lifetime. The goal should be that all resources in a resource group have the same lifecycle: they should all be created, updated, and deleted simultaneously.

These two types of groupings do not always overlap. Let’s look at an example. In figure 11.1, you’ll see the infrastructure for two application components that communicate over a service bus queue. To support this scenario, three resource groups are created:

Figure 11.1 Grouping resources into resource groups

This resource group structure, shown in figure 11.1, aligns very well with the organization that works on this infrastructure: three teams, all of which are responsible for the resources in a single resource group.

However, when we look at the deployment lifecycles, another grouping can be made. For example, you might notice that there are resources in the Producer and Consumer resource groups with possibly different lifetimes, as shown in figure 11.2.

You will recognize the following lifecycles, marked by the grey boxes:

Figure 11.2 Different lifecycles, indicated by dashed lines, in the Producer and Consumer resource groups

This example illustrates how a single resource group, with a sensible grouping of resources, can have resources with mixed lifetimes. This is where deployment stacks come in: they provide a custom grouping concept for organizing resources into a stack of resources with a shared lifetime.

Before we move on to explore deployment stacks in detail, let’s look at why the Complete deployment mode you learned about in chapter 4 is not good enough for dealing with these situations.

11.1.1 Complete deployment mode is not good enough

The first and foremost reason why Complete deployments do not cover all cases is that a Complete deployment assumes that the scope in which resources that are no longer declared should be deleted is the same as the deployment scope. The Venn diagram in figure 11.3 illustrates this.

Figure 11.3 Difference between handling normal and “hidden” resources when deploying in Complete mode

Here you see that Complete deployments do not remove existing “hidden” resources, like alert rules and RBAC assignments. Traditionally, complete deployments ignore some resource types, like role assignments and alert rules, because these resource types already existed before they were exposed as resources that could be managed through IaC. To maintain backward compatibility with how Complete deployments worked before these resources became available in the control pane, they are ignored during Complete deployments. In other words, resources that were not removed during a Complete deployment before will also not be removed during a Complete deployment in the future. Also, Complete deployments are limited to a single scope, so they cannot span multiple resource groups or subscriptions.

These limitations make Complete deployments useless for the following cases:

Deployment stacks do not have these limitations. They can span multiple resource groups or subscriptions and inherently only consider resources that were part of the previous version of the stack for removal.

11.1.2 Deployment stacks to the rescue!

A deployment stack is a new type of Azure resource that acts as a custom grouping around one or more other resources. It is not a hierarchical grouping like resource groups or subscriptions but instead a group you create yourself and to which you add one or more resources using a nested template. A deployment stack can span multiple resource groups or even subscriptions, as shown in figure 11.4.

Figure 11.4 A single deployment stack can span multiple resource groups and subscriptions.

When you later update the deployment stack or the resources that are part of the stack, the stack will infer whether there are resources that were part of the stack in the previous deployment but are not in the current deployment. When that is the case, it will remove or disconnect those resources, depending on your configuration.

This might sound a bit abstract, so let’s explore it using an example. In the following sections you will create, update, and remove a deployment stack to get a feeling for how it works. At first, you will create a service bus namespace with a single queue as part of a deployment stack. Later on, you will change the name of that queue in the template a few times to see how the stack handles these changes. The name of the actual resource cannot be changed, so new resources will be created and old ones will be removed.

11.1.3 Creating a deployment stack

To start exploring deployment stacks, let’s create a stack with a service bus namespace and a queue, as shown in the following listing.

Listing 11.1 Deploying a template stack

{
    "$schema": "https:/ /schema.management.azure.com/schemas/
         2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "resources": [
        {
            "name": "messagingStack",
            "type": "Microsoft.Resources/deploymentStacks",    
            "apiVersion": "2021-05-01-preview",
            "properties": {
                "updateBehavior": "detachResources",
                "template": {                                  
                    "$schema": "https:/ /schema.management.azure.com
                         /schemas/2019-04-01/deploymentTemplate.json#",
                    "contentVersion": "1.0.0.0",
                    "resources": [
                        {
                            "name": "messaging-example-stack",
                            "type": "Microsoft.ServiceBus/namespaces",
                            "apiVersion": "2021-01-01-preview",
                            "location": "[resourceGroup().location]",
                            "sku": {
                                "name": "Standard"
                            },
                            "resources": [
                                {
                                    "name": "exampleQueue",
                                    "type": "queues",
                                    "apiVersion": "2021-01-01-preview",
                                    "dependsOn": [
                                        "[resourceId('Microsoft.ServiceBus/
                                           namespaces', 
                                           'messaging-example-stack')]"
                                    ]
                                }
                            ]
                        }
                    ]
                }
            }
        }
    ]
}

The new resource type for deployment stacks

The resources in a stack are described using an ARM template.

In this template you’ll see a new resource type being used, deploymentStacks, which creates the deployment stack.

Deployment stacks have two type-specific properties: updateBehavior and template. The updateBehavior property is only used when you’re updating a stack, so that will be discussed in the next section. The value for the template property is a nested template that should describe the resources that are part of the stack: in this case, a service bus namespace and a queue within that namespace.

If you deploy this template to any resource group, you will see that both the namespace and the queue are created as declared. Now, let’s explore what happens when you update the stack.

11.1.4 Updating a deployment stack

Continuing on with the template that declares a service bus queue, let’s assume the name of the queue has to be updated from exampleQueue to betterExampleQueue, due to changing requirements within the organization. If you make this change and deploy the template again, you will see that a new queue with the name betterExampleQueue is created, but the original exampleQueue queue is not removed.

This might not be what you expected, given that we introduced deployment stacks as a mechanism for automatically removing resources that are no longer part of the stack. To understand what is going on, let’s take a closer look at the updateBehavior property. This property determines what should be done with resources that are no longer described in the stack, and it can have two values:

To test this, update your template again and set the value for updateBehavior to purgeResources and redeploy the template. What do you see? Do both queues still exist?

Again, you might be surprised that they do, but this is the correct behavior. In the second deployment you did, the first queue (exampleQueue) was removed from the stack. As a result, redeploying the same stack with updateBehavior=purgeResources did not detect the resource as missing from the stack declaration, so it did not remove the resource from Azure.

Now make a fourth and final change to your template: rename the queue for a second time and call it finalExampleQueue. If you redeploy the template one more time, you will see the expected behavior, where finalExampleQueue is created, and betterExampleQueue is removed.

To better understand what you have seen, take a look at figure 11.5. Here you see how, at first, the contents of the resource group are the same as described in the initial stack deployment. Next, the three consecutive updates are shown on the right, updating the resource group. You can see how the exampleQueue is detached from the stack at the first update, which is why it is not removed from the resource group during the third deployment.

Figure 11.5 The four deployments of the same stack and how they affect the resources in a resource group

11.1.5 Removing a deployment stack

When you delete a stack, the resources that were part of the stack are automatically detached: they are no longer part of the stack, but they are not removed from Azure. This is also the case in the last deployment, with updateBehavior set to purgeResources. If you want to also remove the resources, you will have to do two deployments:

  1. First, deploy the stack again with updateBehavior set to purgeResources and with an empty template.

  2. Once the deployment of the now empty stack is complete, you can remove the stack itself.

note This is the behavior at the time of writing and will change in the future. The final design for deployment stacks will allow you to specify either detach or purge behavior. However, it is not yet known when this will be released.

Now that you have created, updated and deleted a deployment stack, let’s take a look at another important use case that you can use deployment stacks for, namely locking resources.

11.2 Provisioning resources for others, but disallowing updates

Imagine you are working in the central cloud team for a large organization. One of your responsibilities is providing all decentralized application teams with connectivity to the central network. In Azure you can do this using a hub-and-spoke network, which is an architecture where each decentralized team gets its own vnet, called a spoke, which is peered with a single central network, called the hub. It would look something like figure 11.6.

Figure 11.6 Hub-and-spoke network architecture

Of course, you’ll want to decide on the IP ranges for the vnets that you create for the application teams, and you’ll want to deny them from changing those. But application teams should have visibility into the vnet configuration and be able to connect VMs and private endpoints to the vnet. The question is, how do you authorize others to work with a resource, yet deny them the authorization to change it?

11.2.1 Azure Blueprints: A first solution

In Azure there is an existing solution for this type of problem: Azure Blueprints. Blueprints are another syntax for IaC on Azure that you can use to describe a group of resource groups, role assignments, policy assignments, or ARM templates. A group of one or more of these resource descriptions can be combined into a single blueprint.

Note This discussion of Blueprints is a high-level overview on purpose. Blueprints is available in Azure today, but it has been announced that they will be replaced in the future. For that reason, we won’t discuss Blueprints in depth.

Once you have written a blueprint, you can deploy it to Azure to create an initial version of the blueprint definition. Any subsequent deployment of an updated blueprint will create a new version of the blueprint.

To use a blueprint version, you assign it to a subscription. This assignment invokes the creation of all the resources described in the blueprint and the deployment of the ARM templates in the blueprint. If the blueprint specifies one or more parameters, the values for these parameters will have to be provided at the time of the assignment. Figure 11.7 shows how these concepts are related.

Figure 11.7 Blueprint definitions, versions, and assignments

The first time you deploy a new blueprint into Azure, you actually create two resources: the blueprint itself, and the first blueprint version. During subsequent updates, you don’t change what already exists, but only add new versions of the blueprint. This is useful if you have tens or even hundreds of blueprint assignments. An assignment is for a specific version, so you can upgrade your assignments whenever you want, instead of having an enforced update due to a change in the blueprint definition. Just like a template, a blueprint can specify one or more parameters, for which values have to be provided as part of the assignment.

A common use case for blueprints is provisioning a virtual network to subscriptions, and connecting that to a central, organization-wide network. In such a situation, you would create a blueprint that creates a resource group, deploys a vnet into that resource group, and peers it to the central network.

But this solves only half of the problem: provisioning resources to a subscription. To solve the other half of the problem, blueprints offer one specific property that ARM templates and Bicep do not: a blueprint assignment can be locked. This is not a resource lock that you can apply to any resource. Instead, these locks protect all the resources that were created by the blueprint assignment from being updated or deleted, and they can only be removed by specific users.

It sounds like a wonderful solution, right? But here comes the catch: Azure Blueprints is still in preview, and it has been so for years. More recently, it has been announced that Azure Blueprints will never be formally released. Instead, deployment stacks will be enhanced with blueprint-like locking functionality so you can deploy resources to be used by others.

This will pose a challenge for you when you are in need of the functionality provided by Blueprints or deployment stacks, as you will have to choose between using a technology that you know will disappear or choosing a technology that is not fully mature yet. Which option is best will differ from context to context. If you decide you need to start using Azure Blueprints, you can read more about it in Microsoft’s “What is Azure Blueprints?” article (http://mng.bz/Pn18).

Finally, it is important to know that the Product Group plans to deliver functionality for converting blueprints into deployment stacks in the future. It is not yet known whether this will be a manual or automated process.

11.3 The future of deployment stacks

When we were writing this chapter, deployment stacks were still in private preview. Preview means that the technology’s implementation details might still change between us writing and you reading this chapter. Also, the capabilities of deployment stacks will continue to grow while the Product Group continues to work on the feature. If you want to keep getting the latest regarding deployment stacks, keep an eye on Microsoft Learn, the Azure Blog, or the GitHub repository accompanying this book.

Summary