AWS CDK Pipelines: Real-World Tips and Tricks (Part 2)
In this article I’ll share with you some useful tips and tricks when using AWS CDK Pipelines that go beyond the simple demos and which can be implemented in your real-world applications. It is the second article in my series on AWS CDK Pipelines. Go here for the first one.
The AWS Cloud Development Kit (AWS CDK) is a powerful open source development framework for creating and deploying cloud application resources and can be considered to be an infrastructure-as-code tool. AWS CDK Pipelines is a library within this framework which brings infrastructure automation to a new level via, as I would call it, self-mutating pipelines-as-code. Such a CDK Pipeline encompasses application code, a corresponding code pipeline, the application’s cloud infrastructure, and an infrastructure pipeline, all defined by code and able to automatically adapt itself. A slightly more in-depth introduction in CDK and Pipelines can be found in the previous article in this series.
Currently, CDK Pipelines is still in developer preview but loads of teams and people are jumping on this fast pacing train of innovation. However, because it is still in a beta stage, best practices are not yet well defined by the development team, nor by the community. Because of this I want to share with you some tips and tricks I have compiled by working on several CDK Pipelines application (running in production!) at my current job.
Similar to the first article, the next sections all answer a specific question which I had — and most likely you will too. All sections follow the simple format “How do I do this?” and can be considered as stand-alone. They are not chronological, nor do they explain the basics of CDK Pipelines. None of the sections describe the sole solution to a specific problem or question but are a solution, based on personal and professional experience. Different from the first article is that this one will use Python as a development language instead of TypeScript because I, and other developers, were more Python-people. Hence, after the first CDK Pipelines application, we switched languages. However, the solutions to CDK Pipelines questions are trans-language and will benefit you regardless of your CDK development language.
Disclaimer: As the CDK development team extends the CDK libraries at a high pace and includes breaking changes, some of the details might not work exactly the way they do by the time you’re reading this.
How do I manage configuration of all environments?
Structuring configuration is always a tough one. I even avoided it in the first article because I hadn’t yet figured out a good approach myself. But now I’m pretty satisfied with my method.
Configuration is defined and managed at two levels:
- A raw config file
- Config objects created by processing of the raw config file
There is a third level if you wish to standardise configuration across multiple CDK Pipelines applications, for which I refer to the section “How do I manage configuration across CDK applications?”.
Raw config file
I distinguish two types of configuration. Configuration that affects the resources in AWS environments and configuration that affects your CDK application itself. Hence, the raw config file consists of those two parts, and application part and an environments part. This is how the high level structure of such a file looks like:
The application part contains information such as the application’s name, the git repository it is watching, the account and region of the central AWS build environment, the version of the CDK libraries, … The environments part contains sections per application environment such as development, staging, and production — not to be confused with an AWS environment which is an account + region combination. In this example, it also contains a default environment section which specifies the default behaviour of all environments unless explicitly overwritten in the respective sections. This is not a requirement and you can choose to not include this section. As a minimum, each environment section needs to contain the account and region the resources have to be deployed to. Additionally, this can contain configuration for your project specific resources (e.g. bucket name, auto-scaling group details, standard resources tags, …).
And example of such a file could look like this:
Config objects created from raw config file
Just as for the config file, make the distinction between application en resources configuration by defining two config classes. An AppConfig class and a ResourceConfig class to create config objects from the raw config file. Actually, a third RawConfig class is useful to parse the raw config files and merge the default config into each environment section.
A quick summary:
RawConfig
Used for processing the entire raw config file into more useable parts.
One instance of this class should be sufficient.AppConfig
Used to convert all raw application configuration into an application config object. This needs input of theRawConfiginstance. One instance of this class should be sufficient.ResourceConfig
Used to convert all raw environment specific configuration into an environment config object. This needs input of theRawConfiginstance.
Multiple instance of this class should be created, one for each AWS environment (account + region combination).
Here are snippets of how such classes can look like
The classes can then be used in the CDK pipeline application. Firstly, creating a PipelineStack instance in the app.py file requires the AWS environment where this pipeline needs to be deployed to. This is exactly the build_environment attribute of the AppConfig instance.
It is also useful the pass the RawConfig and AppConfig instances to the PipelineStack constructor since their content will be needed in that stack creation. E.g. the app configuration contains details of the repository to watch and the raw config instance is required to create environment specific config objects. This is how it all comes together:
Note that all information required at stage and stack level is contained in the config objects.
How do I manage configuration across CDK applications?
To standardise the way of handling CDK Pipelines configuration arccos multiple applications, the config classes discussed in How do I manage configuration of all environments? can be used as base classes. Each application can create more application-specific config classes by inheriting from those base classes. Publishing the base classes as a Python package which will make usability easier too.
In each CDK Pipelines application, create a configuration.py file that contains those classes:
You can now use these more specific config classes in the rest of your application. Moreover, you can customise the config classes as much as needed, e.g. add extra fields, include extra logic, extend the constructor method, …
How do I update my application’s CDK version?
This may sound like an intuitive question with an obvious answer, but due to the self mutating aspect of CDK pipelines, it is far from it. To properly answer this question, let’s first talk about what I mean with the application’s CDK version.
First a bit of background on CDK versions:
You can have more than one CDK version specified in your CDK Pipelines application. There are two main types, one is the CDK CLI version which is used to synthesis CloudFormation templates for your stacks, and the other is the version of the CDK construct (Python) packages which are used to define your construct with code. The former version should always be equal or higher than the latter. Additionally, make sure that all CDK packages have the same version.
Note that the multiple CDK CLI versions can be specified in your pipeline application. The one specified in your package.json is used for local development and for predefined pipeline actions such SimpleSynthAction.standard_npm_synth(). If no standard standard synth action is used, it is better to explicitly state which CDK CLI version your CodeBuild should use. Additionally, CDK CLI commands can also be used by your CodePipeline when it performs certain actions. By default, it uses the latest CDK version but you can specify a particular one.
To avoid surprises, it is good practices to keep all versions of CDK CLI and CDK packages the same:
Upgrading your CDK packages
When you want to upgrade the version of your application’s CDK packages to x.yz, you first need to update your pipeline’s CDK CLI version to be equal or greater than x.yz (wherever <CDK_VERSION>is defined e.g. in the config file and the package.json file) and trigger the pipeline (via commit/merge/…) such that it updates itself during the UpdatePipeline/SelfMutate stage/action. Next, update your application’s CDK packages to version x.yz with pip install aws-cdk.pipelines=x.yz (make sure to update all aws-cdk.abcd packages, not just pipelines). When developing in Python, do not forget to also export an updated environment.yml file (or equivalent) such that those new versions will also be used in the pipeline execution. The next time the pipeline is triggered, the creation of resources will happen with the upgrade CDK version.
If you don’t first update the pipeline’s CDK CLI version, the pipeline will try to synthesis CloudFormation templates with a CDK CLI version lower than your application’s CDK packages versions and this will fail. The synthesising happens during the synth action of the build stage in a CodeBuild instance with the specifications of the pipeline which is already in place, i.e. one with a CDK CLI version lower than the upgraded package versions. Only after the PipelineUpdate stage has succeeded is the CodeBuild instance for the next synth action updated.
How do I recover from being stuck in a failing Build Stage?
It is possible to be stuck in a failing Build stage (Synth action) when misconfiguring the specifications of the CodeBuild instance. Due to misconfiguration (e.g. a typo in the install commands of the SimpleSynthAction) this step can fail. This example has misspelled “install”:
After you realise your typo, you fix it and trigger the pipeline again with a new commit. And yet again, the Build stage failed. You notice that your typo-fix has not yet been included. At first, this seems strange but it is actually the expected behaviour. Your pipeline needs to be updated (the typo-fix) but this only happens in the UpdatePipeline/SelfMutate stage/action which only occurs after the Build stage. As the Build stage still has the configuration with the typo, fixing the pipeline via self-mutation is not possible.
There are two solutions.
- If you (as a developer) have the permissions to deploy a CDK pipeline, you can just redeploy a new pipeline which will directly have the specification with the typo-fix. You can do this from your local machine with
npx cdk deploy. - You can manually fix the typo in the CodeBuild specs (
Build project -> Build details -> Buildspec) and trigger the pipeline once more with theRelease Changebutton, all via the AWS Console.
Personally, I prefer the second option because it is faster.
This problem can occur quite frequently when developing the CDK pipeline in the beginning. Not just typos but bad build commands, wrong variables, … can all occur. To avoid such issues in the future, it is a good practice to specify the details of install/build/… commands outside of your CDK Pipeline’s source code and in a separate file. A useful place is the package.json file and call npm scripts from your pipeline’s CodeBuild. This way, the specifications of the CodeBuild can always remain the same e.g. npm run ci_build meaning the CodeBuild instance does not require an update/SelfMutate. If you make a typo if the ci_build script, you can easily fix it and trigger the pipeline again with a new commit. But this time, the CodeBuild will not be stuck in a failed state because from the CodeBuild’s perspective, the command is still npm run ci_build.
How do I remove my CDK Pipeline?
Just as deploying the pipeline via cdk deploy <my-pipeline-stack>, it can be removed by calling cdk destroy <my-pipeline-stack> from your machine.
Removing a CDK Pipeline via cdk destroy only destroys the pipeline and not the resources it had deployed.
How do I tear down my infrastructure resources?
Because removing a CDK Pipeline via cdk destroy only destroys the pipeline and not the resources it had deployed, the resources have to be destroyed separately.
The easiest way is to do this manually via the AWS Console. Go the CloudFormation in the account where your resources are deployed and press the delete button of all the desired stacks.
You can also try to do this automatically, but I have currently not yet create a way of doing this from within the pipeline. There are, however, some interesting discussions about this on GitHub. The tricky part is that you need to destroy them in the correct (reversed) order and your destroy action will need to assume a role with correct permissions in the deployed accounts in order to perform cdk destroy actions
How do I develop CDK stacks without interfering with the pipeline?
When the development team is large, it might be required that a single developer needs to create/test some AWS resources without breaking/interfering with the development environment. If this is the case, it is possible for that developer to deploy source resources to a separate spike environment without triggering the pipeline, hereby leaving the pipeline untouched.
Assuming the spike environment has been bootstrapped, the developer has to take following steps:
- Create a new stage object in
app.pyof the desired stage construct for the spike environment. Always create personal stages and not stacks at this level because otherwise CDK will not create a newassembly-*directory in the next step but it will produces that content straight intocdk.out/. In this specific example, the configuration of the spike account follows the method described in How do I manage configuration of all environments?
- Build the CDK app and synthesise the CloudFormation templates locally using
npx cdk synth.This will create a foldercdk.out/assembly-Personal, which contains all the files required to deploy the resources. - Deploy the Personal stage resources to the spike account with
npx cdk -a cdk.out/assembly-Personal deploy - Develop the stacks, do some testing, explore the AWS service, …
- When these resources are not needed any more, remove them by destroying the stage with
npx cdk -a cdk.out/assembly-Personal destroy
Note that none of the changes for the personal stage should be committed since the pipeline should not be triggered. This type of development is only relevant for temporary development by a single developer.
How do I avoid using from_lookup()?
In the case of vpc.from_lookup() the documentation already provides the answer:
This function only needs to be used to use VPCs not defined in your CDK application. If you are looking to share a VPC between stacks, you can pass the Vpc object between stacks and use it as normal.
If the required vpc is an external one (other application or other stage/environment), you can create a dummy vpc object of it with vpc.from_vpc_attributes(). This method requires at least that the availability_zones and vpc_id of the external vpc are known. With a dummy object, you can only use the attributes which you provide upon creation. So if you need to pass a SubnetSelection of the private subnets of the vpc to another construct, you need to pass those private_subnet_ids to the vpc.from_vpc_attributes() method.
Here is an example:
This explanation is equivalent for route53.from_lookup(). You need to use the from_hosted_zone_attributes() method and provide it with the hosted_zone_id and zone_name.
from_vpc_attributes() will not fetch information from the actual external resource!
When not using CDK Pipelines, calling this method will lead to a lookup when the CDK CLI is executed. You can therefore not use any values that will only be available at CloudFormation execution time (i.e., Tokens). The vpc information will be cached in cdk.context.json and the same vpc will be used on future runs. To refresh the lookup, you will have to evict the value from the cache using the cdk context command. More information about the context can be found in the documentation.
But because CDK Pipelines does not use context queries, explicitly providing the attributes with from_vpc_attributes() is equivalent as storing them in the cdk.context.json, only now you will store them in the config file.
This is equivalent for other from_xxx_attributes() methods.
How do I avoid using stack.availability_zones()?
The documentation reveals the problem:
Returns the list of AZs that are available in the AWS environment (account/region) associated with this stack.
…
If they are not available in the context, returns a set of dummy values and reports them as missing, and let the CLI resolve them by calling EC2 DescribeAvailabilityZones on the target environment.
Since CDK Pipelines does not support context queries, stack.availability_zones() will return dummy values. The solution is to overwrite the get availability_zones() getter method in the vpc stack with the correct availability zones. When developing in Python, you need to use the correct getter decorator:
Once again, I have shared my personal tips and tricks with you. I do have more, especially some Python specific once. So stay tuned for upcoming articles. In the meanwhile, the AWS CDK development team continues to extend the CDK libraries at a high pace, so some details I have used/explained might not be working any more.
I hope these tips and tricks have helped you in some way!
Do check out the previous article for more tips and trick if you haven’t done so.
