Infrastructure as code (IaC) has an interesting testing problem. It's relatively easy to test that your code produces the configuration you expect, but that doesn't necessarily prove that the cloud provider created the resources you intended.
When testing software, we generally talk about three levels of testing. Unit tests test a small piece of code in isolation. Integration tests verify that multiple pieces of software work together. End-to-end (E2E) tests go a step further and exercise the system through its real-world workflow, including the external systems it depends on.
A unit test might verify that a Pulumi component produces the expected properties. An integration test might verify that several of our components work together. An E2E test can actually deploy those components to Azure and verify that Azure created what we expected.
That's what I wanted to do with my Pulumi components.
Pulumi already provides testing capabilities for infrastructure code. However, those tests didn't fit particularly well into my team's existing TypeScript-based development workflow. Rather than introduce another language and testing framework specifically for infrastructure tests, I decided to use Vitest and Pulumi's Automation API.
Vitest gives us a familiar testing framework with describe, test, expect, beforeAll, and afterAll. Pulumi's Automation API lets us control a Pulumi deployment programmatically. Put those two pieces together, and we can have a test suite that creates a temporary Pulumi stack, deploys real resources to Azure, runs assertions against those resources, and then cleans everything up when the tests are finished.
The result is an infrastructure E2E test that looks much more like the rest of our TypeScript tests than a separate infrastructure testing system.
We don't have to rely on Pulumi alone to tell us what happened. We can verify the actual resources in Azure as well. For that, I use the Azure CLI and Node and compare the Azure output with the outputs returned by Pulumi. Node can spawn a process and return the output stream. I wrote a wrapper for the Azure CLI tool that can pass arguments from the method, spawn the CLI as az [...params] --output json then use that output JSON to run tests against.
The resulting flow looks something like this:
---
config:
theme: 'dark'
---
sequenceDiagram
participant V as Vitest
participant P as Pulumi Automation API
participant A as Azure
participant CLI as Azure CLI
V->>P: Create temporary stack
V->>P: pulumi up
P->>A: Deploy resources
A-->>P: Resources created
P-->>V: Deployment outputs
V->>CLI: Query deployed resources
CLI->>A: Get resource details
A-->>CLI: Resource JSON
CLI-->>V: Azure resource data
V->>V: Run assertions
V->>P: pulumi destroy
P->>A: Delete resources
A-->>P: Resources deleted
P-->>V: Cleanup complete
We're not just testing that Pulumi believes it deployed the correct infrastructure; we're testing that the infrastructure actually exists in Azure in the state we expect.
For example, I wanted to get the details of an Application Insights resource from Azure. I created an interface for the JSON properties I cared about and then called my helper method as shown:
interface ApplicationInsightsResult {
id?: string;
name?: string;
location?: string;
kind?: string;
tags?: Record<string, string>;
ingestionMode?: string;
workspaceResourceId?: string;
connectionString?: string;
}
appInsights = await azJson<ApplicationInsightsResult>([
'monitor',
'app-insights',
'component',
'show',
'--app',
appInsightsName,
'--resource-group',
resourceGroupName,
]);The resulting command would look like this: az monitor app-insights component show --app [appInsightsName] --resource-group [resourceGroupName] --output json
Do be aware that this process can take time because I'm querying real Azure information via the CLI. Depending on your connection or Azure's current conditions, the CLI command could take several seconds to complete.
I'm also calling this method in Vitest's beforeAll method which means that the CLI command will execute each time you run a test suite.
Building the E2E Test
In my team's case, we have several custom resource components with the goal of either covering gaps in Pulumi's out-of-the-box components or a corporate rule. For example, we deploy an app insights and smart detector rules for app services. Or 99% of the time, storage account resources cannot have public access. Neither is a fault in the Pulumi offering, but it is something that we as a team have decided to do.
Test Program
Let's take a look at the Pulumi Automation API program code.
const testProgram = async () => {
const appIn = new applicationInsights.ApplicationInsightsComponent(
'e2e-app-in-test',
{
appInsightsName: 'ai-e2e-test',
resourceGroupName: resourceGroupName,
environment: helpers.AzureEnvironment.Development,
tags: {
Owner: 'Cloud',
Application: 'Pulumi E2E Test',
},
},
);
return {
id: appIn.id,
connectionString: appIn.connectionString,
proactiveDetectionResources: appIn.proactiveDetectionResources,
};
};Some of this is specific to my team, of course, but the idea should carry over. I'm using my custom application insights resource I created applicationInsights.ApplicationInsightsComponent and passing the required parameters. This works just like a native Pulumi component. It's important that the program returns any outputs that you might want to test against.
Before All
beforeAll(
async () => {
const stackName = `e2e-temp-stack-${Date.now()}`;
stack = await LocalWorkspace.createOrSelectStack(
{
stackName,
projectName: 'e2e-app-in-test',
program: testProgram,
},
{pulumiCommand},
);
await stack.setConfig('azure-native:location', {
value: location,
});
const upResult = await stack.up({
onOutput: console.log,
onEvent: console.log,
});
outputs = upResult.outputs;
appInsights = await azJson<ApplicationInsightsResult>([
'monitor',
'app-insights',
'component',
'show',
'--app',
appInsightsName,
'--resource-group',
resourceGroupName,
]);
},
15 * 60 * 1000,
);This section describes an action we want to run before all the tests in this suite run. This is where we tell Pulumi to run, create the stack, and create the resources. It's also where we'll spawn the Azure CLI command to capture the real values. I'll show what azJson looks like in just a bit.
After All Clean Up
afterAll(
async () => {
const skipCleanup = process.env['SKIP_PULUMI_CLEANUP'] === 'true';
if (skipCleanup) {
console.log('Skipping Pulumi cleanup because SKIP_PULUMI_CLEANUP=true');
return;
}
if (stack) {
await stack.destroy({
onOutput: console.log,
});
await stack.workspace.removeStack(stack.name);
}
},
15 * 60 * 1000,
);Unless SKIP_PULUMI_CLEANUP is set in the environment, we want all the resources in Azure to be cleaned up after the test suite is complete. This is extremely useful when developing/debugging the test. You can deploy the infrastructure, skip cleanup, manually inspect Azure, and then clean it up afterward.
We also want the stack removed because it's just a placeholder for the test.
This is the great part about using Pulumi's Automation API. The whole process is completely encapsulated in our test file.
The Actual Tests
Armed with the Pulumi and the Azure CLI outputs, we can test various things.
We can test the Pulumi outputs:
test('Connection string output should be defined', () => {
expect(outputs.connectionString.value).toBeDefined();
});We can test that Azure created the resource with the name we expected:
test('Application Insights should have the correct name', () => {
expect(appInsights.name).toEqual(appInsightsName);
});We can compare the Pulumi output values with the actual Azure resource:
test('Application Insights should have the correct connection string', () => {
expect(appInsights.connectionString).toBeDefined();
expect(appInsights.connectionString).toEqual(outputs.connectionString.value);
});Azure CLI Wrapper
Here's the azJson method I mentioned previously:
const execAsync = promisify(exec);
export async function azJson<T>(args: string[]): Promise<T> {
const flatArgs = [...args, '--output json'].join(' ');
console.log(`Attempting to run 'az ${flatArgs}'`);
const {stdout} = await execAsync(`az ${flatArgs}`);
console.log(stdout);
return JSON.parse(stdout) as T;
}It simply smashes all the arguments into a single, space-separated string and then uses Node's exec to spawn the az program with those arguments
Conclusion
Pulumi makes it easy to unit test your components. They provide the Automation API that makes it easy to create E2E tests. We can build out tests that verify Azure and Pulumi agree with what we asked them to do.
If you'd be interested in a post about setting up a unit or integration test using Vitest with Pulumi, please reach out to me on X or LinkedIn.