Determining the owner of an NHI is extremely important when managing identities in your organization. If the organization encounters an NHI-related incident, knowing which human is responsible can be a great shortcut on the journey to remediation.

Creating NHI ownership in your friendly neighborhood cloud provider is not always easy, but it is pretty straightforward. When a human directly creates or modifies an identity, you can find the relevant event in the logs and see who is responsible. But does this apply to IaC-generated identities as well?

IaC ownership

The technical challenge of identifying ownership for NHIs created by IaC is much harder since the basic question of who is the owner of an IaC resource is much harder.

Let's start small:

‍

Introduction

Infrastructure as Code (IaC) is a major tool for creating scalable environments in the cloud. With a single command, you can create hundreds of accounts, servers, policies, and identities. It is a tough task to monitor identities, especially non-human identities (NHIs), but IaC-generated identities make it even harder.

Most IaC setups include several platforms, making the management process decentralized and shared across multiple technologies. On top of that, IaC-managed environments change much faster, making it harder for security teams to keep up with the pace.

The basic question to be answered when dealing with NHIs is: Who is the owner of an identity? This question is especially tough to answer when the creator is an automated process triggered by commits or deployments.

We tried to answer this question in different ways, one of which was a tag-based approach. In this blog post, I’ll be sharing the challenges in the journey to master this approach, but first, let’s talk about ownership.

IaC Ownership

Ownership

Direct Role Creation

I am a developer at a small company, and I requested my DevOps team (kindly) to create a few resources, including a role. I have a simple Terraform deployment that directly creates an aws_iam_role in my AWS account - let’s call it danz_role.

After the Terraform deployment runs, a dedicated Terraform AWS role will create my new role.

If we try to find the owner of this role as if it were a regular identity, the owner will be the Terraform IaC dedicated role - another NHI.

We need to take a different approach here and find the human in charge of creating this role. So, who is it?

• The DevOps engineer who ran this deployment?
• Or me, the developer who requested it?

Let's make it even more interesting.

CICD-Triggered Deployments

‍

I have a medium-sized company with CI/CD running deployments triggered by commits on relevant files. This CI/CD workflow is in charge of updating the environment, each time the IaC repo is modified - the CI/CD workflow runs the IaC engine to apply the changes in the code on the environment.

Every CI/CD run that results in the creation of a new role raises the question: Who is the human in charge of this role? The deployment ran automatically, so who is responsible?

• Is it the DevOps engineer who configured this CI/CD pipeline?
• Or the developer who committed the code that triggered it?

And what if the role is created using a module?

‍

A module is a feature in IaC that prevents code duplication by acting as a reusable template for resource creation. Developers only need to pass the necessary parameters instead of rewriting the entire configuration.

A developer can use a module to create entire environments. If this module attaches an overprivileged policy to a role, who’s to blame?

• The developer who used this module?
• Or the DevOps engineer who wrote it?

Can a role be co-created by more than one human? CloudTrail would say no (if it could speak), but as we already know, some answers depend on the context.

IaC ownership - How to

The process of creating resources using IaC consists of many different components and services.

‍

The graph above shows how a commit to an IaC repo turns into an actual role in your CSP.

When it comes to IaC, the code is an integral part of the identity. Whoever wrote the declaring lines for a new role is de facto the creator of the identity.

Okay, so let’s simply check which human is in charge of the code for creating new roles, and we could have ownership for all our IaC-generated identities!

We need a few things to do so:

1. For each IaC-generated identity, find the code that created it.
2. Find which human committed or edited this code.
3. Et voilà! We have it!

Step 2 is fairly simple - most code management platforms have endless retention for commit history by design to deal with similar challenges regarding code ownership.

Step 1, and now it gets harder, is where we need to determine which lines of code are responsible for creating new identities.

IaC - Infinite amount (of) Capabilities

As we already know, there is more than one way to create resources in IaC. Our goal is to find a solution that works independently of which IaC implementation concepts the DevOps team decides to use. How do we do that?

IaC projects can have complex structures where data is ingested and diffused into these structures to form resources with the intended names and functionalities.

The ideal solution would involve an IaC reading mechanism, some kind of parser capable of reading a complete infrastructure repository and locating:

1. Configuration files that initiate the IaC process.
2. Module files that are used during the creation of each identity.
3. Variable files that contain the values forming the identity.
4. Every other file relevant to identity creation.

Developing such a mechanism could take a long time and might not be the most efficient solution.

We also explored open-source tools for parsing IaC code. Terraform, for instance, has a few tools that support parsing, but parsing is not enough. We need to go through the complete deployment process and follow the diffusion of values between IaC files, sometimes across multiple repositories.

Tagging approach

There are many approaches to tackle this challenge. We divided them into three:

1. Log-based - Digest logs from all related platforms and infer ownership accordingly.
2. Plan execution - Will explain in a minute.
3. Code analysis - Analyzing the IaC code from different angles without parsing the whole thing.

I chose to elaborate the plan execution approach that uses tags.

For the sake of specific examples, I’ll stop writing about IaC in general and focus on Terraform instead. Terraform is the most common IaC tool, so we figured it would be a good place to start.

My ideal solution was to read the original code and track the resource generation process. And what better way to read Terraform code than using Terraform itself?

Here is my master plan:

The result - an enhanced plan file containing extra information needed to determine which pieces of code are responsible for the creation of each identity.

During this process, I wanted to ensure that no queries or requests were sent to any services in the original environment. I didn’t want to affect it in any way. Running terraform plan made sense to me because this stage of IaC shouldn’t need to actively interact with the actual environment - it simply reads the code and generates the desired state accordingly (will talk about state files later).

IaC - In a Glimpse

The idea of turning code into infrastructure is pretty simple. A declarative language in which each resource has a predefined value and set of attributes where you can write everything you need in the right format and save it in a repo.

Every Terraform project has a state file that keeps an up-to-date state of the environment. This file holds records of every resource created by the IaC process. Whenever a change to the code is made, terraform checks the state file and determines which changes are needed.

The basic stages of running your Terraform project:

1. terraform init - Initializes the Terraform engine based on the version specified in your configuration file.
2. terraform plan - Reads the code and compares it to the current state, calculates the differences and generates an executable format that a dedicated role in your CSP can use.
3. terraform apply - applies the necessary changes suggested by the plan to reach the desired environment as described in the code.

Terraform plan

My first milestone - take an environment and run terraform plan locally. Terraform configurations usually interact with a few external services:

1. Target CSP
   1. State management - Most commonly, the state file is stored in an S3 bucket in your infrastructure account.
   2. Actual execution - The Terraform script interacts with the cloud provider to apply changes.
2. Git repositories - Dedicated Terraform modules are often stored in code management platforms. A main Terraform file that uses a module specifies the module’s path in the source code.

These interactions can be avoided by modifying the configuration to run locally:

1. Local state management - The state file is created and stored locally instead of in a remote bucket.
2. No execution is needed - Since we are only running terraform plan, no changes will be applied to the actual environment.
3. Clone modules locally - Instead of using remote Git repositories, we can modify the configuration to use local copies of the modules.

After minor changes and some trial and error, I successfully executed a Terraform plan on a real repository.

To ensure there were no unexpected interactions with the live environment, I ran the terraform plan command without an internet connection. The entire process was executed locally, without requiring any inputs from external services.

The only part that initially required network access was terraform init, but this step has no interaction with the target environment.

Modify the code - gotta tag em all

If I had a logging mechanism that tracked each step while Terraform formed an identity, it would be perfect. However, Terraform logs don’t focus on this part of resource formation and mainly provide functional monitoring for basic deployment steps.

What I needed from Terraform was to log every file involved in an identity’s creation during execution.

I tried using a feature supported across most platforms for IAM resources - tags. A tag is a string that can be attached to resources.

I wrote a script that scans the repository and adds a tags attribute to each relevant file. If tags already exist, I appended another one.

This includes files that contain:

• IAM resources (users, roles, instance profiles, service accounts, managed identities, etc.).
• Modules that define IAM resources.
• Other files defining identities/policies.

Enhanced plan file

After running terraform plan again, this time after modifying the code, the result is pretty cool.

{
   "address": "module.single_role['danz'].aws_iam_role.role",
   "mode": "managed",
   "type": "aws_iam_role",
   "name": "role",
   "provider_name": "registry.terraform.io/hashicorp/aws",
   "schema_version": 0,
   "values": {
       "description": "Just an AWS role",
       "name": "danz-role",
       "path": "/",
       "tags": {
           "ModuleFilePath": "/Users/danz/modules_repo/single_role/single_role.tf",
           "ResourceFilePath": "/Users/danz/modules_repo/resources/single_role/main.tf",
           "InputFilePath": "/Users/danz/infra_repo/src/us-east-1/my_env/roles/danz.json"
   }
}

The JSON above describes the role named danz-role, which was created by a module named single_role.

• The module is defined in a file named single_role.tf.
• The resources it creates are detailed in a separate file named resources.tf.
• I also used a feature in one of Terraform’s wrappers (Terragrunt) that allows input via .json files. This file is also included in the tagging for this identity.

At first some tags didn’t make it all the way to the plan file. Some modules didn’t handle tag inheritance well since they weren’t expecting tags to be passed down. I had to modify those modules as well and add a tags attribute wherever it was needed.

Single identity, different owners

Now I have a list of files related to each identity. When someone commits changes to them, they will be considered an owner or maintainer of the identity.

I also included main configuration files that define which resources will be created, which modules will be used, and what values will be assigned to variables.

IaC Is a Challenge

Now I have a well-formatted file with every identity created in a single Terraform project.

I can determine the relevant files for each identity and link them to the committing human to identify the owner of IaC-generated identities.

This approach is far from being perfect; along the way I tackled significant challenges and gathered dependencies that must be met for this process to work:

1. Running terraform plan
   This seemed like a passable hurdle at first, but my solution was pretty tailor-made for specific use cases. AWS made it relatively simple to switch state management from remote to local, but in Azure, it’s a whole different ball game.
   For both AWS and Azure, there are multiple points of contact between the code and the actual environment (such as state management for specific resources handled by the CSP outside of the main state file).
   
   • AWS allowed me to continue running without these dependencies, but Azure did not.
   • I only tested a small number of cases - there likely are AWS configurations that wouldn’t let me run terraform plan either.
   • There was the possibility of creating mock environments to interact with the code during plan execution. However, this would require creating a well-suited environment for each IaC deployment I wanted to assign ownership to, making it impractical.
2. Editing the code
   Tagging every file across the repositories was challenging due to the variety of formats and implementations. There are parsers available for HCL, but they mostly allow reading and offer limited writing or editing capabilities.
   • This required handling different file types and implementations, which lost the point of being a generic solution.
3. Tagging Mechanism
   Passing tags through the entire Terraform flow wasn’t seamless. When using a chain of Terraform objects - for instance, Input file → Resource → Module - each object should pass the tags from its predecessor.
   • When tags weren’t passed correctly, it wasn’t enough to simply add tags to each object.
   • Additional validations and modifications were required to ensure proper tag inheritance.
   • This added another layer of complexity.

Conclusion

IaC ownership is a serious challenge - even understanding and explaining it takes time. As part of Token's effort to map all NHIs, we addressed IaC-generated identities because they account for a large portion of all identities and are particularly difficult to manage.

This blog was a small demonstration of the deep research we conducted to expand the variety of identities we can provide ownership for. Ultimately, we ended up implementing other approaches and not the tagging approach due to its many dependencies which made it impractical for large-scale deployment.

However, DevOps teams could still benefit from tagging to better track their IaC identities. Understanding which files create your identities can be extremely useful for troubleshooting issues involving IaC-generated identities.

A DevOps engineer with full control over an IaC repository would not face the same constraints, making this approach a strong candidate for implementation and further exploration.

This effort is one of many that we are conducting at Token Security to map all NHIs and their human owners, a task that can be very challenging and luckily interesting. Token Security is a leading provider of NHI security solutions, securing NHIs across any system and environment. Token offers a comprehensive platform for your daily NHI needs: visibility, posture, and identity lifecycle.

Thanks for reading!

‍