Debugging Terraform like a hacker

Learn how to debug Terraform configurations using built-in logging (TF_LOG), and troubleshoot common errors.

In this post, I talk about Terraform’s debugging capabilities, particularly how to diagnose and resolve issues using logs, error messages, and plan diffs. The goal is to approach Terraform like a hacker.

Enabling detailed logs with TF_LOG

Terraform provides an environment variable called TF_LOG that controls the verbosity of internal logging. This is not enabled by default, but when debugging a stubborn issue—such as a resource failing to provision or an intermittent authentication error—enabling logs can quickly reveal the root cause.

Follow my journey of 100 Days of Red Team on WhatsApp, Telegram or Discord.

To enable logging, set the TF_LOG variable before running your Terraform command:

PowerShell:

$env:TF_LOG = "DEBUG"
$env:TF_LOG_PATH = "terraform-debug.log"

Bash:

export TF_LOG=DEBUG
export TF_LOG_PATH=terraform-debug.log

CMD:

set TF_LOG=DEBUG
set TF_LOG_PATH=terraform-debug.log

The TF_LOG_PATH variable creates a searchable file that can be analyzed line-by-line or even grepped for specific errors, such as "error" or "403".

The output includes detailed traces such as provider operations, data source reads, and dependency graph evaluations.

The available log levels, in increasing order of verbosity, are:

• ERROR – Only serious failures.

• WARN – Warnings that may indicate non-fatal problems.

• INFO – General operational messages.

• DEBUG – Full internal debug output (most detailed).

Most red team operators will find DEBUG the most helpful during active troubleshooting or engagement, as it prints every internal step Terraform takes.

Troubleshooting common errors

Terraform error messages are generally descriptive, but interpreting them still requires context. Common issues red team operators may encounter include:

1. Credential and permission errors
   These show up as HTTP 400/403/401 errors or messages like Error: Error loading zone 'redteam.cloud': 403 Forbidden.

   • Fix: Check your cloud provider credentials and permissions associated with the IAM profile.

2. State locking issues
   Especially relevant when using remote backends, a locked state prevents further runs.

   • Fix: Use terraform force-unlock <LOCK_ID> to release the lock if you're sure no other operation is running.

     terraform force-unlock - Manually removes a state lock. Use with caution—only when you're sure another Terraform process isn't running.

3. Incorrect resource references
   Terraform may complain that a resource doesn't exist or can't be referenced.

   • Fix: Use following commands to visually inspect dependencies:

     terraform graph | Out-File -Encoding ASCII -FilePath graph.dot
     dot -Tsvg graph.dot -o graph.svg

     terraform graph - Creates a DOT-format dependency graph of resources. Useful for visualizing relationships and understanding build order.

     Make sure Graphviz is installed and dot is available in your system PATH. You can install it via choco install graphviz. You will need to run this in an elevated PowerShell or CMD window (Run as Administrator).

4. Invalid count or conditional Logic
   Errors like count cannot be computed or resource cannot be null usually result from misconfigured count or for_each.

   • Fix: Use terraform console to test expressions interactively.

     terraform console - Opens an interactive shell where you can evaluate variables and expressions. Great for debugging conditionals and dynamic logic.

Always read the full stack trace Terraform gives. In most cases, the error message contains a path to the problematic file and line number. Combine this with debug logs to triangulate the issue.

Plan diff debugging tricks

Understanding the output of terraform plan is an art. The terraform plan command shows what changes Terraform will make, but sometimes it suggests changes you don’t expect—or keeps trying to recreate a resource on every run. Interpreting why Terraform thinks a change is needed is key to debugging. This becomes especially relevant when dealing with idempotency.

Idempotency means that running the same Terraform code multiple times will always result in the same infrastructure, as long as nothing has changed in the configuration or the external environment.

1. Run the plan twice

Run terraform plan, wait a few minutes, then run it again without changing any files.

Example:

terraform plan
sleep 300
terraform plan

If the second plan still shows changes, you may be dealing with non-idempotent resource behavior. This often happens with certain cloud resources (like DNS records or instance metadata) that change subtly over time.

2. Use -target to focus on one resource

When debugging a single resource, isolate it:

terraform plan -target=aws_instance.redirector

This avoids noise from other parts of the infrastructure and helps you focus on just one piece.

3. Compare plan outputs

You can export plan outputs and compare them using JSON:

terraform plan -out=plan1.out
# Make changes or wait
terraform plan -out=plan2.out

terraform show -json plan1.out > plan1.json
terraform show -json plan2.out > plan2.json

diff plan1.json plan2.json

terraform show -json - Converts a plan file (.out) to structured JSON. Ideal for comparing plans or extracting data programmatically.

This is especially useful for identifying changes that are subtle—like tag updates or provider-specific behavior (e.g., EBS volumes defaulting to a certain encryption).

4. Look for "forces replacement" warnings

Sometimes a resource is destroyed and recreated because a property can’t be changed in-place. This usually means the attribute is immutable. Replacing a live server or IP during an operation can break the op—so it's important to identify these before they happen.

The importance of debug logs for red team operators

For red teamers, debugging logs help in understanding behavior deeply enough to manipulate it when needed. Whether pivoting through misconfigured IAM policies, misusing default VPCs, or deploying disposable C2 redirectors, the ability to debug Terraform like a hacker translates to faster, stealthier, and more resilient infrastructure.

Furthermore, in advanced scenarios, Terraform logs can expose internal API calls that red teamers can replicate manually or abuse through alternative tooling. This makes TF_LOG not just a debug tool, but a reconnaissance tool in infrastructure-heavy engagements.

TL;DR
- Terraform configurations can be debugged using built-in logging (TF_LOG) 
- Debugging is helpful in troubleshooting common errors like permission issues and state locks
- Additional commands like terraform graph, console, force-unlock and show can help resolve certain issues. 
- Terraform debug logs are a duble edged sword for red team operators. While they enable troubleshooting their own infrastructure, they can be a source of valuable information during an engagement.

Follow my journey of 100 Days of Red Team on WhatsApp, Telegram or Discord.