Or check our Popular Categories...

Simon 01-11-2025 35

Managing Secrets in Terraform: From Bad to Automated

When it comes to Infrastructure as Code (IaC), Terraform is an incredibly powerful tool. It allows us to define, deploy,…

When it comes to Infrastructure as Code (IaC), Terraform is an incredibly powerful tool.

It allows us to define, deploy, and maintain entire environments in a repeatable, auditable, and version-controlled way.

But there’s one topic that’s often misunderstood or completely overlooked: secret management.

Passwords, SSH keys, and connection strings are all essential for our infrastructure, yet they’re also among the easiest things to expose — accidentally or otherwise.

Let’s explore how most setups evolve: from bad, to good, to fully automated.

Stage 1 – The Bad Way

The easiest and most dangerous approach: hardcoding credentials directly into your Terraform code.

resource "azurerm_windows_virtual_machine" "vm" {
  name                  = "vm-demo"
  location              = azurerm_resource_group.rg.location
  resource_group_name   = azurerm_resource_group.rg.name
  admin_username        = "azureadmin"
  admin_password        = "SuperSecret123!"
}

Yes, it works — the VM will deploy, and you can log in.

But now your password exists in:

  • Your .tf files
  • Your Terraform state file
  • Your Git history (if committed)
  • Possibly your CI/CD logs

Even if you move that password to a password manager later, the damage is already done — the secret is stored in plaintext multiple places.

🚫 Why this is bad: Once a secret lands in version control or the Terraform state, it’s exposed permanently.

Rotation becomes painful, and the risk of leakage skyrockets.

Stage 2 – The Decent Way

A slight improvement is using Terraform’s random_password resource to generate credentials dynamically.

resource "random_password" "admin" {
  length  = 20
  special = true
}

resource "azurerm_windows_virtual_machine" "vm" {
  name                = "vm-demo"
  admin_username      = "azureadmin"
  admin_password      = random_password.admin.result
}

This eliminates manually created passwords and ensures strong complexity.

You can even output the password temporarily for Bastion or RDP connections.

But the improvement only goes so far.

⚠️ Still risky: The generated password is still stored unencrypted in the Terraform state file which can and should be stored in a centralized storage account (you can adopt this from a previous post).

But anyone with backend access (like an Azure Storage account or Terraform Cloud workspace) can read it.

It never rotates, either.

You’ve improved reproducibility and security up to certain point — but not usability.

Stage 3 – The Good Way

To improve further, we introduce Azure Key Vault as the dedicated secret store.

Terraform can generate a password and securely store it in Key Vault before using it.

resource "random_password" "admin" {
  length  = 24
  special = true
}

resource "azurerm_key_vault_secret" "vm_password" {
  name         = "vm-admin-password"
  value        = random_password.admin.result
  key_vault_id = azurerm_key_vault.kv.id
}

resource "azurerm_windows_virtual_machine" "vm" {
  name                = "vm-demo"
  admin_username      = "azureadmin"
  admin_password      = azurerm_key_vault_secret.vm_password.value
}

Now the password:

  • Is randomly generated,
  • Stored securely in Key Vault, and
  • Referenced by the VM configuration.

However, even this “good” solution isn’t perfect.

Terraform must still “see” the password to apply it to the resource, meaning it’s still stored in plaintext in the Terraform state file.

And, just like before, the secret never expires.

🔒 Better — but static: Key Vault improves storage, but Terraform doesn’t handle lifecycle management or rotation.

❗️Despite…

…the fact that it still can’t handle automated key rotation the usability for connecting via Bastion got improved by a lot.

There is no need to remember or copy&paste any secrets/ssh keys if it is stored within a Azure Key Vault.

The Core Problem

No matter how you define or reference your secrets, Terraform always stores them in the state file.

This is by design — Terraform must know the value to manage it.

That means:

  1. Secrets are visible in plaintext within the state file.
  2. There’s no automatic rotation or expiration.
  3. Once leaked, the secret remains valid indefinitely.

Terraform is fantastic for provisioning infrastructure — but it’s not meant for managing secret lifecycles.

Stage 4 – The Better Way

A more secure approach is to use ephemeral resources to generate and manage secrets at runtime. With this method, sensitive values such as passwords or SSH keys are never stored in the Terraform state, but instead are directly written to a secure store like Azure Key Vault.

#Example - variables.tf 
variable "adminpassword_ephemeral" {
  type        = string
  sensitive   = true
  ephemeral   = true
}
variable "adminpassword_version" {
  type        = number
}

# ---- tf.vars Example
 adminpassword_ephemeral = "SuperSecurePassword"
 adminpassword_version   = 1



#Example - password generation
ephemeral "random_password" "password" {
  length           = 16
  special          = true
  override_special = "!#$%&*()-_=+[]{}<>:?"
}

#Example - ssh generation
ephemeral "tls_private_key" "ssh" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

These secrets are then pushed to Key Vault using the value_wo argument, which stands for write-only. The values exist only during the apply phase and do not persist in the Terraform state:

resource "azurerm_key_vault_secret" "password" {
  depends_on   = [ ephemeral.random_password.password ]
  name         = "adminuser-pw"
  key_vault_id = azurerm_key_vault.this.id
  
  #ephemeral values
  value_wo     = ephemeral.random_password.password.result
  value_wo_version = 1
}

All secrets are temporary in Terraform but persist securely in Key Vault, reducing the risk of accidental exposure.

Notes and Caveats:

  • Not every AzureRM provider resource fully supports ephemeral/write-only variables.
    • In some cases, the AzAPI provider is required to pass sensitive data without storing it in state.
  • value_wo_version and other lifecycle settings ensure the secret can be rotated without causing unnecessary Terraform diffs.

Benefits:

  • Secrets never appear in Terraform state files.
  • Secrets exist only during runtime, reducing exposure.
  • Provides a clean middle ground between “good” (plain variables) and “automated” (full CI/CD secret injection) approaches.

Tips:

Feel free to read more about this approach on:

https://nedinthecloud.com/2025/07/15/write-only-arguments-in-terraform/

And if you are interested in the approach of using the AzAPI provider for passing sensitive data:

https://blog.kewalaka.nz/post/2025/06/ephemeral_azapi/

Stage 5 – My Automated Way

To overcome Terraform’s limitations, I built an Azure Automation Runbook that handles what Terraform cannot: automatic credential rotation.

This PowerShell runbook uses a Managed Identity to authenticate and then:

  • Rotates Windows passwords using the VMAccessAgent extension.
  • Rotates Linux passwords and SSH keys using the VMAccessForLinux extension.
  • Updates Key Vault secrets with the new credentials.
  • Runs automatically on a defined schedule via Azure Automation.

🧹 Clean lifecycle: Each SSH key has a defined lifespan, rotates on schedule, and old keys are safely discarded.

This ensures that when team members leave, access is automatically removed after a the next run — no manual cleanup required.

How it fits together

Terraform --> Deploys: VMs + Key Vault + Automation Account + Runbook
             ↓
Azure Automation Runbook (Rotate-AzVMSecrets.ps1)
             ↓
Rotates VM Passwords & SSH Keys
             ↓
Stores new secrets securely in Key Vault

Each execution updates both the VMs and the secrets in Key Vault — without manual intervention or re-deployment.

Requirements:

To make sure that Terraform do not change secrets during the next apply you have to implement the following licecycle blocks:

resource "azurerm_key_vault_secret" "password-win" {
  name = "${local.vm_win_name}-${local.admin_username}-pw"
  ...
  expiration_date = timeadd(timestamp(), "168h") #7d
  
  lifecycle {
    ignore_changes = [ value,tags,expiration_date ]
  }
}
resource "azurerm_windows_virtual_machine" "win" {
  name                = local.vm_win_name
  ...
  lifecycle {
    ignore_changes = [ admin_password ]
  }
}

resource "azurerm_linux_virtual_machine" "unix" {
  name                = local.vm_unix_name
  ...
  
  admin_ssh_key {
    username = local.admin_username
    public_key = tls_private_key.ssh.public_key_openssh
  }

  lifecycle {
    ignore_changes = [ admin_ssh_key, admin_password ]
  }
}

View my full terraform test environment down below.

✅ Result:

  • Secrets and Keys rotate automatically.
  • Terraform never touches or stores them.
  • Operations are fully contained within Azure.

Runbook:

🧩 Script: simonvedder/Rotate-AzVMSecrets.ps1 

<#
.SYNOPSIS
    Automated Azure VM credential rotation runbook for Windows and Linux VMs.

.DESCRIPTION
    Rotates admin passwords and SSH keys for Azure VMs based on Key Vault expiration dates.
    Runs daily via Azure Automation Schedule with Managed Identity.

.PARAMETER KeyVaultName
    Name of the Azure Key Vault for storing credentials.

.PARAMETER RotationThresholdDays
    Trigger rotation when expiration date is within this many days (default: 7).

.PARAMETER RetentionPolicyDays
    Set new secret expiration to this many days from now (default: 90).

.PARAMETER SubscriptionScope
    "All" to process all accessible subscriptions, or comma-separated subscription IDs.

.PARAMETER DryRun
    Simulate operations without making changes (default: false).

.NOTES
    File Name      : Rotate-AzVMSecrets.ps1
    Author         : Simon Vedder
    Date           : 01.11.2025

    Version: 1.0.0
    Required Modules: Az.Accounts, Az.Compute, Az.KeyVault, Az.Resources
    Required Roles:
#>

[CmdletBinding()]
param(
    [Parameter(Mandatory = $true)]
    [string]$KeyVaultName,

    [Parameter(Mandatory = $false)]
    [int]$RotationThresholdDays = 7,

    [Parameter(Mandatory = $false)]
    [int]$RetentionPolicyDays = 90,

    [Parameter(Mandatory = $false)]
    [string]$SubscriptionScope = "All",

    [Parameter(Mandatory = $false)]
    [bool]$DryRun = $false
)

#Requires -Modules Az.Accounts, Az.Compute, Az.KeyVault, Az.Resources

# ============================================================================
# HELPER FUNCTIONS
# ============================================================================

function Write-Log {
    param([string]$Message, [string]$Level = 'Info')
    $timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
    Write-Output "$timestamp [$Level] $Message" 
}

function New-StrongPassword {
    param([int]$Length = 24)
    
    $chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!@#$%^&*()-_=+[]{}|;:,.<>?"
    $bytes = New-Object byte[] $Length
    $rng = [System.Security.Cryptography.RNGCryptoServiceProvider]::new()
    $rng.GetBytes($bytes)
    $rng.Dispose()
    
    $password = -join ($bytes | ForEach-Object { $chars[$_ % $chars.Length] })
    
    # Ensure complexity
    if ($password -notmatch '[a-z]') { $password = $password.Insert(0, 'a') }
    if ($password -notmatch '[A-Z]') { $password = $password.Insert(1, 'A') }
    if ($password -notmatch '[0-9]') { $password = $password.Insert(2, '1') }
    if ($password -notmatch '[^a-zA-Z0-9]') { $password = $password.Insert(3, '!') }
    
    return $password
}

function New-SSHKeyPair {
    param([string]$KeyName)

    # RSA Key erstellen
    $rsa = [System.Security.Cryptography.RSA]::Create(4096)
    
    # Private Key exportieren (PEM/PKCS#8)
    $privateKeyBytes = $rsa.ExportPkcs8PrivateKey()
    $privateKeyBase64 = [Convert]::ToBase64String($privateKeyBytes, `
    [System.Base64FormattingOptions]::InsertLineBreaks)
    $privateKey = "-----BEGIN PRIVATE KEY-----`n$privateKeyBase64`n-----END PRIVATE KEY-----`n"
    
    # Public Key im SSH Format
    # Wir müssen den RSA Public Key manuell in SSH Format konvertieren
    $rsaParams = $rsa.ExportParameters($false)
    
    # SSH-RSA Format: type + exponent + modulus
    $typeBytes = [System.Text.Encoding]::ASCII.GetBytes("ssh-rsa")
    $typeLength = [BitConverter]::GetBytes($typeBytes.Length)
    [Array]::Reverse($typeLength)
    
    $exponentLength = [BitConverter]::GetBytes($rsaParams.Exponent.Length)
    [Array]::Reverse($exponentLength)
    
    $modulusLength = [BitConverter]::GetBytes($rsaParams.Modulus.Length + 1)
    [Array]::Reverse($modulusLength)
    
    # SSH Public Key zusammenbauen
    $buffer = @()
    $buffer += $typeLength
    $buffer += $typeBytes
    $buffer += $exponentLength
    $buffer += $rsaParams.Exponent
    $buffer += $modulusLength
    $buffer += 0x00  # Padding byte
    $buffer += $rsaParams.Modulus
    
    $publicKeyBase64 = [Convert]::ToBase64String($buffer)
    $publicKey = "ssh-rsa $publicKeyBase64"
    
    $rsa.Dispose()
    
    return @{ PrivateKey = $privateKey; PublicKey = $publicKey }
}

# ============================================================================
# MAIN SCRIPT
# ============================================================================

$startTime = Get-Date
Write-Log "====== VM Credential Rotation Started ======" -Level Info
Write-Log "KeyVault: $KeyVaultName `
  | Threshold: $RotationThresholdDays days `
  | Retention: $RetentionPolicyDays days" `
  -Level Info

# Authenticate with Managed Identity
try {
    Write-Log "Connecting with Managed Identity..." -Level Info
    $null = Connect-AzAccount -Identity -ErrorAction Stop
    Write-Log "Authentication successful" -Level Success
}
catch {
    Write-Log "Authentication failed: $_" -Level Error
    exit 1
}

# Get subscriptions
try {
    if ($SubscriptionScope -eq "All") {
        $subscriptions = Get-AzSubscription
    }
    else {
        $subscriptions = ($SubscriptionScope -split ",") `
        | ForEach-Object { Get-AzSubscription -SubscriptionId $_.Trim() }
    }
    Write-Log "Found $($subscriptions.Count) subscription(s)" -Level Success
}
catch {
    Write-Log "Failed to get subscriptions: $_" -Level Error
    exit 1
}

# Statistics
$stats = @{
    TotalVMs = 0
    Rotated = 0
    Failed = 0
    Skipped = 0
}

# Process each subscription
foreach ($sub in $subscriptions) {
    Write-Log "Processing subscription: $($sub.Name)" -Level Info
    $null = Set-AzContext -SubscriptionId $sub.Id
    
    $vms = Get-AzVM
    $stats.TotalVMs += $vms.Count
    Write-Log "Found $($vms.Count) VMs" -Level Info
    
    foreach ($vm in $vms) {
        try {
            Write-Log "----------------------------------------" -Level Info
            Write-Log "VM: $($vm.Name) | OS: $($vm.StorageProfile.OsDisk.OsType)" -Level Info
            
            $adminUsername = $vm.OSProfile.AdminUsername
            $osType = $vm.StorageProfile.OsDisk.OsType
            $needsRotation = $false
            
            # ========== WINDOWS VM ==========
            if ($osType -eq "Windows") {
                $secretName = "$($vm.Name)-$adminUsername-pw"
                
                # Check if rotation needed
                try {
                    $secret = Get-AzKeyVaultSecret -VaultName $KeyVaultName `
                    -Name $secretName -ErrorAction Stop
                    
                    if ($null -eq $secret.Attributes.Expires) {
                        Write-Log "No expiration date, needs rotation" -Level Warning
                        $needsRotation = $true
                    }
                    else {
                        $daysUntilExpiration = ($secret.Attributes.Expires - (Get-Date)).Days
                        Write-Log "Expires in $daysUntilExpiration days" -Level Info
                        
                        if ($daysUntilExpiration -le $RotationThresholdDays) {
                            $needsRotation = $true
                        }
                    }
                }
                catch {
                    Write-Log "Secret not found, needs creation" -Level Warning
                    $needsRotation = $true
                }
                
                # Rotate if needed
                if ($needsRotation) {
                    Write-Log "Rotating Windows password..." -Level Info
                    
                    # Check VM power state
                    $vmStatus = Get-AzVM -ResourceGroupName $vm.ResourceGroupName `
                    -Name $vm.Name -Status
                    $powerState = $vmStatus.Statuses `
                    | Where-Object { $_.Code -like "PowerState/*" } `
                    | Select-Object -ExpandProperty Code
                    $isRunning = $powerState -eq "PowerState/running"
                    
                    if (-not $isRunning) {
                        Write-Log "VM is not running (State: $powerState), skipping rotation" `
                         -Level Warning
                        $stats.Skipped++
                        continue
                    }
                    
                    if ($DryRun) {
                        Write-Log "[DRY-RUN] Would rotate password" -Level Warning
                        $stats.Skipped++
                        continue
                    }
                    
                    # Generate new password
                    $newPassword = New-StrongPassword
                    $securePassword = ConvertTo-SecureString `
                    -String $newPassword `
                    -AsPlainText -Force
                    
                    # Update VM via extension FIRST (before storing in Key Vault)
                    $bstr = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($securePassword)
                    $plainPassword = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($bstr)
                    [System.Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr)
                    
                    try {
                        # Install extension
                        $null = Set-AzVMExtension `
                            -ResourceGroupName $vm.ResourceGroupName `
                            -VMName $vm.Name `
                            -Name "VMAccessAgent" `
                            -Publisher "Microsoft.Compute" `
                            -ExtensionType "VMAccessAgent" `
                            -TypeHandlerVersion "2.4" `
                            -Settings @{ UserName = $adminUsername } `
                            -ProtectedSettings @{ Password = $plainPassword } `
                            -ErrorAction Stop `
                            -Location $vm.Location `
                            -ForceRerun (New-Guid).Guid

                        Write-Log "VM password updated successfully" -Level Success
                        
                        # Store in Key Vault ONLY after successful VM update
                        $expirationDate = (Get-Date).AddDays($RetentionPolicyDays)
                        $null = Set-AzKeyVaultSecret `
                            -VaultName $KeyVaultName `
                            -Name $secretName `
                            -SecretValue $securePassword `
                            -Expires $expirationDate `
                            -Tag @{ 
                                VMName = $vm.Name
                                AdminName = $adminUsername
                                Type = "Password"
                                OSType = "Windows"
                                LastRotated = (Get-Date -Format "yyyy-MM-dd")
                            }
                        Write-Log "Password stored in Key Vault" -Level Success
                        $stats.Rotated++
                    }
                    catch {
                        Write-Log "Failed to update VM password: $_" -Level Error
                        $stats.Failed++
                    }
                    finally {
                        # Clear memory
                        $newPassword = $null
                        $plainPassword = $null
                        $securePassword = $null
                        [System.GC]::Collect()
                    }
                }
                else {
                    Write-Log "No rotation needed" -Level Info
                    $stats.Skipped++
                }
            }
            
            # ========== LINUX VM ==========
            elseif ($osType -eq "Linux") {
                $passwordSecretName = "$($vm.Name)-$adminUsername-pw"
                $sshSecretName = "$($vm.Name)-$adminUsername-ssh-priv"
                $sshSecretPubName = "$($vm.Name)-$adminUsername-ssh-pub"
                $passwordAuthEnabled = -not $vm.OSProfile.LinuxConfiguration.DisablePasswordAuthentication
                
                # Rotate password if enabled
                if ($passwordAuthEnabled) {
                    Write-Log "Password authentication enabled" -Level Info
                    
                    try {
                        $secret = Get-AzKeyVaultSecret `
                        -VaultName $KeyVaultName `
                        -Name $passwordSecretName `
                        -ErrorAction Stop
                        
                        if ($null -eq $secret.Attributes.Expires) {
                            $needsRotation = $true
                        }
                        else {
                            $daysUntilExpiration = ($secret.Attributes.Expires - (Get-Date)).Days
                            Write-Log "Password expires in $daysUntilExpiration days" `
                            -Level Info
                            $needsRotation = ($daysUntilExpiration -le $RotationThresholdDays)
                        }
                    }
                    catch {
                        $needsRotation = $true
                    }
                    
                    if ($needsRotation) {
                        Write-Log "Rotating Linux password..." -Level Info
                        
                        # Check VM power state
                        $vmStatus = Get-AzVM `
                        -ResourceGroupName $vm.ResourceGroupName `
                        -Name $vm.Name `
                        -Status
                        
                        $powerState = $vmStatus.Statuses `
                        | Where-Object { $_.Code -like "PowerState/*" } `
                        | Select-Object -ExpandProperty Code
                        $isRunning = $powerState -eq "PowerState/running"
                        
                        if (-not $isRunning) {
                            Write-Log "VM is not running (State: $powerState), skipping rotation" `
                            -Level Warning
                            $stats.Skipped++
                            continue
                        }
                        
                        if ($DryRun) {
                            Write-Log "[DRY-RUN] Would rotate password" -Level Warning
                        }
                        else {
                            $newPassword = New-StrongPassword
                            $securePassword = ConvertTo-SecureString -String $newPassword `
                            -AsPlainText -Force
                            
                            $bstr = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($securePassword)
                            $plainPassword = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($bstr)
                            [System.Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr)
                            
                            try {
                                
                                $protectedSettings = @{
                                    username = $adminUsername
                                    password = $plainPassword
                                    reset_ssh = $false
                                }

                                $null = Set-AzVMExtension `
                                    -ResourceGroupName $vm.ResourceGroupName `
                                    -VMName $vm.Name `
                                    -Name "VMAccessForLinux" `
                                    -Publisher "Microsoft.OSTCExtensions" `
                                    -ExtensionType "VMAccessForLinux" `
                                    -TypeHandlerVersion "1.5" `
                                    -ProtectedSettings $protectedSettings `
                                    -ErrorAction Stop `
                                    -Location $vm.Location `
                                    -ForceRerun (New-Guid).Guid
                                
                                Write-Log "Linux password updated" -Level Success
                                
                                # Store in Key Vault AFTER successful VM update
                                $expirationDate = (Get-Date).AddDays($RetentionPolicyDays)
                                $null = Set-AzKeyVaultSecret `
                                    -VaultName $KeyVaultName `
                                    -Name $passwordSecretName `
                                    -SecretValue $securePassword `
                                    -Expires $expirationDate `
                                    -Tag @{ 
                                         VMName = $vm.Name
                                         AdminName = $adminUsername
                                         OSType = "Linux"
                                         Type = "Password"
                                         LastRotated = (Get-Date -Format "yyyy-MM-dd")
                                     }
                                
                                $stats.Rotated++
                            }
                            catch {
                                Write-Log "Failed to update Linux password: $_" -Level Error
                                $stats.Failed++
                            }
                            finally {
                                $newPassword = $null
                                $plainPassword = $null
                                $securePassword = $null
                                [System.GC]::Collect()
                            }
                        }
                    }
                    else {
                        Write-Log "No rotation needed" -Level Info
                        $stats.Skipped++
                    }
                }
                
                # Rotate SSH key
                Write-Log "Checking SSH key..." -Level Info
                $needsSSHRotation = $false
                
                try {
                    $sshSecret = Get-AzKeyVaultSecret `
                    -VaultName $KeyVaultName `
                    -Name $sshSecretName `
                    -ErrorAction Stop
                    
                    if ($null -eq $sshSecret.Attributes.Expires) {
                        $needsSSHRotation = $true
                    }
                    else {
                        $daysUntilExpiration = ($sshSecret.Attributes.Expires - (Get-Date)).Days
                        Write-Log "SSH key expires in $daysUntilExpiration days" -Level Info
                        $needsSSHRotation = ($daysUntilExpiration -le $RotationThresholdDays)
                    }
                }
                catch {
                    $needsSSHRotation = $true
                }
                
                if ($needsSSHRotation) {
                    Write-Log "Rotating SSH key..." -Level Info
                    
                    # Check VM power state
                    $vmStatus = Get-AzVM `
                    -ResourceGroupName $vm.ResourceGroupName `
                    -Name $vm.Name `
                    -Status
                    
                    $powerState = $vmStatus.Statuses `
                    | Where-Object { $_.Code -like "PowerState/*" } `
                    | Select-Object -ExpandProperty Code
                    $isRunning = $powerState -eq "PowerState/running"
                    
                    if (-not $isRunning) {
                        Write-Log "VM is not running (State: $powerState), skipping SSH rotation" `
                        -Level Warning
                        $stats.Skipped++
                        continue
                    }
                    
                    if ($DryRun) {
                        Write-Log "[DRY-RUN] Would rotate SSH key" -Level Warning
                        $stats.Skipped++
                    }
                    else {
                        $keyPair = New-SSHKeyPair -KeyName "$($vm.Name)-$adminUsername"
                        $securePrivateKey = ConvertTo-SecureString `
                        -String $keyPair.PrivateKey `
                        -AsPlainText -Force
                        $securePublicKey = ConvertTo-SecureString `
                        -String $keyPair.PublicKey `
                        -AsPlainText -Force
                        
                        try {
                            $protectedSettings = @{
                                username = $adminUsername
                                ssh_key = $keyPair.PublicKey
                                reset_ssh = $true               
                                remove_prior_keys = $true       
                            }

                            $null = Set-AzVMExtension `
                                -ResourceGroupName $vm.ResourceGroupName `
                                -VMName $vm.Name `
                                -Name "VMAccessForLinux" `
                                -Publisher "Microsoft.OSTCExtensions" `
                                -ExtensionType "VMAccessForLinux" `
                                -TypeHandlerVersion "1.5" `
                                -ProtectedSettings $protectedSettings `
                                -ErrorAction Stop `
                                -Location $vm.Location `
                                -ForceRerun (New-Guid).Guid
                            
                            Write-Log "SSH key updated" -Level Success
                            
                            # Store in Key Vault AFTER successful VM update
                            $expirationDate = (Get-Date).AddDays($RetentionPolicyDays)
                            $null = Set-AzKeyVaultSecret `
                                -VaultName $KeyVaultName `
                                -Name $sshSecretName `
                                -SecretValue $securePrivateKey `
                                -Expires $expirationDate `
                                -Tag @{ 
                                    VMName = $vm.Name
                                    AdminName = $adminUsername
                                    OSType = "Linux"
                                    Type = "SSHKey"
                                    LastRotated = (Get-Date -Format "yyyy-MM-dd") 
                                }

                            $null = Set-AzKeyVaultSecret `
                                -VaultName $KeyVaultName `
                                -Name $sshSecretPubName `
                                -SecretValue $securePublicKey `
                                -Expires $expirationDate `
                                -Tag @{ 
                                    VMName = $vm.Name
                                    AdminName = $adminUsername
                                    OSType = "Linux"
                                    Type = "SSHPublicKey"
                                    LastRotated = (Get-Date -Format "yyyy-MM-dd") 
                                }
                            
                            $stats.Rotated++
                        }
                        catch {
                            Write-Log "Failed to update SSH key: $_" -Level Error
                            $stats.Failed++
                        }
                        finally {
                            $keyPair = $null
                            $securePrivateKey = $null
                            [System.GC]::Collect()
                        }
                    }
                }
                else {
                    Write-Log "SSH key does not need rotation" -Level Info
                    $stats.Skipped++
                }
            }
        }
        catch {
            Write-Log "Error processing VM $($vm.Name): $_" -Level Error
            $stats.Failed++
        }
    }
}

# Summary
$duration = (Get-Date) - $startTime
Write-Log "----------------------------------------" -Level Info
Write-Log "=============== Summary ===============" -Level Info
Write-Log "Duration: $($duration.ToString('mm\:ss'))" -Level Info
Write-Log "Total VMs: $($stats.TotalVMs)" -Level Info
Write-Log "Rotated: $($stats.Rotated)" -Level Success
Write-Log "Skipped: $($stats.Skipped)" -Level Info
Write-Log "Failed: $($stats.Failed)" -Level $(if ($stats.Failed -gt 0) { 'Error' } else { 'Info' })

if ($DryRun) {
    Write-Log "DRY-RUN MODE - No changes made" -Level Warning
}

Write-Log "Completed" -Level Success
exit $(if ($stats.Failed -gt 0) { 1 } else { 0 })

Seamless Access via Azure Bastion

One of the biggest benefits of this design is that you can connect to your VMs using Azure Bastion without ever handling a password or private key.

No local secrets.

No copy-pasting.

No insecure file sharing.

Bastion pulls credentials directly from Key Vault when needed, providing a fully managed and secure experience.

This is the perfect balance of security and usability — something rarely achieved in secret management.

Conclusion

Terraform is an excellent provisioning tool — but not a secure secret manager.

Its state-handling model inherently exposes sensitive data and doesn’t support key rotation or expiration.

Even with Azure Key Vault integration, the secrets still end up stored unencrypted in the Terraform state.

That’s why the right approach is to separate provisioning from operations.

By combining:

  • Terraform for infrastructure deployment,
  • Azure Key Vault for secure storage, and
  • Azure Automation Runbooks for lifecycle and rotation,

you achieve a cloud-native, fully automated, and highly secure architecture.

From my perspective, this separation of duties is the cleanest and safest way to manage credentials in modern Azure environments.

It ensures that Terraform does what it’s best at — building infrastructure — while Automation handles what Terraform cannot: continuous secret lifecycle management.

💡 Final takeaway:

Use Terraform to deploy.

Use Key Vault to store.

Use Azure Automation to rotate.

That’s the trio that finally makes your secrets safe.

So you’d even be able to use the bad way initially if you combine it with my automated way 😉

Terraform Example:

🧩 main.tf: simonvedder/main.tf

If you want to explore the Terraform configuration that I used to test, check out my example project:

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = ">=4.51.0"
    }
    random = {
      source = "hashicorp/random"
      version = ">=3.7.2"
    }
  }
}


provider "azurerm" {
  features {
  }
  subscription_id = "<subscriptionID>"
}

# Get current subscription
data "azurerm_client_config" "current" {}




###########################################
###             Locals                 ###
###########################################
locals {
  resource_group_name = "securetfsecrets"
  location = "westeurope"

  kv_name = "${local.resource_group_name}-kv"

  vm_prefix = "demo-vm"
  vm_win_name = "${local.vm_prefix}-win"
  vm_unix_name = "${local.vm_prefix}-unix"
  
  admin_username = "superman"
}

###########################################
###          Resourcegroup              ###
###########################################
# Resource Group
resource "azurerm_resource_group" "this" {
  name     = local.resource_group_name
  location = local.location
}

###########################################
###             Network                 ###
###########################################
# Networking
resource "azurerm_virtual_network" "this" {
  name                = "vnet"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name

}

# Subnet
resource "azurerm_subnet" "this" {
  name                 = "main"
  resource_group_name  = azurerm_resource_group.this.name
  virtual_network_name = azurerm_virtual_network.this.name
  address_prefixes     = ["10.0.1.0/24"]
}

###########################################
###          KV & Secrets & Keys        ###
###########################################

# Key Vault
resource "azurerm_key_vault" "this" {
  tenant_id = data.azurerm_client_config.current.tenant_id
  resource_group_name = azurerm_resource_group.this.name
  location = azurerm_resource_group.this.location
  name = local.kv_name
  sku_name = "standard"
  rbac_authorization_enabled = true
}

resource "azurerm_role_assignment" "kv_admin" {
  scope = azurerm_key_vault.this.id
  principal_id = "<userid>"
  role_definition_name = "Key Vault Administrator"
}

resource "random_password" "password" {
length = 16
special = true
override_special = "!#$%&*()-_=+[]{}<>:?"
}

resource "azurerm_key_vault_secret" "password-win" {
  depends_on = [ random_password.password, azurerm_role_assignment.kv_admin ]
  name = "${local.vm_win_name}-${local.admin_username}-pw"
  key_vault_id = azurerm_key_vault.this.id
  value = random_password.password.result
  expiration_date = timeadd(timestamp(), "168h")
  tags = {
    VMName      = local.vm_win_name
    AdminName   = local.admin_username
    OSType      = "Windows"
    Type        = "Password"
    LastRotated = formatdate("YYYY-MM-DD", timestamp())
  }
  lifecycle {
    ignore_changes = [ value,tags,expiration_date ]
  }
}

resource "azurerm_key_vault_secret" "password-unix" {
  depends_on = [ random_password.password, azurerm_role_assignment.kv_admin  ]
  name = "${local.vm_unix_name}-${local.admin_username}-pw"
  key_vault_id = azurerm_key_vault.this.id
  value = random_password.password.result
  expiration_date = timeadd(timestamp(), "168h")
  tags = {
    VMName      = local.vm_unix_name
    AdminName   = local.admin_username
    OSType      = "Linux"
    Type        = "Password"
    LastRotated = formatdate("YYYY-MM-DD", timestamp())
  }
  lifecycle {
    ignore_changes = [ value,tags,expiration_date ]
  }
}

resource "tls_private_key" "ssh" {
  algorithm = "RSA"
  rsa_bits = 4096
}

resource "azurerm_key_vault_secret" "ssh-priv" {
  depends_on = [ tls_private_key.ssh, azurerm_role_assignment.kv_admin  ]
  name = "${local.vm_unix_name}-${local.admin_username}-ssh-priv"
  key_vault_id = azurerm_key_vault.this.id
  value = tls_private_key.ssh.private_key_pem
  expiration_date = timeadd(timestamp(), "168h")
  tags = {
    VMName      = local.vm_unix_name
    AdminName   = local.admin_username
    OSType      = "Linux"
    Type        = "SSHKey"
    LastRotated = formatdate("YYYY-MM-DD", timestamp())
  }
  lifecycle {
    ignore_changes = [ value,tags,expiration_date ]
  }
}

resource "azurerm_key_vault_secret" "ssh-pub" {
  depends_on = [ tls_private_key.ssh, azurerm_role_assignment.kv_admin  ]
  name = "${local.vm_unix_name}-${local.admin_username}-ssh-pub"
  key_vault_id = azurerm_key_vault.this.id
  value = tls_private_key.ssh.public_key_openssh
  expiration_date = timeadd(timestamp(), "168h")
  tags = {
    VMName      = local.vm_unix_name
    AdminName   = local.admin_username
    OSType      = "Linux"
    Type        = "SSHPublicKey"
    LastRotated = formatdate("YYYY-MM-DD", timestamp())
  }
  lifecycle {
    ignore_changes = [ value,tags,expiration_date ]
  }
}

###########################################
###            VMs & NICs               ###
###########################################

### Windows
resource "azurerm_windows_virtual_machine" "win" {
  name                = local.vm_win_name
  resource_group_name = azurerm_resource_group.this.name
  location            = azurerm_resource_group.this.location
  size                = "Standard_B2ms"
  admin_username      = local.admin_username
  admin_password      = azurerm_key_vault_secret.password-win.value

  network_interface_ids = [
    azurerm_network_interface.win.id,
  ]

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Premium_LRS"
  }

  source_image_reference {
    publisher = "MicrosoftWindowsServer"
    offer     = "WindowsServer"
    sku       = "2022-datacenter-g2"
    version   = "latest"
  }
  lifecycle {
    ignore_changes = [ admin_password ]
  }
}
resource "azurerm_network_interface" "win" {
  name                = "${local.vm_win_name}-nic"
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name

  ip_configuration {
    name                          = "this"
    subnet_id                     = azurerm_subnet.this.id
    private_ip_address_allocation = "Dynamic"
  }
}


### Linux
resource "azurerm_linux_virtual_machine" "unix" {
  name                = local.vm_unix_name
  resource_group_name = azurerm_resource_group.this.name
  location            = azurerm_resource_group.this.location
  size                = "Standard_B2ms"
  admin_username      = local.admin_username
  admin_password      = azurerm_key_vault_secret.password-unix.value
  disable_password_authentication = false

  admin_ssh_key {
    username = local.admin_username
    public_key = tls_private_key.ssh.public_key_openssh
  }

  network_interface_ids = [
    azurerm_network_interface.unix.id,
  ]

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Premium_LRS"
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "ubuntu-24_04-lts"
    sku       = "server"
    version   = "latest"
  }
  lifecycle {
    ignore_changes = [ admin_ssh_key, admin_password ]
  }
}
resource "azurerm_network_interface" "unix" {
  name                = "${local.vm_unix_name}-nic"
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name

  ip_configuration {
    name                          = "this"
    subnet_id                     = azurerm_subnet.this.id
    private_ip_address_allocation = "Dynamic"
  }
}

Simon

Cloud Engineer focused on Azure, Terraform & PowerShell. Passionate about automation, efficient solutions, and sharing real-world cloud projects and insights.⸻Let me know if you’d like to make it more casual, technical, or personalized.
Write a Comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts

1 Min Read
Simon 15-06-2025

Self Service VM & AVD Order via Web Form

In one of my projects, I developed an Azure-based solution that enables users to provision a virtual machine (VM) or…
Discover More
1 Min Read
Simon 02-06-2025

Deallocate VM based on Activity Log

In many organizations, virtual machines in Azure are kept running even when they’re no longer needed – especially during business…
Discover More
1 Min Read
Simon 19-07-2025

Collecting GPU Metrics on Azure VMs with Telegraf

When working with GPU-enabled Azure virtual machines — such as those used for AI training, inference, or video rendering —…
Discover More
1 Min Read
Simon 04-10-2025

Automated Terraform Deployments with GitHub

Managing infrastructure through Terraform is powerful — but doing it securely and automatically is where real efficiency begins. In this…
Discover More
1 Min Read
Simon 12-07-2025

Azure Template Specs: Reusable ARM Templates

Infrastructure as Code (IaC) is undoubtedly the gold standard when it comes to modern cloud infrastructure. Tools like Bicep, Terraform,…
Discover More
1 Min Read
Simon 05-07-2025

Using Terraform to Generate Local Zip Files

Terraform is most commonly used to provision infrastructure — VMs, storage accounts, networking, etc. But sometimes, your infrastructure code also…
Discover More
1 Min Read
Simon 27-07-2025

VM Power Management in Azure Using Tags, Runbooks, and a Custom GUI

As a Cloud Solutions Engineer, optimizing costs and automating operations is a daily priority. One of the common challenges in…
Discover More