infra-automation/docs/security/vault-management.md

# Ansible Vault Management Guide

This document describes how to manage encrypted secrets using Ansible Vault in this infrastructure.

## Overview

Ansible Vault is used to encrypt sensitive data such as passwords, API tokens, and private keys. All vault files are stored in `inventories/<environment>/group_vars/all/vault.yml`.

## Table of Contents

- [Quick Start](#quick-start)
- [Vault File Locations](#vault-file-locations)
- [Creating Vault Files](#creating-vault-files)
- [Encrypting and Decrypting](#encrypting-and-decrypting)
- [Editing Vault Files](#editing-vault-files)
- [Using Vault Variables](#using-vault-variables)
- [Vault Password Management](#vault-password-management)
- [Best Practices](#best-practices)
- [Troubleshooting](#troubleshooting)

## Quick Start

```bash
# 1. Create vault file from example
cp inventories/production/group_vars/all/vault.yml.example \
   inventories/production/group_vars/all/vault.yml

# 2. Edit and fill in secrets
vi inventories/production/group_vars/all/vault.yml

# 3. Encrypt the vault file
ansible-vault encrypt inventories/production/group_vars/all/vault.yml

# 4. Run playbook with vault password
ansible-playbook site.yml --ask-vault-pass
```

## Vault File Locations

Vault files are organized by environment:

```
inventories/
├── production/
│   └── group_vars/
│       └── all/
│           ├── vault.yml.example  # Template
│           └── vault.yml          # Encrypted (gitignored)
├── staging/
│   └── group_vars/
│       └── all/
│           ├── vault.yml.example
│           └── vault.yml
└── development/
    └── group_vars/
        └── all/
            ├── vault.yml.example
            └── vault.yml
```

**Important**: `vault.yml` files should be added to `.gitignore` to prevent accidental commits.

## Creating Vault Files

### From Example Template

```bash
# Copy example template
cp inventories/production/group_vars/all/vault.yml.example \
   inventories/production/group_vars/all/vault.yml

# Edit and replace CHANGEME placeholders
vi inventories/production/group_vars/all/vault.yml

# Encrypt the file
ansible-vault encrypt inventories/production/group_vars/all/vault.yml
```

### Create New Vault File

```bash
# Create and encrypt in one step
ansible-vault create inventories/production/group_vars/all/vault.yml
```

This opens your editor to add vault contents, then automatically encrypts on save.

## Encrypting and Decrypting

### Encrypt a File

```bash
ansible-vault encrypt inventories/production/group_vars/all/vault.yml
```

You'll be prompted to create a vault password.

### Decrypt a File

```bash
# Decrypt to view/edit (dangerous - creates plaintext file)
ansible-vault decrypt inventories/production/group_vars/all/vault.yml

# View without decrypting
ansible-vault view inventories/production/group_vars/all/vault.yml
```

**Warning**: Decrypting a file leaves it in plaintext. Always re-encrypt after editing.

### Encrypt Multiple Files

```bash
ansible-vault encrypt inventories/*/group_vars/all/vault.yml
```

## Editing Vault Files

### Edit Encrypted File

```bash
# Edit encrypted file directly (recommended)
ansible-vault edit inventories/production/group_vars/all/vault.yml
```

This decrypts the file in memory, opens your editor, and re-encrypts on save.

### Change Vault Password

```bash
ansible-vault rekey inventories/production/group_vars/all/vault.yml
```

You'll be prompted for the old password, then the new password.

## Using Vault Variables

### In Playbooks

Reference vault variables like normal variables:

```yaml
---
- name: Configure database
  hosts: databases
  tasks:
    - name: Set MySQL root password
      mysql_user:
        name: root
        password: "{{ vault_mysql_root_password }}"
        host: localhost
```

### In Templates

```jinja2
# /etc/my.cnf
[client]
user = root
password = {{ vault_mysql_root_password }}
```

### In Role Defaults

```yaml
# roles/mysql/defaults/main.yml
---
mysql_root_password: "{{ vault_mysql_root_password }}"
```

## Vault Password Management

### Option 1: Interactive Password Prompt (Most Secure)

```bash
ansible-playbook site.yml --ask-vault-pass
```

### Option 2: Password File

Create a password file:

```bash
# Create password file (gitignored)
echo "YourVaultPassword123!" > .vault_pass
chmod 600 .vault_pass
```

Add to `.gitignore`:
```
.vault_pass
```

Update `ansible.cfg`:
```ini
[defaults]
vault_password_file = .vault_pass
```

Run playbooks without prompt:
```bash
ansible-playbook site.yml
```

### Option 3: Environment Variable

```bash
export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
ansible-playbook site.yml
```

### Option 4: Script-Based Password (Advanced)

Create a script that retrieves the password from a secure source:

```bash
#!/bin/bash
# vault-password.sh
# Retrieve password from AWS Secrets Manager, HashiCorp Vault, etc.
aws secretsmanager get-secret-value \
  --secret-id ansible-vault-password \
  --query SecretString \
  --output text
```

Make it executable:
```bash
chmod +x vault-password.sh
```

Use in `ansible.cfg`:
```ini
[defaults]
vault_password_file = ./vault-password.sh
```

## Best Practices

### Security

1. **Never commit unencrypted vault files** to version control
2. **Use different vault passwords** for each environment
3. **Rotate vault passwords** every 90 days
4. **Restrict access** to vault password files (`chmod 600`)
5. **Use strong passwords** (minimum 20 characters, mixed case, numbers, symbols)
6. **Store production passwords** in a secure password manager (1Password, LastPass, etc.)

### Organization

1. **Prefix vault variables** with `vault_` for clarity:
   ```yaml
   vault_mysql_root_password: "secret123"
   vault_api_token: "token456"
   ```

2. **Use vault variables in role defaults**:
   ```yaml
   # roles/mysql/defaults/main.yml
   mysql_root_password: "{{ vault_mysql_root_password }}"
   ```

3. **Document all vault variables** in `vault.yml.example`

4. **One vault file per environment** for easier management

### Git Management

Add to `.gitignore`:
```
# Vault passwords
.vault_pass
vault-password.sh

# Unencrypted vault files
**/vault.yml
!**/vault.yml.example
```

Verify vault files are encrypted before committing:
```bash
# Check if file is encrypted
head -1 inventories/production/group_vars/all/vault.yml
# Should output: $ANSIBLE_VAULT;1.1;AES256
```

## Multiple Vault Passwords

For environments with different vault passwords:

### Using Vault IDs

```bash
# Encrypt with vault ID
ansible-vault encrypt \
  --vault-id production@prompt \
  inventories/production/group_vars/all/vault.yml

ansible-vault encrypt \
  --vault-id staging@prompt \
  inventories/staging/group_vars/all/vault.yml

# Run playbook with multiple vault IDs
ansible-playbook site.yml \
  --vault-id production@.vault_pass_production \
  --vault-id staging@.vault_pass_staging
```

## Common Vault Variables

### User Credentials
```yaml
vault_ansible_user_ssh_key: "ssh-rsa AAAA..."
vault_root_password: "password"
vault_ansible_become_password: "password"
```

### API Tokens
```yaml
vault_aws_access_key_id: "AKIA..."
vault_aws_secret_access_key: "secret"
vault_netbox_api_token: "token"
```

### Database Credentials
```yaml
vault_mysql_root_password: "password"
vault_postgresql_postgres_password: "password"
```

### Application Secrets
```yaml
vault_app_secret_key: "secret_key"
vault_app_api_key: "api_key"
```

## Troubleshooting

### Wrong Vault Password

**Error**: `Decryption failed (no vault secrets were found that could decrypt)`

**Solution**: Verify you're using the correct vault password for that environment.

### Vault File Not Found

**Error**: `ERROR! Attempting to decrypt but no vault secrets found`

**Solution**: Create the vault file or check the path is correct.

### Permission Denied

**Error**: `Permission denied: 'vault.yml'`

**Solution**: Check file permissions:
```bash
ls -la inventories/production/group_vars/all/vault.yml
chmod 600 inventories/production/group_vars/all/vault.yml
```

### Forgot Vault Password

**Solution**: Unfortunately, there's no way to recover a forgotten vault password. You'll need to:
1. Re-create the vault file from scratch
2. Re-enter all secrets
3. Re-encrypt with a new password

**Prevention**: Store vault passwords in a secure password manager.

### Check Vault File Integrity

```bash
# Verify file can be decrypted
ansible-vault view inventories/production/group_vars/all/vault.yml

# Check encryption format
file inventories/production/group_vars/all/vault.yml
# Should output: ASCII text (encrypted vault file)
```

## Emergency Procedures

### Compromised Vault Password

1. **Immediately change the vault password**:
   ```bash
   ansible-vault rekey inventories/production/group_vars/all/vault.yml
   ```

2. **Rotate all secrets** stored in the vault

3. **Audit access logs** to determine scope of compromise

4. **Update vault password** in all secure storage locations

### Lost Access to Production Vault

1. Use backup vault password from secure password manager
2. If no backup exists, rotate all production credentials
3. Create new vault file with new credentials
4. Update all systems with new credentials

## References

- [Ansible Vault Documentation](https://docs.ansible.com/ansible/latest/user_guide/vault.html)
- [Ansible Best Practices - Vault](https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html#variables-and-vaults)
- Internal: [CLAUDE.md - Secrets Management](../CLAUDE.md)

---

**Document Version**: 1.0
**Last Updated**: 2025-11-11
**Maintainer**: Infrastructure Team