infra-automation/docs/inventory.md

# Ansible Inventory Configuration

This document describes the dynamic and static inventory configurations for the Ansible infrastructure.

## Table of Contents

1. [Overview](#overview)
2. [Inventory Structure](#inventory-structure)
3. [Dynamic Inventory Solutions](#dynamic-inventory-solutions)
4. [Static/Hybrid Inventory](#statichybrid-inventory)
5. [Usage Examples](#usage-examples)
6. [Troubleshooting](#troubleshooting)

---

## Overview

Per the CLAUDE.md guidelines, this infrastructure uses **dynamic inventories** as the primary inventory source, with static inventories permitted for development environments only.

### Available Inventory Solutions

| Solution | Type | Use Case | Status |
|----------|------|----------|--------|
| SSH Config Parser | Dynamic | Quick discovery from SSH config | ✅ Active |
| Libvirt/KVM Plugin | Dynamic | Real-time VM discovery | ✅ Active |
| Static YAML | Static/Hybrid | Development environment | ✅ Active |

---

## Inventory Structure

```
inventories/
├── production/              # Production environment (dynamic only)
│   ├── group_vars/
│   │   └── all.yml
│   └── [dynamic inventory configs]
├── staging/                 # Staging environment (dynamic only)
│   ├── group_vars/
│   │   └── all.yml
│   └── [dynamic inventory configs]
└── development/             # Development environment
    ├── hosts.yml           # Static/hybrid inventory
    ├── libvirt_kvm.yml     # Libvirt dynamic config
    ├── group_vars/
    │   ├── all.yml
    │   ├── kvm_guests.yml
    │   └── hypervisors.yml
    └── host_vars/
```

---

## Dynamic Inventory Solutions

### 1. SSH Config Parser (`ssh_config_inventory.py`)

**Location:** `/opt/ansible/plugins/inventory/ssh_config_inventory.py`

#### Description
Parses `~/.ssh/config` to automatically generate Ansible inventory with proper grouping and connection parameters.

#### Features
- Automatic host discovery from SSH config
- Intelligent grouping by host characteristics
- ProxyJump support for nested VM access
- No external dependencies (pure Python)

#### Usage

```bash
# List all hosts and groups
python3 plugins/inventory/ssh_config_inventory.py --list

# Get variables for specific host
python3 plugins/inventory/ssh_config_inventory.py --host pihole

# Use with ansible commands
ansible all -i plugins/inventory/ssh_config_inventory.py --list-hosts

# Use with playbooks
ansible-playbook -i plugins/inventory/ssh_config_inventory.py site.yml
```

#### Host Categorization Logic

| Category | Criteria |
|----------|----------|
| `external_hosts` | Public IPs, no ProxyJump, non-ansible user |
| `hypervisors` | ForwardAgent enabled, specific users (grok) |
| `dns_servers` | ansible user + ProxyJump + hostname contains 'pihole'/'dns' |
| `mail_servers` | ansible user + ProxyJump + hostname contains 'mail'/'mx' |
| `development` | ansible user + ProxyJump + hostname contains 'dev'/'test'/'derp' |

#### Generated Inventory Structure

```json
{
  "all": {
    "children": ["external_hosts", "hypervisors", "kvm_guests"]
  },
  "external_hosts": {
    "hosts": ["odin"]
  },
  "hypervisors": {
    "hosts": ["grokbox"]
  },
  "kvm_guests": {
    "children": ["dns_servers", "mail_servers", "development", "uncategorized"],
    "vars": {
      "ansible_user": "ansible",
      "ansible_ssh_common_args": "-o StrictHostKeyChecking=accept-new"
    }
  },
  "_meta": {
    "hostvars": { ... }
  }
}
```

---

### 2. Libvirt/KVM Dynamic Inventory (`libvirt_kvm.py`)

**Location:** `/opt/ansible/plugins/inventory/libvirt_kvm.py`

#### Description
Queries libvirt hypervisors directly to discover KVM guest VMs in real-time, including their state, resources, and network configuration.

#### Features
- Real-time VM discovery via libvirt API
- VM state detection (running, stopped, paused)
- Automatic IP address detection
- Resource information (vCPUs, memory, networks)
- Multiple hypervisor support
- ProxyJump configuration

#### Requirements

```bash
# Debian/Ubuntu
apt-get install python3-libvirt

# RHEL/Fedora/Rocky
dnf install python3-libvirt
```

#### Configuration

Set environment variables or use configuration file:

```bash
# Environment variables
export LIBVIRT_DEFAULT_URI="qemu+ssh://grok@grok.home.serneels.xyz/system"
export LIBVIRT_HYPERVISOR_NAME="grokbox"
```

Or use YAML configuration file: `inventories/development/libvirt_kvm.yml`

#### Usage

```bash
# List all VMs
python3 plugins/inventory/libvirt_kvm.py --list

# Get specific VM details
python3 plugins/inventory/libvirt_kvm.py --host mymx

# Use with ansible
ansible running_vms -i plugins/inventory/libvirt_kvm.py -m ping

# Use with playbooks
ansible-playbook -i plugins/inventory/libvirt_kvm.py playbooks/update.yml
```

#### Generated Groups

| Group | Description |
|-------|-------------|
| `hypervisors` | KVM hypervisor hosts |
| `kvm_guests` | All guest VMs |
| `running_vms` | VMs in running state |
| `stopped_vms` | VMs not running (shutoff, paused, etc.) |

#### Host Variables

Each VM includes:
- `vm_name`: VM hostname
- `vm_uuid`: Libvirt UUID
- `vm_state`: Current state (running, shutoff, etc.)
- `vm_vcpus`: Number of virtual CPUs
- `vm_memory_mb`: Memory allocation in MB
- `vm_networks`: Network interface details
- `ansible_host`: IP address (if available)
- `ansible_ssh_common_args`: ProxyJump configuration
- `hypervisor`: Parent hypervisor name

#### Example Output

```json
{
  "running_vms": {
    "hosts": ["mymx", "pihole", "derp"]
  },
  "_meta": {
    "hostvars": {
      "pihole": {
        "vm_name": "pihole",
        "vm_uuid": "6d714c93-16fb-41c8-8ef8-9001f9066b3a",
        "vm_state": "running",
        "vm_vcpus": 2,
        "vm_memory_mb": 2048,
        "ansible_host": "192.168.122.12",
        "ansible_ssh_common_args": "-o ProxyJump=grokbox -o StrictHostKeyChecking=accept-new",
        "hypervisor": "grokbox"
      }
    }
  }
}
```

---

## Static/Hybrid Inventory

**Location:** `/opt/ansible/inventories/development/hosts.yml`

### Description
Manually maintained static inventory for the development environment with detailed host metadata and configuration.

### Structure

```yaml
all:
  children:
    external_hosts:     # Public-facing hosts
    hypervisors:        # KVM hypervisor hosts
    kvm_guests:         # Virtual machine guests
      children:
        dns_servers:
        mail_servers:
        development:
        uncategorized:
```

### Group Variables

Variables are defined in `group_vars/` directory:

- **`all.yml`**: Global variables for all hosts
- **`kvm_guests.yml`**: Common VM configuration (LVM, networking, ProxyJump)
- **`hypervisors.yml`**: Hypervisor-specific settings (libvirt, QEMU)

### Host Variables

Host-specific variables can be placed in `host_vars/` directory:

```
host_vars/
├── pihole.yml
├── mymx.yml
└── derp.yml
```

### Usage

```bash
# List hosts
ansible all -i inventories/development/hosts.yml --list-hosts

# Run playbook
ansible-playbook -i inventories/development/hosts.yml site.yml

# Target specific group
ansible dns_servers -i inventories/development/hosts.yml -m ping
```

---

## Usage Examples

### Example 1: List All Hosts (Dynamic)

```bash
# Using SSH config parser
ansible all -i plugins/inventory/ssh_config_inventory.py --list-hosts

# Using libvirt inventory
ansible all -i plugins/inventory/libvirt_kvm.py --list-hosts
```

### Example 2: Ping All Running VMs

```bash
ansible running_vms -i plugins/inventory/libvirt_kvm.py -m ping
```

### Example 3: Run Playbook Against KVM Guests

```bash
ansible-playbook -i inventories/development/hosts.yml \
  --limit kvm_guests \
  playbooks/system-update.yml
```

### Example 4: Check Host Variables

```bash
# Using dynamic inventory
ansible-inventory -i plugins/inventory/libvirt_kvm.py --host pihole

# Using static inventory
ansible-inventory -i inventories/development/hosts.yml --host pihole --yaml
```

### Example 5: Multiple Inventory Sources

You can combine multiple inventory sources:

```bash
ansible-playbook -i inventories/development/hosts.yml \
  -i plugins/inventory/libvirt_kvm.py \
  site.yml
```

### Example 6: Filter by Group

```bash
# Target only mail servers
ansible mail_servers -i plugins/inventory/ssh_config_inventory.py -m setup

# Target only hypervisors
ansible hypervisors -i inventories/development/hosts.yml -m shell -a "virsh list --all"
```

---

## Ansible Configuration

Configure default inventory in `ansible.cfg`:

```ini
[defaults]
inventory = ./inventories/development/hosts.yml
# Or use dynamic:
# inventory = ./plugins/inventory/libvirt_kvm.py

# Enable multiple inventory sources
# inventory = ./inventories/development/hosts.yml,./plugins/inventory/libvirt_kvm.py

# Inventory plugins path
inventory_plugins = ./plugins/inventory

# Enable fact caching for performance
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_facts
fact_caching_timeout = 86400
```

---

## Troubleshooting

### SSH Config Parser Issues

**Problem:** Hosts not appearing in inventory

**Solution:**
- Check `~/.ssh/config` exists and is readable
- Verify Host declarations are properly formatted
- Run with `--list` to see parsed output
- Check for Python syntax errors

**Problem:** Incorrect host categorization

**Solution:**
- Review categorization logic in `_categorize_host()` method
- Add custom categorization rules
- Use static inventory for specific grouping needs

### Libvirt Inventory Issues

**Problem:** `python3-libvirt` not installed

**Solution:**
```bash
# Debian/Ubuntu
sudo apt-get install python3-libvirt

# RHEL/Rocky/Fedora
sudo dnf install python3-libvirt
```

**Problem:** Connection to hypervisor fails

**Solution:**
- Verify SSH access to hypervisor: `ssh grok@grok.home.serneels.xyz`
- Check libvirt URI: `virsh -c qemu+ssh://grok@grok.home.serneels.xyz/system list`
- Ensure SSH keys are properly configured
- Check SSH agent forwarding if needed

**Problem:** VMs discovered but no IP addresses

**Solution:**
- VMs may not have DHCP leases yet
- Check VM is fully booted: `virsh dominfo <vm_name>`
- Manually query: `virsh domifaddr <vm_name>`
- Use static inventory with known IP addresses

### Static Inventory Issues

**Problem:** YAML syntax errors

**Solution:**
- Validate YAML syntax: `yamllint inventories/development/hosts.yml`
- Check indentation (use 2 spaces)
- Verify with: `ansible-inventory -i inventories/development/hosts.yml --list`

**Problem:** Variables not being applied

**Solution:**
- Check variable precedence order
- Verify `group_vars/` and `host_vars/` file names match group/host names
- Use `ansible-inventory --host <hostname>` to debug variable merging

### General Debugging

```bash
# Verify inventory parsing
ansible-inventory -i <inventory_source> --list

# Check specific host variables
ansible-inventory -i <inventory_source> --host <hostname>

# Graph inventory structure
ansible-inventory -i <inventory_source> --graph

# Test connectivity
ansible all -i <inventory_source> -m ping -vvv

# Dry run playbook
ansible-playbook -i <inventory_source> site.yml --check --diff
```

---

## Security Considerations

### SSH Config Parser
- ✅ No credentials stored in inventory
- ✅ Uses existing SSH configuration
- ⚠️ Ensure `~/.ssh/config` has proper permissions (600)

### Libvirt Inventory
- ✅ Uses SSH key authentication
- ✅ No passwords in configuration
- ⚠️ Requires SSH access to hypervisor
- ⚠️ Libvirt connection string may be logged

### Static Inventory
- ✅ Version controlled and auditable
- ⚠️ Use Ansible Vault for sensitive variables
- ⚠️ Never commit unencrypted credentials

### Best Practices
- Use Ansible Vault for secrets: `ansible-vault encrypt group_vars/all/vault.yml`
- Rotate SSH keys regularly (90-180 days per CLAUDE.md)
- Use ProxyJump/bastion hosts for nested VM access
- Enable SSH ControlMaster for connection reuse
- Implement inventory caching for large infrastructures

---

## Performance Optimization

### Caching
Enable fact caching in `ansible.cfg`:
```ini
[defaults]
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_facts
fact_caching_timeout = 86400
```

### Parallelism
Adjust fork count:
```ini
[defaults]
forks = 20
```

### SSH Connection Reuse
Configure ControlMaster in `~/.ssh/config`:
```
Host *
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%r@%h-%p
  ControlPersist 600s
```

---

## References

- [Ansible Dynamic Inventory](https://docs.ansible.com/ansible/latest/user_guide/intro_dynamic_inventory.html)
- [Libvirt Python API](https://libvirt.org/python.html)
- [SSH Config Documentation](https://man.openbsd.org/ssh_config)
- [CLAUDE.md Guidelines](/opt/ansible/CLAUDE.md)

---

**Document Version:** 1.0.0
**Last Updated:** 2025-11-10
**Maintainer:** Ansible Infrastructure Team