infra-automation/docs/roles/deploy_linux_vm.md

# Deploy Linux VM Role Documentation

## Overview

The `deploy_linux_vm` role provides enterprise-grade automated deployment of Linux virtual machines on KVM/libvirt hypervisors. It implements comprehensive security hardening, LVM storage management, and multi-distribution support aligned with CLAUDE.md infrastructure guidelines.

## Purpose

- **Automated VM Provisioning**: Unattended deployment using cloud-init for consistent infrastructure
- **Security-First Design**: Built-in SSH hardening, SELinux/AppArmor enforcement, firewall configuration
- **LVM Storage Management**: Automated LVM setup with CLAUDE.md-compliant partition schema
- **Multi-Distribution Support**: Debian, Ubuntu, RHEL, AlmaLinux, Rocky Linux, openSUSE
- **Production Ready**: Idempotent, well-tested, and suitable for production environments

## Architecture

### Deployment Flow

```
┌──────────────────────┐
│  Ansible Controller  │
│  (Control Node)      │
└──────────┬───────────┘
           │
           │ SSH (port 22)
           ▼
┌──────────────────────┐
│  KVM Hypervisor      │
│  (grokbox, etc.)     │
└──────────┬───────────┘
           │
           │ 1. Download cloud image
           │ 2. Create VM disks
           │ 3. Generate cloud-init ISO
           │ 4. Define & start VM
           ▼
┌──────────────────────┐
│  Guest VM            │
│  ┌────────────────┐  │
│  │ Cloud-Init     │──┼──▶ User creation
│  │ First Boot     │  │    SSH keys
│  │                │  │    Package installation
│  └────────┬───────┘  │    Security hardening
│           │          │
│           ▼          │
│  ┌────────────────┐  │
│  │ Post-Deploy    │──┼──▶ LVM configuration
│  │ Configuration  │  │    Data migration
│  │                │  │    Fstab updates
│  └────────────────┘  │
└──────────────────────┘
```

### Storage Architecture

```
Hypervisor: /var/lib/libvirt/images/
├── ubuntu-22.04-cloud.qcow2           # Base cloud image (shared)
├── vm_name.qcow2                      # Primary disk (30GB default)
│   ├── /dev/vda1 → /boot (2GB)
│   ├── /dev/vda2 → / (root, 8GB)
│   └── /dev/vda3 → swap (1GB)
├── vm_name-lvm.qcow2                  # LVM disk (30GB default)
│   └── /dev/vdb → Physical Volume
│       └── vg_system (Volume Group)
│           ├── lv_opt → /opt (3GB)
│           ├── lv_tmp → /tmp (1GB, noexec)
│           ├── lv_home → /home (2GB)
│           ├── lv_var → /var (5GB)
│           ├── lv_var_log → /var/log (2GB)
│           ├── lv_var_tmp → /var/tmp (5GB, noexec)
│           ├── lv_var_audit → /var/log/audit (1GB)
│           └── lv_swap → swap (2GB)
└── vm_name-cloud-init.iso             # Cloud-init configuration
```

### Task Organization

The role follows modular task organization:

```
roles/deploy_linux_vm/tasks/
├── main.yml                    # Orchestration and task flow
├── preflight.yml               # Pre-deployment validation
├── install.yml                 # Hypervisor package installation
├── download_image.yml          # Cloud image download and verification
├── create_storage.yml          # VM disk creation
├── cloud-init.yml              # Cloud-init configuration generation
├── deploy_vm.yml               # VM definition and deployment
├── post_deploy_lvm.yml         # LVM configuration on guest
└── cleanup.yml                 # Temporary file cleanup
```

## Integration Points

### With Infrastructure

The role integrates seamlessly with:

- **Dynamic Inventories**: Works with AWS, Azure, Proxmox, VMware inventory sources
- **Configuration Management**: Post-deployment hooks for additional role application
- **Monitoring Integration**: Collects deployment metrics for tracking
- **CMDB Sync**: Can export VM metadata to NetBox, ServiceNow

### With Other Roles

**Typical Workflow:**

```yaml
# 1. Deploy VM infrastructure
- role: deploy_linux_vm

# 2. Gather system information
- role: system_info

# 3. Apply application-specific configuration
- role: webserver
  # or
- role: database
  # or
- role: kubernetes_node
```

### Cloud-Init Integration

The role generates comprehensive cloud-init configuration:

- **User Data**: User creation, SSH keys, package installation
- **Meta Data**: Instance ID, hostname, network configuration
- **Vendor Data**: Distribution-specific customizations

Cloud-init handles:
- Ansible user creation with sudo access
- SSH key deployment
- Essential package installation (vim, htop, git, python3, etc.)
- Security package installation (aide, auditd, chrony)
- SSH hardening configuration
- Firewall setup
- SELinux/AppArmor configuration
- Automatic security updates

## Data Model

### Role Variables

#### Required Variables

| Variable | Type | Description | Example |
|----------|------|-------------|---------|
| `deploy_linux_vm_os_distribution` | string | Target distribution identifier | `ubuntu-22.04`, `almalinux-9` |

#### VM Configuration Variables

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `deploy_linux_vm_name` | string | `linux-guest` | VM name in libvirt |
| `deploy_linux_vm_hostname` | string | `linux-vm` | Guest hostname |
| `deploy_linux_vm_domain` | string | `localdomain` | Domain name (FQDN = hostname.domain) |
| `deploy_linux_vm_vcpus` | integer | `2` | Number of virtual CPUs |
| `deploy_linux_vm_memory_mb` | integer | `2048` | RAM allocation in MB |
| `deploy_linux_vm_disk_size_gb` | integer | `30` | Primary disk size in GB |

#### LVM Configuration Variables

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `deploy_linux_vm_use_lvm` | boolean | `true` | Enable LVM configuration |
| `deploy_linux_vm_lvm_vg_name` | string | `vg_system` | Volume group name |
| `deploy_linux_vm_lvm_pv_device` | string | `/dev/vdb` | Physical volume device |
| `deploy_linux_vm_lvm_volumes` | list | (see below) | Logical volume definitions |

**Default LVM Volumes (CLAUDE.md Compliant):**

```yaml
deploy_linux_vm_lvm_volumes:
  - name: lv_opt
    size: 3G
    mount: /opt
    fstype: ext4
  - name: lv_tmp
    size: 1G
    mount: /tmp
    fstype: ext4
    mount_options: noexec,nosuid,nodev
  - name: lv_home
    size: 2G
    mount: /home
    fstype: ext4
  - name: lv_var
    size: 5G
    mount: /var
    fstype: ext4
  - name: lv_var_log
    size: 2G
    mount: /var/log
    fstype: ext4
  - name: lv_var_tmp
    size: 5G
    mount: /var/tmp
    fstype: ext4
    mount_options: noexec,nosuid,nodev
  - name: lv_var_audit
    size: 1G
    mount: /var/log/audit
    fstype: ext4
  - name: lv_swap
    size: 2G
    mount: none
    fstype: swap
```

#### Security Configuration Variables

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `deploy_linux_vm_enable_firewall` | boolean | `true` | Enable UFW (Debian) or firewalld (RHEL) |
| `deploy_linux_vm_enable_selinux` | boolean | `true` | Enable SELinux enforcing (RHEL family) |
| `deploy_linux_vm_enable_apparmor` | boolean | `true` | Enable AppArmor (Debian family) |
| `deploy_linux_vm_enable_auditd` | boolean | `true` | Enable audit daemon |
| `deploy_linux_vm_enable_automatic_updates` | boolean | `true` | Enable automatic security updates |
| `deploy_linux_vm_automatic_reboot` | boolean | `false` | Auto-reboot after updates (not recommended) |

#### SSH Hardening Variables

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `deploy_linux_vm_ssh_permit_root_login` | string | `no` | Allow root SSH login |
| `deploy_linux_vm_ssh_password_authentication` | string | `no` | Allow password authentication |
| `deploy_linux_vm_ssh_gssapi_authentication` | string | `no` | **GSSAPI disabled per requirements** |
| `deploy_linux_vm_ssh_gssapi_cleanup_credentials` | string | `no` | GSSAPI credential cleanup |
| `deploy_linux_vm_ssh_max_auth_tries` | integer | `3` | Maximum authentication attempts |
| `deploy_linux_vm_ssh_client_alive_interval` | integer | `300` | SSH keepalive interval (seconds) |
| `deploy_linux_vm_ssh_client_alive_count_max` | integer | `2` | Maximum keepalive probes |

#### User Configuration Variables

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `deploy_linux_vm_ansible_user` | string | `ansible` | Service account username |
| `deploy_linux_vm_ansible_user_ssh_key` | string | (generated) | SSH public key for ansible user |
| `deploy_linux_vm_root_password` | string | `ChangeMe123!` | Root password (console only) |

### Distribution Support Matrix

| Distribution | Versions | Cloud Image Source | Tested |
|--------------|----------|-------------------|--------|
| **Debian** | 11 (Bullseye)<br>12 (Bookworm) | https://cloud.debian.org/images/cloud/ | ✓ |
| **Ubuntu** | 20.04 LTS (Focal)<br>22.04 LTS (Jammy)<br>24.04 LTS (Noble) | https://cloud-images.ubuntu.com/ | ✓ |
| **RHEL** | 8, 9 | Red Hat Customer Portal | ✓ |
| **AlmaLinux** | 8, 9 | https://repo.almalinux.org/almalinux/ | ✓ |
| **Rocky Linux** | 8, 9 | https://download.rockylinux.org/pub/rocky/ | ✓ |
| **CentOS Stream** | 8, 9 | https://cloud.centos.org/centos/ | ✓ |
| **openSUSE Leap** | 15.5, 15.6 | https://download.opensuse.org/distribution/ | ✓ |

## Use Cases

### Use Case 1: Development Environment

**Scenario**: Create development VMs for a development team.

```yaml
---
- name: Deploy Development VMs
  hosts: hypervisor_dev
  become: yes
  vars:
    dev_vms:
      - { name: dev01, user: alice, distro: ubuntu-22.04 }
      - { name: dev02, user: bob, distro: debian-12 }
      - { name: dev03, user: charlie, distro: almalinux-9 }
  tasks:
    - name: Deploy developer VMs
      include_role:
        name: deploy_linux_vm
      vars:
        deploy_linux_vm_name: "{{ item.name }}"
        deploy_linux_vm_hostname: "{{ item.name }}"
        deploy_linux_vm_os_distribution: "{{ item.distro }}"
        deploy_linux_vm_vcpus: 2
        deploy_linux_vm_memory_mb: 4096
        deploy_linux_vm_use_lvm: false  # Skip LVM for dev environments
      loop: "{{ dev_vms }}"
```

**Benefits**:
- Rapid provisioning of consistent dev environments
- Easy destruction and recreation
- Reduced LVM overhead for ephemeral VMs

### Use Case 2: Production Web Application Stack

**Scenario**: Deploy a 3-tier web application (load balancer, app servers, database).

```yaml
---
- name: Deploy Production Web Stack
  hosts: hypervisor_prod
  become: yes
  serial: 1  # Deploy one at a time for safety
  tasks:
    # Load Balancer
    - name: Deploy load balancer
      include_role:
        name: deploy_linux_vm
      vars:
        deploy_linux_vm_name: "lb01"
        deploy_linux_vm_hostname: "lb01"
        deploy_linux_vm_domain: "production.example.com"
        deploy_linux_vm_os_distribution: "ubuntu-22.04"
        deploy_linux_vm_vcpus: 2
        deploy_linux_vm_memory_mb: 4096
        deploy_linux_vm_use_lvm: true

    # Application Servers
    - name: Deploy application servers
      include_role:
        name: deploy_linux_vm
      vars:
        deploy_linux_vm_name: "app{{ '%02d' | format(item) }}"
        deploy_linux_vm_hostname: "app{{ '%02d' | format(item) }}"
        deploy_linux_vm_domain: "production.example.com"
        deploy_linux_vm_os_distribution: "almalinux-9"
        deploy_linux_vm_vcpus: 4
        deploy_linux_vm_memory_mb: 8192
        deploy_linux_vm_disk_size_gb: 50
      loop: [1, 2, 3]

    # Database Server
    - name: Deploy database server
      include_role:
        name: deploy_linux_vm
      vars:
        deploy_linux_vm_name: "db01"
        deploy_linux_vm_hostname: "db01"
        deploy_linux_vm_domain: "production.example.com"
        deploy_linux_vm_os_distribution: "almalinux-9"
        deploy_linux_vm_vcpus: 8
        deploy_linux_vm_memory_mb: 32768
        deploy_linux_vm_disk_size_gb: 200
        deploy_linux_vm_lvm_volumes:
          - { name: lv_opt, size: 5G, mount: /opt, fstype: ext4 }
          - { name: lv_tmp, size: 2G, mount: /tmp, fstype: ext4, mount_options: noexec,nosuid,nodev }
          - { name: lv_home, size: 2G, mount: /home, fstype: ext4 }
          - { name: lv_var, size: 10G, mount: /var, fstype: ext4 }
          - { name: lv_var_log, size: 5G, mount: /var/log, fstype: ext4 }
          - { name: lv_pgsql, size: 100G, mount: /var/lib/pgsql, fstype: xfs }
          - { name: lv_swap, size: 4G, mount: none, fstype: swap }
```

**Benefits**:
- Consistent infrastructure across tiers
- Customized resources per tier
- LVM allows for database storage expansion
- Security hardening applied uniformly

### Use Case 3: CI/CD Build Agents

**Scenario**: Deploy ephemeral build agents for CI/CD pipeline.

```yaml
---
- name: Deploy CI/CD Build Agents
  hosts: hypervisor_ci
  become: yes
  vars:
    agent_count: 5
  tasks:
    - name: Deploy build agents
      include_role:
        name: deploy_linux_vm
      vars:
        deploy_linux_vm_name: "ci-agent-{{ item }}"
        deploy_linux_vm_hostname: "ci-agent-{{ item }}"
        deploy_linux_vm_os_distribution: "ubuntu-22.04"
        deploy_linux_vm_vcpus: 4
        deploy_linux_vm_memory_mb: 8192
        deploy_linux_vm_use_lvm: false
        deploy_linux_vm_enable_automatic_updates: false  # Controlled updates
      loop: "{{ range(1, agent_count + 1) | list }}"
```

**Benefits**:
- Quick provisioning of build capacity
- Easy horizontal scaling
- Consistent build environment
- Simple cleanup after job completion

### Use Case 4: Disaster Recovery Testing

**Scenario**: Create replica VMs for DR testing without impacting production.

```yaml
---
- name: Deploy DR Test Environment
  hosts: hypervisor_dr
  become: yes
  tasks:
    - name: Deploy DR replicas
      include_role:
        name: deploy_linux_vm
      vars:
        deploy_linux_vm_name: "dr-{{ item.name }}"
        deploy_linux_vm_hostname: "dr-{{ item.name }}"
        deploy_linux_vm_domain: "dr.example.com"
        deploy_linux_vm_os_distribution: "{{ item.distro }}"
        deploy_linux_vm_vcpus: "{{ item.vcpus }}"
        deploy_linux_vm_memory_mb: "{{ item.memory }}"
      loop:
        - { name: web01, distro: ubuntu-22.04, vcpus: 4, memory: 8192 }
        - { name: db01, distro: almalinux-9, vcpus: 8, memory: 16384 }
```

**Benefits**:
- Isolated DR testing environment
- Production-like configuration
- Quick teardown after testing

## Security Implementation

### Security Controls Mapping

| Control Area | Implementation | Compliance |
|-------------|---------------|------------|
| **Access Control** | SSH key-only authentication, root login disabled | CIS 5.2.10, 5.2.9 |
| **Network Security** | Firewall enabled, minimal services exposed | CIS 3.5.x |
| **Audit & Logging** | auditd enabled, centralized logging ready | CIS 4.1.x, NIST AU family |
| **Cryptography** | SSH v2 only, strong ciphers | CIS 5.2.11 |
| **Least Privilege** | Non-root ansible user, sudo with logging | CIS 5.3.x |
| **Patch Management** | Automatic security updates | NIST SI-2 |
| **Mandatory Access Control** | SELinux enforcing / AppArmor enabled | CIS 1.6.x, NIST AC-3 |
| **File Integrity** | AIDE installed and configured | CIS 1.3.2, NIST SI-7 |
| **Time Sync** | chrony configured | CIS 2.2.1.1, NIST AU-8 |
| **Storage Security** | /tmp noexec, separate /var/log | CIS 1.1.x |

### SSH Hardening Details

The role implements comprehensive SSH hardening per CLAUDE.md requirements:

**Configuration File**: `/etc/ssh/sshd_config.d/99-security.conf`

```ini
# Authentication
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
ChallengeResponseAuthentication no
KerberosAuthentication no
GSSAPIAuthentication no               # Explicitly disabled per requirements
GSSAPICleanupCredentials no

# Connection limits
MaxAuthTries 3
MaxSessions 10
ClientAliveInterval 300
ClientAliveCountMax 2

# Security hardening
PermitEmptyPasswords no
X11Forwarding no
Protocol 2
```

### Firewall Configuration

**Debian/Ubuntu (UFW)**:
```bash
# Default policies
ufw default deny incoming
ufw default allow outgoing

# Allow SSH
ufw allow 22/tcp

# Enable
ufw --force enable
```

**RHEL/AlmaLinux (firewalld)**:
```bash
# Default zone: drop
firewall-cmd --set-default-zone=drop

# Allow SSH in public zone
firewall-cmd --zone=public --add-service=ssh --permanent

# Reload
firewall-cmd --reload
```

### SELinux/AppArmor

**RHEL Family (SELinux)**:
- Mode: `enforcing`
- Policy: `targeted`
- Status check: `getenforce`
- Troubleshooting: `ausearch -m avc -ts recent`

**Debian Family (AppArmor)**:
- Status: `enabled`
- Mode: `enforce`
- Status check: `aa-status`
- Profiles: All default profiles enabled

### Automatic Updates Configuration

**Debian/Ubuntu (unattended-upgrades)**:
```conf
# /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security";
};
Unattended-Upgrade::Automatic-Reboot "false";
```

**RHEL/AlmaLinux (dnf-automatic)**:
```conf
# /etc/dnf/automatic.conf
[commands]
upgrade_type = security
apply_updates = yes
reboot = never
```

## Performance Considerations

### Execution Time

Typical deployment timeline:
- **Pre-flight checks**: 5-10 seconds
- **Package installation**: 10-30 seconds (first run only)
- **Cloud image download**: 30-120 seconds (first run only, cached thereafter)
- **VM deployment**: 30-60 seconds
- **Cloud-init first boot**: 60-180 seconds
- **LVM configuration**: 30-60 seconds
- **Total**: 3-7 minutes per VM

Factors affecting performance:
- Internet connection speed (image download)
- Hypervisor disk I/O (VM creation)
- VM boot time (distribution-dependent)
- Cloud-init package installation count

### Optimization Strategies

1. **Pre-cache cloud images**:
   ```bash
   ansible-playbook site.yml -t deploy_linux_vm,download
   ```

2. **Parallel deployment**:
   ```bash
   ansible-playbook site.yml -t deploy_linux_vm -f 5
   ```

3. **Skip slow operations**:
   ```bash
   ansible-playbook site.yml -t deploy_linux_vm --skip-tags install,download
   ```

4. **Disable LVM for faster provisioning**:
   ```yaml
   deploy_linux_vm_use_lvm: false
   ```

### Resource Requirements

**Hypervisor Requirements**:
- CPU: 2+ cores per VM recommended
- RAM: 2GB base + (VM memory allocation * concurrent VMs)
- Disk: 100GB+ available in `/var/lib/libvirt/images`
- Network: 10 Mbps+ for cloud image downloads

**Control Node Requirements**:
- Minimal (Ansible controller overhead)
- Disk: <1MB per VM for cloud-init config storage

## Troubleshooting Guide

### Common Issues

#### Issue: Cloud image download fails

**Symptoms**: Task fails during image download
**Causes**:
- No internet connectivity from hypervisor
- Image URL changed or unavailable
- Insufficient disk space

**Solutions**:
```bash
# Test internet connectivity
ansible hypervisor -m shell -a "ping -c 3 8.8.8.8"

# Check disk space
ansible hypervisor -m shell -a "df -h /var/lib/libvirt/images"

# Manual download and verification
ansible hypervisor -m shell -a "wget -O /tmp/test.img <cloud_image_url>"

# Check image URL validity
ansible hypervisor -m shell -a "curl -I <cloud_image_url>"
```

#### Issue: VM fails to start

**Symptoms**: VM shows as "shut off" immediately after creation
**Causes**:
- Insufficient resources on hypervisor
- Cloud-init ISO creation failed
- libvirt permission issues

**Solutions**:
```bash
# Check VM status and errors
ansible hypervisor -m shell -a "virsh list --all"
ansible hypervisor -m shell -a "virsh start <vm_name>"
ansible hypervisor -m shell -a "journalctl -u libvirtd -n 50"

# Check libvirt logs
ansible hypervisor -m shell -a "tail -50 /var/log/libvirt/qemu/<vm_name>.log"

# Verify cloud-init ISO exists
ansible hypervisor -m shell -a "ls -lh /var/lib/libvirt/images/<vm_name>-cloud-init.iso"

# Check resource availability
ansible hypervisor -m shell -a "free -h && df -h"
```

#### Issue: Cannot SSH to VM

**Symptoms**: SSH connection refused or times out
**Causes**:
- Cloud-init not completed
- Firewall blocking SSH
- Wrong IP address
- SSH key mismatch

**Solutions**:
```bash
# Get VM IP address
ansible hypervisor -m shell -a "virsh domifaddr <vm_name>"

# Check if VM is responsive (via console)
ansible hypervisor -m shell -a "virsh console <vm_name>"
# (Press Ctrl+] to exit console)

# Wait for cloud-init completion
ssh ansible@<VM_IP> "cloud-init status --wait"

# Check cloud-init logs
ssh ansible@<VM_IP> "tail -100 /var/log/cloud-init-output.log"

# Verify SSH service
ssh ansible@<VM_IP> "systemctl status sshd"

# Check firewall rules
ssh ansible@<VM_IP> "sudo ufw status" # Debian/Ubuntu
ssh ansible@<VM_IP> "sudo firewall-cmd --list-all" # RHEL
```

#### Issue: LVM configuration fails

**Symptoms**: Post-deployment LVM tasks fail
**Causes**:
- Second disk not attached
- LVM packages not installed
- Insufficient disk space

**Solutions**:
```bash
# Check if second disk exists
ssh ansible@<VM_IP> "lsblk"

# Verify LVM packages
ssh ansible@<VM_IP> "which lvm"

# Check physical volumes
ssh ansible@<VM_IP> "sudo pvs"

# Check volume groups
ssh ansible@<VM_IP> "sudo vgs"

# Check logical volumes
ssh ansible@<VM_IP> "sudo lvs"

# Manually re-run LVM configuration
ansible-playbook site.yml -t deploy_linux_vm,lvm,post-deploy \
  -e "deploy_linux_vm_name=<vm_name>"
```

#### Issue: Slow VM performance

**Symptoms**: VM is sluggish or unresponsive
**Causes**:
- Overcommitted hypervisor resources
- Disk I/O bottleneck
- Memory swapping

**Solutions**:
```bash
# Check hypervisor load
ansible hypervisor -m shell -a "top -bn1 | head -20"

# Check VM resource allocation
ansible hypervisor -m shell -a "virsh dominfo <vm_name>"

# Check disk I/O
ansible hypervisor -m shell -a "iostat -x 1 5"

# Inside VM: check memory
ssh ansible@<VM_IP> "free -h"

# Inside VM: check disk I/O
ssh ansible@<VM_IP> "iostat -x 1 5"
```

### Debug Mode

Run with increased verbosity:

```bash
# Standard verbose
ansible-playbook site.yml -t deploy_linux_vm -v

# More verbose (connections)
ansible-playbook site.yml -t deploy_linux_vm -vv

# Very verbose (debugging)
ansible-playbook site.yml -t deploy_linux_vm -vvv

# Extreme verbose (all data)
ansible-playbook site.yml -t deploy_linux_vm -vvvv
```

### Log Locations

**Hypervisor**:
- libvirt logs: `/var/log/libvirt/qemu/<vm_name>.log`
- System logs: `journalctl -u libvirtd`

**Guest VM**:
- Cloud-init output: `/var/log/cloud-init-output.log`
- Cloud-init logs: `/var/log/cloud-init.log`
- System logs: `journalctl` or `/var/log/syslog` (Debian) / `/var/log/messages` (RHEL)
- SSH logs: `/var/log/auth.log` (Debian) / `/var/log/secure` (RHEL)
- Audit logs: `/var/log/audit/audit.log`

## Maintenance

### Regular Updates

**Quarterly Tasks**:
- Review cloud image URLs for updates
- Test role with latest distribution versions
- Update documentation for new features
- Review security controls and compliance

**Testing Checklist**:
```bash
# 1. Syntax validation
ansible-playbook site.yml --syntax-check

# 2. Dry-run
ansible-playbook site.yml -t deploy_linux_vm --check

# 3. Deploy test VM
ansible-playbook site.yml -t deploy_linux_vm \
  -e "deploy_linux_vm_name=test-vm-$(date +%s)"

# 4. Verify deployment
ansible hypervisor -m shell -a "virsh list --all"

# 5. SSH connectivity
ssh -J hypervisor ansible@<test_vm_ip> "hostname"

# 6. Security validation
ssh ansible@<test_vm_ip> "sudo getenforce" # RHEL
ssh ansible@<test_vm_ip> "sudo aa-status" # Debian

# 7. Cleanup
ansible hypervisor -m shell -a "virsh destroy test-vm-*"
ansible hypervisor -m shell -a "virsh undefine test-vm-* --remove-all-storage"
```

### Monitoring

Track deployment metrics:
- Deployment success rate
- Average deployment time
- Cloud-init failure rate
- SSH connectivity success rate

### Backup Strategy

**VM Backups**:
```bash
# Create VM snapshot
virsh snapshot-create-as <vm_name> backup-$(date +%Y%m%d) "Pre-update backup"

# Export VM configuration
virsh dumpxml <vm_name> > <vm_name>.xml

# Backup VM disk
qemu-img convert -O qcow2 /var/lib/libvirt/images/<vm_name>.qcow2 \
  /backup/<vm_name>-$(date +%Y%m%d).qcow2
```

## Advanced Usage

### Custom Cloud-Init Configuration

Override default cloud-init with custom configuration:

```yaml
deploy_linux_vm_cloud_init_user_data: |
  #cloud-config
  package_update: true
  package_upgrade: true
  packages:
    - custom-package
    - another-package
  runcmd:
    - [sh, -c, "echo 'Custom configuration' > /root/custom.txt"]
```

### Integration with Terraform

Use Ansible role within Terraform provisioner:

```hcl
resource "null_resource" "deploy_vm" {
  provisioner "local-exec" {
    command = <<EOT
      ansible-playbook site.yml -t deploy_linux_vm \
        -e "deploy_linux_vm_name=${var.vm_name}" \
        -e "deploy_linux_vm_os_distribution=${var.distro}"
    EOT
  }
}
```

### CI/CD Integration

Jenkins pipeline example:

```groovy
pipeline {
    agent any
    stages {
        stage('Deploy VM') {
            steps {
                ansiblePlaybook(
                    playbook: 'site.yml',
                    tags: 'deploy_linux_vm',
                    extraVars: [
                        deploy_linux_vm_name: "${env.VM_NAME}",
                        deploy_linux_vm_os_distribution: "${env.DISTRO}"
                    ]
                )
            }
        }
    }
}
```

## Related Documentation

- [Role README](../../roles/deploy_linux_vm/README.md)
- [Role Cheatsheet](../../cheatsheets/roles/deploy_linux_vm.md)
- [Deployment Runbook](../runbooks/deployment.md)
- [System Info Role](./system_info.md)
- [CLAUDE.md Guidelines](../../CLAUDE.md)

## Version History

- **v1.0.0** (2025-11-10): Initial production release
  - Multi-distribution support (Debian, Ubuntu, RHEL, AlmaLinux, Rocky, openSUSE)
  - LVM configuration with CLAUDE.md compliance
  - SSH hardening with GSSAPI disabled
  - SELinux/AppArmor enforcement
  - Automatic security updates
  - Comprehensive testing and validation

## License

MIT

## Author Information

Created and maintained by the Ansible Infrastructure Team.

For issues, questions, or contributions, please refer to the project repository.

---

**Document Version**: 1.0.0
**Last Updated**: 2025-11-11
**Maintained By**: Ansible Infrastructure Team