infra-automation/docs/debian12-vm-deployment.md

Debian 12 VM Deployment Documentation

Overview

This document describes the automated deployment process for Debian 12 virtual machines on the grokbox KVM/libvirt hypervisor. The deployment uses cloud-init for unattended configuration and follows the security-first principles outlined in CLAUDE.md.

Table of Contents

1. Architecture
2. Prerequisites
3. Deployment Process
4. Configuration
5. Security Features
6. Post-Deployment
7. Troubleshooting
8. Maintenance

Architecture

Infrastructure Components

┌─────────────────────────────────────────────┐
│ grokbox (KVM Hypervisor)                    │
│                                             │
│  ┌──────────────────────────────────────┐  │
│  │ libvirt/QEMU                         │  │
│  │                                      │  │
│  │  ┌────────────────────────────────┐ │  │
│  │  │ Debian 12 Guest VM             │ │  │
│  │  │                                │ │  │
│  │  │ - 2 vCPUs / 2GB RAM           │ │  │
│  │  │ - 20GB qcow2 disk             │ │  │
│  │  │ - cloud-init configured       │ │  │
│  │  │ - ansible user ready          │ │  │
│  │  │ - Security hardened           │ │  │
│  │  └────────────────────────────────┘ │  │
│  │                                      │  │
│  │  Network: virbr0 (192.168.122.0/24) │  │
│  └──────────────────────────────────────┘  │
└─────────────────────────────────────────────┘

Deployment Workflow

[Ansible Control Node]
         │
         │ 1. SSH to grokbox
         ▼
   [grokbox hypervisor]
         │
         │ 2. Download Debian cloud image
         ├─ 3. Verify checksums
         ├─ 4. Create VM disk (qcow2)
         ├─ 5. Generate cloud-init ISO
         ├─ 6. Create VM with virt-install
         │
         │ 7. VM boots with cloud-init
         ▼
   [Debian 12 VM]
         │
         ├─ 8. Create ansible user
         ├─ 9. Configure SSH
         ├─ 10. Install packages
         ├─ 11. Security hardening
         └─ 12. System ready

Prerequisites

Hypervisor Requirements

On grokbox, ensure the following are present:

1. Virtualization Support

   # Verify CPU virtualization
   egrep -c '(vmx|svm)' /proc/cpuinfo  # Should be > 0
   
   # Verify KVM module loaded
   lsmod | grep kvm
   

2. Required Packages

   • libvirt-daemon-system
   • libvirt-clients
   • virtinst
   • qemu-kvm
   • qemu-utils
   • cloud-image-utils
   • genisoimage
   • python3-libvirt

3. Sufficient Resources

   • Storage: ~25GB available in /var/lib/libvirt/images/
   • Memory: Enough free RAM for VM allocation
   • Network: libvirt default network configured

4. libvirtd Service Running

   systemctl status libvirtd
   

Ansible Control Node Requirements

1. Ansible 2.9 or newer
2. SSH access to grokbox hypervisor
3. SSH key configured for grok user
4. Python 3.x installed

Network Requirements

• Connectivity to Debian cloud image repository
• DNS resolution working
• Default libvirt network (virbr0) configured and active

Deployment Process

Step 1: Pre-flight Checks

The playbook performs the following validations:

• VM Name Uniqueness: Ensures no VM with the same name exists
• Virtualization Support: Validates QEMU/KVM capabilities
• Package Installation: Installs required tools if missing
• Service Status: Verifies libvirtd is running

Step 2: Image Management

Download Debian Cloud Image

• Source: https://cloud.debian.org/images/cloud/bookworm/latest/
• Image: debian-12-generic-amd64.qcow2
• Cache Location: /var/lib/libvirt/images/debian-12-generic-amd64.qcow2
• Checksum Verification: SHA512SUMS validated

The base image is downloaded once and cached for subsequent deployments.

Create VM Disk

A new copy-on-write (CoW) disk is created using qemu-img:

qemu-img create -f qcow2 \
  -F qcow2 \
  -b /var/lib/libvirt/images/debian-12-generic-amd64.qcow2 \
  /var/lib/libvirt/images/debian12-guest.qcow2 \
  20G

This creates a thin-provisioned disk backed by the cloud image.

Step 3: Cloud-Init Configuration

Two configuration files are generated:

meta-data

instance-id: debian12-guest
local-hostname: debian12

user-data

Comprehensive cloud-init configuration including:

• User Management: Creates ansible user with SSH keys
• Security Configuration: SSH hardening, firewall setup
• Package Installation: Essential and security packages
• System Configuration: Time sync, locale, timezone
• Automatic Updates: Unattended security upgrades

ISO Generation

The configuration files are packaged into a bootable ISO:

genisoimage -output debian12-guest-cloud-init.iso \
  -volid cidata -joliet -rock \
  user-data meta-data

Step 4: VM Creation

VM is created using virt-install:

virt-install \
  --name debian12-guest \
  --memory 2048 \
  --vcpus 2 \
  --disk path=/var/lib/libvirt/images/debian12-guest.qcow2,format=qcow2,bus=virtio \
  --disk path=/var/lib/libvirt/images/debian12-guest-cloud-init.iso,device=cdrom \
  --network network=default,model=virtio \
  --os-variant debian11 \
  --graphics none \
  --console pty,target_type=serial \
  --import \
  --noautoconsole

Step 5: Boot and Initialization

1. VM Boots: Starts from the qcow2 disk
2. Cloud-Init Runs: Reads configuration from ISO
3. System Configuration: Applies all settings
4. Network Configuration: Obtains IP via DHCP
5. Package Updates: Downloads and installs updates
6. Service Initialization: Starts all configured services

Typical boot time: 60-90 seconds

Step 6: Validation

The playbook validates:

• VM is running and accessible
• IP address assigned
• SSH port (22) accepting connections
• cloud-init completed successfully
• System resources available

Step 7: Post-Deployment Configuration

Optional second play that:

• Waits for cloud-init completion
• Gathers system facts
• Displays system information
• Validates disk and memory usage

Configuration

Default Configuration

# VM Specifications
vm_name: "debian12-guest"
vm_hostname: "debian12"
vm_domain: "localdomain"
vm_vcpus: 2
vm_memory_mb: 2048
vm_disk_size_gb: 20

# Network
vm_network: "default"
vm_bridge: "virbr0"

# Storage
vm_disk_path: "/var/lib/libvirt/images/{{ vm_name }}.qcow2"
cloud_init_iso_path: "/var/lib/libvirt/images/{{ vm_name }}-cloud-init.iso"

Customization Examples

High-Performance VM

ansible-playbook plays/deploy-debian12-vm.yml \
  -e "vm_name=app-server" \
  -e "vm_vcpus=8" \
  -e "vm_memory_mb=16384" \
  -e "vm_disk_size_gb=100"

Development VM

ansible-playbook plays/deploy-debian12-vm.yml \
  -e "vm_name=dev-workstation" \
  -e "vm_vcpus=4" \
  -e "vm_memory_mb=8192" \
  -e "vm_disk_size_gb=50" \
  -e "vm_hostname=devbox" \
  -e "vm_domain=dev.local"

Custom SSH Key

ansible-playbook plays/deploy-debian12-vm.yml \
  -e "vm_name=secure-vm" \
  -e "ansible_user_ssh_key='ssh-ed25519 AAAA...'"

Variable Precedence

Variables can be set in order of precedence:

1. Command-line (-e flag) - Highest
2. Playbook vars section
3. Inventory host_vars
4. Inventory group_vars
5. Defaults in playbook - Lowest

Security Features

User Management

• ansible user: Non-root service account

  • Passwordless sudo access
  • SSH key authentication only
  • Member of sudo group
  • Home directory: /home/ansible

• root user: Console access only

  • SSH login disabled
  • Password set for emergency console access
  • Remote access blocked

SSH Hardening

Configuration in /etc/ssh/sshd_config.d/99-security.conf:

PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
MaxAuthTries 3
MaxSessions 10
ClientAliveInterval 300
ClientAliveCountMax 2

Firewall Configuration

• UFW (Uncomplicated Firewall) enabled by default
• Default policy: deny incoming, allow outgoing
• SSH (port 22) allowed
• Additional rules can be added post-deployment

Automatic Security Updates

Unattended-upgrades configured for:

• Automatic installation of security updates
• Daily update checks
• Automatic cleanup of old kernels
• Email notifications (if configured)
• No automatic reboot (requires manual intervention)

Audit and Monitoring

• auditd: System call auditing enabled
• aide: File integrity monitoring installed
• chrony: Time synchronization configured
• Logging: All cloud-init output logged

Compliance Features

Aligned with CLAUDE.md security requirements:

• ✅ Principle of least privilege
• ✅ Encryption in transit (SSH)
• ✅ Key-based authentication
• ✅ Automated security updates
• ✅ System auditing enabled
• ✅ Time synchronization
• ✅ Firewall enabled by default

Post-Deployment

Adding to Inventory

Update your Ansible inventory:

# inventories/development/hosts.yml
kvm_guests:
  children:
    application_servers:
      hosts:
        debian12-guest:
          ansible_host: 192.168.122.X
          ansible_user: ansible
          ansible_ssh_common_args: '-o ProxyJump=grokbox -o StrictHostKeyChecking=accept-new'
          ansible_python_interpreter: /usr/bin/python3
          host_description: "Application Server - Debian 12"
          host_role: application
          host_type: virtual_machine
          hypervisor: grokbox
          vm_vcpus: 2
          vm_memory_mb: 2048
          autostart: true

Initial Access

# Get VM IP address
ssh grokbox "virsh domifaddr debian12-guest"

# SSH to VM via ProxyJump
ssh -J grokbox ansible@192.168.122.X

# Or add to ~/.ssh/config
Host debian12-guest
    HostName 192.168.122.X
    User ansible
    ProxyJump grokbox
    StrictHostKeyChecking accept-new

Configuration Management

Run additional roles or playbooks:

# Example: Configure web server
ansible-playbook -i inventories/development/hosts.yml \
  playbooks/configure-webserver.yml \
  -l debian12-guest

# Example: Security hardening
ansible-playbook -i inventories/development/hosts.yml \
  playbooks/security-hardening.yml \
  -l debian12-guest

VM Management Commands

# Start VM
virsh start debian12-guest

# Shutdown VM gracefully
virsh shutdown debian12-guest

# Force shutdown
virsh destroy debian12-guest

# Reboot VM
virsh reboot debian12-guest

# Enable autostart
virsh autostart debian12-guest

# Disable autostart
virsh autostart debian12-guest --disable

# VM status
virsh dominfo debian12-guest

# VM resource usage
virsh domstats debian12-guest

# Console access
virsh console debian12-guest

Troubleshooting

Common Issues

1. VM Already Exists

Error: VM with name already exists

Solution:

# Check existing VMs
virsh list --all

# Remove existing VM
virsh destroy debian12-guest  # if running
virsh undefine debian12-guest --remove-all-storage

2. Image Download Fails

Error: Failed to download cloud image

Causes:

• Network connectivity issues
• Proxy configuration
• DNS resolution problems

Solution:

# Test connectivity
curl -I https://cloud.debian.org

# Manual download
cd /var/lib/libvirt/images
wget https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-generic-amd64.qcow2

# Re-run playbook
ansible-playbook plays/deploy-debian12-vm.yml -t deploy

3. VM Won't Get IP Address

Error: IP address not assigned after 10 retries

Causes:

• DHCP server not running
• Network misconfiguration
• VM network interface issues

Solution:

# Check libvirt network
virsh net-list --all
virsh net-info default
virsh net-start default  # if not started

# Check VM network interface
virsh domiflist debian12-guest

# Check DHCP leases
virsh net-dhcp-leases default

# Access console to troubleshoot
virsh console debian12-guest
# Check: ip addr, systemctl status networking

4. SSH Connection Failed

Error: SSH connection timeout or refused

Causes:

• SSH service not started
• Firewall blocking
• Wrong IP address
• cloud-init not completed

Solution:

# Verify VM is running
virsh list

# Check cloud-init status via console
virsh console debian12-guest
# Run: cloud-init status --wait

# Check SSH service
# Via console: systemctl status ssh

# Check firewall
# Via console: ufw status

# Verify SSH key
ssh-add -l

5. Insufficient Resources

Error: Failed to allocate memory or storage

Solution:

# Check available resources
free -h
df -h /var/lib/libvirt/images/

# Adjust VM resources
ansible-playbook plays/deploy-debian12-vm.yml \
  -e "vm_memory_mb=1024" \
  -e "vm_disk_size_gb=10"

Debug Mode

Enable verbose logging:

# Ansible verbose mode
ansible-playbook plays/deploy-debian12-vm.yml -vvv

# Check cloud-init logs on VM
virsh console debian12-guest
# Then: tail -f /var/log/cloud-init-output.log

# Check libvirt logs
journalctl -u libvirtd -f

Health Checks

# Verify VM health
virsh dominfo debian12-guest
virsh domstats debian12-guest

# Network connectivity
ping $(virsh domifaddr debian12-guest | grep -oP '(\d{1,3}\.){3}\d{1,3}' | head -1)

# SSH connectivity
ssh -J grokbox ansible@$(virsh domifaddr debian12-guest | grep -oP '(\d{1,3}\.){3}\d{1,3}' | head -1) "echo 'VM is accessible'"

Maintenance

Updating the Base Image

Periodically update the cached Debian cloud image:

# Remove old image
ssh grokbox "rm /var/lib/libvirt/images/debian-12-generic-amd64.qcow2"

# Download latest
ansible-playbook plays/deploy-debian12-vm.yml -t download,verify

VM Snapshots

Create snapshots before major changes:

# Create snapshot
virsh snapshot-create-as debian12-guest \
  snapshot1 \
  "Before application deployment"

# List snapshots
virsh snapshot-list debian12-guest

# Revert to snapshot
virsh snapshot-revert debian12-guest snapshot1

# Delete snapshot
virsh snapshot-delete debian12-guest snapshot1

Backup and Restore

Backup VM

# Stop VM
virsh shutdown debian12-guest

# Backup disk
cp /var/lib/libvirt/images/debian12-guest.qcow2 \
   /backup/debian12-guest-$(date +%Y%m%d).qcow2

# Backup XML config
virsh dumpxml debian12-guest > /backup/debian12-guest.xml

# Start VM
virsh start debian12-guest

Restore VM

# Copy disk back
cp /backup/debian12-guest-20241110.qcow2 \
   /var/lib/libvirt/images/debian12-guest.qcow2

# Define VM from XML
virsh define /backup/debian12-guest.xml

# Start VM
virsh start debian12-guest

Resize VM Disk

# Shutdown VM
virsh shutdown debian12-guest

# Resize disk
qemu-img resize /var/lib/libvirt/images/debian12-guest.qcow2 +10G

# Start VM
virsh start debian12-guest

# On VM: resize partition and filesystem
growpart /dev/vda 1
resize2fs /dev/vda1

Resource Adjustment

Modify VM resources:

# Set maximum memory (requires shutdown)
virsh setmaxmem debian12-guest 4194304 --config

# Set current memory (can be done live)
virsh setmem debian12-guest 4194304

# Set vCPUs (requires shutdown)
virsh setvcpus debian12-guest 4 --config --maximum
virsh setvcpus debian12-guest 4 --config

Best Practices

1. Naming Convention: Use descriptive VM names indicating purpose
2. Resource Planning: Right-size VMs to avoid waste
3. Documentation: Document VM purpose and configuration
4. Monitoring: Set up monitoring for critical VMs
5. Backups: Regular backups of important VMs
6. Updates: Keep VMs updated with security patches
7. Inventory: Maintain accurate Ansible inventory
8. Tags: Use libvirt tags for organization
9. Networking: Use appropriate network isolation
10. Testing: Test deployment process in development first

References

• CLAUDE.md - Infrastructure guidelines
• Cheatsheet - Quick reference
• Debian Cloud Images
• cloud-init Documentation
• libvirt Documentation
• virt-install man page

Support and Contact

For issues or questions:

1. Check troubleshooting section above
2. Review cloud-init logs: /var/log/cloud-init.log
3. Review libvirt logs: journalctl -u libvirtd
4. Consult Ansible playbook: plays/deploy-debian12-vm.yml

---

Document Version: 1.0 Last Updated: 2025-11-10 Maintained By: Ansible Infrastructure Team