infra-automation/docs/troubleshooting.md

Troubleshooting Guide

Overview

Comprehensive troubleshooting guide for common issues encountered in Ansible-managed infrastructure.

Last Updated: 2025-11-11 Document Owner: Operations Team

---

Table of Contents

1. Ansible Execution Issues
2. SSH and Connectivity
3. VM Deployment Issues
4. System Information Collection
5. Storage and LVM Issues
6. Security and Firewall
7. Performance Issues
8. General Diagnostics

---

Ansible Execution Issues

Issue: "Failed to connect to the host via SSH"

Symptoms: Cannot connect to target hosts

Causes:

• SSH key not authorized
• Wrong SSH user
• Host unreachable
• SSH service not running

Solutions:

# 1. Test connectivity
ping <target_host>

# 2. Test SSH manually
ssh ansible@<target_host>

# 3. Verify SSH service on target
ansible <target_host> -m shell -a "systemctl status sshd" -u root --ask-pass

# 4. Check SSH key is authorized
ansible <target_host> -m authorized_key \
  -a "user=ansible key='{{ lookup('file', '~/.ssh/id_rsa.pub') }}' state=present" \
  -u root --ask-pass

# 5. Verify ansible user exists
ansible <target_host> -m shell -a "id ansible" -u root --ask-pass

Issue: "Permission denied" or "sudo: a password is required"

Symptoms: Tasks fail due to insufficient permissions

Causes:

• ansible user lacks sudo permissions
• become: yes not specified
• Incorrect sudo configuration

Solutions:

# 1. Verify sudo permissions
ssh ansible@<target_host> "sudo -l"

# 2. Check sudoers configuration
ssh ansible@<target_host> "sudo cat /etc/sudoers.d/ansible"

# 3. Fix sudoers if needed (as root)
ssh root@<target_host> "cat > /etc/sudoers.d/ansible <<'EOF'
ansible ALL=(ALL) NOPASSWD: ALL
Defaults:ansible !requiretty
EOF"

# 4. Ensure become is set in playbook
# Add to playbook:
# become: yes

Issue: "Module not found" or "No module named..."

Symptoms: Python module import errors

Causes:

• Python dependencies missing on control node or target
• Wrong Python interpreter

Solutions:

# On control node
pip3 install -r requirements.txt

# On target hosts
ansible all -m package -a "name=python3,python3-pip state=present" --become

# Specify Python interpreter
ansible all -m setup -a "filter=ansible_python_version" \
  -e "ansible_python_interpreter=/usr/bin/python3"

---

SSH and Connectivity

Issue: "UNREACHABLE!" for all hosts

Symptoms: Cannot reach any hosts in inventory

Causes:

• Network connectivity issues
• DNS resolution failures
• Firewall blocking SSH
• Incorrect inventory configuration

Solutions:

# 1. Verify inventory syntax
ansible-inventory --list -i inventories/production

# 2. Test DNS resolution
ansible all -m shell -a "hostname -f" -i inventories/production

# 3. Test network connectivity
ansible all -m ping -i inventories/production

# 4. Check SSH port accessibility
nmap -p 22 <target_host>

# 5. Verify inventory file paths
ansible all --list-hosts -i inventories/production

Issue: SSH connection hangs or times out

Symptoms: SSH attempts timeout or hang indefinitely

Causes:

• Network latency
• SSH idle timeout
• Firewall dropping connections
• MTU issues

Solutions:

# 1. Increase SSH timeout in ansible.cfg
[defaults]
timeout = 60

# 2. Enable SSH keepalive
[ssh_connection]
ssh_args = -o ServerAliveInterval=60 -o ServerAliveCountMax=3

# 3. Test with verbose SSH
ssh -vvv ansible@<target_host>

# 4. Check MTU issues
ping -M do -s 1472 <target_host>  # Should not fragment

---

VM Deployment Issues

Issue: VM fails to start after creation

Symptoms: VM shows "shut off" immediately after deployment

Causes:

• Insufficient resources on hypervisor
• Cloud-init ISO creation failed
• Invalid VM configuration

Solutions:

# 1. Check hypervisor resources
ansible hypervisor -m shell -a "free -h && df -h /var/lib/libvirt/images"

# 2. Check VM definition
ansible hypervisor -m shell -a "virsh dumpxml <vm_name>"

# 3. View libvirt logs
ansible hypervisor -m shell -a "tail -50 /var/log/libvirt/qemu/<vm_name>.log"

# 4. Start VM manually and check errors
ansible hypervisor -m shell -a "virsh start <vm_name>"

# 5. Check cloud-init ISO exists
ansible hypervisor -m shell -a "ls -lh /var/lib/libvirt/images/<vm_name>-cloud-init.iso"

Issue: Cloud-init fails on first boot

Symptoms: Cannot SSH to VM, cloud-init errors in logs

Causes:

• Cloud-init configuration errors
• Network connectivity issues in VM
• Package installation failures

Solutions:

# 1. Access VM console
ansible hypervisor -m shell -a "virsh console <vm_name>"
# Press Enter, login as root (if console password set)

# 2. Check cloud-init status
ssh ansible@<vm_ip> "cloud-init status --long"

# 3. View cloud-init logs
ssh ansible@<vm_ip> "tail -100 /var/log/cloud-init-output.log"

# 4. Re-run cloud-init modules
ssh ansible@<vm_ip> "sudo cloud-init clean && sudo cloud-init init"

# 5. Verify network connectivity in VM
ssh ansible@<vm_ip> "ping -c 3 8.8.8.8 && nslookup google.com"

Issue: Cannot get VM IP address

Symptoms: virsh domifaddr returns no IP

Causes:

• VM networking not configured
• DHCP not working
• VM not fully booted

Solutions:

# 1. Wait for VM to boot completely
sleep 60

# 2. Check all network sources
ansible hypervisor -m shell -a "virsh domifaddr <vm_name> --source lease"
ansible hypervisor -m shell -a "virsh domifaddr <vm_name> --source agent"

# 3. Check DHCP leases
ansible hypervisor -m shell -a "virsh net-dhcp-leases default"

# 4. Check VM network configuration
ansible hypervisor -m shell -a "virsh domif-getlink <vm_name> vnet0"

# 5. Access via console to configure networking
ansible hypervisor -m shell -a "virsh console <vm_name>"

---

System Information Collection

Issue: system_info role fails with "command not found"

Symptoms: Tasks fail due to missing commands (lshw, dmidecode, etc.)

Causes:

• Required packages not installed
• Package installation skipped

Solutions:

# 1. Run installation tasks
ansible-playbook site.yml -t system_info,install

# 2. Manually install packages
ansible all -m package -a "name=lshw,dmidecode,pciutils,usbutils state=present" --become

# 3. Verify packages installed
ansible all -m shell -a "which lshw dmidecode lspci"

Issue: Statistics files not created

Symptoms: No JSON files in ./stats/machines/

Causes:

• Directory permissions issues on control node
• Disk space full
• Export tasks not executed

Solutions:

# 1. Check directory exists and is writable
ls -la ./stats/machines/
mkdir -p ./stats/machines
chmod 755 ./stats/machines

# 2. Check disk space
df -h .

# 3. Run export tasks explicitly
ansible-playbook site.yml -t system_info,export

# 4. Check for errors in Ansible output
ansible-playbook site.yml -t system_info -vvv | tee ansible_debug.log

---

Storage and LVM Issues

Issue: LVM configuration fails on deployed VM

Symptoms: LVM post-deployment tasks fail

Causes:

• Second disk not attached to VM
• LVM tools not installed
• Physical volume creation failed

Solutions:

# 1. Verify second disk exists
ssh ansible@<vm_ip> "lsblk"

# 2. Check for /dev/vdb
ssh ansible@<vm_ip> "ls -l /dev/vdb"

# 3. Verify LVM packages
ssh ansible@<vm_ip> "which pvcreate vgcreate lvcreate"

# 4. Manually create PV if needed
ssh ansible@<vm_ip> "sudo pvcreate /dev/vdb"

# 5. Re-run LVM configuration
ansible-playbook site.yml -t deploy_linux_vm,lvm,post-deploy \
  -e "deploy_linux_vm_name=<vm_name>"

Issue: Disk full on hypervisor

Symptoms: VM deployment fails, "No space left on device"

Causes:

• Insufficient disk space in /var/lib/libvirt/images
• Too many cached cloud images
• Old VM disks not cleaned up

Solutions:

# 1. Check disk space
ansible hypervisor -m shell -a "df -h /var/lib/libvirt/images"

# 2. List all VM disks
ansible hypervisor -m shell -a "ls -lh /var/lib/libvirt/images/*.qcow2"

# 3. Remove old cloud images
ansible hypervisor -m shell -a "find /var/lib/libvirt/images -name '*-cloud*.qcow2' -mtime +90 -delete"

# 4. Remove unused VM disks (CAREFUL!)
# Verify VM is deleted first
ansible hypervisor -m shell -a "virsh list --all"
ansible hypervisor -m shell -a "rm /var/lib/libvirt/images/old-vm-*.qcow2"

# 5. Clean up libvirt storage pools
ansible hypervisor -m shell -a "virsh pool-refresh default && virsh vol-list default"

---

Security and Firewall

Issue: Cannot SSH to VM after deployment

Symptoms: SSH connection refused or times out

Causes:

• Firewall blocking SSH
• SSH service not running
• SSH keys not deployed correctly

Solutions:

# 1. Check if VM is running
ansible hypervisor -m shell -a "virsh list --all"

# 2. Access via hypervisor console
ansible hypervisor -m shell -a "virsh console <vm_name>"

# 3. From console, check sshd status
systemctl status sshd

# 4. Check firewall rules
sudo ufw status  # Debian/Ubuntu
sudo firewall-cmd --list-all  # RHEL/AlmaLinux

# 5. Temporarily allow SSH (for troubleshooting)
sudo ufw allow 22/tcp  # Debian/Ubuntu
sudo firewall-cmd --add-service=ssh --permanent && sudo firewall-cmd --reload  # RHEL

# 6. Verify SSH key authorized
cat ~/.ssh/authorized_keys

Issue: SELinux denials preventing operations

Symptoms: Operations fail with "Permission denied" even with sudo

Causes:

• SELinux blocking operations
• Incorrect file contexts
• Missing SELinux policies

Solutions:

# 1. Check SELinux status
ssh ansible@<host> "getenforce"

# 2. Check for denials
ssh ansible@<host> "sudo ausearch -m avc -ts recent"

# 3. Generate policy from denials
ssh ansible@<host> "sudo ausearch -m avc -ts recent | audit2allow -M mypolicy"
ssh ansible@<host> "sudo semodule -i mypolicy.pp"

# 4. Fix file contexts
ssh ansible@<host> "sudo restorecon -Rv /<path>"

# 5. Temporarily set to permissive for testing (NOT PRODUCTION)
ssh ansible@<host> "sudo setenforce 0"
# After testing, re-enable
ssh ansible@<host> "sudo setenforce 1"

---

Performance Issues

Issue: Ansible playbook execution is very slow

Symptoms: Playbooks take excessive time to complete

Causes:

• Fact gathering on many hosts
• Serial execution instead of parallel
• Slow network connections
• Large inventory

Solutions:

# 1. Enable fact caching in ansible.cfg
[defaults]
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_facts
fact_caching_timeout = 3600

# 2. Increase parallelism
ansible-playbook site.yml -f 20

# 3. Skip fact gathering if not needed
ansible-playbook site.yml -e "gather_facts=false"

# 4. Use strategy plugin
[defaults]
strategy = free  # In ansible.cfg

# 5. Enable pipelining
[ssh_connection]
pipelining = True

# 6. Profile task execution
ansible-playbook site.yml --timing

Issue: High CPU usage on hypervisor

Symptoms: Hypervisor CPU at 100%, VMs slow

Causes:

• CPU overcommitment
• Runaway processes in VMs
• Insufficient resources

Solutions:

# 1. Check hypervisor load
ansible hypervisor -m shell -a "top -bn1 | head -20"

# 2. Check VM CPU allocation
ansible hypervisor -m shell -a "virsh vcpuinfo <vm_name>"

# 3. List VMs by CPU usage
ansible hypervisor -m shell -a "virsh domstats --cpu-total"

# 4. Inside VMs, check processes
ssh ansible@<vm_ip> "top -bn1 | head -20"

# 5. Reduce VM vCPU allocation if needed
ansible hypervisor -m shell -a "virsh setvcpus <vm_name> <new_count> --config"
ansible hypervisor -m shell -a "virsh shutdown <vm_name> && virsh start <vm_name>"

---

General Diagnostics

Diagnostic Commands

# Ansible inventory
ansible-inventory --list
ansible-inventory --graph

# Connectivity test
ansible all -m ping

# Gather facts from hosts
ansible all -m setup

# Check disk space across all hosts
ansible all -m shell -a "df -h"

# Check memory across all hosts
ansible all -m shell -a "free -h"

# Check system load
ansible all -m shell -a "uptime"

# List running services
ansible all -m shell -a "systemctl list-units --type=service --state=running"

# Check for failed services
ansible all -m shell -a "systemctl --failed"

# Review system logs
ansible all -m shell -a "journalctl -p err -n 50"

Debug Mode

# Verbose output (level 1)
ansible-playbook site.yml -v

# More verbose (level 2 - shows module arguments)
ansible-playbook site.yml -vv

# Very verbose (level 3 - shows connection attempts)
ansible-playbook site.yml -vvv

# Maximum verbosity (level 4 - shows everything)
ansible-playbook site.yml -vvvv

Log Locations

Control Node:

• Ansible log: /var/log/ansible.log (if configured)
• Command history: ~/.bash_history

Target Hosts:

• System logs: /var/log/syslog (Debian) or /var/log/messages (RHEL)
• Auth logs: /var/log/auth.log (Debian) or /var/log/secure (RHEL)
• Audit logs: /var/log/audit/audit.log
• Cloud-init: /var/log/cloud-init-output.log
• Journal: journalctl

---

Getting Help

Internal Resources

• CLAUDE.md Guidelines
• Architecture Overview
• Role Documentation
• Cheatsheets

External Resources

• Ansible Documentation
• KVM/libvirt Documentation
• Distribution-specific documentation

Support Channels

• Internal issue tracker: https://git.mymx.me
• Operations team: ops@example.com
• On-call escalation: +1-XXX-XXX-XXXX

---

Document Version: 1.0.0 Last Updated: 2025-11-11 Maintained By: Operations Team