infra-automation/cheatsheets/playbooks/disaster_recovery.md

Disaster Recovery Playbook Cheatsheet

Quick reference for using the disaster recovery playbook.

⚠️ WARNING

This playbook performs DESTRUCTIVE OPERATIONS. Only use when recovering from a disaster or system failure.

Quick Start

# Assess damage only (safe)
ansible-playbook playbooks/disaster_recovery.yml --limit failed_host --tags assess

# Full recovery
ansible-playbook playbooks/disaster_recovery.yml --limit failed_host \
  --extra-vars "dr_backup_date=2025-01-11"

Prerequisites

1. Backups available - Ensure backups exist in /var/backups/
2. System accessible - Host must be reachable via SSH
3. Confirmation ready - You'll need to type "RECOVER" to proceed

Common Usage

Assessment Phase (Safe)

# Assess system damage without making changes
ansible-playbook playbooks/disaster_recovery.yml \
  --limit failed_host \
  --tags assess

# Multiple hosts
ansible-playbook playbooks/disaster_recovery.yml \
  --limit "host1,host2,host3" \
  --tags assess

Configuration Recovery

# Restore configuration files only
ansible-playbook playbooks/disaster_recovery.yml \
  --limit failed_host \
  --tags restore_config \
  --extra-vars "dr_backup_date=2025-01-11"

Data Recovery

# Restore application data only
ansible-playbook playbooks/disaster_recovery.yml \
  --limit failed_host \
  --tags restore_data \
  --extra-vars "dr_backup_date=2025-01-11"

Full Recovery

# Complete system recovery
ansible-playbook playbooks/disaster_recovery.yml \
  --limit failed_host \
  --extra-vars "dr_backup_date=2025-01-11"

Available Tags

Tag	Description	Destructive?
assess	Assess system state	No ✅
prepare	Prepare for recovery	Yes ⚠️
restore_config	Restore configuration	Yes ⚠️
restore_data	Restore data	Yes ⚠️
services	Restart services	No ✅
verify	Verify restoration	No ✅

Extra Variables

Variable	Default	Description
dr_backup_date	latest	Backup date to restore (format: YYYY-MM-DD)
dr_verify_only	false	Assessment mode only (no changes)

Recovery Phases

1. Assessment

ansible-playbook playbooks/disaster_recovery.yml \
  --limit host \
  --tags assess

Checks:

• System accessibility
• Filesystem status
• Service status
• System errors

2. Preparation

ansible-playbook playbooks/disaster_recovery.yml \
  --limit host \
  --tags prepare

Actions:

• Stops non-critical services
• Creates pre-recovery backup
• Syncs filesystems

3. Restoration

ansible-playbook playbooks/disaster_recovery.yml \
  --limit host \
  --tags restore_config,restore_data

Restores:

• System configuration (/etc)
• SSH configuration
• Application data
• Database dumps

4. Service Restart

ansible-playbook playbooks/disaster_recovery.yml \
  --limit host \
  --tags services

Restarts:

• SSH daemon
• Time synchronization
• Auditd
• Firewall

5. Verification

ansible-playbook playbooks/disaster_recovery.yml \
  --limit host \
  --tags verify

Verifies:

• SSH connectivity
• Critical services running
• Filesystem integrity
• NTP synchronization

Recovery Scenarios

Scenario 1: Configuration Corruption

# Restore only configuration files
ansible-playbook playbooks/disaster_recovery.yml \
  --limit webserver01 \
  --tags assess,restore_config,verify \
  --extra-vars "dr_backup_date=2025-01-11"

Scenario 2: Failed System Upgrade

# Full recovery from pre-upgrade backup
ansible-playbook playbooks/disaster_recovery.yml \
  --limit dbserver01 \
  --extra-vars "dr_backup_date=2025-01-10"

Scenario 3: Data Loss

# Restore application data only
ansible-playbook playbooks/disaster_recovery.yml \
  --limit appserver01 \
  --tags restore_data \
  --extra-vars "dr_backup_date=latest"

Scenario 4: Complete System Failure

# 1. Rebuild OS (manual or automated provisioning)
# 2. Ensure SSH access works
# 3. Run full recovery
ansible-playbook playbooks/disaster_recovery.yml \
  --limit new_replacement_host \
  --extra-vars "dr_backup_date=2025-01-11"

Finding Available Backups

# List all available backups for a host
ansible failed_host -m shell -a "ls -lh /var/backups/config/"

# Check backup dates
ansible failed_host -m shell -a "ls /var/backups/*/backup_manifest_*.txt"

# View backup manifest
ansible failed_host -m shell -a "cat /var/backups/backup_manifest_2025-01-11_0230.txt"

Logs and Reports

Recovery logs: ./logs/disaster_recovery/<date>/<hostname>_recovery.log

Example Output

=========================================
!! DISASTER RECOVERY MODE !!
=========================================
Host: webserver01
Environment: production
Timestamp: 2025-01-11T10:00:00Z
Backup Date: 2025-01-11

WARNING: This playbook performs destructive operations!
=========================================

[Pause for confirmation - type 'RECOVER']

=== System Assessment ===
OS: Ubuntu 22.04
Uptime: 2 hours
Filesystems: OK

=== Restoration Status ===
Configuration restored: Yes
Data restored: Yes
Services restarted: Yes

=== Service Status ===
SSH: Running
Firewall: Running
NTP: Synchronized

=== Next Steps ===
1. Verify application-specific services
2. Test application functionality
3. Monitor system logs for errors
4. Update documentation
5. Conduct post-recovery review
=========================================

Troubleshooting

Backup not found

# Check backup location
ansible failed_host -m shell -a "ls -la /var/backups/"

# Restore from remote backup server
ansible failed_host -m synchronize \
  -a "src=/mnt/backup-server/backups/ dest=/var/backups/ mode=pull"

SSH connection lost during recovery

The SSH service restart is designed to maintain connections. If lost:

# Wait 60 seconds for SSH to restart
# Retry connection

ansible failed_host -m ping

Service won't start after recovery

# Check service status
ansible failed_host -m shell -a "systemctl status service_name"

# Check service logs
ansible failed_host -m shell -a "journalctl -u service_name -n 50"

SELinux blocking services

# Relabel SELinux contexts
ansible failed_host -m shell -a "restorecon -R /etc /var"

Post-Recovery Checklist

• Verify all services running
• Test application functionality
• Check disk space
• Review system logs
• Verify backups are current
• Update documentation
• Notify stakeholders
• Conduct lessons learned review

Best Practices

1. Test recovery procedures regularly - Monthly DR drills
2. Document recovery time objectives (RTO) - Know your targets
3. Keep backups off-site - Don't rely on local backups only
4. Verify backup integrity - Test restores before disasters
5. Maintain runbooks - Document specific recovery procedures
6. Practice on staging - Test recovery in non-production first
7. Have communication plan - Know who to notify

Quick Reference Commands

# Assess damage only
ansible-playbook playbooks/disaster_recovery.yml \
  --limit host --tags assess

# Full recovery with latest backup
ansible-playbook playbooks/disaster_recovery.yml \
  --limit host

# Specific backup date
ansible-playbook playbooks/disaster_recovery.yml \
  --limit host \
  --extra-vars "dr_backup_date=2025-01-11"

# Configuration only
ansible-playbook playbooks/disaster_recovery.yml \
  --limit host \
  --tags restore_config

# Verify recovery
ansible-playbook playbooks/disaster_recovery.yml \
  --limit host \
  --tags verify

# Assessment mode (no changes)
ansible-playbook playbooks/disaster_recovery.yml \
  --limit host \
  --extra-vars "dr_verify_only=true"

Emergency Contacts

Keep this information updated:

• Infrastructure Team Lead: [Contact]
• On-Call Engineer: [Contact]
• Backup System Admin: [Contact]
• Management Escalation: [Contact]

See Also

• Disaster Recovery Playbook
• Backup Playbook
• Disaster Recovery Runbook