infra-automation/docs/network-access-patterns.md

Network Access Patterns and SSH Jump Host Configuration

Document Version: 1.0 Last Updated: 2025-11-11 Status: Production Ready

---

Overview

This document describes the network access patterns and SSH jump host (ProxyJump/bastion) configuration for accessing infrastructure resources through intermediary hosts.

---

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Control Node (Ansible)                        │
│                  (Your Local Machine)                            │
└────────────────┬────────────────────────────────────────────────┘
                 │
                 │ SSH Direct Access
                 │
         ┌───────▼────────┐
         │   grokbox      │ ◄──── Hypervisor / Jump Host
         │  (Jump Host)   │       grok.home.serneels.xyz
         │   KVM/libvirt  │       SSH Port: 22
         └───────┬────────┘
                 │
                 │ ProxyJump / Bastion
                 │ Internal Network: 192.168.122.0/24
                 │
     ┌───────────┼───────────┬───────────┐
     │           │           │           │
┌────▼────┐ ┌───▼────┐ ┌────▼────┐ ┌───▼────┐
│ pihole  │ │  mymx  │ │  derp   │ │  ...   │
│ .12     │ │  .119  │ │  .99    │ │        │
└─────────┘ └────────┘ └─────────┘ └────────┘
   Guest VMs (Private Network - virbr0)

---

Jump Host Configuration

What is a Jump Host (Bastion)?

A jump host (also called a bastion host) is an intermediary server used to access machines on a private network from an external network. In our infrastructure:

• Jump Host: grokbox (grok.home.serneels.xyz)
• Target Hosts: KVM guest VMs on private network (192.168.122.0/24)
• Access Method: SSH ProxyJump

Benefits

1. Security: Guest VMs don't need public internet exposure
2. Centralized Access Control: All access goes through grokbox
3. Simplified Network: Private network behind NAT
4. Audit Trail: All connections logged on jump host
5. Reduced Attack Surface: Single hardened entry point

---

Implementation

1. Group Variables Configuration

File: inventories/development/group_vars/kvm_guests.yml

# SSH & Connectivity
ansible_user: ansible
ansible_become_password: null  # Passwordless sudo configured

# Connection via ProxyJump through hypervisor
ansible_ssh_common_args: >-
  -o ProxyJump=grokbox
  -o StrictHostKeyChecking=accept-new
  -o ServerAliveInterval=45
  -o ServerAliveCountMax=3
  -o ControlMaster=auto
  -o ControlPersist=600s

SSH Options Explained:

• ProxyJump=grokbox: Route connection through grokbox
• StrictHostKeyChecking=accept-new: Auto-accept new host keys (development)
• ServerAliveInterval=45: Send keepalive every 45 seconds
• ServerAliveCountMax=3: Disconnect after 3 failed keepalives
• ControlMaster=auto: Enable SSH connection multiplexing
• ControlPersist=600s: Keep master connection for 10 minutes

2. Host Inventory Configuration

File: inventories/development/hosts.yml

kvm_guests:
  vars:
    ansible_user: ansible
    ansible_python_interpreter: /usr/bin/python3
    ansible_ssh_common_args: '-o StrictHostKeyChecking=accept-new'
    host_type: virtual_machine
    hypervisor: grokbox
    network: "virbr0 (192.168.122.0/24)"

  children:
    dns_servers:
      hosts:
        pihole:
          ansible_host: 192.168.122.12
          ansible_ssh_common_args: '-o ProxyJump=grokbox -o StrictHostKeyChecking=accept-new'
          host_description: "Pi-hole DNS/DHCP Server"

3. SSH Config Integration (Optional)

For manual SSH access, add to ~/.ssh/config:

# Jump Host
Host grokbox
    HostName grok.home.serneels.xyz
    User grok
    ForwardAgent yes
    ControlMaster auto
    ControlPersist 10m
    ControlPath ~/.ssh/control-%C

# KVM Guest VMs via Jump Host
Host pihole mymx derp
    User ansible
    ProxyJump grokbox
    StrictHostKeyChecking accept-new
    ServerAliveInterval 45
    ServerAliveCountMax 3

# Specific VM configurations
Host pihole
    HostName 192.168.122.12

Host mymx
    HostName 192.168.122.119

Host derp
    HostName 192.168.122.99

---

Usage Examples

Ansible Ad-Hoc Commands

# Ping all KVM guest VMs through jump host
ansible kvm_guests -m ping

# Ping specific VM
ansible pihole -m ping

# Run command on all guests
ansible kvm_guests -m shell -a "uptime"

# Gather facts from pihole through jump host
ansible pihole -m setup

Playbook Execution

# Run system_info playbook on KVM guests
ansible-playbook playbooks/gather_system_info.yml --limit kvm_guests

# Deploy to specific VM through jump host
ansible-playbook playbooks/deploy_vm.yml --limit pihole

# Security audit all guests
ansible-playbook playbooks/security_audit.yml --limit kvm_guests

Manual SSH Access

# SSH to guest VM (using ~/.ssh/config)
ssh pihole

# SSH with explicit ProxyJump
ssh -J grokbox ansible@192.168.122.12

# SSH with verbose output for debugging
ssh -vvv -J grokbox ansible@192.168.122.12

# Copy files through jump host
scp -o ProxyJump=grokbox file.txt ansible@192.168.122.12:/tmp/

---

Connection Flow

1. Standard Ansible Connection

Control Node → grokbox → Guest VM
     |             |          |
     └─ SSH ───────┴─ SSH ───┘
   (public)      (private)

2. Detailed SSH Handshake

1. Control Node connects to grokbox
   ssh grok@grok.home.serneels.xyz

2. Control Node requests ProxyJump
   ProxyJump=grokbox initiates second connection

3. grokbox connects to Guest VM
   ssh ansible@192.168.122.12

4. Two-hop SSH tunnel established
   Control Node ←→ grokbox ←→ Guest VM

5. Ansible executes tasks through tunnel
   SFTP transfers, module execution, etc.

---

Troubleshooting

Issue 1: Connection Timeout

Symptom: Connection timed out or Operation timed out

Debug:

# Test jump host connectivity
ansible grokbox -m ping

# Test with verbose SSH
ansible pihole -m ping -vvv

Solution:

• Verify jump host (grokbox) is reachable
• Check jump host firewall allows SSH
• Verify guest VM IP address is correct
• Ensure guest VM is running: ssh grokbox "virsh list --all"

Issue 2: Authentication Failed

Symptom: Permission denied (publickey)

Debug:

# Test SSH key forwarding
ssh -A grokbox "ssh -T ansible@192.168.122.12"

# Check authorized_keys on guest
ssh grokbox "ssh ansible@192.168.122.12 'cat ~/.ssh/authorized_keys'"

Solution:

• Verify SSH public key is in guest VM's ~/.ssh/authorized_keys
• Check file permissions: authorized_keys (600), .ssh/ (700)
• Ensure ansible user exists on guest VM
• Verify SSH key format (RSA, ED25519, etc.)

Issue 3: ProxyJump Not Working

Symptom: ProxyJump not supported or Bad configuration option

Debug:

# Check SSH version (ProxyJump requires OpenSSH 7.3+)
ssh -V

# Test with ProxyCommand instead
ssh -o ProxyCommand="ssh -W %h:%p grokbox" ansible@192.168.122.12

Solution:

• Update OpenSSH to version 7.3 or higher
• Use -o ProxyCommand as fallback for older SSH versions
• Check SSH config syntax

Issue 4: Connection Multiplexing Issues

Symptom: Control socket connect: Connection refused

Debug:

# Clear SSH control sockets
rm -rf ~/.ssh/control-*

# Disable ControlMaster temporarily
ansible pihole -m ping -e 'ansible_ssh_common_args=""'

Solution:

• Remove stale control sockets
• Increase ControlPersist timeout
• Check control socket directory permissions

---

Security Considerations

Jump Host Hardening

The jump host (grokbox) must be properly secured:

1. SSH Hardening (applied via deploy_linux_vm role):

   • Disable root login
   • Key-based authentication only
   • Fail2ban for brute-force protection
   • Rate limiting on SSH port

2. Firewall Configuration:

   # Allow SSH from specific IPs only (recommended)
   ufw allow from YOUR_IP to any port 22
   
   # Or allow SSH globally (less secure)
   ufw allow 22/tcp
   

3. Audit Logging:

   • Enable auditd for SSH session logging
   • Monitor /var/log/auth.log for access attempts
   • Set up alerts for failed authentication

4. Network Segmentation:

   • Keep guest VM network (virbr0) isolated from public internet
   • Use firewall rules to restrict guest-to-guest communication if needed

SSH Key Management

1. Use Strong Key Types:

   # Generate ED25519 key (recommended)
   ssh-keygen -t ed25519 -C "ansible@infrastructure"
   
   # Or RSA 4096-bit
   ssh-keygen -t rsa -b 4096 -C "ansible@infrastructure"
   

2. Key Rotation:

   • Rotate SSH keys every 90-180 days
   • Maintain key inventory
   • Remove old/unused keys from authorized_keys

3. SSH Agent Forwarding (Use with caution):

   # In group_vars if needed
   ansible_ssh_extra_args: '-o ForwardAgent=yes'
   

   ⚠️ Warning: Agent forwarding can expose keys if jump host is compromised

---

Performance Optimization

SSH ControlMaster

ControlMaster (already configured) reuses SSH connections:

ansible_ssh_common_args: >-
  -o ControlMaster=auto
  -o ControlPersist=600s

Benefits:

• Reduces connection overhead by 80-90%
• Faster playbook execution
• Lower resource usage on jump host

Connection Pooling

ansible.cfg settings:

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=60s

Parallel Execution

# Run playbook with increased forks (default: 5)
ansible-playbook site.yml --forks 20

# Or set in ansible.cfg
[defaults]
forks = 20

---

Monitoring and Logging

Jump Host Access Logs

View SSH connection logs:

# On grokbox (jump host)
sudo tail -f /var/log/auth.log | grep sshd

# Filter for ansible user connections
sudo grep "ansible" /var/log/auth.log | grep "Accepted"

Ansible Logging

Enable Ansible logging in ansible.cfg:

[defaults]
log_path = /var/log/ansible/ansible.log

Connection Metrics

# Show active SSH connections on jump host
ssh grokbox "ss -tn | grep :22"

# Show Ansible control sockets
ls -lh ~/.ssh/control-*

---

Alternative Access Patterns

Pattern 1: SSH Config Only

Rely solely on ~/.ssh/config without Ansible inventory settings:

# ~/.ssh/config
Host pihole
    HostName 192.168.122.12
    User ansible
    ProxyJump grokbox

Then use simple inventory:

pihole:
  ansible_host: 192.168.122.12
  # No ansible_ssh_common_args needed

Pattern 2: Dynamic ProxyJump

Set jump host dynamically based on group membership:

# group_vars/kvm_guests.yml
ansible_ssh_common_args: "-o ProxyJump={{ hypervisor_host }}"
hypervisor_host: grokbox

Pattern 3: Multi-Hop Jump Hosts

For infrastructure with multiple tiers:

ansible_ssh_common_args: >-
  -o ProxyJump=bastion1,bastion2,grokbox

Or in SSH config:

Host target
    ProxyJump bastion1,bastion2,grokbox

---

Testing and Validation

Test Suite

# 1. Test jump host connectivity
ansible grokbox -m ping

# 2. Test guest connectivity through jump host
ansible kvm_guests -m ping

# 3. Test command execution
ansible kvm_guests -m shell -a "hostname"

# 4. Test file transfer
ansible pihole -m copy -a "content='test' dest=/tmp/test.txt"

# 5. Test privilege escalation
ansible pihole -m shell -a "id" --become

# 6. Verbose connection test
ansible pihole -m ping -vvv

Validation Checklist

• Jump host (grokbox) is reachable from control node
• SSH key authentication works on jump host
• Jump host can reach guest VMs on private network
• Guest VMs have ansible user with SSH key configured
• Guest VMs have passwordless sudo for ansible user
• ControlMaster connection multiplexing works
• Playbook execution completes successfully
• Connection logs show proper ProxyJump usage

---

References

• SSH ProxyJump Documentation: man ssh_config (search for ProxyJump)
• Ansible SSH Connection Plugin: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/ssh_connection.html
• OpenSSH Bastion Hosts: https://www.openssh.com/
• CLAUDE.md Guidelines: Section on secure access patterns
• CIS Benchmark: SSH hardening recommendations

---

Change Log

Version	Date	Changes
1.0	2025-11-11	Initial documentation for jump host configuration

---

Document Maintained By: Infrastructure Team Review Frequency: Quarterly Next Review: 2025-02-11