infra-automation/docs/network-access-patterns.md

# Network Access Patterns and SSH Jump Host Configuration

**Document Version**: 1.0
**Last Updated**: 2025-11-11
**Status**: Production Ready

---

## Overview

This document describes the network access patterns and SSH jump host (ProxyJump/bastion) configuration for accessing infrastructure resources through intermediary hosts.

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                    Control Node (Ansible)                        │
│                  (Your Local Machine)                            │
└────────────────┬────────────────────────────────────────────────┘
                 │
                 │ SSH Direct Access
                 │
         ┌───────▼────────┐
         │   grokbox      │ ◄──── Hypervisor / Jump Host
         │  (Jump Host)   │       grok.home.serneels.xyz
         │   KVM/libvirt  │       SSH Port: 22
         └───────┬────────┘
                 │
                 │ ProxyJump / Bastion
                 │ Internal Network: 192.168.122.0/24
                 │
     ┌───────────┼───────────┬───────────┐
     │           │           │           │
┌────▼────┐ ┌───▼────┐ ┌────▼────┐ ┌───▼────┐
│ pihole  │ │  mymx  │ │  derp   │ │  ...   │
│ .12     │ │  .119  │ │  .99    │ │        │
└─────────┘ └────────┘ └─────────┘ └────────┘
   Guest VMs (Private Network - virbr0)
```

---

## Jump Host Configuration

### What is a Jump Host (Bastion)?

A **jump host** (also called a **bastion host**) is an intermediary server used to access machines on a private network from an external network. In our infrastructure:

- **Jump Host**: `grokbox` (grok.home.serneels.xyz)
- **Target Hosts**: KVM guest VMs on private network (192.168.122.0/24)
- **Access Method**: SSH ProxyJump

### Benefits

1. **Security**: Guest VMs don't need public internet exposure
2. **Centralized Access Control**: All access goes through grokbox
3. **Simplified Network**: Private network behind NAT
4. **Audit Trail**: All connections logged on jump host
5. **Reduced Attack Surface**: Single hardened entry point

---

## Implementation

### 1. Group Variables Configuration

**File**: `inventories/development/group_vars/kvm_guests.yml`

```yaml
# SSH & Connectivity
ansible_user: ansible
ansible_become_password: null  # Passwordless sudo configured

# Connection via ProxyJump through hypervisor
ansible_ssh_common_args: >-
  -o ProxyJump=grokbox
  -o StrictHostKeyChecking=accept-new
  -o ServerAliveInterval=45
  -o ServerAliveCountMax=3
  -o ControlMaster=auto
  -o ControlPersist=600s
```

**SSH Options Explained**:
- `ProxyJump=grokbox`: Route connection through grokbox
- `StrictHostKeyChecking=accept-new`: Auto-accept new host keys (development)
- `ServerAliveInterval=45`: Send keepalive every 45 seconds
- `ServerAliveCountMax=3`: Disconnect after 3 failed keepalives
- `ControlMaster=auto`: Enable SSH connection multiplexing
- `ControlPersist=600s`: Keep master connection for 10 minutes

### 2. Host Inventory Configuration

**File**: `inventories/development/hosts.yml`

```yaml
kvm_guests:
  vars:
    ansible_user: ansible
    ansible_python_interpreter: /usr/bin/python3
    ansible_ssh_common_args: '-o StrictHostKeyChecking=accept-new'
    host_type: virtual_machine
    hypervisor: grokbox
    network: "virbr0 (192.168.122.0/24)"

  children:
    dns_servers:
      hosts:
        pihole:
          ansible_host: 192.168.122.12
          ansible_ssh_common_args: '-o ProxyJump=grokbox -o StrictHostKeyChecking=accept-new'
          host_description: "Pi-hole DNS/DHCP Server"
```

### 3. SSH Config Integration (Optional)

For manual SSH access, add to `~/.ssh/config`:

```ssh-config
# Jump Host
Host grokbox
    HostName grok.home.serneels.xyz
    User grok
    ForwardAgent yes
    ControlMaster auto
    ControlPersist 10m
    ControlPath ~/.ssh/control-%C

# KVM Guest VMs via Jump Host
Host pihole mymx derp
    User ansible
    ProxyJump grokbox
    StrictHostKeyChecking accept-new
    ServerAliveInterval 45
    ServerAliveCountMax 3

# Specific VM configurations
Host pihole
    HostName 192.168.122.12

Host mymx
    HostName 192.168.122.119

Host derp
    HostName 192.168.122.99
```

---

## Usage Examples

### Ansible Ad-Hoc Commands

```bash
# Ping all KVM guest VMs through jump host
ansible kvm_guests -m ping

# Ping specific VM
ansible pihole -m ping

# Run command on all guests
ansible kvm_guests -m shell -a "uptime"

# Gather facts from pihole through jump host
ansible pihole -m setup
```

### Playbook Execution

```bash
# Run system_info playbook on KVM guests
ansible-playbook playbooks/gather_system_info.yml --limit kvm_guests

# Deploy to specific VM through jump host
ansible-playbook playbooks/deploy_vm.yml --limit pihole

# Security audit all guests
ansible-playbook playbooks/security_audit.yml --limit kvm_guests
```

### Manual SSH Access

```bash
# SSH to guest VM (using ~/.ssh/config)
ssh pihole

# SSH with explicit ProxyJump
ssh -J grokbox ansible@192.168.122.12

# SSH with verbose output for debugging
ssh -vvv -J grokbox ansible@192.168.122.12

# Copy files through jump host
scp -o ProxyJump=grokbox file.txt ansible@192.168.122.12:/tmp/
```

---

## Connection Flow

### 1. Standard Ansible Connection
```
Control Node → grokbox → Guest VM
     |             |          |
     └─ SSH ───────┴─ SSH ───┘
   (public)      (private)
```

### 2. Detailed SSH Handshake
```
1. Control Node connects to grokbox
   ssh grok@grok.home.serneels.xyz

2. Control Node requests ProxyJump
   ProxyJump=grokbox initiates second connection

3. grokbox connects to Guest VM
   ssh ansible@192.168.122.12

4. Two-hop SSH tunnel established
   Control Node ←→ grokbox ←→ Guest VM

5. Ansible executes tasks through tunnel
   SFTP transfers, module execution, etc.
```

---

## Troubleshooting

### Issue 1: Connection Timeout

**Symptom**: `Connection timed out` or `Operation timed out`

**Debug**:
```bash
# Test jump host connectivity
ansible grokbox -m ping

# Test with verbose SSH
ansible pihole -m ping -vvv
```

**Solution**:
- Verify jump host (grokbox) is reachable
- Check jump host firewall allows SSH
- Verify guest VM IP address is correct
- Ensure guest VM is running: `ssh grokbox "virsh list --all"`

### Issue 2: Authentication Failed

**Symptom**: `Permission denied (publickey)`

**Debug**:
```bash
# Test SSH key forwarding
ssh -A grokbox "ssh -T ansible@192.168.122.12"

# Check authorized_keys on guest
ssh grokbox "ssh ansible@192.168.122.12 'cat ~/.ssh/authorized_keys'"
```

**Solution**:
- Verify SSH public key is in guest VM's `~/.ssh/authorized_keys`
- Check file permissions: `authorized_keys` (600), `.ssh/` (700)
- Ensure ansible user exists on guest VM
- Verify SSH key format (RSA, ED25519, etc.)

### Issue 3: ProxyJump Not Working

**Symptom**: `ProxyJump not supported` or `Bad configuration option`

**Debug**:
```bash
# Check SSH version (ProxyJump requires OpenSSH 7.3+)
ssh -V

# Test with ProxyCommand instead
ssh -o ProxyCommand="ssh -W %h:%p grokbox" ansible@192.168.122.12
```

**Solution**:
- Update OpenSSH to version 7.3 or higher
- Use `-o ProxyCommand` as fallback for older SSH versions
- Check SSH config syntax

### Issue 4: Connection Multiplexing Issues

**Symptom**: `Control socket connect: Connection refused`

**Debug**:
```bash
# Clear SSH control sockets
rm -rf ~/.ssh/control-*

# Disable ControlMaster temporarily
ansible pihole -m ping -e 'ansible_ssh_common_args=""'
```

**Solution**:
- Remove stale control sockets
- Increase ControlPersist timeout
- Check control socket directory permissions

---

## Security Considerations

### Jump Host Hardening

The jump host (grokbox) must be properly secured:

1. **SSH Hardening** (applied via `deploy_linux_vm` role):
   - Disable root login
   - Key-based authentication only
   - Fail2ban for brute-force protection
   - Rate limiting on SSH port

2. **Firewall Configuration**:
   ```bash
   # Allow SSH from specific IPs only (recommended)
   ufw allow from YOUR_IP to any port 22

   # Or allow SSH globally (less secure)
   ufw allow 22/tcp
   ```

3. **Audit Logging**:
   - Enable `auditd` for SSH session logging
   - Monitor `/var/log/auth.log` for access attempts
   - Set up alerts for failed authentication

4. **Network Segmentation**:
   - Keep guest VM network (virbr0) isolated from public internet
   - Use firewall rules to restrict guest-to-guest communication if needed

### SSH Key Management

1. **Use Strong Key Types**:
   ```bash
   # Generate ED25519 key (recommended)
   ssh-keygen -t ed25519 -C "ansible@infrastructure"

   # Or RSA 4096-bit
   ssh-keygen -t rsa -b 4096 -C "ansible@infrastructure"
   ```

2. **Key Rotation**:
   - Rotate SSH keys every 90-180 days
   - Maintain key inventory
   - Remove old/unused keys from `authorized_keys`

3. **SSH Agent Forwarding** (Use with caution):
   ```yaml
   # In group_vars if needed
   ansible_ssh_extra_args: '-o ForwardAgent=yes'
   ```
   ⚠️ **Warning**: Agent forwarding can expose keys if jump host is compromised

---

## Performance Optimization

### SSH ControlMaster

ControlMaster (already configured) reuses SSH connections:

```yaml
ansible_ssh_common_args: >-
  -o ControlMaster=auto
  -o ControlPersist=600s
```

**Benefits**:
- Reduces connection overhead by 80-90%
- Faster playbook execution
- Lower resource usage on jump host

### Connection Pooling

ansible.cfg settings:

```ini
[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=60s
```

### Parallel Execution

```bash
# Run playbook with increased forks (default: 5)
ansible-playbook site.yml --forks 20

# Or set in ansible.cfg
[defaults]
forks = 20
```

---

## Monitoring and Logging

### Jump Host Access Logs

**View SSH connection logs**:
```bash
# On grokbox (jump host)
sudo tail -f /var/log/auth.log | grep sshd

# Filter for ansible user connections
sudo grep "ansible" /var/log/auth.log | grep "Accepted"
```

### Ansible Logging

Enable Ansible logging in `ansible.cfg`:

```ini
[defaults]
log_path = /var/log/ansible/ansible.log
```

### Connection Metrics

```bash
# Show active SSH connections on jump host
ssh grokbox "ss -tn | grep :22"

# Show Ansible control sockets
ls -lh ~/.ssh/control-*
```

---

## Alternative Access Patterns

### Pattern 1: SSH Config Only

Rely solely on `~/.ssh/config` without Ansible inventory settings:

```ssh-config
# ~/.ssh/config
Host pihole
    HostName 192.168.122.12
    User ansible
    ProxyJump grokbox
```

Then use simple inventory:
```yaml
pihole:
  ansible_host: 192.168.122.12
  # No ansible_ssh_common_args needed
```

### Pattern 2: Dynamic ProxyJump

Set jump host dynamically based on group membership:

```yaml
# group_vars/kvm_guests.yml
ansible_ssh_common_args: "-o ProxyJump={{ hypervisor_host }}"
hypervisor_host: grokbox
```

### Pattern 3: Multi-Hop Jump Hosts

For infrastructure with multiple tiers:

```yaml
ansible_ssh_common_args: >-
  -o ProxyJump=bastion1,bastion2,grokbox
```

Or in SSH config:
```ssh-config
Host target
    ProxyJump bastion1,bastion2,grokbox
```

---

## Testing and Validation

### Test Suite

```bash
# 1. Test jump host connectivity
ansible grokbox -m ping

# 2. Test guest connectivity through jump host
ansible kvm_guests -m ping

# 3. Test command execution
ansible kvm_guests -m shell -a "hostname"

# 4. Test file transfer
ansible pihole -m copy -a "content='test' dest=/tmp/test.txt"

# 5. Test privilege escalation
ansible pihole -m shell -a "id" --become

# 6. Verbose connection test
ansible pihole -m ping -vvv
```

### Validation Checklist

- [ ] Jump host (grokbox) is reachable from control node
- [ ] SSH key authentication works on jump host
- [ ] Jump host can reach guest VMs on private network
- [ ] Guest VMs have ansible user with SSH key configured
- [ ] Guest VMs have passwordless sudo for ansible user
- [ ] ControlMaster connection multiplexing works
- [ ] Playbook execution completes successfully
- [ ] Connection logs show proper ProxyJump usage

---

## References

- **SSH ProxyJump Documentation**: `man ssh_config` (search for ProxyJump)
- **Ansible SSH Connection Plugin**: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/ssh_connection.html
- **OpenSSH Bastion Hosts**: https://www.openssh.com/
- **CLAUDE.md Guidelines**: Section on secure access patterns
- **CIS Benchmark**: SSH hardening recommendations

---

## Change Log

| Version | Date | Changes |
|---------|------|---------|
| 1.0 | 2025-11-11 | Initial documentation for jump host configuration |

---

**Document Maintained By**: Infrastructure Team
**Review Frequency**: Quarterly
**Next Review**: 2025-02-11