docs: update project instructions
This commit is contained in:
Username
331
CLAUDE.md
331
CLAUDE.md
@@ -1,96 +1,295 @@
|
|||||||
# PPF Project Instructions
|
# PPF Project Instructions
|
||||||
|
|
||||||
## Current State
|
## Architecture
|
||||||
|
|
||||||
PPF is a Python 2 proxy scraping and validation framework with:
|
|
||||||
- Multi-target validation (2/3 majority voting)
|
|
||||||
- SSL/TLS proxy testing with MITM detection
|
|
||||||
- Web dashboard with electric cyan theme
|
|
||||||
- Interactive world map (/map endpoint)
|
|
||||||
- Memory profiling (/api/memory endpoint)
|
|
||||||
- Tor connection pooling with health monitoring
|
|
||||||
|
|
||||||
## Deployment to Odin
|
|
||||||
|
|
||||||
When deploying PPF to odin server, follow these specifications exactly:
|
|
||||||
|
|
||||||
### Production Deployment
|
|
||||||
|
|
||||||
```
|
```
|
||||||
Host: odin
|
┌──────────┬─────────────┬────────────────────────────────────────────────────────┐
|
||||||
Container User: podman
|
│ Host │ Role │ Notes
|
||||||
Source Path: /home/podman/ppf/src/
|
├──────────┼─────────────┼────────────────────────────────────────────────────────┤
|
||||||
Data Path: /home/podman/ppf/data/
|
│ odin │ Master │ Scrapes proxy lists, verifies conflicts, port 8081
|
||||||
Config: /home/podman/ppf/config.ini
|
│ forge │ Worker │ Tests proxies, reports to master via WireGuard
|
||||||
HTTP Port: 8081
|
│ hermes │ Worker │ Tests proxies, reports to master via WireGuard
|
||||||
Container: ppf (localhost/ppf:latest)
|
│ janus │ Worker │ Tests proxies, reports to master via WireGuard
|
||||||
|
└──────────┴─────────────┴────────────────────────────────────────────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
### Deployment Steps
|
### Role Separation
|
||||||
|
|
||||||
1. **Validate syntax** - Run `python3 -m py_compile` on all .py files
|
- **Odin (Master)**: Scrapes proxy sources, does verification tests only. No routine testing. Local Tor only.
|
||||||
2. **Sync to staging** - rsync Python files to `odin:/tmp/ppf-update/`
|
- **Workers**: All routine proxy testing. Each uses only local Tor (127.0.0.1:9050).
|
||||||
3. **Copy to container source** - `sudo -u podman cp /tmp/ppf-update/*.py /home/podman/ppf/src/`
|
|
||||||
4. **Restart container** - `sudo -u podman podman restart ppf`
|
|
||||||
5. **Clean staging** - Remove `/tmp/ppf-update/`
|
|
||||||
6. **Verify** - Check container logs for successful startup
|
|
||||||
|
|
||||||
### Test Pod (Development)
|
## CRITICAL: Directory Structure Differences
|
||||||
|
|
||||||
If running a test container, use:
|
```
|
||||||
- Different port: 8082
|
┌──────────┬─────────────────────────┬──────────────────────────────────────────┐
|
||||||
- Different container name: ppf-test
|
│ Host │ Code Location │ Container Mount
|
||||||
- Separate data directory: /home/podman/ppf-test/data/
|
├──────────┼─────────────────────────┼──────────────────────────────────────────┤
|
||||||
|
│ odin │ /home/podman/ppf/*.py │ Mounts ppf/ directly to /app
|
||||||
|
│ workers │ /home/podman/ppf/src/ │ Mounts ppf/src/ to /app (via systemd)
|
||||||
|
└──────────┴─────────────────────────┴──────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
**ODIN uses root ppf/ directory. WORKERS use ppf/src/ subdirectory.**
|
||||||
|
|
||||||
|
## Host Access
|
||||||
|
|
||||||
|
**ALWAYS use Ansible from `/opt/ansible` with venv activated:**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Test container example
|
cd /opt/ansible && source venv/bin/activate
|
||||||
sudo -u podman podman run -d --name ppf-test \
|
|
||||||
-p 8082:8081 \
|
|
||||||
-v /home/podman/ppf/src:/app/src:ro \
|
|
||||||
-v /home/podman/ppf-test/data:/app/data \
|
|
||||||
localhost/ppf:latest
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Files to Deploy
|
### Quick Reference Commands
|
||||||
|
|
||||||
All Python files in project root:
|
```bash
|
||||||
- proxywatchd.py (main daemon)
|
# Check worker status
|
||||||
- httpd.py (web dashboard)
|
ANSIBLE_REMOTE_TMP=/tmp/.ansible ansible forge,hermes,janus -m shell -a "hostname"
|
||||||
- config.py, dbs.py, fetch.py, etc.
|
|
||||||
|
|
||||||
### Do NOT Deploy
|
# Check worker config
|
||||||
|
ANSIBLE_REMOTE_TMP=/tmp/.ansible ansible forge,hermes,janus -m shell -a "grep -E 'threads|timeout|ssl' /home/podman/ppf/config.ini"
|
||||||
|
|
||||||
- config.ini (server-specific)
|
# Check worker logs
|
||||||
- data/ directory contents
|
ANSIBLE_REMOTE_TMP=/tmp/.ansible ansible forge -m shell -a "sudo -u podman journalctl --user -u ppf-worker -n 20"
|
||||||
- *.sqlite files
|
|
||||||
|
# Modify config option
|
||||||
|
ANSIBLE_REMOTE_TMP=/tmp/.ansible ansible forge,hermes,janus -m lineinfile -a "path=/home/podman/ppf/config.ini line='ssl_only = 1' insertafter='ssl_first'"
|
||||||
|
|
||||||
|
# Restart workers (different UIDs!)
|
||||||
|
ANSIBLE_REMOTE_TMP=/tmp/.ansible ansible janus,forge -m raw -a "sudo -u podman XDG_RUNTIME_DIR=/run/user/996 systemctl --user restart ppf-worker"
|
||||||
|
ANSIBLE_REMOTE_TMP=/tmp/.ansible ansible hermes -m raw -a "sudo -u podman XDG_RUNTIME_DIR=/run/user/1001 systemctl --user restart ppf-worker"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Full Deployment Procedure
|
||||||
|
|
||||||
|
### Step 1: Validate Syntax Locally
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /home/user/git/ppf
|
||||||
|
for f in *.py; do python3 -m py_compile "$f" && echo "OK: $f"; done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2: Deploy to ALL Hosts
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /opt/ansible && source venv/bin/activate
|
||||||
|
|
||||||
|
# Deploy to ODIN (root ppf/ directory)
|
||||||
|
ANSIBLE_REMOTE_TMP=/tmp/.ansible ansible odin -m synchronize \
|
||||||
|
-a "src=/home/user/git/ppf/ dest=/home/podman/ppf/ rsync_opts='--include=*.py,--include=servers.txt,--exclude=*'"
|
||||||
|
|
||||||
|
# Deploy to WORKERS (ppf/src/ subdirectory)
|
||||||
|
ANSIBLE_REMOTE_TMP=/tmp/.ansible ansible forge,hermes,janus -m synchronize \
|
||||||
|
-a "src=/home/user/git/ppf/ dest=/home/podman/ppf/src/ rsync_opts='--include=*.py,--include=servers.txt,--exclude=*'"
|
||||||
|
|
||||||
|
# CRITICAL: Fix ownership on ALL hosts (rsync uses ansible user, containers need podman)
|
||||||
|
ANSIBLE_REMOTE_TMP=/tmp/.ansible ansible odin,forge,hermes,janus -m raw \
|
||||||
|
-a "chown -R podman:podman /home/podman/ppf/"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Note:** Ownership must be fixed after every deploy. rsync runs as ansible user, but containers run as podman user. Missing ownership fix causes `ImportError: No module named X` errors.
|
||||||
|
|
||||||
|
### Step 3: Restart Services
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Restart ODIN (UID 1005)
|
||||||
|
ansible odin -m raw \
|
||||||
|
-a "cd /tmp && XDG_RUNTIME_DIR=/run/user/1005 runuser -u podman -- podman restart ppf"
|
||||||
|
|
||||||
|
# Restart WORKERS (note different UIDs)
|
||||||
|
ansible janus,forge -m raw \
|
||||||
|
-a "sudo -u podman XDG_RUNTIME_DIR=/run/user/996 systemctl --user restart ppf-worker"
|
||||||
|
ansible hermes -m raw \
|
||||||
|
-a "sudo -u podman XDG_RUNTIME_DIR=/run/user/1001 systemctl --user restart ppf-worker"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 4: Verify All Running
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check odin (UID 1005)
|
||||||
|
ansible odin -m raw \
|
||||||
|
-a "cd /tmp && XDG_RUNTIME_DIR=/run/user/1005 runuser -u podman -- podman ps"
|
||||||
|
|
||||||
|
# Check workers
|
||||||
|
ansible janus,forge -m raw \
|
||||||
|
-a "sudo -u podman XDG_RUNTIME_DIR=/run/user/996 systemctl --user is-active ppf-worker"
|
||||||
|
ansible hermes -m raw \
|
||||||
|
-a "sudo -u podman XDG_RUNTIME_DIR=/run/user/1001 systemctl --user is-active ppf-worker"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Podman User IDs
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────┬───────┬─────────────────────────────┐
|
||||||
|
│ Host │ UID │ XDG_RUNTIME_DIR
|
||||||
|
├──────────┼───────┼─────────────────────────────┤
|
||||||
|
│ odin │ 1005 │ /run/user/1005
|
||||||
|
│ hermes │ 1001 │ /run/user/1001
|
||||||
|
│ janus │ 996 │ /run/user/996
|
||||||
|
│ forge │ 996 │ /run/user/996
|
||||||
|
└──────────┴───────┴─────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Odin config.ini
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[common]
|
||||||
|
tor_hosts = 127.0.0.1:9050 # Local Tor ONLY
|
||||||
|
|
||||||
|
[watchd]
|
||||||
|
threads = 0 # NO routine testing
|
||||||
|
database = data/ppf.sqlite
|
||||||
|
|
||||||
|
[scraper]
|
||||||
|
threads = 10
|
||||||
|
```
|
||||||
|
|
||||||
|
### Worker config.ini
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[common]
|
||||||
|
tor_hosts = 127.0.0.1:9050 # Local Tor ONLY
|
||||||
|
|
||||||
|
[watchd]
|
||||||
|
threads = 35
|
||||||
|
timeout = 9
|
||||||
|
ssl_first = 1 # Try SSL handshake first
|
||||||
|
ssl_only = 0 # Set to 1 to skip secondary check on SSL failure
|
||||||
|
checktype = head # Secondary check type: head, irc, judges
|
||||||
|
```
|
||||||
|
|
||||||
|
### Config Options
|
||||||
|
|
||||||
|
```
|
||||||
|
┌───────────────┬─────────┬────────────────────────────────────────────────────┐
|
||||||
|
│ Option │ Default │ Description
|
||||||
|
├───────────────┼─────────┼────────────────────────────────────────────────────┤
|
||||||
|
│ ssl_first │ 1 │ Try SSL handshake first, fallback to checktype
|
||||||
|
│ ssl_only │ 0 │ Skip secondary check when SSL fails (faster)
|
||||||
|
│ checktype │ head │ Secondary check: head, irc, judges
|
||||||
|
│ threads │ 20 │ Number of test threads
|
||||||
|
│ timeout │ 15 │ Socket timeout in seconds
|
||||||
|
└───────────────┴─────────┴────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Work Distribution
|
||||||
|
|
||||||
|
Fair distribution algorithm (httpd.py):
|
||||||
|
|
||||||
|
```
|
||||||
|
fair_share = (due_proxies / active_workers) * 1.2
|
||||||
|
batch_size = clamp(fair_share, min=100, max=1000)
|
||||||
|
```
|
||||||
|
|
||||||
|
- Master calculates batch size based on queue and active workers
|
||||||
|
- Workers shuffle their batch locally to avoid testing same proxies simultaneously
|
||||||
|
- Claims expire after 5 minutes if not completed
|
||||||
|
|
||||||
|
## Worker systemd Unit
|
||||||
|
|
||||||
|
Located at `/home/podman/.config/systemd/user/ppf-worker.service`:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[Unit]
|
||||||
|
Description=PPF Worker Container
|
||||||
|
After=network-online.target tor.service
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
Restart=on-failure
|
||||||
|
RestartSec=10
|
||||||
|
WorkingDirectory=%h
|
||||||
|
ExecStartPre=-/usr/bin/podman stop -t 10 ppf-worker
|
||||||
|
ExecStartPre=-/usr/bin/podman rm -f ppf-worker
|
||||||
|
ExecStart=/usr/bin/podman run \
|
||||||
|
--name ppf-worker --rm --log-driver=journald --network=host \
|
||||||
|
-v %h/ppf/src:/app:ro \
|
||||||
|
-v %h/ppf/data:/app/data \
|
||||||
|
-v %h/ppf/config.ini:/app/config.ini:ro \
|
||||||
|
-e PYTHONUNBUFFERED=1 \
|
||||||
|
localhost/ppf-worker:latest \
|
||||||
|
python -u ppf.py --worker --server http://10.200.1.250:8081
|
||||||
|
ExecStop=/usr/bin/podman stop -t 10 ppf-worker
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=default.target
|
||||||
|
```
|
||||||
|
|
||||||
|
## Rebuilding Images
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Workers - from ppf/ directory (Dockerfile copies from src/)
|
||||||
|
ansible forge,hermes,janus -m raw \
|
||||||
|
-a "cd /home/podman/ppf && sudo -u podman podman build -t localhost/ppf-worker:latest ."
|
||||||
|
|
||||||
|
# Odin - from ppf/ directory
|
||||||
|
ansible odin -m raw \
|
||||||
|
-a "cd /home/podman/ppf && sudo -u podman podman build -t localhost/ppf:latest ."
|
||||||
|
```
|
||||||
|
|
||||||
## API Endpoints
|
## API Endpoints
|
||||||
|
|
||||||
```
|
```
|
||||||
/dashboard Web UI with live statistics
|
/dashboard Web UI with live statistics
|
||||||
/map Interactive world map (Leaflet.js)
|
/map Interactive world map
|
||||||
/health Health check: {"status": "ok"}
|
/health Health check
|
||||||
/api/stats Runtime statistics (JSON)
|
/api/stats Runtime statistics (JSON)
|
||||||
/api/memory Memory profiling data (JSON)
|
/api/workers Connected worker status
|
||||||
|
/api/memory Memory profiling data
|
||||||
/api/countries Proxy counts by country
|
/api/countries Proxy counts by country
|
||||||
/api/locations Precise proxy locations (requires DB5)
|
/proxies Working proxies list
|
||||||
/proxies Working proxies (limit, proto, country params)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Memory Analysis
|
## Troubleshooting
|
||||||
|
|
||||||
Query production memory state:
|
### Missing servers.txt
|
||||||
|
|
||||||
|
Workers need `servers.txt` in src/:
|
||||||
```bash
|
```bash
|
||||||
ssh odin "curl -s localhost:8081/api/memory" | python3 -m json.tool
|
ansible forge,hermes,janus -m copy \
|
||||||
|
-a "src=/home/user/git/ppf/servers.txt dest=/home/podman/ppf/src/servers.txt owner=podman group=podman"
|
||||||
```
|
```
|
||||||
|
|
||||||
Key metrics:
|
### Exit Code 126 (Permission/Storage)
|
||||||
- `start_rss` / `process.VmRSS` - memory growth
|
|
||||||
- `objgraph_common` - top object types by count
|
|
||||||
- `samples` - RSS history over time
|
|
||||||
- `gc.objects` - total GC-tracked objects
|
|
||||||
|
|
||||||
Current baseline (~260k queue):
|
```bash
|
||||||
- Start: 442 MB
|
sudo -u podman podman system reset --force
|
||||||
- Running: 1.6 GB
|
# Then rebuild image
|
||||||
- Per-job overhead: ~4.5 KB
|
```
|
||||||
|
|
||||||
|
### Dashboard Shows NaN or Missing Data
|
||||||
|
|
||||||
|
Odin likely running old code. Redeploy to odin:
|
||||||
|
```bash
|
||||||
|
ansible odin -m synchronize \
|
||||||
|
-a "src=/home/user/git/ppf/ dest=/home/podman/ppf/ rsync_opts='--include=*.py,--include=servers.txt,--exclude=*'"
|
||||||
|
ansible odin -m raw -a "chown -R podman:podman /home/podman/ppf/"
|
||||||
|
ansible odin -m raw -a "cd /tmp; sudo -u podman podman restart ppf"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Worker Keeps Crashing
|
||||||
|
|
||||||
|
1. Check systemd status with correct UID
|
||||||
|
2. Verify servers.txt exists in src/
|
||||||
|
3. Check ownership
|
||||||
|
4. Run manually to see error:
|
||||||
|
```bash
|
||||||
|
sudo -u podman podman run --rm --network=host \
|
||||||
|
-v /home/podman/ppf/src:/app:ro \
|
||||||
|
-v /home/podman/ppf/data:/app/data \
|
||||||
|
-v /home/podman/ppf/config.ini:/app/config.ini:ro \
|
||||||
|
localhost/ppf-worker:latest \
|
||||||
|
python -u ppf.py --worker --server http://10.200.1.250:8081
|
||||||
|
```
|
||||||
|
|
||||||
|
## Files to Deploy
|
||||||
|
|
||||||
|
- All *.py files
|
||||||
|
- servers.txt
|
||||||
|
|
||||||
|
## Do NOT Deploy
|
||||||
|
|
||||||
|
- config.ini (server-specific)
|
||||||
|
- data/ contents
|
||||||
|
- *.sqlite files
|
||||||
|
|||||||
Reference in New Issue
Block a user