username/ppf
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
8208670fc1ed62a12fa1933de22f6097acb59de9
ppf/CLAUDE.md
Username 12f6b1d8eb docs: update CLAUDE.md for operations toolkit 
Replace verbose ansible deployment commands with ppf-deploy,
ppf-logs, and ppf-service references. Keep raw ansible only
for ad-hoc config operations not covered by tools.
2026-02-17 22:52:54 +01:00

264 lines
9.5 KiB
Markdown
Raw Blame History

# PPF Project Instructions
## Architecture
```
┌──────────┬─────────────┬────────────────────────────────────────────────────────┐
│ Host │ Role │ Notes
├──────────┼─────────────┼────────────────────────────────────────────────────────┤
│ odin │ Master │ Scrapes proxy lists, verifies conflicts, port 8081
│ cassius │ Worker │ Tests proxies, reports to master via WireGuard
│ edge │ Worker │ Tests proxies, reports to master via WireGuard
│ sentinel │ Worker │ Tests proxies, reports to master via WireGuard
└──────────┴─────────────┴────────────────────────────────────────────────────────┘
```
### Role Separation
- **Odin (Master)**: Scrapes proxy sources, does verification tests only. No routine testing. Local Tor only.
- **Workers**: All routine proxy testing. Each uses only local Tor (127.0.0.1:9050).
## CRITICAL: Directory Structure Differences
```
┌──────────┬─────────────────────────┬──────────────────────────────────────────┐
│ Host │ Code Location │ Container Mount
├──────────┼─────────────────────────┼──────────────────────────────────────────┤
│ odin │ /home/podman/ppf/*.py │ Mounts ppf/ directly to /app
│ workers │ /home/podman/ppf/src/ │ Mounts ppf/src/ to /app (via systemd)
└──────────┴─────────────────────────┴──────────────────────────────────────────┘
```
**ODIN uses root ppf/ directory. WORKERS use ppf/src/ subdirectory.**
## Operations Toolkit
All deployment and service management is handled by `tools/`:
```
tools/
lib/ppf-common.sh shared library (hosts, wrappers, colors)
ppf-deploy deploy code to nodes
ppf-logs view container logs
ppf-service manage containers (status/start/stop/restart)
```
Symlinked to `~/.local/bin/` for direct use.
### Deployment
```bash
ppf-deploy # all nodes: validate, sync, restart
ppf-deploy odin # master only
ppf-deploy workers # cassius, edge, sentinel
ppf-deploy cassius edge # specific hosts
ppf-deploy --no-restart # sync only, skip restart
```
Steps performed automatically:
1. Validate Python syntax locally
2. Rsync `*.py` + `servers.txt` (root for odin, `src/` for workers)
3. Copy compose file per role (`compose.master.yml` / `compose.worker.yml`)
4. Fix ownership (`chown -R podman:podman`)
5. Restart containers and show status
### Container Logs
```bash
ppf-logs # last 40 lines from odin
ppf-logs cassius # specific worker
ppf-logs -f edge # follow mode
ppf-logs -n 100 sentinel # last N lines
```
### Service Management
```bash
ppf-service status # all nodes: compose ps + health
ppf-service status workers # workers only
ppf-service restart odin # restart master
ppf-service stop cassius # stop specific worker
ppf-service start workers # start all workers
```
### Direct Ansible (for operations not covered by tools)
Tools use `/opt/ansible` venv and `ANSIBLE_REMOTE_TMP=/tmp/.ansible`
internally. For ad-hoc operations:
```bash
cd /opt/ansible && source venv/bin/activate
# Check worker config
ANSIBLE_REMOTE_TMP=/tmp/.ansible ansible cassius,edge,sentinel \
-m shell -a "grep -E 'threads|timeout|ssl' /home/podman/ppf/config.ini"
# Modify config option
ANSIBLE_REMOTE_TMP=/tmp/.ansible ansible cassius,edge,sentinel \
-m lineinfile -a "path=/home/podman/ppf/config.ini line='ssl_only = 1' insertafter='ssl_first'"
```
## Podman User IDs
```
┌──────────┬───────┬─────────────────────────────┐
│ Host │ UID │ XDG_RUNTIME_DIR
├──────────┼───────┼─────────────────────────────┤
│ odin │ 1005 │ /run/user/1005
│ cassius │ 993 │ /run/user/993
│ edge │ 993 │ /run/user/993
│ sentinel │ 992 │ /run/user/992
└──────────┴───────┴─────────────────────────────┘
```
**Prefer dynamic UID discovery** (`uid=$(id -u podman)`) over hardcoded values.
## Configuration
### Odin config.ini
```ini
[common]
tor_hosts = 127.0.0.1:9050 # Local Tor ONLY
[watchd]
threads = 0 # NO routine testing
database = data/ppf.sqlite
[scraper]
threads = 10
```
### Worker config.ini
```ini
[common]
tor_hosts = 127.0.0.1:9050 # Local Tor ONLY
[watchd]
threads = 35
timeout = 9
ssl_first = 1 # Try SSL handshake first
ssl_only = 0 # Set to 1 to skip secondary check on SSL failure
checktype = head # Secondary check: head, irc, judges, none (SSL-only)
```
### Config Options
```
┌───────────────┬─────────┬────────────────────────────────────────────────────┐
│ Option │ Default │ Description
├───────────────┼─────────┼────────────────────────────────────────────────────┤
│ ssl_first │ 1 │ Try SSL handshake first, fallback to checktype
│ ssl_only │ 0 │ Skip secondary check when SSL fails (faster)
│ checktype │ head │ Secondary check: head, irc, judges, none/false
│ threads │ 20 │ Number of test threads
│ timeout │ 15 │ Socket timeout in seconds
└───────────────┴─────────┴────────────────────────────────────────────────────┘
```
## Work Distribution
Fair distribution algorithm (httpd.py):
```
fair_share = (due_proxies / active_workers) * 1.2
batch_size = clamp(fair_share, min=100, max=1000)
```
- Master calculates batch size based on queue and active workers
- Workers shuffle their batch locally to avoid testing same proxies simultaneously
- Claims expire after 5 minutes if not completed
## Worker Container
Workers run as podman containers with `--restart=unless-stopped`:
```bash
podman run -d --name ppf-worker --network=host --restart=unless-stopped \
-e PYTHONUNBUFFERED=1 \
-v /home/podman/ppf/src:/app:ro,Z \
-v /home/podman/ppf/data:/app/data:Z \
-v /home/podman/ppf/config.ini:/app/config.ini:ro,Z \
-v /home/podman/ppf/servers.txt:/app/servers.txt:ro,Z \
localhost/ppf-worker:latest \
python -u ppf.py --worker --server http://10.200.1.250:8081
```
## Rebuilding Images
```bash
# Workers - from ppf/ directory (Dockerfile copies from src/)
ansible cassius,edge,sentinel -m raw \
-a "cd /home/podman/ppf && sudo -u podman podman build -t localhost/ppf-worker:latest ."
# Odin - from ppf/ directory
ansible odin -m raw \
-a "cd /home/podman/ppf && sudo -u podman podman build -t localhost/ppf:latest ."
```
## API Endpoints
```
/dashboard Web UI with live statistics
/map Interactive world map
/health Health check
/api/stats Runtime statistics (JSON)
/api/workers Connected worker status
/api/countries Proxy counts by country
/api/claim-urls Claim URL batch for worker-driven fetching (GET)
/api/report-urls Report URL fetch results (POST)
/api/report-proxies Report working proxies (POST)
/proxies Working proxies list
```
## Troubleshooting
### Missing servers.txt
Redeploy syncs `servers.txt` automatically:
```bash
ppf-deploy workers
```
### Exit Code 126 (Permission/Storage)
```bash
sudo -u podman podman system reset --force
# Then rebuild image
```
### Dashboard Shows NaN or Missing Data
Odin likely running old code:
```bash
ppf-deploy odin
```
### Worker Keeps Crashing
1. Check status: `ppf-service status workers`
2. Check logs: `ppf-logs -n 50 cassius`
3. Redeploy (fixes ownership + servers.txt): `ppf-deploy cassius`
4. If still failing, run manually on the host to see error:
```bash
sudo -u podman podman run --rm --network=host \
-v /home/podman/ppf/src:/app:ro,Z \
-v /home/podman/ppf/data:/app/data:Z \
-v /home/podman/ppf/config.ini:/app/config.ini:ro,Z \
-v /home/podman/ppf/servers.txt:/app/servers.txt:ro,Z \
localhost/ppf-worker:latest \
python -u ppf.py --worker --server http://10.200.1.250:8081
```
## Files to Deploy
- All *.py files
- servers.txt
## Do NOT Deploy
- config.ini (server-specific)
- data/ contents
- *.sqlite files
Reference in New Issue View Git Blame Copy Permalink