PPF Project Instructions

Current State

PPF is a Python 2 proxy scraping and validation framework with:

Multi-target validation (2/3 majority voting)
SSL/TLS proxy testing with MITM detection
Web dashboard with electric cyan theme
Interactive world map (/map endpoint)
Memory profiling (/api/memory endpoint)
Tor connection pooling with health monitoring

Deployment to Odin

When deploying PPF to odin server, follow these specifications exactly:

Production Deployment

Host:           odin
Container User: podman
Source Path:    /home/podman/ppf/src/
Data Path:      /home/podman/ppf/data/
Config:         /home/podman/ppf/config.ini
HTTP Port:      8081
Container:      ppf (localhost/ppf:latest)

Deployment Steps

Validate syntax - Run python3 -m py_compile on all .py files
Sync to staging - rsync Python files to odin:/tmp/ppf-update/
Copy to container source - sudo -u podman cp /tmp/ppf-update/*.py /home/podman/ppf/src/
Restart container - sudo -u podman podman restart ppf
Clean staging - Remove /tmp/ppf-update/
Verify - Check container logs for successful startup

Test Pod (Development)

If running a test container, use:

Different port: 8082
Different container name: ppf-test
Separate data directory: /home/podman/ppf-test/data/

# Test container example
sudo -u podman podman run -d --name ppf-test \
    -p 8082:8081 \
    -v /home/podman/ppf/src:/app/src:ro \
    -v /home/podman/ppf-test/data:/app/data \
    localhost/ppf:latest

Files to Deploy

All Python files in project root:

proxywatchd.py (main daemon)
httpd.py (web dashboard)
config.py, dbs.py, fetch.py, etc.

Do NOT Deploy

config.ini (server-specific)
data/ directory contents
*.sqlite files

API Endpoints

/dashboard       Web UI with live statistics
/map             Interactive world map (Leaflet.js)
/health          Health check: {"status": "ok"}
/api/stats       Runtime statistics (JSON)
/api/memory      Memory profiling data (JSON)
/api/countries   Proxy counts by country
/api/locations   Precise proxy locations (requires DB5)
/proxies         Working proxies (limit, proto, country params)

Memory Analysis

Query production memory state:

ssh odin "curl -s localhost:8081/api/memory" | python3 -m json.tool

Key metrics:

start_rss / process.VmRSS - memory growth
objgraph_common - top object types by count
samples - RSS history over time
gc.objects - total GC-tracked objects

Current baseline (~260k queue):

Start: 442 MB
Running: 1.6 GB
Per-job overhead: ~4.5 KB

2.6 KiB Raw Blame History