97 lines
2.6 KiB
Markdown
97 lines
2.6 KiB
Markdown
# PPF Project Instructions
|
|
|
|
## Current State
|
|
|
|
PPF is a Python 2 proxy scraping and validation framework with:
|
|
- Multi-target validation (2/3 majority voting)
|
|
- SSL/TLS proxy testing with MITM detection
|
|
- Web dashboard with electric cyan theme
|
|
- Interactive world map (/map endpoint)
|
|
- Memory profiling (/api/memory endpoint)
|
|
- Tor connection pooling with health monitoring
|
|
|
|
## Deployment to Odin
|
|
|
|
When deploying PPF to odin server, follow these specifications exactly:
|
|
|
|
### Production Deployment
|
|
|
|
```
|
|
Host: odin
|
|
Container User: podman
|
|
Source Path: /home/podman/ppf/src/
|
|
Data Path: /home/podman/ppf/data/
|
|
Config: /home/podman/ppf/config.ini
|
|
HTTP Port: 8081
|
|
Container: ppf (localhost/ppf:latest)
|
|
```
|
|
|
|
### Deployment Steps
|
|
|
|
1. **Validate syntax** - Run `python3 -m py_compile` on all .py files
|
|
2. **Sync to staging** - rsync Python files to `odin:/tmp/ppf-update/`
|
|
3. **Copy to container source** - `sudo -u podman cp /tmp/ppf-update/*.py /home/podman/ppf/src/`
|
|
4. **Restart container** - `sudo -u podman podman restart ppf`
|
|
5. **Clean staging** - Remove `/tmp/ppf-update/`
|
|
6. **Verify** - Check container logs for successful startup
|
|
|
|
### Test Pod (Development)
|
|
|
|
If running a test container, use:
|
|
- Different port: 8082
|
|
- Different container name: ppf-test
|
|
- Separate data directory: /home/podman/ppf-test/data/
|
|
|
|
```bash
|
|
# Test container example
|
|
sudo -u podman podman run -d --name ppf-test \
|
|
-p 8082:8081 \
|
|
-v /home/podman/ppf/src:/app/src:ro \
|
|
-v /home/podman/ppf-test/data:/app/data \
|
|
localhost/ppf:latest
|
|
```
|
|
|
|
### Files to Deploy
|
|
|
|
All Python files in project root:
|
|
- proxywatchd.py (main daemon)
|
|
- httpd.py (web dashboard)
|
|
- config.py, dbs.py, fetch.py, etc.
|
|
|
|
### Do NOT Deploy
|
|
|
|
- config.ini (server-specific)
|
|
- data/ directory contents
|
|
- *.sqlite files
|
|
|
|
## API Endpoints
|
|
|
|
```
|
|
/dashboard Web UI with live statistics
|
|
/map Interactive world map (Leaflet.js)
|
|
/health Health check: {"status": "ok"}
|
|
/api/stats Runtime statistics (JSON)
|
|
/api/memory Memory profiling data (JSON)
|
|
/api/countries Proxy counts by country
|
|
/api/locations Precise proxy locations (requires DB5)
|
|
/proxies Working proxies (limit, proto, country params)
|
|
```
|
|
|
|
## Memory Analysis
|
|
|
|
Query production memory state:
|
|
```bash
|
|
ssh odin "curl -s localhost:8081/api/memory" | python3 -m json.tool
|
|
```
|
|
|
|
Key metrics:
|
|
- `start_rss` / `process.VmRSS` - memory growth
|
|
- `objgraph_common` - top object types by count
|
|
- `samples` - RSS history over time
|
|
- `gc.objects` - total GC-tracked objects
|
|
|
|
Current baseline (~260k queue):
|
|
- Start: 442 MB
|
|
- Running: 1.6 GB
|
|
- Per-job overhead: ~4.5 KB
|