docs: document connection limit, async fetch, and connection pool

Update USAGE, CHEATSHEET, README, PROJECT, and TASKS for the three
performance improvements: max_connections semaphore, async HTTP
source fetching, and first-hop TCP connection pool.
This commit is contained in:
user
2026-02-15 17:56:08 +01:00
parent 248f5c3306
commit 40560ef6dd
5 changed files with 63 additions and 2 deletions

View File

@@ -24,6 +24,8 @@ Client -------> s5p -------> Hop 1 -------> Hop 2 -------> Target
- **config.py** -- YAML config loading, proxy URL parsing, pool config
- **pool.py** -- managed proxy pool (multi-source, health-tested, persistent)
- **source.py** -- legacy proxy source (single HTTP API, kept for backward compat)
- **http.py** -- minimal async HTTP/1.1 client (GET/POST JSON, no external deps)
- **connpool.py** -- pre-warmed TCP connection pool to first chain hop
- **cli.py** -- argparse CLI, logging setup, cProfile support
## Deployment
@@ -61,3 +63,6 @@ All other functionality uses Python stdlib (`asyncio`, `socket`, `struct`).
- **Warm start** -- quick-test alive subset on restart, defer full test to background
- **SIGHUP reload** -- re-read config, update pool settings, re-fetch sources
- **Dead reporting** -- POST evicted proxies to upstream API for list quality feedback
- **Connection semaphore** -- cap concurrent connections to prevent fd exhaustion
- **Async HTTP** -- native asyncio HTTP client replaces blocking urllib, parallel fetches
- **First-hop pool** -- pre-warmed TCP connections to chain[0], stale-evicted, auto-refilled

View File

@@ -18,6 +18,9 @@ through configurable chains of SOCKS4, SOCKS5, and HTTP CONNECT proxies.
- Dead proxy reporting to upstream API (optional `report_url`)
- SIGHUP hot reload (timeout, retries, log_level, pool config)
- Connection metrics with pool stats (logged periodically and on shutdown)
- Concurrent connection limit with backpressure (`max_connections`)
- Async HTTP client for proxy source fetching (parallel, no threads)
- First-hop TCP connection pool (pre-warmed, stale-evicted)
- Container-ready (Alpine-based, podman/docker)
- Graceful shutdown (SIGTERM/SIGINT)
- Pure Python, asyncio-based, minimal dependencies
@@ -65,6 +68,8 @@ cp config/example.yaml config/s5p.yaml
listen: 127.0.0.1:1080
timeout: 10
retries: 3
max_connections: 256 # concurrent connection limit
pool_size: 8 # pre-warmed connections to first hop
chain:
- socks5://127.0.0.1:9050 # Tor
@@ -84,7 +89,7 @@ proxy_pool:
## CLI Reference
```
s5p [-c FILE] [-l [HOST:]PORT] [-C URL[,URL,...]] [-S URL] [-t SEC] [-r N] [-v|-q]
s5p [-c FILE] [-l [HOST:]PORT] [-C URL[,URL,...]] [-S URL] [-t SEC] [-r N] [-m N] [-v|-q]
Options:
-c, --config FILE YAML config file
@@ -93,6 +98,7 @@ Options:
-S, --proxy-source URL Proxy source API URL
-t, --timeout SEC Per-hop timeout (default: 10)
-r, --retries N Max attempts per connection (default: 3, proxy_source only)
-m, --max-connections N Max concurrent connections (default: 256)
-v, --verbose Debug logging
-q, --quiet Errors only
--cprofile [FILE] Enable cProfile, dump to FILE (default: s5p.prof)

View File

@@ -27,6 +27,9 @@
- [x] Static chain health check (skip pool tests if chain unreachable)
- [x] SIGHUP hot config reload (timeout, retries, log_level, pool config)
- [x] Dead proxy reporting (`report_url` POST evicted proxies to API)
- [x] Concurrent connection semaphore (`max_connections`, CLI `-m`)
- [x] Async HTTP client (replace blocking urllib, parallel source fetch)
- [x] First-hop TCP connection pool (`pool_size`, `pool_max_idle`)
## Next
- [ ] Integration tests with mock proxy server

View File

@@ -13,6 +13,7 @@ s5p -v # debug logging
s5p -q # errors only
s5p -S http://api:8081/proxies # proxy source API
s5p -r 5 # retry up to 5 proxies
s5p -m 512 # max concurrent connections
s5p --cprofile # profile to s5p.prof
s5p --cprofile out.prof # profile to custom file
```
@@ -34,6 +35,14 @@ Volumes: `./src` (ro), `./config/s5p.yaml` (ro), `~/.cache/s5p` → `/data` (poo
cp config/example.yaml config/s5p.yaml # create live config (gitignored)
```
## Performance Tuning (config)
```yaml
max_connections: 256 # concurrent connection cap
pool_size: 8 # pre-warmed TCP conns to first hop (0 = off)
pool_max_idle: 30 # evict idle pooled conns (seconds)
```
## Proxy Pool (config)
```yaml

View File

@@ -43,6 +43,9 @@ listen: 127.0.0.1:1080
timeout: 10
retries: 3
log_level: info
max_connections: 256 # concurrent connection limit (backpressure)
pool_size: 0 # pre-warmed TCP connections to first hop (0 = disabled)
pool_max_idle: 30 # max idle time for pooled connections (seconds)
chain:
- socks5://127.0.0.1:9050
@@ -245,9 +248,10 @@ Settings reloaded on SIGHUP:
| `timeout` | Per-connection timeout |
| `retries` | Max retry attempts |
| `log_level` | Logging verbosity |
| `max_connections` | Concurrent connection limit |
| `proxy_pool.*` | Sources, intervals, thresholds |
Settings that require a restart: `listen`, `chain`.
Settings that require a restart: `listen`, `chain`, `pool_size`, `pool_max_idle`.
Requires `-c` / `--config` to know which file to re-read. Without a
config file, SIGHUP is ignored with a warning.
@@ -304,6 +308,40 @@ curl --proxy socks5h://127.0.0.1:1080 https://example.com
Note: use `socks5h://` (not `socks5://`) with curl to send DNS through the proxy.
## Connection Limit
s5p caps concurrent connections with an `asyncio.Semaphore`. When all
slots are taken, new clients backpressure at TCP accept (the connection
is accepted but the handler waits for a slot).
```yaml
max_connections: 256 # default
```
```bash
s5p -m 512 # override via CLI
```
The `max_connections` value is reloaded on SIGHUP. Connections already
in flight are not affected -- new limits take effect as active connections
close.
## First-Hop Connection Pool
Pre-warms TCP connections to the first hop in the chain, avoiding the
TCP handshake latency on each client request. Only the raw TCP
connection is pooled -- once SOCKS/HTTP CONNECT negotiation begins,
the connection is consumed.
```yaml
pool_size: 8 # 0 = disabled (default)
pool_max_idle: 30 # seconds before a pooled connection is evicted
```
Connections idle longer than `pool_max_idle` are discarded and replaced.
A background task tops up the pool at half the idle interval. Requires
at least one hop in `chain` -- ignored if chain is empty.
## Chain Order
Hops are traversed left-to-right: