docs: document connection limit, async fetch, and connection pool
Update USAGE, CHEATSHEET, README, PROJECT, and TASKS for the three performance improvements: max_connections semaphore, async HTTP source fetching, and first-hop TCP connection pool.
This commit is contained in:
@@ -24,6 +24,8 @@ Client -------> s5p -------> Hop 1 -------> Hop 2 -------> Target
|
||||
- **config.py** -- YAML config loading, proxy URL parsing, pool config
|
||||
- **pool.py** -- managed proxy pool (multi-source, health-tested, persistent)
|
||||
- **source.py** -- legacy proxy source (single HTTP API, kept for backward compat)
|
||||
- **http.py** -- minimal async HTTP/1.1 client (GET/POST JSON, no external deps)
|
||||
- **connpool.py** -- pre-warmed TCP connection pool to first chain hop
|
||||
- **cli.py** -- argparse CLI, logging setup, cProfile support
|
||||
|
||||
## Deployment
|
||||
@@ -61,3 +63,6 @@ All other functionality uses Python stdlib (`asyncio`, `socket`, `struct`).
|
||||
- **Warm start** -- quick-test alive subset on restart, defer full test to background
|
||||
- **SIGHUP reload** -- re-read config, update pool settings, re-fetch sources
|
||||
- **Dead reporting** -- POST evicted proxies to upstream API for list quality feedback
|
||||
- **Connection semaphore** -- cap concurrent connections to prevent fd exhaustion
|
||||
- **Async HTTP** -- native asyncio HTTP client replaces blocking urllib, parallel fetches
|
||||
- **First-hop pool** -- pre-warmed TCP connections to chain[0], stale-evicted, auto-refilled
|
||||
|
||||
@@ -18,6 +18,9 @@ through configurable chains of SOCKS4, SOCKS5, and HTTP CONNECT proxies.
|
||||
- Dead proxy reporting to upstream API (optional `report_url`)
|
||||
- SIGHUP hot reload (timeout, retries, log_level, pool config)
|
||||
- Connection metrics with pool stats (logged periodically and on shutdown)
|
||||
- Concurrent connection limit with backpressure (`max_connections`)
|
||||
- Async HTTP client for proxy source fetching (parallel, no threads)
|
||||
- First-hop TCP connection pool (pre-warmed, stale-evicted)
|
||||
- Container-ready (Alpine-based, podman/docker)
|
||||
- Graceful shutdown (SIGTERM/SIGINT)
|
||||
- Pure Python, asyncio-based, minimal dependencies
|
||||
@@ -65,6 +68,8 @@ cp config/example.yaml config/s5p.yaml
|
||||
listen: 127.0.0.1:1080
|
||||
timeout: 10
|
||||
retries: 3
|
||||
max_connections: 256 # concurrent connection limit
|
||||
pool_size: 8 # pre-warmed connections to first hop
|
||||
|
||||
chain:
|
||||
- socks5://127.0.0.1:9050 # Tor
|
||||
@@ -84,7 +89,7 @@ proxy_pool:
|
||||
## CLI Reference
|
||||
|
||||
```
|
||||
s5p [-c FILE] [-l [HOST:]PORT] [-C URL[,URL,...]] [-S URL] [-t SEC] [-r N] [-v|-q]
|
||||
s5p [-c FILE] [-l [HOST:]PORT] [-C URL[,URL,...]] [-S URL] [-t SEC] [-r N] [-m N] [-v|-q]
|
||||
|
||||
Options:
|
||||
-c, --config FILE YAML config file
|
||||
@@ -93,6 +98,7 @@ Options:
|
||||
-S, --proxy-source URL Proxy source API URL
|
||||
-t, --timeout SEC Per-hop timeout (default: 10)
|
||||
-r, --retries N Max attempts per connection (default: 3, proxy_source only)
|
||||
-m, --max-connections N Max concurrent connections (default: 256)
|
||||
-v, --verbose Debug logging
|
||||
-q, --quiet Errors only
|
||||
--cprofile [FILE] Enable cProfile, dump to FILE (default: s5p.prof)
|
||||
|
||||
3
TASKS.md
3
TASKS.md
@@ -27,6 +27,9 @@
|
||||
- [x] Static chain health check (skip pool tests if chain unreachable)
|
||||
- [x] SIGHUP hot config reload (timeout, retries, log_level, pool config)
|
||||
- [x] Dead proxy reporting (`report_url` POST evicted proxies to API)
|
||||
- [x] Concurrent connection semaphore (`max_connections`, CLI `-m`)
|
||||
- [x] Async HTTP client (replace blocking urllib, parallel source fetch)
|
||||
- [x] First-hop TCP connection pool (`pool_size`, `pool_max_idle`)
|
||||
|
||||
## Next
|
||||
- [ ] Integration tests with mock proxy server
|
||||
|
||||
@@ -13,6 +13,7 @@ s5p -v # debug logging
|
||||
s5p -q # errors only
|
||||
s5p -S http://api:8081/proxies # proxy source API
|
||||
s5p -r 5 # retry up to 5 proxies
|
||||
s5p -m 512 # max concurrent connections
|
||||
s5p --cprofile # profile to s5p.prof
|
||||
s5p --cprofile out.prof # profile to custom file
|
||||
```
|
||||
@@ -34,6 +35,14 @@ Volumes: `./src` (ro), `./config/s5p.yaml` (ro), `~/.cache/s5p` → `/data` (poo
|
||||
cp config/example.yaml config/s5p.yaml # create live config (gitignored)
|
||||
```
|
||||
|
||||
## Performance Tuning (config)
|
||||
|
||||
```yaml
|
||||
max_connections: 256 # concurrent connection cap
|
||||
pool_size: 8 # pre-warmed TCP conns to first hop (0 = off)
|
||||
pool_max_idle: 30 # evict idle pooled conns (seconds)
|
||||
```
|
||||
|
||||
## Proxy Pool (config)
|
||||
|
||||
```yaml
|
||||
|
||||
@@ -43,6 +43,9 @@ listen: 127.0.0.1:1080
|
||||
timeout: 10
|
||||
retries: 3
|
||||
log_level: info
|
||||
max_connections: 256 # concurrent connection limit (backpressure)
|
||||
pool_size: 0 # pre-warmed TCP connections to first hop (0 = disabled)
|
||||
pool_max_idle: 30 # max idle time for pooled connections (seconds)
|
||||
|
||||
chain:
|
||||
- socks5://127.0.0.1:9050
|
||||
@@ -245,9 +248,10 @@ Settings reloaded on SIGHUP:
|
||||
| `timeout` | Per-connection timeout |
|
||||
| `retries` | Max retry attempts |
|
||||
| `log_level` | Logging verbosity |
|
||||
| `max_connections` | Concurrent connection limit |
|
||||
| `proxy_pool.*` | Sources, intervals, thresholds |
|
||||
|
||||
Settings that require a restart: `listen`, `chain`.
|
||||
Settings that require a restart: `listen`, `chain`, `pool_size`, `pool_max_idle`.
|
||||
|
||||
Requires `-c` / `--config` to know which file to re-read. Without a
|
||||
config file, SIGHUP is ignored with a warning.
|
||||
@@ -304,6 +308,40 @@ curl --proxy socks5h://127.0.0.1:1080 https://example.com
|
||||
|
||||
Note: use `socks5h://` (not `socks5://`) with curl to send DNS through the proxy.
|
||||
|
||||
## Connection Limit
|
||||
|
||||
s5p caps concurrent connections with an `asyncio.Semaphore`. When all
|
||||
slots are taken, new clients backpressure at TCP accept (the connection
|
||||
is accepted but the handler waits for a slot).
|
||||
|
||||
```yaml
|
||||
max_connections: 256 # default
|
||||
```
|
||||
|
||||
```bash
|
||||
s5p -m 512 # override via CLI
|
||||
```
|
||||
|
||||
The `max_connections` value is reloaded on SIGHUP. Connections already
|
||||
in flight are not affected -- new limits take effect as active connections
|
||||
close.
|
||||
|
||||
## First-Hop Connection Pool
|
||||
|
||||
Pre-warms TCP connections to the first hop in the chain, avoiding the
|
||||
TCP handshake latency on each client request. Only the raw TCP
|
||||
connection is pooled -- once SOCKS/HTTP CONNECT negotiation begins,
|
||||
the connection is consumed.
|
||||
|
||||
```yaml
|
||||
pool_size: 8 # 0 = disabled (default)
|
||||
pool_max_idle: 30 # seconds before a pooled connection is evicted
|
||||
```
|
||||
|
||||
Connections idle longer than `pool_max_idle` are discarded and replaced.
|
||||
A background task tops up the pool at half the idle interval. Requires
|
||||
at least one hop in `chain` -- ignored if chain is empty.
|
||||
|
||||
## Chain Order
|
||||
|
||||
Hops are traversed left-to-right:
|
||||
|
||||
Reference in New Issue
Block a user