From 40560ef6dde232c67afb40e9e1a753f162294f46 Mon Sep 17 00:00:00 2001 From: user Date: Sun, 15 Feb 2026 17:56:08 +0100 Subject: [PATCH] docs: document connection limit, async fetch, and connection pool Update USAGE, CHEATSHEET, README, PROJECT, and TASKS for the three performance improvements: max_connections semaphore, async HTTP source fetching, and first-hop TCP connection pool. --- PROJECT.md | 5 +++++ README.md | 8 +++++++- TASKS.md | 3 +++ docs/CHEATSHEET.md | 9 +++++++++ docs/USAGE.md | 40 +++++++++++++++++++++++++++++++++++++++- 5 files changed, 63 insertions(+), 2 deletions(-) diff --git a/PROJECT.md b/PROJECT.md index 3720e37..15952a7 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -24,6 +24,8 @@ Client -------> s5p -------> Hop 1 -------> Hop 2 -------> Target - **config.py** -- YAML config loading, proxy URL parsing, pool config - **pool.py** -- managed proxy pool (multi-source, health-tested, persistent) - **source.py** -- legacy proxy source (single HTTP API, kept for backward compat) +- **http.py** -- minimal async HTTP/1.1 client (GET/POST JSON, no external deps) +- **connpool.py** -- pre-warmed TCP connection pool to first chain hop - **cli.py** -- argparse CLI, logging setup, cProfile support ## Deployment @@ -61,3 +63,6 @@ All other functionality uses Python stdlib (`asyncio`, `socket`, `struct`). - **Warm start** -- quick-test alive subset on restart, defer full test to background - **SIGHUP reload** -- re-read config, update pool settings, re-fetch sources - **Dead reporting** -- POST evicted proxies to upstream API for list quality feedback +- **Connection semaphore** -- cap concurrent connections to prevent fd exhaustion +- **Async HTTP** -- native asyncio HTTP client replaces blocking urllib, parallel fetches +- **First-hop pool** -- pre-warmed TCP connections to chain[0], stale-evicted, auto-refilled diff --git a/README.md b/README.md index 55c83d4..fe6337a 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,9 @@ through configurable chains of SOCKS4, SOCKS5, and HTTP CONNECT proxies. - Dead proxy reporting to upstream API (optional `report_url`) - SIGHUP hot reload (timeout, retries, log_level, pool config) - Connection metrics with pool stats (logged periodically and on shutdown) +- Concurrent connection limit with backpressure (`max_connections`) +- Async HTTP client for proxy source fetching (parallel, no threads) +- First-hop TCP connection pool (pre-warmed, stale-evicted) - Container-ready (Alpine-based, podman/docker) - Graceful shutdown (SIGTERM/SIGINT) - Pure Python, asyncio-based, minimal dependencies @@ -65,6 +68,8 @@ cp config/example.yaml config/s5p.yaml listen: 127.0.0.1:1080 timeout: 10 retries: 3 +max_connections: 256 # concurrent connection limit +pool_size: 8 # pre-warmed connections to first hop chain: - socks5://127.0.0.1:9050 # Tor @@ -84,7 +89,7 @@ proxy_pool: ## CLI Reference ``` -s5p [-c FILE] [-l [HOST:]PORT] [-C URL[,URL,...]] [-S URL] [-t SEC] [-r N] [-v|-q] +s5p [-c FILE] [-l [HOST:]PORT] [-C URL[,URL,...]] [-S URL] [-t SEC] [-r N] [-m N] [-v|-q] Options: -c, --config FILE YAML config file @@ -93,6 +98,7 @@ Options: -S, --proxy-source URL Proxy source API URL -t, --timeout SEC Per-hop timeout (default: 10) -r, --retries N Max attempts per connection (default: 3, proxy_source only) + -m, --max-connections N Max concurrent connections (default: 256) -v, --verbose Debug logging -q, --quiet Errors only --cprofile [FILE] Enable cProfile, dump to FILE (default: s5p.prof) diff --git a/TASKS.md b/TASKS.md index f09d6e1..2e4279f 100644 --- a/TASKS.md +++ b/TASKS.md @@ -27,6 +27,9 @@ - [x] Static chain health check (skip pool tests if chain unreachable) - [x] SIGHUP hot config reload (timeout, retries, log_level, pool config) - [x] Dead proxy reporting (`report_url` POST evicted proxies to API) +- [x] Concurrent connection semaphore (`max_connections`, CLI `-m`) +- [x] Async HTTP client (replace blocking urllib, parallel source fetch) +- [x] First-hop TCP connection pool (`pool_size`, `pool_max_idle`) ## Next - [ ] Integration tests with mock proxy server diff --git a/docs/CHEATSHEET.md b/docs/CHEATSHEET.md index a44d6dc..0ab354f 100644 --- a/docs/CHEATSHEET.md +++ b/docs/CHEATSHEET.md @@ -13,6 +13,7 @@ s5p -v # debug logging s5p -q # errors only s5p -S http://api:8081/proxies # proxy source API s5p -r 5 # retry up to 5 proxies +s5p -m 512 # max concurrent connections s5p --cprofile # profile to s5p.prof s5p --cprofile out.prof # profile to custom file ``` @@ -34,6 +35,14 @@ Volumes: `./src` (ro), `./config/s5p.yaml` (ro), `~/.cache/s5p` → `/data` (poo cp config/example.yaml config/s5p.yaml # create live config (gitignored) ``` +## Performance Tuning (config) + +```yaml +max_connections: 256 # concurrent connection cap +pool_size: 8 # pre-warmed TCP conns to first hop (0 = off) +pool_max_idle: 30 # evict idle pooled conns (seconds) +``` + ## Proxy Pool (config) ```yaml diff --git a/docs/USAGE.md b/docs/USAGE.md index d6dc42c..e0d5566 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -43,6 +43,9 @@ listen: 127.0.0.1:1080 timeout: 10 retries: 3 log_level: info +max_connections: 256 # concurrent connection limit (backpressure) +pool_size: 0 # pre-warmed TCP connections to first hop (0 = disabled) +pool_max_idle: 30 # max idle time for pooled connections (seconds) chain: - socks5://127.0.0.1:9050 @@ -245,9 +248,10 @@ Settings reloaded on SIGHUP: | `timeout` | Per-connection timeout | | `retries` | Max retry attempts | | `log_level` | Logging verbosity | +| `max_connections` | Concurrent connection limit | | `proxy_pool.*` | Sources, intervals, thresholds | -Settings that require a restart: `listen`, `chain`. +Settings that require a restart: `listen`, `chain`, `pool_size`, `pool_max_idle`. Requires `-c` / `--config` to know which file to re-read. Without a config file, SIGHUP is ignored with a warning. @@ -304,6 +308,40 @@ curl --proxy socks5h://127.0.0.1:1080 https://example.com Note: use `socks5h://` (not `socks5://`) with curl to send DNS through the proxy. +## Connection Limit + +s5p caps concurrent connections with an `asyncio.Semaphore`. When all +slots are taken, new clients backpressure at TCP accept (the connection +is accepted but the handler waits for a slot). + +```yaml +max_connections: 256 # default +``` + +```bash +s5p -m 512 # override via CLI +``` + +The `max_connections` value is reloaded on SIGHUP. Connections already +in flight are not affected -- new limits take effect as active connections +close. + +## First-Hop Connection Pool + +Pre-warms TCP connections to the first hop in the chain, avoiding the +TCP handshake latency on each client request. Only the raw TCP +connection is pooled -- once SOCKS/HTTP CONNECT negotiation begins, +the connection is consumed. + +```yaml +pool_size: 8 # 0 = disabled (default) +pool_max_idle: 30 # seconds before a pooled connection is evicted +``` + +Connections idle longer than `pool_max_idle` are discarded and replaced. +A background task tops up the pool at half the idle interval. Requires +at least one hop in `chain` -- ignored if chain is empty. + ## Chain Order Hops are traversed left-to-right: