feat: worker-driven discovery and validation tightening #1

username · 2026-02-17T17:39:38Z

username commented

2026-02-17 17:39:38 +00:00

Summary

V2 worker mode: URL-driven proxy discovery (claim URLs from master, fetch via Tor, extract and report proxies)
Score-based URL scheduling with EMA tracking
Tighten secondary check validation: neutral judge blocks, HTTP response validation, reject private IPs, remove weak HEAD regexes
Configurable secondary check: checktype = none disables Phase 2 (SSL-only mode)
Compose files for master and worker container management

Commits

dbs: add last_seen, url scoring columns
httpd: claim-urls, report-urls, report-proxies endpoints
config: worker V2 flags, checktype none/false
ppf: worker_v2_main(), HTTP client functions
fetch: encode unicode URLs for HTTP/SOCKS
watchd: tighten secondary check validation
compose: rewrite master and worker compose files

## Summary - V2 worker mode: URL-driven proxy discovery (claim URLs from master, fetch via Tor, extract and report proxies) - Score-based URL scheduling with EMA tracking - Tighten secondary check validation: neutral judge blocks, HTTP response validation, reject private IPs, remove weak HEAD regexes - Configurable secondary check: `checktype = none` disables Phase 2 (SSL-only mode) - Compose files for master and worker container management ## Commits - `dbs`: add last_seen, url scoring columns - `httpd`: claim-urls, report-urls, report-proxies endpoints - `config`: worker V2 flags, checktype none/false - `ppf`: worker_v2_main(), HTTP client functions - `fetch`: encode unicode URLs for HTTP/SOCKS - `watchd`: tighten secondary check validation - `compose`: rewrite master and worker compose files

username added 24 commits 2026-02-17 17:39:38 +00:00

httpd: fix wsgi /proxies route ignoring query params 2960458825

The WSGI _handle_route had a hardcoded LIMIT 100 query for /proxies,
ignoring limit, proto, country, asn, and format parameters. Align
with the BaseHTTPRequestHandler path that already supported them.

httpd: exclude untested proxies from results 6ba4b3e1e9

Filter out entries with proto IS NULL from /proxies and /proxies/count
endpoints. These are proxies added to the database but never validated,
leaking into results with null proto, asn, and zero latency.

httpd: add /proxies/all endpoint for unlimited proxy list 00afd141ae

docs: remove completed items from TODO and ROADMAP e6b736a577

dbs: add 19 proxy sources from 7 new repositories c19959cda2

Expand PROXY_SOURCES with proxifly, vakhov, prxchk, sunny9577,
officialputuid, hookzof, and iplocate lists. Add source_proto
and protos_working schema columns for protocol intelligence.
Remove completed proxy source expansion task from roadmap.

docs: add worker-driven discovery design doc 96e6f06e0d

Architecture proposal to move proxy list fetching from master to
workers. Workers claim URLs, fetch lists, extract and test proxies,
report working proxies and URL health back to master. Trust-based
model: workers report working proxies only, no consensus needed.

dbs: add last_seen column to proxylist da832d94b7

httpd: pass url database to api server 5197c3b7e6

httpd: add /api/claim-urls endpoint 3162c65549

httpd: add /api/report-urls endpoint 66157b5216

httpd: add /api/report-proxies endpoint 6c111af630

docs: update worker hosts to cassius, edge, sentinel 18c7118ed8

config: add worker V2 config items and --worker-v2 flag 4a5210f9f7

ppf: add HTTP client functions for V2 worker endpoints 0685c2bc4c

ppf: add worker_v2_main() for URL-driven discovery 862eeed5c8

dbs: add url scoring columns to uris table 66441f9292

Migration functions for check_interval, working_ratio, avg_fetch_time,
last_worker, and yield_rate columns with sensible defaults.

httpd: add score-based url scheduling with EMA tracking c5287073bf

Replace ORDER BY RANDOM() in claim_urls with composite score:
age/interval ratio, yield bonus, quality bonus, error/stale penalties.

Rewrite submit_url_reports with adaptive check_interval and EMA for
avg_fetch_time and yield_rate. Add working_ratio correlation in
submit_proxy_reports via pending count tracking.

ppf: pass url scoring config to httpd module c710555aad

ppf: fix worker_id undefined when using --worker-key e74782ad3f

fetch: encode unicode URLs to bytes before HTTP/SOCKS ops 0311abb46a

When URLs arrive as unicode (e.g. from JSON API responses), the unicode
type propagates through _parse_url into the SOCKS5 packet construction
in rocksock. Port bytes > 127 formatted via %c in a unicode string
produce non-ASCII characters that fail on socket sendall() implicit
ASCII encode.

Encode URLs to UTF-8 bytes at fetch entry points to keep the entire
request pipeline in str (bytes) domain.

add compose files for container management 1236ddbd2d

Replace raw podman run with declarative compose.yml per host type.
Master (odin) gets compose.master.yml, workers get compose.worker.yml.

watchd: tighten secondary check validation 2e3ce149f9

- judge blocks record as neutral (judge_block category), not success;
  evaluate() filters them out so they affect neither pass nor fail count
- require HTTP/1.x response line for non-IRC checks; non-HTTP garbage
  (captive portals, proxy error pages) fails immediately
- add is_public_ip() rejecting RFC 1918, loopback, link-local, and
  multicast ranges from judge exit IP extraction
- remove 5 weak HEAD regex targets whose fingerprint headers appear on
  error pages and captive portals (p3p, X-XSS-Protection,
  x-frame-options, referrer-policy, X-UA-Compatible)

config: allow checktype = none to disable secondary check 716d60898b

Accepts none/false/off/disabled as checktype value, normalized to
'none' internally. When set, ssl_first is forced on and no Phase 2
check runs -- only successful TLS handshakes count as working.

compose: rewrite master and worker compose files

CI / syntax-check (pull_request) Failing after 0s

Details

CI / syntax-check (push) Failing after 0s

Details

CI / memory-leak-check (pull_request) Failing after 16s

Details

CI / memory-leak-check (push) Successful in 16s

Details

fab1e1d110

Drop deprecated version key, add SELinux volume labels, SIGTERM
handling with 30s grace period, configurable master URL via
PPF_MASTER_URL env var, and usage documentation in headers.

username merged commit fab1e1d110 into master

2026-02-17 17:39:49 +00:00

username deleted branch feature/worker-driven-discovery

2026-02-17 17:39:49 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: username/ppf#1