When URLs arrive as unicode (e.g. from JSON API responses), the unicode
type propagates through _parse_url into the SOCKS5 packet construction
in rocksock. Port bytes > 127 formatted via %c in a unicode string
produce non-ASCII characters that fail on socket sendall() implicit
ASCII encode.
Encode URLs to UTF-8 bytes at fetch entry points to keep the entire
request pipeline in str (bytes) domain.
Replace ORDER BY RANDOM() in claim_urls with composite score:
age/interval ratio, yield bonus, quality bonus, error/stale penalties.
Rewrite submit_url_reports with adaptive check_interval and EMA for
avg_fetch_time and yield_rate. Add working_ratio correlation in
submit_proxy_reports via pending count tracking.
Architecture proposal to move proxy list fetching from master to
workers. Workers claim URLs, fetch lists, extract and test proxies,
report working proxies and URL health back to master. Trust-based
model: workers report working proxies only, no consensus needed.
Filter out entries with proto IS NULL from /proxies and /proxies/count
endpoints. These are proxies added to the database but never validated,
leaking into results with null proto, asn, and zero latency.
The WSGI _handle_route had a hardcoded LIMIT 100 query for /proxies,
ignoring limit, proto, country, asn, and format parameters. Align
with the BaseHTTPRequestHandler path that already supported them.
- add debian-security archive repository
- run apt-get upgrade for all available patches
- upgrade pip/setuptools/wheel to latest py2.7 versions
reduces container vulnerabilities from 293 to 130
- ssl_only mode: skip secondary check when SSL handshake fails
- _build_due_sql(): unified query for proxies due testing
- working_checktime/fail_retry_interval: new schedule formula
- fail_retry_backoff: linear backoff option for failing proxies
- CDN_PREFIXES: filter Cloudflare, Fastly, Akamai, CloudFront, Google
- is_cdn_ip(): check if IP belongs to known CDN ranges
- insert_proxies(): skip CDN IPs with count in log message
- verification tables: worker_results, verification_queue, worker_trust
- queue_verification(): add proxies for manager re-testing
- get_verification_stats(): queue size and trigger breakdown
- get_all_worker_trust(): trust scores for all workers
- /api/dashboard: single endpoint returning stats + workers + countries
- dashboard.js: use batch endpoint (2 requests -> 1 per poll cycle)
- _get_workers_data: refactored from /api/workers for code reuse
- worker verification: trust scoring based on result accuracy
- fair distribution: dynamic batch sizing based on queue and workers
- queue tracking: session progress, due/claimed/pending counts
pytest-based test suite with fixtures for database testing.
Covers misc.py utilities, dbs.py operations, and fetch.py validation.
Includes mock_network.py for future network testing.
