- threading.local() caches proxy_db and url_db per greenlet (eliminates
~2.7k redundant sqlite3.connect + PRAGMA calls per session on odin)
- ASN database now lazy-loaded on first lookup (defers ~3.6s startup cost)
- URL claim error penalty increased from 0.3*error(cap 2) to 0.5*error(cap 4)
and stale penalty from 0.1*stale(cap 1) to 0.2*stale(cap 1.5) to reduce
worker cycles wasted on erroring URLs (71% of 7,158 URLs erroring)
Boost SOCKS sources in claim_urls scoring when SOCKS proxies
are underrepresented (<40% of pool). Dynamic 0-1.0 boost based
on current protocol distribution.
Add urls section with total/healthy/dead/erroring counts, fetch
activity, productive source count, aggregate yield, and top sources
ranked by working_ratio.
Load pyasn database in httpd and look up ASN when workers report
working proxies. Falls back to a pure-Python ipasn.dat reader when
the pyasn C extension is unavailable (Python 2.7 containers).
Backfills existing proxies with null ASN on startup.
Drop _work_claims tracking, claim_work(), submit_results(),
get_due_proxy_count(), calculate_fair_batch_size(), and the
/api/work + /api/results endpoint handlers.
Replace ORDER BY RANDOM() in claim_urls with composite score:
age/interval ratio, yield bonus, quality bonus, error/stale penalties.
Rewrite submit_url_reports with adaptive check_interval and EMA for
avg_fetch_time and yield_rate. Add working_ratio correlation in
submit_proxy_reports via pending count tracking.
Filter out entries with proto IS NULL from /proxies and /proxies/count
endpoints. These are proxies added to the database but never validated,
leaking into results with null proto, asn, and zero latency.
The WSGI _handle_route had a hardcoded LIMIT 100 query for /proxies,
ignoring limit, proto, country, asn, and format parameters. Align
with the BaseHTTPRequestHandler path that already supported them.
- /api/dashboard: single endpoint returning stats + workers + countries
- dashboard.js: use batch endpoint (2 requests -> 1 per poll cycle)
- _get_workers_data: refactored from /api/workers for code reuse
- worker verification: trust scoring based on result accuracy
- fair distribution: dynamic batch sizing based on queue and workers
- queue tracking: session progress, due/claimed/pending counts
- Add from __future__ import division
- Change "Tests This Session" to "Tests (Cumulative)"
- Fix Tor status to show IDLE when available but 0% success
- Add memory leak detection: RSS growth, peak, GC stats
- Add DEAD_PROXY=-1 constant for permanently dead proxies
- Mark proxy dead when: failed >= max_fail*2, or max_fail with fatal error
- Fatal errors: refused, unreachable, auth (proxy definitely not working)
- Dead proxies excluded from testing (failed >= 0 query)
- Cleanup_stale also removes old dead proxies
- Dashboard shows separate dead vs failing counts
- dark theme with CSS custom properties
- responsive grid layout for stats cards
- sparkline charts for rate/success history
- gevent pywsgi server with BaseHTTPServer fallback
- new API endpoints: /api/stats, /api/judges, /api/protocols
- protocol breakdown, latency percentiles, geo distribution
- tor pool health, judge availability, engine stats
- auto-refresh with configurable interval
