Commit Graph

128 Commits

Author SHA1 Message Date
Username
e985f52fe6 watchd: add target health tracking for all target pools
Generalizes JudgeStats into TargetStats with cooldown-based filtering
for head targets, SSL targets, and IRC servers. Targets that repeatedly
block or fail are temporarily avoided, preventing unfair proxy failures
when a target goes down. Exposes per-pool health via /api/stats.
2026-02-18 18:21:53 +01:00
Username
3e5c486e7e watchd: adaptive ssl for secondary checks
Some checks failed
CI / syntax-check (push) Failing after 1s
CI / memory-leak-check (push) Successful in 18s
Use SSL error reason from primary handshake to decide whether
the secondary check should use SSL or plain HTTP. Protocol errors
(proxy can't TLS) fall back to plain HTTP; other failures retry
with SSL sans cert verification.
2026-02-18 09:49:40 +01:00
Username
304830e382 watchd: delete proxies immediately on max_fail instead of marking stale
Some checks failed
CI / syntax-check (push) Failing after 1s
CI / memory-leak-check (push) Successful in 17s
2026-02-18 00:50:00 +01:00
Username
4c5f4fa01d watchd: add protocol fingerprint probes and fix nullable counters
Add lightweight SOCKS5/SOCKS4/HTTP handshake probes to identify proxy
protocol before full testing. Guard consecutive_success, success_count,
and total_duration against NoneType from worker-reported upserts.
Track last_check and last_target for test provenance.
2026-02-17 21:06:16 +01:00
Username
2e3ce149f9 watchd: tighten secondary check validation
- judge blocks record as neutral (judge_block category), not success;
  evaluate() filters them out so they affect neither pass nor fail count
- require HTTP/1.x response line for non-IRC checks; non-HTTP garbage
  (captive portals, proxy error pages) fails immediately
- add is_public_ip() rejecting RFC 1918, loopback, link-local, and
  multicast ranges from judge exit IP extraction
- remove 5 weak HEAD regex targets whose fingerprint headers appear on
  error pages and captive portals (p3p, X-XSS-Protection,
  x-frame-options, referrer-policy, X-UA-Compatible)
2026-02-17 18:37:38 +01:00
Username
c710555aad ppf: pass url scoring config to httpd module 2026-02-17 15:20:15 +01:00
Username
5197c3b7e6 httpd: pass url database to api server 2026-02-17 13:42:01 +01:00
Username
1cb7d93a5f proxywatchd: add ssl_only mode and schedule improvements
- ssl_only mode: skip secondary check when SSL handshake fails
- _build_due_sql(): unified query for proxies due testing
- working_checktime/fail_retry_interval: new schedule formula
- fail_retry_backoff: linear backoff option for failing proxies
2026-01-08 09:05:25 +01:00
Username
dfb4739b66 proxywatchd: add __slots__ to hot objects for memory reduction 2025-12-28 17:23:51 +01:00
Username
5a797a9b97 proxywatchd: use context manager for all DB operations 2025-12-28 17:11:56 +01:00
Username
0d7d2dce70 refactor: extract modules from proxywatchd.py
All checks were successful
CI / syntax-check (push) Successful in 3s
CI / memory-leak-check (push) Successful in 11s
Extract focused modules to reduce proxywatchd.py complexity:
- stats.py: JudgeStats, Stats, regexes, ssl_targets (557 lines)
- mitm.py: MITMCertStats, cert extraction functions (239 lines)
- dns.py: socks4_resolve with TTL caching (86 lines)
- job.py: PriorityJobQueue, calculate_priority (103 lines)

proxywatchd.py reduced from 2488 to 1591 lines (-36%).
2025-12-28 15:45:24 +01:00
Username
9b44043237 add ssl_first: try SSL handshake before secondary check
All checks were successful
CI / syntax-check (push) Successful in 3s
CI / memory-leak-check (push) Successful in 11s
When ssl_first=1 (default), proxy validation first attempts an SSL
handshake. If it fails, falls back to the configured secondary check
(head, judges, or irc). This separates SSL capability detection from
basic connectivity testing.

New config options:
- ssl_first: enable SSL-first pattern (default: 1)
- checktype: secondary check type (head, judges, irc)
2025-12-28 14:56:46 +01:00
Username
1c8e3062b7 add configurable thread scaling and queue counter reset
All checks were successful
CI / syntax-check (push) Successful in 3s
CI / memory-leak-check (push) Successful in 12s
- Add scale_cooldown and scale_threshold config options
- ThreadScaler now reads scaling params from config
- Reset priority queue counter when queue empties to prevent unbounded growth
2025-12-28 14:37:32 +01:00
Username
d219cc567f phase 2: code cleanup and simplification
All checks were successful
CI / syntax-check (push) Successful in 3s
CI / memory-leak-check (push) Successful in 11s
- Remove unused result_queue from WorkerThread and worker mode
- Remove num_targets abstraction, simplify to single-target mode
- Add _db_context() context manager for database connections
- Refactor 5 call sites to use context manager (finish, init, cleanup_stale, periodic saves)
- Mark _prep_db/_close_db as deprecated
- Add __version__ = '2.0.0' to ppf.py
- Add thread spawn stagger (0-100ms) in worker mode for Tor-friendly startup
2025-12-28 14:31:37 +01:00
Username
e46a3f6ef1 proxywatchd: add DNS cache TTL, IP validation, debug logging
- DNS cache entries now expire after 1 hour (DNS_CACHE_TTL)
- Add is_valid_ip() to validate extracted IPs have valid octets
- Add debug logging to silent exception blocks (ASN lookup, socket disconnect)
- Convert rwip() to static method with cleaner int() implementation
- Add _sample_dbg() for sampled diagnostic logging
- Add set_config() for worker mode imports
- Remove unused tor_targets (tor checktype was removed)
- Move tor_proxy_url to misc.py for shared use
2025-12-28 14:12:50 +01:00
Username
d9205614fc proxywatchd: secondary check only for SSL handshake failures
All checks were successful
CI / syntax-check (push) Successful in 3s
CI / memory-leak-check (push) Successful in 11s
2025-12-26 21:36:45 +01:00
Username
03dd277c54 dashboard: fix ssl badge after use_ssl config removal
All checks were successful
CI / syntax-check (push) Successful in 2s
CI / memory-leak-check (push) Successful in 11s
2025-12-26 21:30:23 +01:00
Username
9f81e1e4af watchd: remove use_ssl config, use HTTP for non-ssl checks
All checks were successful
CI / syntax-check (push) Successful in 2s
CI / memory-leak-check (push) Successful in 11s
2025-12-26 21:25:52 +01:00
Username
f7a762331a watchd: add tor checktype, use Tor API for secondary check
All checks were successful
CI / syntax-check (push) Successful in 3s
CI / memory-leak-check (push) Successful in 11s
2025-12-26 21:20:16 +01:00
Username
6ea201805f watchd: configurable adaptive timeout per failure 2025-12-26 20:57:21 +01:00
Username
07262e8b50 proxywatchd: cap threads by queue size, faster scale-down 2025-12-26 18:12:39 +01:00
Username
560230988d support multiple checktypes with random selection 2025-12-25 20:23:05 +01:00
Username
0424527e84 proxywatchd: add 0.3s delay before SSL fallback to HTTP 2025-12-25 19:49:59 +01:00
Username
3ac7305954 proxywatchd: persist MITM certificate stats across restarts
Add save_state/load_state to MITMCertStats for JSON persistence.
Stats saved periodically (5min) and at shutdown, loaded at startup.
2025-12-25 19:47:51 +01:00
Username
68e8b88afa tor: use random credentials for circuit isolation
All checks were successful
CI / syntax-check (push) Successful in 6s
CI / memory-leak-check (push) Successful in 14s
2025-12-25 19:18:25 +01:00
Username
9ba965c87f proxywatchd: ensure socket cleanup before SSL fallback
All checks were successful
CI / syntax-check (push) Successful in 6s
CI / memory-leak-check (push) Successful in 14s
2025-12-25 19:13:47 +01:00
Username
b41d077905 proxywatchd: make peak_rate per-session, not persistent
All checks were successful
CI / syntax-check (push) Successful in 6s
CI / memory-leak-check (push) Successful in 14s
2025-12-25 18:54:05 +01:00
Username
92b4036c37 proxywatchd: delay peak measurement until after startup
All checks were successful
CI / syntax-check (push) Successful in 6s
CI / memory-leak-check (push) Successful in 14s
2025-12-25 18:48:27 +01:00
Username
a8d06666b7 proxywatchd: fallback to HTTP when SSL fails
All checks were successful
CI / syntax-check (push) Successful in 6s
CI / memory-leak-check (push) Successful in 14s
2025-12-25 18:39:51 +01:00
Username
c459736561 proxywatchd: add sanity checks to rate/peak calculations
All checks were successful
CI / syntax-check (push) Successful in 6s
CI / memory-leak-check (push) Successful in 14s
2025-12-25 18:11:22 +01:00
Username
2201515b10 proxywatchd: track failures by protocol and SSL category 2025-12-25 02:51:47 +01:00
Username
38fb16a439 proxywatchd: fix mitm reset logic, track ssl_mitm category 2025-12-24 01:26:37 +01:00
Username
4f17e7c546 proxywatchd: treat MITM as working, mark fatal errors as dead
All checks were successful
CI / syntax-check (push) Successful in 6s
CI / memory-leak-check (push) Successful in 14s
2025-12-24 01:17:02 +01:00
Username
de750a1312 proxywatchd: fix division, uptime, and scaler
- Add from __future__ import division for correct percentages
- Remove start_time restoration so uptime reflects current session
- Remove success_rate threshold for scaling (scale on queue depth only)
2025-12-24 00:17:47 +01:00
Username
f83733dd46 proxywatchd: mark confirmed-dead proxies as permanently dead
- Add DEAD_PROXY=-1 constant for permanently dead proxies
- Mark proxy dead when: failed >= max_fail*2, or max_fail with fatal error
- Fatal errors: refused, unreachable, auth (proxy definitely not working)
- Dead proxies excluded from testing (failed >= 0 query)
- Cleanup_stale also removes old dead proxies
- Dashboard shows separate dead vs failing counts
2025-12-23 18:03:01 +01:00
Username
1e2054bec5 dashboard: fix missing tor pool and db stats fields 2025-12-23 17:56:12 +01:00
Username
c0dbba7b45 proxywatchd: gevent concurrency and comprehensive stats
Major features:
- gevent monkey-patching for cooperative concurrency
- Stats class with latency percentiles, histograms, geo tracking
- session state persistence across restarts
- JudgeStats with cooldown for blocked/rate-limited judges
- ThreadScaler for dynamic greenlet pool sizing
- SIGHUP handler for config hot-reload
- SSL/TLS tracking with MITM detection
- anonymity detection (transparent/anonymous/elite)

Bug fixes:
- rwip(): fix string comparison (b[0] == '0' not == 0)
- rwip(): fix strip logic (b[1:] not b[:1])
- use string.ascii_letters for Py2/3 compatibility
2025-12-23 17:23:45 +01:00
Username
95bafcacff proxywatchd: add startup logging, fix geolocation error handling 2025-12-21 23:37:19 +01:00
Username
8718d33276 fix: use canonical schema from dbs.py in proxywatchd 2025-12-21 10:30:31 +01:00
Username
79475c2bff add latency tracking and dynamic thread scaling
- dbs.py: add avg_latency, latency_samples columns with migration
- dbs.py: update_proxy_latency() with exponential moving average
- proxywatchd.py: ThreadScaler class for dynamic thread count
- proxywatchd.py: calculate/record latency for successful proxies
- proxywatchd.py: _spawn_thread(), _remove_thread(), _adjust_threads()
- scaler reports status alongside periodic stats
2025-12-21 00:08:19 +01:00
Username
e24f68500c style: normalize indentation and improve code style
- convert tabs to 4-space indentation
- add docstrings to modules and classes
- remove unused import (copy)
- use explicit object inheritance
- use 'while True' over 'while 1'
- use 'while args' over 'while len(args)'
- use '{}' over 'dict()'
- consistent string formatting
- Python 2/3 compatible Queue import
2025-12-20 23:18:45 +01:00
Username
a694e441a4 proxywatchd: add priority queue for job scheduling 2025-12-20 23:11:49 +01:00
Username
af5e1ce4b0 proxywatchd: integrate tor connection pool 2025-12-20 23:02:26 +01:00
Username
2f2ff9a385 proxywatchd: add stats tracking and httpd integration
- Stats class with failure category tracking
- Configurable stats_interval for periodic reports
- Optional httpd server startup when enabled
- cleanup_stale() for removing dead proxies
2025-12-20 22:28:23 +01:00
Username
86cabd1562 standardize code style: shebangs, class definitions, comments 2025-12-20 18:05:41 +01:00
Username
68d6a8e15f proxywatchd: implement multi-target validation with work-stealing queue 2025-12-20 16:46:09 +01:00
Your Name
d7db366857 split to ip/port, "cleanse" ips and ports, bugfixes 2021-08-22 20:39:50 +02:00
Your Name
c3bb49d229 proxywatchd: make use of verifycert 2021-07-27 22:36:24 +02:00
Your Name
d4dd2a42ea proxywatchd.py: randomly choose from available regex keys 2021-07-05 16:54:17 +02:00
Your Name
d78212ac50 proxywatchd.py: add more websites 2021-07-05 16:45:02 +02:00