514 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Username	6cc903c924	httpd: add batch API endpoint and worker improvements ... - /api/dashboard: single endpoint returning stats + workers + countries - dashboard.js: use batch endpoint (2 requests -> 1 per poll cycle) - _get_workers_data: refactored from /api/workers for code reuse - worker verification: trust scoring based on result accuracy - fair distribution: dynamic batch sizing based on queue and workers - queue tracking: session progress, due/claimed/pending counts	2026-01-08 09:02:56 +01:00
Username	44604f1ce3	tests: add unit test infrastructure ... pytest-based test suite with fixtures for database testing. Covers misc.py utilities, dbs.py operations, and fetch.py validation. Includes mock_network.py for future network testing.	2026-01-08 01:42:38 +01:00
Username	c1ec5d593b	worker: check tor every 30s instead of exponential backoff	2025-12-28 18:41:05 +01:00
Username	966c0d641d	docs: mark low-effort tasks as completed	2025-12-28 17:25:06 +01:00
Username	dfb4739b66	proxywatchd: add __slots__ to hot objects for memory reduction	2025-12-28 17:23:51 +01:00
Username	480a652889	httpd: add stats export endpoint with CSV/JSON support	2025-12-28 17:23:44 +01:00
Username	1f09c75345	docs: add database context manager to completed work	2025-12-28 17:13:24 +01:00
Username	5a797a9b97	proxywatchd: use context manager for all DB operations	2025-12-28 17:11:56 +01:00
Username	9e2fc3e09d	docs: update roadmap and todo with recent changes	2025-12-28 17:00:52 +01:00
Username	e758ce7178	dashboard: add keyboard shortcuts and optimize polling ... - fetch.py: convert proxy validation cache to LRU with OrderedDict - thread-safe lock, move_to_end() on hits, evict oldest when full - dashboard.js: add keyboard shortcuts (r=refresh, 1-9=tabs, t=theme, p=pause) - dashboard.js: skip chart rendering for inactive tabs (reduces CPU)	2025-12-28 16:52:52 +01:00
Username	18ae73bfb8	httpd: add worker test rate tracking ... Track per-worker test rates using 120s sliding window. Display combined rate in dashboard and individual rates in worker cards.	2025-12-28 16:43:53 +01:00
Username	2bc00d3ebd	worker: check tor before claiming work	2025-12-28 16:09:40 +01:00
Username	0d7d2dce70	refactor: extract modules from proxywatchd.py ... Extract focused modules to reduce proxywatchd.py complexity: - stats.py: JudgeStats, Stats, regexes, ssl_targets (557 lines) - mitm.py: MITMCertStats, cert extraction functions (239 lines) - dns.py: socks4_resolve with TTL caching (86 lines) - job.py: PriorityJobQueue, calculate_priority (103 lines) proxywatchd.py reduced from 2488 to 1591 lines (-36%).	2025-12-28 15:45:24 +01:00
Username	35f24bb8b0	dashboard: refactor layout and add worker stats	2025-12-28 15:19:50 +01:00
Username	e89db20f5b	scraper: add Bing and Yahoo engines	2025-12-28 15:19:39 +01:00
Username	0fbfee2855	httpd: add worker registration and distributed testing API	2025-12-28 15:19:08 +01:00
Username	3b361916fa	fetch, dbs: minor refactoring	2025-12-28 15:18:42 +01:00
Username	50bc13a615	misc: add SSL protocol error detection and diag logging	2025-12-28 15:18:29 +01:00
Username	f4286ea515	ppf: remove num_targets param (removed in phase 2)	2025-12-28 15:16:52 +01:00
Username	9b44043237	add ssl_first: try SSL handshake before secondary check ... When ssl_first=1 (default), proxy validation first attempts an SSL handshake. If it fails, falls back to the configured secondary check (head, judges, or irc). This separates SSL capability detection from basic connectivity testing. New config options: - ssl_first: enable SSL-first pattern (default: 1) - checktype: secondary check type (head, judges, irc)	2025-12-28 14:56:46 +01:00
Username	9f782c3222	config: default to head checktype only	2025-12-28 14:45:27 +01:00
Username	1c8e3062b7	add configurable thread scaling and queue counter reset ... - Add scale_cooldown and scale_threshold config options - ThreadScaler now reads scaling params from config - Reset priority queue counter when queue empties to prevent unbounded growth	2025-12-28 14:37:32 +01:00
Username	d219cc567f	phase 2: code cleanup and simplification ... - Remove unused result_queue from WorkerThread and worker mode - Remove num_targets abstraction, simplify to single-target mode - Add _db_context() context manager for database connections - Refactor 5 call sites to use context manager (finish, init, cleanup_stale, periodic saves) - Mark _prep_db/_close_db as deprecated - Add __version__ = '2.0.0' to ppf.py - Add thread spawn stagger (0-100ms) in worker mode for Tor-friendly startup	2025-12-28 14:31:37 +01:00
Username	72a2dcdaf4	ppf: add worker mode with distributed testing ... - Add --worker mode for distributed proxy testing - Workers claim batches from manager, test via local Tor, submit results - Add --register to register new workers with manager - Add thread spawn stagger (0-100ms) to avoid overwhelming Tor - Verify Tor connectivity before claiming work - Add heartbeat and batch timeout handling - Track worker profiling state for dashboard display	2025-12-28 14:12:59 +01:00
Username	e46a3f6ef1	proxywatchd: add DNS cache TTL, IP validation, debug logging ... - DNS cache entries now expire after 1 hour (DNS_CACHE_TTL) - Add is_valid_ip() to validate extracted IPs have valid octets - Add debug logging to silent exception blocks (ASN lookup, socket disconnect) - Convert rwip() to static method with cleaner int() implementation - Add _sample_dbg() for sampled diagnostic logging - Add set_config() for worker mode imports - Remove unused tor_targets (tor checktype was removed) - Move tor_proxy_url to misc.py for shared use	2025-12-28 14:12:50 +01:00
Username	58d83ae52f	dashboard: always show SSL badge	2025-12-26 21:37:37 +01:00
Username	d9205614fc	proxywatchd: secondary check only for SSL handshake failures	2025-12-26 21:36:45 +01:00
Username	03dd277c54	dashboard: fix ssl badge after use_ssl config removal	2025-12-26 21:30:23 +01:00
Username	9f81e1e4af	watchd: remove use_ssl config, use HTTP for non-ssl checks	2025-12-26 21:25:52 +01:00
Username	f7a762331a	watchd: add tor checktype, use Tor API for secondary check	2025-12-26 21:20:16 +01:00
Username	d2bd7d4f34	fetch: retry with different Tor circuit on failure	2025-12-26 20:57:28 +01:00
Username	6ea201805f	watchd: configurable adaptive timeout per failure	2025-12-26 20:57:21 +01:00
Username	7232846b0f	ppf: add --reset flag to clear all state	2025-12-26 20:57:15 +01:00
Username	906d1b33ae	fetch: cache is_usable_proxy results	2025-12-26 20:04:01 +01:00
Username	1f41f3df5c	dashboard: pause polling when tab is hidden	2025-12-26 19:58:20 +01:00
Username	ad89eb262e	httpd: add rate limiting and security headers	2025-12-26 19:52:45 +01:00
Username	59fe2c4a14	dashboard: support multiple checktype badges	2025-12-26 19:34:23 +01:00
Username	adbe20dae3	cleanup: remove unused imports and dead code	2025-12-26 19:34:23 +01:00
Username	a20b5525f0	ppf: handle confidence field in proxy tuples	2025-12-26 19:34:22 +01:00
Username	481dc514fb	fetch: add IPv6, auth proxy, and confidence scoring support	2025-12-26 19:13:36 +01:00
Username	50f49a20ff	httpd: restrict /api/memory to localhost only	2025-12-26 19:12:39 +01:00
Username	07262e8b50	proxywatchd: cap threads by queue size, faster scale-down	2025-12-26 18:12:39 +01:00
Username	8ae639cb94	httpd: cache gc.get_objects and get_db_health calls	2025-12-26 18:12:31 +01:00
Username	b955281b51	config: set checktype to judges,head	2025-12-25 20:25:43 +01:00
Username	560230988d	support multiple checktypes with random selection	2025-12-25 20:23:05 +01:00
Username	755abc7f6e	dashboard: add queue ETA countdown to Worker Pool pane	2025-12-25 19:58:49 +01:00
Username	0424527e84	proxywatchd: add 0.3s delay before SSL fallback to HTTP	2025-12-25 19:49:59 +01:00
Username	3ac7305954	proxywatchd: persist MITM certificate stats across restarts ... Add save_state/load_state to MITMCertStats for JSON persistence. Stats saved periodically (5min) and at shutdown, loaded at startup.	2025-12-25 19:47:51 +01:00
Username	272eba0f05	scraper: reuse connections, cycle circuit on block	2025-12-25 19:26:23 +01:00
Username	68e8b88afa	tor: use random credentials for circuit isolation	2025-12-25 19:18:25 +01:00