426 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Username	92d6e57fb8	dockerfile: apply debian 10 security updates ... - add debian-security archive repository - run apt-get upgrade for all available patches - upgrade pip/setuptools/wheel to latest py2.7 versions reduces container vulnerabilities from 293 to 130	2026-01-18 09:14:48 +01:00
Username	d87ff73d95	httpd: remove memory profiling code ... Remove objgraph/pympler imports, gc.get_objects() caching, and memory_samples tracking. Keep basic RSS tracking for dashboard.	2026-01-17 19:25:33 +01:00
Username	12174b0d9d	fetch: fix LRU cache for python 2 compatibility	2026-01-08 09:05:59 +01:00
Username	8b606efa6d	docs: update project instructions	2026-01-08 09:05:44 +01:00
Username	ae0b11d60f	docs: update roadmap with completed items	2026-01-08 09:05:39 +01:00
Username	5da5f3025d	dashboard: update UI for queue status display	2026-01-08 09:05:34 +01:00
Username	2156319bad	ppf: worker heartbeat includes thread count	2026-01-08 09:05:30 +01:00
Username	1cb7d93a5f	proxywatchd: add ssl_only mode and schedule improvements ... - ssl_only mode: skip secondary check when SSL handshake fails - _build_due_sql(): unified query for proxies due testing - working_checktime/fail_retry_interval: new schedule formula - fail_retry_backoff: linear backoff option for failing proxies	2026-01-08 09:05:25 +01:00
Username	8272cf06e0	config: add verification and schedule settings ... - [verification] section: enabled, threads, batch_size, interval, max_queue - working_checktime: retest interval for working proxies (default: 300s) - fail_retry_interval: retry interval for failing proxies (default: 60s) - fail_retry_backoff: linear backoff for failures (default: True) - ssl_only: skip secondary check on SSL failure (default: False)	2026-01-08 09:05:20 +01:00
Username	64b3629585	dbs: add CDN filtering and verification tables ... - CDN_PREFIXES: filter Cloudflare, Fastly, Akamai, CloudFront, Google - is_cdn_ip(): check if IP belongs to known CDN ranges - insert_proxies(): skip CDN IPs with count in log message - verification tables: worker_results, verification_queue, worker_trust - queue_verification(): add proxies for manager re-testing - get_verification_stats(): queue size and trigger breakdown - get_all_worker_trust(): trust scores for all workers	2026-01-08 09:05:13 +01:00
Username	721a602dd9	misc: simplify tor proxy URL to avoid circuit exhaustion	2026-01-08 09:05:03 +01:00
Username	39731e25b3	docs: document batch API endpoint	2026-01-08 09:03:01 +01:00
Username	6cc903c924	httpd: add batch API endpoint and worker improvements ... - /api/dashboard: single endpoint returning stats + workers + countries - dashboard.js: use batch endpoint (2 requests -> 1 per poll cycle) - _get_workers_data: refactored from /api/workers for code reuse - worker verification: trust scoring based on result accuracy - fair distribution: dynamic batch sizing based on queue and workers - queue tracking: session progress, due/claimed/pending counts	2026-01-08 09:02:56 +01:00
Username	44604f1ce3	tests: add unit test infrastructure ... pytest-based test suite with fixtures for database testing. Covers misc.py utilities, dbs.py operations, and fetch.py validation. Includes mock_network.py for future network testing.	2026-01-08 01:42:38 +01:00
Username	c1ec5d593b	worker: check tor every 30s instead of exponential backoff	2025-12-28 18:41:05 +01:00
Username	966c0d641d	docs: mark low-effort tasks as completed	2025-12-28 17:25:06 +01:00
Username	dfb4739b66	proxywatchd: add __slots__ to hot objects for memory reduction	2025-12-28 17:23:51 +01:00
Username	480a652889	httpd: add stats export endpoint with CSV/JSON support	2025-12-28 17:23:44 +01:00
Username	1f09c75345	docs: add database context manager to completed work	2025-12-28 17:13:24 +01:00
Username	5a797a9b97	proxywatchd: use context manager for all DB operations	2025-12-28 17:11:56 +01:00
Username	9e2fc3e09d	docs: update roadmap and todo with recent changes	2025-12-28 17:00:52 +01:00
Username	e758ce7178	dashboard: add keyboard shortcuts and optimize polling ... - fetch.py: convert proxy validation cache to LRU with OrderedDict - thread-safe lock, move_to_end() on hits, evict oldest when full - dashboard.js: add keyboard shortcuts (r=refresh, 1-9=tabs, t=theme, p=pause) - dashboard.js: skip chart rendering for inactive tabs (reduces CPU)	2025-12-28 16:52:52 +01:00
Username	18ae73bfb8	httpd: add worker test rate tracking ... Track per-worker test rates using 120s sliding window. Display combined rate in dashboard and individual rates in worker cards.	2025-12-28 16:43:53 +01:00
Username	2bc00d3ebd	worker: check tor before claiming work	2025-12-28 16:09:40 +01:00
Username	0d7d2dce70	refactor: extract modules from proxywatchd.py ... Extract focused modules to reduce proxywatchd.py complexity: - stats.py: JudgeStats, Stats, regexes, ssl_targets (557 lines) - mitm.py: MITMCertStats, cert extraction functions (239 lines) - dns.py: socks4_resolve with TTL caching (86 lines) - job.py: PriorityJobQueue, calculate_priority (103 lines) proxywatchd.py reduced from 2488 to 1591 lines (-36%).	2025-12-28 15:45:24 +01:00
Username	35f24bb8b0	dashboard: refactor layout and add worker stats	2025-12-28 15:19:50 +01:00
Username	e89db20f5b	scraper: add Bing and Yahoo engines	2025-12-28 15:19:39 +01:00
Username	0fbfee2855	httpd: add worker registration and distributed testing API	2025-12-28 15:19:08 +01:00
Username	3b361916fa	fetch, dbs: minor refactoring	2025-12-28 15:18:42 +01:00
Username	50bc13a615	misc: add SSL protocol error detection and diag logging	2025-12-28 15:18:29 +01:00
Username	f4286ea515	ppf: remove num_targets param (removed in phase 2)	2025-12-28 15:16:52 +01:00
Username	9b44043237	add ssl_first: try SSL handshake before secondary check ... When ssl_first=1 (default), proxy validation first attempts an SSL handshake. If it fails, falls back to the configured secondary check (head, judges, or irc). This separates SSL capability detection from basic connectivity testing. New config options: - ssl_first: enable SSL-first pattern (default: 1) - checktype: secondary check type (head, judges, irc)	2025-12-28 14:56:46 +01:00
Username	9f782c3222	config: default to head checktype only	2025-12-28 14:45:27 +01:00
Username	1c8e3062b7	add configurable thread scaling and queue counter reset ... - Add scale_cooldown and scale_threshold config options - ThreadScaler now reads scaling params from config - Reset priority queue counter when queue empties to prevent unbounded growth	2025-12-28 14:37:32 +01:00
Username	d219cc567f	phase 2: code cleanup and simplification ... - Remove unused result_queue from WorkerThread and worker mode - Remove num_targets abstraction, simplify to single-target mode - Add _db_context() context manager for database connections - Refactor 5 call sites to use context manager (finish, init, cleanup_stale, periodic saves) - Mark _prep_db/_close_db as deprecated - Add __version__ = '2.0.0' to ppf.py - Add thread spawn stagger (0-100ms) in worker mode for Tor-friendly startup	2025-12-28 14:31:37 +01:00
Username	72a2dcdaf4	ppf: add worker mode with distributed testing ... - Add --worker mode for distributed proxy testing - Workers claim batches from manager, test via local Tor, submit results - Add --register to register new workers with manager - Add thread spawn stagger (0-100ms) to avoid overwhelming Tor - Verify Tor connectivity before claiming work - Add heartbeat and batch timeout handling - Track worker profiling state for dashboard display	2025-12-28 14:12:59 +01:00
Username	e46a3f6ef1	proxywatchd: add DNS cache TTL, IP validation, debug logging ... - DNS cache entries now expire after 1 hour (DNS_CACHE_TTL) - Add is_valid_ip() to validate extracted IPs have valid octets - Add debug logging to silent exception blocks (ASN lookup, socket disconnect) - Convert rwip() to static method with cleaner int() implementation - Add _sample_dbg() for sampled diagnostic logging - Add set_config() for worker mode imports - Remove unused tor_targets (tor checktype was removed) - Move tor_proxy_url to misc.py for shared use	2025-12-28 14:12:50 +01:00
Username	58d83ae52f	dashboard: always show SSL badge	2025-12-26 21:37:37 +01:00
Username	d9205614fc	proxywatchd: secondary check only for SSL handshake failures	2025-12-26 21:36:45 +01:00
Username	03dd277c54	dashboard: fix ssl badge after use_ssl config removal	2025-12-26 21:30:23 +01:00
Username	9f81e1e4af	watchd: remove use_ssl config, use HTTP for non-ssl checks	2025-12-26 21:25:52 +01:00
Username	f7a762331a	watchd: add tor checktype, use Tor API for secondary check	2025-12-26 21:20:16 +01:00
Username	d2bd7d4f34	fetch: retry with different Tor circuit on failure	2025-12-26 20:57:28 +01:00
Username	6ea201805f	watchd: configurable adaptive timeout per failure	2025-12-26 20:57:21 +01:00
Username	7232846b0f	ppf: add --reset flag to clear all state	2025-12-26 20:57:15 +01:00
Username	906d1b33ae	fetch: cache is_usable_proxy results	2025-12-26 20:04:01 +01:00
Username	1f41f3df5c	dashboard: pause polling when tab is hidden	2025-12-26 19:58:20 +01:00
Username	ad89eb262e	httpd: add rate limiting and security headers	2025-12-26 19:52:45 +01:00
Username	59fe2c4a14	dashboard: support multiple checktype badges	2025-12-26 19:34:23 +01:00
Username	adbe20dae3	cleanup: remove unused imports and dead code	2025-12-26 19:34:23 +01:00