diff --git a/ROADMAP.md b/ROADMAP.md index 4c32f9b..fd95607 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -179,14 +179,14 @@ PPF (Proxy Fetcher) is a Python 2 proxy scraping and validation framework design │ │ │ │ [x] Unify _known_proxies │ [x] Connection pooling │ │ [x] Graceful DB errors │ [x] Dynamic thread scaling │ -│ [x] Batch inserts │ [ ] Unit test infrastructure │ +│ [x] Batch inserts │ [x] Unit test infrastructure │ │ [x] WAL mode for SQLite │ [x] Latency tracking │ │ │ │ ├──────────────────────────┼──────────────────────────────────────────────────┤ │ LOW IMPACT / LOW EFFORT │ LOW IMPACT / HIGH EFFORT │ │ │ │ │ [x] Standardize logging │ [x] Geographic validation │ -│ [x] Config validation │ [x] Additional scrapers │ +│ [x] Config validation │ [x] Additional scrapers (Bing, Yahoo, Mojeek) │ │ [x] Export functionality │ [ ] API sources │ │ [x] Status output │ [ ] Protocol fingerprinting │ │ │ │ @@ -342,6 +342,7 @@ PPF (Proxy Fetcher) is a Python 2 proxy scraping and validation framework design - [x] Tab-aware chart rendering - skip expensive renders for hidden tabs - [x] Visibility API - pause polling when browser tab hidden - [x] Dark/muted-dark/light theme cycling +- [x] Stats export endpoint (/api/stats/export?format=json|csv) ### Proxy Validation Cache (Done) - [x] LRU cache for is_usable_proxy() using OrderedDict @@ -354,6 +355,20 @@ PPF (Proxy Fetcher) is a Python 2 proxy scraping and validation framework design - [x] Removed deprecated `_prep_db()` and `_close_db()` methods - [x] `fetch_rows()` now accepts db parameter for cleaner dependency injection +### Additional Search Engines (Done) +- [x] Bing and Yahoo engine implementations in scraper.py +- [x] Engine rotation for rate limit avoidance +- [x] engines.py module with SearchEngine base class + +### Worker Health Improvements (Done) +- [x] Tor connectivity check before workers claim work +- [x] Fixed interval Tor check (30s) instead of exponential backoff +- [x] Graceful handling when Tor unavailable + +### Memory Optimization (Done) +- [x] `__slots__` on ProxyTestState (27 attrs) and TargetTestJob (4 attrs) +- [x] Reduced per-object memory overhead for hot path objects + --- ## Technical Debt