feat: worker-driven discovery and validation tightening #1
362
ROADMAP.md
362
ROADMAP.md
@@ -45,83 +45,15 @@ PPF (Proxy Fetcher) is a Python 2 proxy scraping and validation framework design
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 1: Stability & Code Quality
|
## Open Work
|
||||||
|
|
||||||
**Objective:** Establish a solid, maintainable codebase
|
### Validation
|
||||||
|
|
||||||
### 1.1 Error Handling Improvements
|
|
||||||
|
|
||||||
| Task | Description | File(s) |
|
| Task | Description | File(s) |
|
||||||
|------|-------------|---------|
|
|------|-------------|---------|
|
||||||
| Add connection retry logic | Implement exponential backoff for failed connections | rocksock.py, fetch.py |
|
|
||||||
| Graceful database errors | Handle SQLite lock/busy errors with retry | mysqlite.py |
|
|
||||||
| Timeout standardization | Consistent timeout handling across all network ops | proxywatchd.py, fetch.py |
|
|
||||||
| Exception logging | Log exceptions with context, not just silently catch | all files |
|
|
||||||
|
|
||||||
### 1.2 Code Consolidation
|
|
||||||
|
|
||||||
| Task | Description | File(s) |
|
|
||||||
|------|-------------|---------|
|
|
||||||
| Unify _known_proxies | Single source of truth for known proxy cache | ppf.py, fetch.py |
|
|
||||||
| Extract proxy utils | Create proxy_utils.py with cleanse/validate functions | new file |
|
|
||||||
| Remove global config pattern | Pass config explicitly instead of set_config() | fetch.py |
|
|
||||||
| Standardize logging | Consistent _log() usage with levels across all modules | all files |
|
|
||||||
|
|
||||||
### 1.3 Testing Infrastructure
|
|
||||||
|
|
||||||
| Task | Description | File(s) |
|
|
||||||
|------|-------------|---------|
|
|
||||||
| Add unit tests | Test proxy parsing, URL extraction, IP validation | tests/ |
|
|
||||||
| Mock network layer | Allow testing without live network/Tor | tests/ |
|
|
||||||
| Validation test suite | Verify multi-target voting logic | tests/ |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 2: Performance Optimization
|
|
||||||
|
|
||||||
**Objective:** Improve throughput and resource efficiency
|
|
||||||
|
|
||||||
### 2.1 Connection Pooling
|
|
||||||
|
|
||||||
| Task | Description | File(s) |
|
|
||||||
|------|-------------|---------|
|
|
||||||
| Tor connection reuse | Pool Tor SOCKS connections instead of reconnecting | proxywatchd.py |
|
|
||||||
| HTTP keep-alive | Reuse connections to same target servers | http2.py |
|
|
||||||
| Connection warm-up | Pre-establish connections before job assignment | proxywatchd.py |
|
|
||||||
|
|
||||||
### 2.2 Database Optimization
|
|
||||||
|
|
||||||
| Task | Description | File(s) |
|
|
||||||
|------|-------------|---------|
|
|
||||||
| Batch inserts | Group INSERT operations (already partial) | dbs.py |
|
|
||||||
| Index optimization | Add indexes for frequent query patterns | dbs.py |
|
|
||||||
| WAL mode | Enable Write-Ahead Logging for better concurrency | mysqlite.py |
|
|
||||||
| Prepared statements | Cache compiled SQL statements | mysqlite.py |
|
|
||||||
|
|
||||||
### 2.3 Threading Improvements
|
|
||||||
|
|
||||||
| Task | Description | File(s) |
|
|
||||||
|------|-------------|---------|
|
|
||||||
| Dynamic thread scaling | Adjust thread count based on success rate | proxywatchd.py |
|
|
||||||
| Priority queue | Test high-value proxies (low fail count) first | proxywatchd.py |
|
|
||||||
| Stale proxy cleanup | Background thread to remove long-dead proxies | proxywatchd.py |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 3: Reliability & Accuracy
|
|
||||||
|
|
||||||
**Objective:** Improve proxy validation accuracy and system reliability
|
|
||||||
|
|
||||||
### 3.1 Enhanced Validation
|
|
||||||
|
|
||||||
| Task | Description | File(s) |
|
|
||||||
|------|-------------|---------|
|
|
||||||
| Latency tracking | Store and use connection latency metrics | proxywatchd.py, dbs.py |
|
|
||||||
| Geographic validation | Verify proxy actually routes through claimed location | proxywatchd.py |
|
|
||||||
| Protocol fingerprinting | Better SOCKS4/SOCKS5/HTTP detection | rocksock.py |
|
| Protocol fingerprinting | Better SOCKS4/SOCKS5/HTTP detection | rocksock.py |
|
||||||
| HTTPS/SSL testing | Validate SSL proxy capabilities | proxywatchd.py |
|
|
||||||
|
|
||||||
### 3.2 Target Management
|
### Target Management
|
||||||
|
|
||||||
| Task | Description | File(s) |
|
| Task | Description | File(s) |
|
||||||
|------|-------------|---------|
|
|------|-------------|---------|
|
||||||
@@ -129,245 +61,11 @@ PPF (Proxy Fetcher) is a Python 2 proxy scraping and validation framework design
|
|||||||
| Target health tracking | Remove unresponsive targets from pool | proxywatchd.py |
|
| Target health tracking | Remove unresponsive targets from pool | proxywatchd.py |
|
||||||
| Geographic target spread | Ensure targets span multiple regions | config.py |
|
| Geographic target spread | Ensure targets span multiple regions | config.py |
|
||||||
|
|
||||||
### 3.3 Failure Analysis
|
### Proxy Source Expansion
|
||||||
|
|
||||||
| Task | Description | File(s) |
|
| Task | Description | File(s) |
|
||||||
|------|-------------|---------|
|
|------|-------------|---------|
|
||||||
| Failure categorization | Distinguish timeout vs refused vs auth-fail | proxywatchd.py |
|
|
||||||
| Retry strategies | Different retry logic per failure type | proxywatchd.py |
|
|
||||||
| Dead proxy quarantine | Separate storage for likely-dead proxies | dbs.py |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 4: Features & Usability
|
|
||||||
|
|
||||||
**Objective:** Add useful features while maintaining simplicity
|
|
||||||
|
|
||||||
### 4.1 Reporting & Monitoring
|
|
||||||
|
|
||||||
| Task | Description | File(s) |
|
|
||||||
|------|-------------|---------|
|
|
||||||
| Statistics collection | Track success rates, throughput, latency | proxywatchd.py |
|
|
||||||
| Periodic status output | Log summary stats every N minutes | ppf.py, proxywatchd.py |
|
|
||||||
| Export functionality | Export working proxies to file (txt, json) | new: export.py |
|
|
||||||
|
|
||||||
### 4.2 Configuration
|
|
||||||
|
|
||||||
| Task | Description | File(s) |
|
|
||||||
|------|-------------|---------|
|
|
||||||
| Config validation | Validate config.ini on startup | config.py |
|
|
||||||
| Runtime reconfiguration | Reload config without restart (SIGHUP) | proxywatchd.py |
|
|
||||||
| Sensible defaults | Document and improve default values | config.py |
|
|
||||||
|
|
||||||
### 4.3 Proxy Source Expansion
|
|
||||||
|
|
||||||
| Task | Description | File(s) |
|
|
||||||
|------|-------------|---------|
|
|
||||||
| Additional scrapers | Support more search engines beyond Searx | scraper.py |
|
|
||||||
| API sources | Integrate free proxy API endpoints | new: api_sources.py |
|
| API sources | Integrate free proxy API endpoints | new: api_sources.py |
|
||||||
| Import formats | Support various proxy list formats | ppf.py |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Implementation Priority
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
||||||
│ Priority Matrix │
|
|
||||||
├──────────────────────────┬──────────────────────────────────────────────────┤
|
|
||||||
│ HIGH IMPACT / LOW EFFORT │ HIGH IMPACT / HIGH EFFORT │
|
|
||||||
│ │ │
|
|
||||||
│ [x] Unify _known_proxies │ [x] Connection pooling │
|
|
||||||
│ [x] Graceful DB errors │ [x] Dynamic thread scaling │
|
|
||||||
│ [x] Batch inserts │ [x] Unit test infrastructure │
|
|
||||||
│ [x] WAL mode for SQLite │ [x] Latency tracking │
|
|
||||||
│ │ │
|
|
||||||
├──────────────────────────┼──────────────────────────────────────────────────┤
|
|
||||||
│ LOW IMPACT / LOW EFFORT │ LOW IMPACT / HIGH EFFORT │
|
|
||||||
│ │ │
|
|
||||||
│ [x] Standardize logging │ [x] Geographic validation │
|
|
||||||
│ [x] Config validation │ [x] Additional scrapers (Bing, Yahoo, Mojeek) │
|
|
||||||
│ [x] Export functionality │ [ ] API sources │
|
|
||||||
│ [x] Status output │ [ ] Protocol fingerprinting │
|
|
||||||
│ │ │
|
|
||||||
└──────────────────────────┴──────────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Completed Work
|
|
||||||
|
|
||||||
### Multi-Target Validation (Done)
|
|
||||||
- [x] Work-stealing queue with shared Queue.Queue()
|
|
||||||
- [x] Multi-target validation (2/3 majority voting)
|
|
||||||
- [x] Interleaved testing (jobs shuffled across proxies)
|
|
||||||
- [x] ProxyTestState and TargetTestJob classes
|
|
||||||
|
|
||||||
### Code Cleanup (Done)
|
|
||||||
- [x] Removed dead HTTP server code from ppf.py
|
|
||||||
- [x] Removed dead gumbo code from soup_parser.py
|
|
||||||
- [x] Removed test code from comboparse.py
|
|
||||||
- [x] Removed unused functions from misc.py
|
|
||||||
- [x] Fixed IP/port cleansing in ppf.py extract_proxies()
|
|
||||||
- [x] Updated .gitignore, removed .pyc files
|
|
||||||
|
|
||||||
### Database Optimization (Done)
|
|
||||||
- [x] Enable SQLite WAL mode for better concurrency
|
|
||||||
- [x] Add indexes for common query patterns (failed, tested, proto, error, check_time)
|
|
||||||
- [x] Optimize batch inserts (remove redundant SELECT before INSERT OR IGNORE)
|
|
||||||
|
|
||||||
### Dependency Reduction (Done)
|
|
||||||
- [x] Make lxml optional (removed from requirements)
|
|
||||||
- [x] Make IP2Location optional (graceful fallback)
|
|
||||||
- [x] Add --nobs flag for stdlib HTMLParser fallback (bs4 optional)
|
|
||||||
|
|
||||||
### Rate Limiting & Stability (Done)
|
|
||||||
- [x] InstanceTracker class in scraper.py with exponential backoff
|
|
||||||
- [x] Configurable backoff_base, backoff_max, fail_threshold
|
|
||||||
- [x] Exception logging with context (replaced bare except blocks)
|
|
||||||
- [x] Unified _known_proxies cache in fetch.py
|
|
||||||
|
|
||||||
### Monitoring & Maintenance (Done)
|
|
||||||
- [x] Stats class in proxywatchd.py (tested/passed/failed tracking)
|
|
||||||
- [x] Periodic stats reporting (configurable stats_interval)
|
|
||||||
- [x] Stale proxy cleanup (cleanup_stale() with configurable stale_days)
|
|
||||||
- [x] Timeout config options (timeout_connect, timeout_read)
|
|
||||||
|
|
||||||
### Connection Pooling (Done)
|
|
||||||
- [x] TorHostState class tracking per-host health and latency
|
|
||||||
- [x] TorConnectionPool with worker affinity for circuit reuse
|
|
||||||
- [x] Exponential backoff (5s, 10s, 20s, 40s, max 60s) on failures
|
|
||||||
- [x] Pool warmup and health status reporting
|
|
||||||
|
|
||||||
### Priority Queue (Done)
|
|
||||||
- [x] PriorityJobQueue class with heap-based ordering
|
|
||||||
- [x] calculate_priority() assigns priority 0-4 by proxy state
|
|
||||||
- [x] New proxies tested first, high-fail proxies last
|
|
||||||
|
|
||||||
### Dynamic Thread Scaling (Done)
|
|
||||||
- [x] ThreadScaler class adjusts thread count dynamically
|
|
||||||
- [x] Scales up when queue deep and success rate acceptable
|
|
||||||
- [x] Scales down when queue shallow or success rate drops
|
|
||||||
- [x] Respects min/max bounds with cooldown period
|
|
||||||
|
|
||||||
### Latency Tracking (Done)
|
|
||||||
- [x] avg_latency, latency_samples columns in proxylist
|
|
||||||
- [x] Exponential moving average calculation
|
|
||||||
- [x] Migration function for existing databases
|
|
||||||
- [x] Latency recorded for successful proxy tests
|
|
||||||
|
|
||||||
### Container Support (Done)
|
|
||||||
- [x] Dockerfile with Python 2.7-slim base
|
|
||||||
- [x] docker-compose.yml for local development
|
|
||||||
- [x] Rootless podman deployment documentation
|
|
||||||
- [x] Volume mounts for persistent data
|
|
||||||
|
|
||||||
### Code Style (Done)
|
|
||||||
- [x] Normalized indentation (4-space, no tabs)
|
|
||||||
- [x] Removed dead code and unused imports
|
|
||||||
- [x] Added docstrings to classes and functions
|
|
||||||
- [x] Python 2/3 compatible imports (Queue/queue)
|
|
||||||
|
|
||||||
### Geographic Validation (Done)
|
|
||||||
- [x] IP2Location integration for country lookup
|
|
||||||
- [x] pyasn integration for ASN lookup
|
|
||||||
- [x] Graceful fallback when database files missing
|
|
||||||
- [x] Country codes displayed in test output: `(US)`, `(IN)`, etc.
|
|
||||||
- [x] Data files: IP2LOCATION-LITE-DB1.BIN, ipasn.dat
|
|
||||||
|
|
||||||
### SSL Proxy Testing (Done)
|
|
||||||
- [x] Default checktype changed to 'ssl'
|
|
||||||
- [x] ssl_targets list with major HTTPS sites
|
|
||||||
- [x] TLS handshake validation with certificate verification
|
|
||||||
- [x] Detects MITM proxies that intercept SSL connections
|
|
||||||
|
|
||||||
### Export Functionality (Done)
|
|
||||||
- [x] export.py CLI tool for exporting working proxies
|
|
||||||
- [x] Multiple formats: txt, json, csv, len (length-prefixed)
|
|
||||||
- [x] Filters: proto, country, anonymity, max_latency
|
|
||||||
- [x] Sort options: latency, added, tested, success
|
|
||||||
- [x] Output to stdout or file
|
|
||||||
|
|
||||||
### Web Dashboard (Done)
|
|
||||||
- [x] /dashboard endpoint with dark theme HTML UI
|
|
||||||
- [x] /api/stats endpoint for JSON runtime statistics
|
|
||||||
- [x] Auto-refresh with JavaScript fetch every 3 seconds
|
|
||||||
- [x] Stats provider callback from proxywatchd.py to httpd.py
|
|
||||||
- [x] Displays: tested/passed/success rate, thread count, uptime
|
|
||||||
- [x] Tor pool health: per-host latency, success rate, availability
|
|
||||||
- [x] Failure categories breakdown: timeout, proxy, ssl, closed
|
|
||||||
|
|
||||||
### Dashboard Enhancements v2 (Done)
|
|
||||||
- [x] Prominent check type badge in header (SSL/JUDGES/HTTP/IRC)
|
|
||||||
- [x] System monitor bar: load, memory, disk, process RSS
|
|
||||||
- [x] Anonymity breakdown: elite/anonymous/transparent counts
|
|
||||||
- [x] Database health: size, tested/hour, added/day, dead count
|
|
||||||
- [x] Enhanced Tor pool stats: requests, success rate, healthy nodes, latency
|
|
||||||
- [x] SQLite ANALYZE/VACUUM functions for query optimization
|
|
||||||
- [x] Lightweight design: client-side polling, minimal DOM updates
|
|
||||||
|
|
||||||
### Dashboard Enhancements v3 (Done)
|
|
||||||
- [x] Electric cyan theme with translucent glass-morphism effects
|
|
||||||
- [x] Unified wrapper styling (.chart-wrap, .histo-wrap, .stats-wrap, .lb-wrap, .pie-wrap)
|
|
||||||
- [x] Consistent backdrop-filter blur and electric glow borders
|
|
||||||
- [x] Tor Exit Nodes cards with hover effects (.tor-card)
|
|
||||||
- [x] Lighter background/tile color scheme (#1e2738 bg, #181f2a card)
|
|
||||||
- [x] Map endpoint restyled to match dashboard (electric cyan theme)
|
|
||||||
- [x] Map markers updated from gold to cyan for approximate locations
|
|
||||||
|
|
||||||
### Memory Profiling & Analysis (Done)
|
|
||||||
- [x] /api/memory endpoint with comprehensive memory stats
|
|
||||||
- [x] objgraph integration for object type counting
|
|
||||||
- [x] pympler integration for memory summaries
|
|
||||||
- [x] Memory sample history tracking (RSS over time)
|
|
||||||
- [x] Process memory from /proc/self/status (VmRSS, VmPeak, VmData, etc.)
|
|
||||||
- [x] GC statistics (collections, objects, thresholds)
|
|
||||||
|
|
||||||
### MITM Detection Optimization (Done)
|
|
||||||
- [x] MITM re-test skip optimization - avoid redundant SSL checks for known MITM proxies
|
|
||||||
- [x] mitm_retest_skipped stats counter for tracking optimization effectiveness
|
|
||||||
- [x] Content hash deduplication for stale proxy list detection
|
|
||||||
- [x] stale_count reset when content hash changes
|
|
||||||
|
|
||||||
### Distributed Workers (Done)
|
|
||||||
- [x] Worker registration and heartbeat system
|
|
||||||
- [x] /api/workers endpoint for worker status monitoring
|
|
||||||
- [x] Tor connectivity check before workers claim work
|
|
||||||
- [x] Worker test rate tracking with sliding window calculation
|
|
||||||
- [x] Combined rate aggregation across all workers
|
|
||||||
- [x] Dashboard worker cards showing per-worker stats
|
|
||||||
|
|
||||||
### Dashboard Performance (Done)
|
|
||||||
- [x] Keyboard shortcuts: r=refresh, 1-9=tabs, t=theme, p=pause
|
|
||||||
- [x] Tab-aware chart rendering - skip expensive renders for hidden tabs
|
|
||||||
- [x] Visibility API - pause polling when browser tab hidden
|
|
||||||
- [x] Dark/muted-dark/light theme cycling
|
|
||||||
- [x] Stats export endpoint (/api/stats/export?format=json|csv)
|
|
||||||
|
|
||||||
### Proxy Validation Cache (Done)
|
|
||||||
- [x] LRU cache for is_usable_proxy() using OrderedDict
|
|
||||||
- [x] Thread-safe with lock for concurrent access
|
|
||||||
- [x] Proper LRU eviction (move_to_end on hits, popitem oldest when full)
|
|
||||||
|
|
||||||
### Database Context Manager (Done)
|
|
||||||
- [x] Refactored all DB operations to use `_db_context()` context manager
|
|
||||||
- [x] Connections guaranteed to close even on exceptions
|
|
||||||
- [x] Removed deprecated `_prep_db()` and `_close_db()` methods
|
|
||||||
- [x] `fetch_rows()` now accepts db parameter for cleaner dependency injection
|
|
||||||
|
|
||||||
### Additional Search Engines (Done)
|
|
||||||
- [x] Bing and Yahoo engine implementations in scraper.py
|
|
||||||
- [x] Engine rotation for rate limit avoidance
|
|
||||||
- [x] engines.py module with SearchEngine base class
|
|
||||||
|
|
||||||
### Worker Health Improvements (Done)
|
|
||||||
- [x] Tor connectivity check before workers claim work
|
|
||||||
- [x] Fixed interval Tor check (30s) instead of exponential backoff
|
|
||||||
- [x] Graceful handling when Tor unavailable
|
|
||||||
|
|
||||||
### Memory Optimization (Done)
|
|
||||||
- [x] `__slots__` on ProxyTestState (27 attrs) and TargetTestJob (4 attrs)
|
|
||||||
- [x] Reduced per-object memory overhead for hot path objects
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -375,37 +73,33 @@ PPF (Proxy Fetcher) is a Python 2 proxy scraping and validation framework design
|
|||||||
|
|
||||||
| Item | Description | Risk |
|
| Item | Description | Risk |
|
||||||
|------|-------------|------|
|
|------|-------------|------|
|
||||||
| ~~Dual _known_proxies~~ | ~~ppf.py and fetch.py maintain separate caches~~ | **Resolved** |
|
|
||||||
| Global config in fetch.py | set_config() pattern is fragile | Low - works but not clean |
|
| Global config in fetch.py | set_config() pattern is fragile | Low - works but not clean |
|
||||||
| ~~No input validation~~ | ~~Proxy strings parsed without validation~~ | **Resolved** |
|
|
||||||
| ~~Silent exception catching~~ | ~~Some except: pass patterns hide errors~~ | **Resolved** |
|
|
||||||
| ~~Hardcoded timeouts~~ | ~~Various timeout values scattered in code~~ | **Resolved** |
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## File Reference
|
## File Reference
|
||||||
|
|
||||||
| File | Purpose | Status |
|
| File | Purpose |
|
||||||
|------|---------|--------|
|
|------|---------|
|
||||||
| ppf.py | Main URL harvester daemon | Active, cleaned |
|
| ppf.py | Main URL harvester daemon |
|
||||||
| proxywatchd.py | Proxy validation daemon | Active, enhanced |
|
| proxywatchd.py | Proxy validation daemon |
|
||||||
| scraper.py | Searx search integration | Active, cleaned |
|
| scraper.py | Searx search integration |
|
||||||
| fetch.py | HTTP fetching with proxy support | Active, LRU cache |
|
| fetch.py | HTTP fetching with proxy support |
|
||||||
| dbs.py | Database schema and inserts | Active |
|
| dbs.py | Database schema and inserts |
|
||||||
| mysqlite.py | SQLite wrapper | Active |
|
| mysqlite.py | SQLite wrapper |
|
||||||
| rocksock.py | Socket/proxy abstraction (3rd party) | Stable |
|
| rocksock.py | Socket/proxy abstraction (3rd party) |
|
||||||
| http2.py | HTTP client implementation | Stable |
|
| http2.py | HTTP client implementation |
|
||||||
| httpd.py | Web dashboard and REST API server | Active, enhanced |
|
| httpd.py | Web dashboard and REST API server |
|
||||||
| config.py | Configuration management | Active |
|
| config.py | Configuration management |
|
||||||
| comboparse.py | Config/arg parser framework | Stable, cleaned |
|
| comboparse.py | Config/arg parser framework |
|
||||||
| soup_parser.py | BeautifulSoup wrapper | Stable, cleaned |
|
| soup_parser.py | BeautifulSoup wrapper |
|
||||||
| misc.py | Utilities (timestamp, logging) | Stable, cleaned |
|
| misc.py | Utilities (timestamp, logging) |
|
||||||
| export.py | Proxy export CLI tool | Active |
|
| export.py | Proxy export CLI tool |
|
||||||
| engines.py | Search engine implementations | Active |
|
| engines.py | Search engine implementations |
|
||||||
| connection_pool.py | Tor connection pooling | Active |
|
| connection_pool.py | Tor connection pooling |
|
||||||
| network_stats.py | Network statistics tracking | Active |
|
| network_stats.py | Network statistics tracking |
|
||||||
| dns.py | DNS resolution with caching | Active |
|
| dns.py | DNS resolution with caching |
|
||||||
| mitm.py | MITM certificate detection | Active |
|
| mitm.py | MITM certificate detection |
|
||||||
| job.py | Priority job queue | Active |
|
| job.py | Priority job queue |
|
||||||
| static/dashboard.js | Dashboard frontend logic | Active, enhanced |
|
| static/dashboard.js | Dashboard frontend logic |
|
||||||
| static/dashboard.html | Dashboard HTML template | Active |
|
| static/dashboard.html | Dashboard HTML template |
|
||||||
|
|||||||
875
TODO.md
875
TODO.md
@@ -1,866 +1,73 @@
|
|||||||
# PPF Implementation Tasks
|
# PPF TODO
|
||||||
|
|
||||||
## Legend
|
## Optimization
|
||||||
|
|
||||||
```
|
### [ ] JSON Stats Response Caching
|
||||||
[ ] Not started
|
|
||||||
[~] In progress
|
|
||||||
[x] Completed
|
|
||||||
[!] Blocked/needs discussion
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
- Cache serialized JSON response with short TTL (1-2s)
|
||||||
|
|
||||||
## Immediate Priority (Next Sprint)
|
|
||||||
|
|
||||||
### [x] 1. Unify _known_proxies Cache
|
|
||||||
|
|
||||||
**Completed.** Added `init_known_proxies()`, `add_known_proxies()`, `is_known_proxy()`
|
|
||||||
to fetch.py. Updated ppf.py to use these functions instead of local cache.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 2. Graceful SQLite Error Handling
|
|
||||||
|
|
||||||
**Completed.** mysqlite.py now retries on "locked" errors with exponential backoff.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 3. Enable SQLite WAL Mode
|
|
||||||
|
|
||||||
**Completed.** mysqlite.py enables WAL mode and NORMAL synchronous on init.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 4. Batch Database Inserts
|
|
||||||
|
|
||||||
**Completed.** dbs.py uses executemany() for batch inserts.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 5. Add Database Indexes
|
|
||||||
|
|
||||||
**Completed.** dbs.py creates indexes on failed, tested, proto, error, check_time.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Short Term (This Month)
|
|
||||||
|
|
||||||
### [x] 6. Log Level Filtering
|
|
||||||
|
|
||||||
**Completed.** Added log level filtering with -q/--quiet and -v/--verbose CLI flags.
|
|
||||||
- misc.py: LOG_LEVELS dict, set_log_level(), get_log_level()
|
|
||||||
- config.py: Added -q/--quiet and -v/--verbose arguments
|
|
||||||
- Log levels: debug=0, info=1, warn=2, error=3
|
|
||||||
- --quiet: only show warn/error
|
|
||||||
- --verbose: show debug messages
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 7. Connection Timeout Standardization
|
|
||||||
|
|
||||||
**Completed.** Added timeout_connect and timeout_read to [common] section in config.py.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 8. Failure Categorization
|
|
||||||
|
|
||||||
**Completed.** Added failure categorization for proxy errors.
|
|
||||||
- misc.py: categorize_error() function, FAIL_* constants
|
|
||||||
- Categories: timeout, refused, auth, unreachable, dns, ssl, closed, proxy, other
|
|
||||||
- proxywatchd.py: Stats.record() now accepts category parameter
|
|
||||||
- Stats.report() shows failure breakdown by category
|
|
||||||
- ProxyTestState.evaluate() returns (success, category) tuple
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 9. Priority Queue for Proxy Testing
|
|
||||||
|
|
||||||
**Completed.** Added priority-based job scheduling for proxy tests.
|
|
||||||
- PriorityJobQueue class with heap-based ordering
|
|
||||||
- calculate_priority() assigns priority 0-4 based on proxy state
|
|
||||||
- Priority 0: New proxies (never tested)
|
|
||||||
- Priority 1: Working proxies (no failures)
|
|
||||||
- Priority 2: Low fail count (< 3)
|
|
||||||
- Priority 3-4: Medium/high fail count
|
|
||||||
- Integrated into prepare_jobs() for automatic prioritization
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 10. Periodic Statistics Output
|
|
||||||
|
|
||||||
**Completed.** Added Stats class to proxywatchd.py with record(), should_report(),
|
|
||||||
and report() methods. Integrated into main loop with configurable stats_interval.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Medium Term (Next Quarter)
|
|
||||||
|
|
||||||
### [x] 11. Tor Connection Pooling
|
|
||||||
|
|
||||||
**Completed.** Added connection pooling with worker-Tor affinity and health monitoring.
|
|
||||||
- connection_pool.py: TorHostState class tracks per-host health, latency, backoff
|
|
||||||
- connection_pool.py: TorConnectionPool with worker affinity, warmup, statistics
|
|
||||||
- proxywatchd.py: Workers get consistent Tor host assignment for circuit reuse
|
|
||||||
- Success/failure tracking with exponential backoff (5s, 10s, 20s, 40s, max 60s)
|
|
||||||
- Latency tracking with rolling averages
|
|
||||||
- Pool status reported alongside periodic stats
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 12. Dynamic Thread Scaling
|
|
||||||
|
|
||||||
**Completed.** Added dynamic thread scaling based on queue depth and success rate.
|
|
||||||
- ThreadScaler class in proxywatchd.py with should_scale(), status_line()
|
|
||||||
- Scales up when queue is deep (2x target) and success rate > 10%
|
|
||||||
- Scales down when queue is shallow or success rate drops
|
|
||||||
- Min/max threads derived from config.watchd.threads (1/4x to 2x)
|
|
||||||
- 30-second cooldown between scaling decisions
|
|
||||||
- _spawn_thread(), _remove_thread(), _adjust_threads() helper methods
|
|
||||||
- Scaler status reported alongside periodic stats
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 13. Latency Tracking
|
|
||||||
|
|
||||||
**Completed.** Added per-proxy latency tracking with exponential moving average.
|
|
||||||
- dbs.py: avg_latency, latency_samples columns added to proxylist schema
|
|
||||||
- dbs.py: _migrate_latency_columns() for backward-compatible migration
|
|
||||||
- dbs.py: update_proxy_latency() with EMA (alpha = 2/(samples+1))
|
|
||||||
- proxywatchd.py: ProxyTestState.last_latency_ms field
|
|
||||||
- proxywatchd.py: evaluate() calculates average latency from successful tests
|
|
||||||
- proxywatchd.py: submit_collected() records latency for passing proxies
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 14. Export Functionality
|
|
||||||
|
|
||||||
**Completed.** Added export.py CLI tool for exporting working proxies.
|
|
||||||
- Formats: txt (default), json, csv, len (length-prefixed)
|
|
||||||
- Filters: --proto, --country, --anonymity, --max-latency
|
|
||||||
- Options: --sort (latency, added, tested, success), --limit, --pretty
|
|
||||||
- Output: stdout or --output file
|
|
||||||
- Usage: `python export.py --proto http --country US --sort latency --limit 100`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 15. Unit Test Infrastructure
|
|
||||||
|
|
||||||
**Completed.** Added pytest-based test suite with comprehensive coverage.
|
|
||||||
- tests/conftest.py: Shared fixtures (temp_db, proxy_db, sample_proxies, etc.)
|
|
||||||
- tests/test_dbs.py: 40 tests for database operations (CDN filtering, latency, stats)
|
|
||||||
- tests/test_fetch.py: 60 tests for proxy validation (skipped in Python 3)
|
|
||||||
- tests/test_misc.py: 39 tests for utilities (timestamp, log levels, SSL errors)
|
|
||||||
- tests/mock_network.py: Network mocking infrastructure
|
|
||||||
|
|
||||||
```
|
|
||||||
Test Results: 79 passed, 60 skipped (Python 2 only)
|
|
||||||
Run with: python3 -m pytest tests/ -v
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Long Term (Future)
|
|
||||||
|
|
||||||
### [x] 16. Geographic Validation
|
|
||||||
|
|
||||||
**Completed.** Added IP2Location and pyasn for proxy geolocation.
|
|
||||||
- requirements.txt: Added IP2Location package
|
|
||||||
- proxywatchd.py: IP2Location for country lookup, pyasn for ASN lookup
|
|
||||||
- proxywatchd.py: Fixed ValueError handling when database files missing
|
|
||||||
- data/: IP2LOCATION-LITE-DB1.BIN (2.7M), ipasn.dat (23M)
|
|
||||||
- Output shows country codes: `http://1.2.3.4:8080 (US)` or `(IN)`, `(DE)`, etc.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 17. SSL Proxy Testing
|
|
||||||
|
|
||||||
**Completed.** Added SSL checktype for TLS handshake validation.
|
|
||||||
- config.py: Default checktype changed to 'ssl'
|
|
||||||
- proxywatchd.py: ssl_targets list with major HTTPS sites
|
|
||||||
- Validates TLS handshake with certificate verification
|
|
||||||
- Detects MITM proxies that intercept SSL connections
|
|
||||||
|
|
||||||
### [x] 18. Additional Search Engines
|
|
||||||
|
|
||||||
**Completed.** Added modular search engine architecture.
|
|
||||||
- engines.py: SearchEngine base class with build_url(), extract_urls(), is_rate_limited()
|
|
||||||
- Engines: DuckDuckGo, Startpage, Mojeek (UK), Qwant (FR), Yandex (RU), Ecosia, Brave
|
|
||||||
- Git hosters: GitHub, GitLab, Codeberg, Gitea
|
|
||||||
- scraper.py: EngineTracker class for multi-engine rate limiting
|
|
||||||
- Config: [scraper] engines, max_pages settings
|
|
||||||
- searx.instances: Updated with 51 active SearXNG instances
|
|
||||||
|
|
||||||
### [x] 19. REST API
|
|
||||||
|
|
||||||
**Completed.** Added HTTP API server for querying working proxies.
|
|
||||||
- httpd.py: ProxyAPIServer class with BaseHTTPServer
|
|
||||||
- Endpoints: /proxies, /proxies/count, /health
|
|
||||||
- Params: limit, proto, country, format (json/plain)
|
|
||||||
- Integrated into proxywatchd.py (starts when httpd.enabled=True)
|
|
||||||
- Config: [httpd] section with listenip, port, enabled
|
|
||||||
|
|
||||||
### [x] 20. Web Dashboard
|
|
||||||
|
|
||||||
**Completed.** Added web dashboard with live statistics.
|
|
||||||
- httpd.py: DASHBOARD_HTML template with dark theme UI
|
|
||||||
- Endpoint: /dashboard (HTML page with auto-refresh)
|
|
||||||
- Endpoint: /api/stats (JSON runtime statistics)
|
|
||||||
- Stats include: tested/passed counts, success rate, thread count, uptime
|
|
||||||
- Tor pool health: per-host latency, success rate, availability
|
|
||||||
- Failure categories: timeout, proxy, ssl, closed, etc.
|
|
||||||
- proxywatchd.py: get_runtime_stats() method provides stats callback
|
|
||||||
|
|
||||||
### [x] 21. Dashboard Enhancements (v2)
|
|
||||||
|
|
||||||
**Completed.** Major dashboard improvements for better visibility.
|
|
||||||
- Prominent check type badge in header (SSL/JUDGES/HTTP/IRC with color coding)
|
|
||||||
- System monitor bar: load average, memory usage, disk usage, process RSS
|
|
||||||
- Anonymity breakdown: elite/anonymous/transparent proxy counts
|
|
||||||
- Database health indicators: size, tested/hour, added/day, dead count
|
|
||||||
- Enhanced Tor pool: total requests, success rate, healthy nodes, avg latency
|
|
||||||
- SQLite ANALYZE/VACUUM functions for query optimization (dbs.py)
|
|
||||||
- Database statistics API (get_database_stats())
|
|
||||||
|
|
||||||
### [x] 22. Completion Queue Optimization
|
|
||||||
|
|
||||||
**Completed.** Eliminated polling bottleneck in proxy test collection.
|
|
||||||
- Added `completion_queue` for event-driven state signaling
|
|
||||||
- `ProxyTestState.record_result()` signals when all targets complete
|
|
||||||
- `collect_work()` drains queue instead of polling all pending states
|
|
||||||
- Changed `pending_states` from list to dict for O(1) removal
|
|
||||||
- Result: `is_complete()` eliminated from hot path, `collect_work()` 54x faster
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 23. Batch API Endpoint
|
|
||||||
|
|
||||||
**Completed.** Added `/api/dashboard` batch endpoint combining stats, workers, and countries.
|
|
||||||
|
|
||||||
**Implementation:**
|
|
||||||
- `httpd.py`: New `/api/dashboard` endpoint returns combined data in single response
|
|
||||||
- `httpd.py`: Refactored `/api/workers` to use `_get_workers_data()` helper method
|
|
||||||
- `dashboard.js`: Updated `fetchStats()` to use batch endpoint instead of multiple calls
|
|
||||||
|
|
||||||
**Response Structure:**
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"stats": { /* same as /api/stats */ },
|
|
||||||
"workers": { /* same as /api/workers */ },
|
|
||||||
"countries": { /* same as /api/countries */ }
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Benefit:**
|
|
||||||
- Reduces dashboard polling from 2 HTTP requests to 1 per poll cycle
|
|
||||||
- Lower RTT impact over SSH tunnels and high-latency connections
|
|
||||||
- Single database connection serves all data
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Profiling-Based Performance Optimizations
|
|
||||||
|
|
||||||
**Baseline:** 30-minute profiling session, 25.6M function calls, 1842s runtime
|
|
||||||
|
|
||||||
The following optimizations were identified through cProfile analysis. Each is
|
|
||||||
assessed for real-world impact based on measured data.
|
|
||||||
|
|
||||||
### [x] 1. SQLite Query Batching
|
|
||||||
|
|
||||||
**Completed.** Added batch update functions and optimized submit_collected().
|
|
||||||
|
|
||||||
**Implementation:**
|
|
||||||
- `batch_update_proxy_latency()`: Single SELECT with IN clause, compute EMA in Python,
|
|
||||||
batch UPDATE with executemany()
|
|
||||||
- `batch_update_proxy_anonymity()`: Batch all anonymity updates in single executemany()
|
|
||||||
- `submit_collected()`: Uses batch functions instead of per-proxy loops
|
|
||||||
|
|
||||||
**Previous State:**
|
|
||||||
- 18,182 execute() calls consuming 50.6s (2.7% of runtime)
|
|
||||||
- Individual UPDATE for each proxy latency and anonymity
|
|
||||||
|
|
||||||
**Improvement:**
|
|
||||||
- Reduced from N execute() + N commit() to 1 SELECT + 1 executemany() per batch
|
|
||||||
- Estimated 15-25% reduction in SQLite overhead
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 2. Proxy Validation Caching
|
|
||||||
|
|
||||||
**Completed.** Converted is_usable_proxy() cache to proper LRU with OrderedDict.
|
|
||||||
|
|
||||||
**Implementation:**
|
|
||||||
- fetch.py: Changed _proxy_valid_cache from dict to OrderedDict
|
|
||||||
- Added thread-safe _proxy_valid_cache_lock
|
|
||||||
- move_to_end() on cache hits to maintain LRU order
|
|
||||||
- Evict oldest entries when cache reaches max size (10,000)
|
|
||||||
- Proper LRU eviction instead of stopping inserts when full
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] 3. Regex Pattern Pre-compilation
|
|
||||||
|
|
||||||
**Completed.** Pre-compiled proxy extraction pattern at module load.
|
|
||||||
|
|
||||||
**Implementation:**
|
|
||||||
- `fetch.py`: Added `PROXY_PATTERN = re.compile(r'...')` at module level
|
|
||||||
- `extract_proxies()`: Changed `re.findall(pattern, ...)` to `PROXY_PATTERN.findall(...)`
|
|
||||||
- Pattern compiled once at import, not on each call
|
|
||||||
|
|
||||||
**Previous State:**
|
|
||||||
- `extract_proxies()`: 166 calls, 2.87s total (17.3ms each)
|
|
||||||
- Pattern recompiled on each call
|
|
||||||
|
|
||||||
**Improvement:**
|
|
||||||
- Eliminated per-call regex compilation overhead
|
|
||||||
- Estimated 30-50% reduction in extract_proxies() time
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [ ] 4. JSON Stats Response Caching
|
|
||||||
|
|
||||||
**Current State:**
|
|
||||||
- 1.9M calls to JSON encoder functions
|
|
||||||
- `_iterencode_dict`: 1.4s, `_iterencode_list`: 0.8s
|
|
||||||
- Dashboard polls every 3 seconds = 600 requests per 30min
|
|
||||||
- Most stats data unchanged between requests
|
|
||||||
|
|
||||||
**Proposed Change:**
|
|
||||||
- Cache serialized JSON response with short TTL (1-2 seconds)
|
|
||||||
- Only regenerate when underlying stats change
|
- Only regenerate when underlying stats change
|
||||||
- Use ETag/If-None-Match for client-side caching
|
- Use ETag/If-None-Match for client-side caching
|
||||||
|
- Savings: ~7-9s/hour. Low priority, only matters with frequent dashboard access.
|
||||||
|
|
||||||
**Assessment:**
|
### [ ] Object Pooling for Test States
|
||||||
```
|
|
||||||
Current cost: ~5.5s per 30min (JSON encoding overhead)
|
|
||||||
Potential saving: 60-80% = 3.3-4.4s per 30min = 6.6-8.8s/hour
|
|
||||||
Effort: Medium (add caching layer to httpd.py)
|
|
||||||
Risk: Low (stale stats for 1-2 seconds acceptable)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Verdict:** LOW PRIORITY. Only matters with frequent dashboard access.
|
- Pool ProxyTestState and TargetTestJob, reset and reuse
|
||||||
|
- Savings: ~11-15s/hour. **Not recommended** - high effort, medium risk, modest gain.
|
||||||
|
|
||||||
|
### [ ] SQLite Connection Reuse
|
||||||
|
|
||||||
|
- Persistent connection per thread with health checks
|
||||||
|
- Savings: ~0.3s/hour. **Not recommended** - negligible benefit.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### [ ] 5. Object Pooling for Test States
|
## Dashboard
|
||||||
|
|
||||||
**Current State:**
|
### [ ] Performance
|
||||||
- `__new__` calls: 43,413 at 10.1s total
|
|
||||||
- `ProxyTestState.__init__`: 18,150 calls, 0.87s
|
|
||||||
- `TargetTestJob` creation: similar overhead
|
|
||||||
- Objects created and discarded each test cycle
|
|
||||||
|
|
||||||
**Proposed Change:**
|
- Cache expensive DB queries (top countries, protocol breakdown)
|
||||||
- Implement object pool for ProxyTestState and TargetTestJob
|
- Lazy-load historical data (only when scrolled into view)
|
||||||
- Reset and reuse objects instead of creating new
|
- WebSocket option for push updates (reduce polling overhead)
|
||||||
- Pool size: 2x thread count
|
- Configurable refresh interval via URL param or localStorage
|
||||||
|
|
||||||
**Assessment:**
|
### [ ] Features
|
||||||
```
|
|
||||||
Current cost: ~11s per 30min = 22s/hour = 14.7min/day
|
|
||||||
Potential saving: 50-70% = 5.5-7.7s per 30min = 11-15s/hour = 7-10min/day
|
|
||||||
Effort: High (significant refactoring, reset logic needed)
|
|
||||||
Risk: Medium (state leakage bugs if reset incomplete)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Verdict:** NOT RECOMMENDED. High effort, medium risk, modest gain.
|
- Historical graphs (24h, 7d) using stats_history table
|
||||||
Python's object creation is already optimized. Focus elsewhere.
|
- Per-ASN performance analysis
|
||||||
|
- Alert thresholds (success rate < X%, MITM detected)
|
||||||
|
- Mobile-responsive improvements
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### [ ] 6. SQLite Connection Reuse
|
## Memory
|
||||||
|
|
||||||
**Current State:**
|
|
||||||
- 718 connection opens in 30min session
|
|
||||||
- Each open: 0.26ms (total 0.18s for connects)
|
|
||||||
- Connection per operation pattern in mysqlite.py
|
|
||||||
|
|
||||||
**Proposed Change:**
|
|
||||||
- Maintain persistent connection per thread
|
|
||||||
- Implement connection pool with health checks
|
|
||||||
- Reuse connections across operations
|
|
||||||
|
|
||||||
**Assessment:**
|
|
||||||
```
|
|
||||||
Current cost: 0.18s per 30min (connection overhead only)
|
|
||||||
Potential saving: 90% = 0.16s per 30min = 0.32s/hour
|
|
||||||
Effort: Medium (thread-local storage, lifecycle management)
|
|
||||||
Risk: Medium (connection state, locking issues)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Verdict:** NOT RECOMMENDED. Negligible time savings (0.16s per 30min).
|
|
||||||
SQLite's lightweight connections don't justify pooling complexity.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Summary: Optimization Priority Matrix
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────┬────────┬────────┬─────────┬───────────┐
|
|
||||||
│ Optimization │ Effort │ Risk │ Savings │ Status
|
|
||||||
├─────────────────────────────────────┼────────┼────────┼─────────┼───────────┤
|
|
||||||
│ 1. SQLite Query Batching │ Low │ Low │ 20-34s/h│ DONE
|
|
||||||
│ 2. Proxy Validation Caching │ V.Low │ None │ 5-8s/h │ DONE
|
|
||||||
│ 3. Regex Pre-compilation │ Low │ None │ 5-8s/h │ DONE
|
|
||||||
│ 4. JSON Response Caching │ Medium │ Low │ 7-9s/h │ Later
|
|
||||||
│ 5. Object Pooling │ High │ Medium │ 11-15s/h│ Skip
|
|
||||||
│ 6. SQLite Connection Reuse │ Medium │ Medium │ 0.3s/h │ Skip
|
|
||||||
└─────────────────────────────────────┴────────┴────────┴─────────┴───────────┘
|
|
||||||
|
|
||||||
Completed: 1 (SQLite Batching), 2 (Proxy Caching), 3 (Regex Pre-compilation)
|
|
||||||
Remaining: 4 (JSON Caching - Later)
|
|
||||||
|
|
||||||
Realized savings from completed optimizations:
|
|
||||||
Per hour: 25-42 seconds saved
|
|
||||||
Per day: 10-17 minutes saved
|
|
||||||
Per week: 1.2-2.0 hours saved
|
|
||||||
|
|
||||||
Note: 68.7% of runtime is socket I/O (recv/send) which cannot be optimized
|
|
||||||
without changing the fundamental network architecture. The optimizations
|
|
||||||
above target the remaining 31.3% of CPU-bound operations.
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Potential Dashboard Improvements
|
|
||||||
|
|
||||||
### [ ] Dashboard Performance Optimizations
|
|
||||||
|
|
||||||
**Goal:** Ensure dashboard remains lightweight and doesn't impact system performance.
|
|
||||||
|
|
||||||
**Current safeguards:**
|
|
||||||
- No polling on server side (client-initiated via fetch)
|
|
||||||
- 3-second refresh interval (configurable)
|
|
||||||
- Minimal DOM updates (targeted element updates, not full re-render)
|
|
||||||
- Static CSS/JS (no server-side templating per request)
|
|
||||||
- No persistent connections (stateless HTTP)
|
|
||||||
|
|
||||||
**Future considerations:**
|
|
||||||
- [x] Add rate limiting on /api/stats endpoint (300 req/60s sliding window)
|
|
||||||
- [ ] Cache expensive DB queries (top countries, protocol breakdown)
|
|
||||||
- [ ] Lazy-load historical data (only when scrolled into view)
|
|
||||||
- [ ] WebSocket option for push updates (reduce polling overhead)
|
|
||||||
- [ ] Configurable refresh interval via URL param or localStorage
|
|
||||||
- [x] Pause polling when browser tab not visible (Page Visibility API)
|
|
||||||
- [x] Skip chart rendering for inactive dashboard tabs (reduces CPU)
|
|
||||||
- [ ] Batch API endpoints - combine /api/stats, /api/workers, /api/countries into
|
|
||||||
single /api/dashboard call to reduce round-trips (helps SSH tunnel latency)
|
|
||||||
|
|
||||||
### [ ] Dashboard Feature Ideas
|
|
||||||
|
|
||||||
**Low priority - consider when time permits:**
|
|
||||||
- [x] Geographic map visualization - /map endpoint with Leaflet.js
|
|
||||||
- [x] Dark/light/muted theme toggle - t key cycles themes
|
|
||||||
- [x] Export stats as CSV/JSON from dashboard (/api/stats/export?format=json|csv)
|
|
||||||
- [ ] Historical graphs (24h, 7d) using stats_history table
|
|
||||||
- [ ] Per-ASN performance analysis
|
|
||||||
- [ ] Alert thresholds (success rate < X%, MITM detected)
|
|
||||||
- [ ] Mobile-responsive improvements
|
|
||||||
- [x] Keyboard shortcuts (r=refresh, 1-9=tabs, t=theme, p=pause)
|
|
||||||
|
|
||||||
### [x] Local JS Library Serving
|
|
||||||
|
|
||||||
**Completed.** All JavaScript libraries now served locally from /static/lib/ endpoint.
|
|
||||||
|
|
||||||
**Bundled libraries (static/lib/):**
|
|
||||||
- Leaflet.js 1.9.4 (leaflet.js, leaflet.css)
|
|
||||||
- Leaflet.markercluster (MarkerCluster.Default.css)
|
|
||||||
- Chart.js 4.x (chart.umd.min.js)
|
|
||||||
- uPlot (uPlot.iife.min.js, uPlot.min.css)
|
|
||||||
|
|
||||||
**Candidate libraries for future enhancements:**
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────────┬─────────┬───────────────────────────────────────────────┐
|
|
||||||
│ Library │ Size │ Use Case
|
|
||||||
├─────────────────┼─────────┼───────────────────────────────────────────────┤
|
|
||||||
│ Chart.js │ 65 KB │ Line/bar/pie charts (simpler API than D3)
|
|
||||||
│ uPlot │ 15 KB │ Fast time-series charts (minimal, performant)
|
|
||||||
│ ApexCharts │ 125 KB │ Modern charts with animations
|
|
||||||
│ Frappe Charts │ 25 KB │ Simple, modern SVG charts
|
|
||||||
│ Sparkline │ 2 KB │ Tiny inline charts (already have custom impl)
|
|
||||||
├─────────────────┼─────────┼───────────────────────────────────────────────┤
|
|
||||||
│ D3.js │ 85 KB │ Full control, complex visualizations
|
|
||||||
│ D3-geo │ 30 KB │ Geographic projections (alternative to Leaflet)
|
|
||||||
├─────────────────┼─────────┼───────────────────────────────────────────────┤
|
|
||||||
│ Leaflet │ 40 KB │ Interactive maps (already using)
|
|
||||||
│ Leaflet.heat │ 5 KB │ Heatmap layer for proxy density
|
|
||||||
│ Leaflet.cluster │ 10 KB │ Marker clustering for many points
|
|
||||||
└─────────────────┴─────────┴───────────────────────────────────────────────┘
|
|
||||||
|
|
||||||
Recommendations:
|
|
||||||
● uPlot - Best for time-series (rate history, success rate history)
|
|
||||||
● Chart.js - Best for pie/bar charts (failure breakdown, protocol stats)
|
|
||||||
● Leaflet - Keep for maps, add heatmap plugin for density viz
|
|
||||||
```
|
|
||||||
|
|
||||||
**Current custom implementations (no library):**
|
|
||||||
- Sparkline charts (Test Rate History, Success Rate History) - inline SVG
|
|
||||||
- Histogram bars (Response Time Distribution) - CSS divs
|
|
||||||
- Pie charts (Failure Breakdown, Protocol Stats) - CSS conic-gradient
|
|
||||||
|
|
||||||
**Decision:** Current custom implementations are lightweight and sufficient.
|
|
||||||
Add libraries only when custom becomes unmaintainable or new features needed.
|
|
||||||
|
|
||||||
### [ ] Memory Optimization Candidates
|
|
||||||
|
|
||||||
**Based on memory analysis (production metrics):**
|
|
||||||
```
|
|
||||||
Current State (260k queue):
|
|
||||||
Start RSS: 442 MB
|
|
||||||
Current RSS: 1,615 MB
|
|
||||||
Per-job: ~4.5 KB overhead
|
|
||||||
|
|
||||||
Object Distribution:
|
|
||||||
259,863 TargetTestJob (1 per job)
|
|
||||||
259,863 ProxyTestState (1 per job)
|
|
||||||
259,950 LockType (1 per job - threading locks)
|
|
||||||
523,395 dict (2 per job - state + metadata)
|
|
||||||
522,807 list (2 per job - results + targets)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Potential optimizations:**
|
|
||||||
- [ ] Lock consolidation - reduce per-proxy locks (260k LockType objects)
|
- [ ] Lock consolidation - reduce per-proxy locks (260k LockType objects)
|
||||||
- [ ] Leaner state objects - reduce dict/list count per job
|
- [ ] Leaner state objects - reduce dict/list count per job
|
||||||
- [x] Slot-based classes - use `__slots__` on ProxyTestState (27 attrs), TargetTestJob (4 attrs)
|
|
||||||
- [ ] Object pooling - reuse ProxyTestState/TargetTestJob objects (not recommended)
|
|
||||||
|
|
||||||
**Verdict:** Memory scales linearly with queue (~4.5 KB/job). No leaks detected.
|
Memory scales linearly with queue (~4.5 KB/job). No leaks detected.
|
||||||
Current usage acceptable for production workloads. Optimize only if memory
|
Optimize only if memory becomes a constraint.
|
||||||
becomes a constraint.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Completed
|
## Known Issues
|
||||||
|
|
||||||
### [x] Work-Stealing Queue
|
### [!] Podman Container Metadata Disappears
|
||||||
- Implemented shared Queue.Queue() for job distribution
|
|
||||||
- Workers pull from shared queue instead of pre-assigned lists
|
|
||||||
- Better utilization across threads
|
|
||||||
|
|
||||||
### [x] Multi-Target Validation
|
`podman ps -a` shows empty even though process is running. Service functions
|
||||||
- Test each proxy against 3 random targets
|
correctly despite missing metadata. Monitor via `ss -tlnp`, `ps aux`, or
|
||||||
- 2/3 majority required for success
|
`curl localhost:8081/health`. Low impact.
|
||||||
- Reduces false negatives from single target failures
|
|
||||||
|
|
||||||
### [x] Interleaved Testing
|
|
||||||
- Jobs shuffled across all proxies before queueing
|
|
||||||
- Prevents burst of 3 connections to same proxy
|
|
||||||
- ProxyTestState accumulates results from TargetTestJobs
|
|
||||||
|
|
||||||
### [x] Code Cleanup
|
|
||||||
- Removed 93 lines dead HTTP server code (ppf.py)
|
|
||||||
- Removed dead gumbo parser (soup_parser.py)
|
|
||||||
- Removed test code (comboparse.py)
|
|
||||||
- Removed unused functions (misc.py)
|
|
||||||
- Fixed IP/port cleansing (ppf.py)
|
|
||||||
- Updated .gitignore
|
|
||||||
|
|
||||||
### [x] Rate Limiting & Instance Tracking (scraper.py)
|
|
||||||
- InstanceTracker class with exponential backoff
|
|
||||||
- Configurable backoff_base, backoff_max, fail_threshold
|
|
||||||
- Instance cycling when rate limited
|
|
||||||
|
|
||||||
### [x] Exception Logging with Context
|
|
||||||
- Replaced bare `except:` with typed exceptions across all files
|
|
||||||
- Added context logging to exception handlers (e.g., URL, error message)
|
|
||||||
|
|
||||||
### [x] Timeout Standardization
|
|
||||||
- Added timeout_connect, timeout_read to [common] config section
|
|
||||||
- Added stale_days, stats_interval to [watchd] config section
|
|
||||||
|
|
||||||
### [x] Periodic Stats & Stale Cleanup (proxywatchd.py)
|
|
||||||
- Stats class tracks tested/passed/failed with thread-safe counters
|
|
||||||
- Configurable stats_interval (default: 300s)
|
|
||||||
- cleanup_stale() removes dead proxies older than stale_days (default: 30)
|
|
||||||
|
|
||||||
### [x] Unified Proxy Cache
|
|
||||||
- Moved _known_proxies to fetch.py with helper functions
|
|
||||||
- init_known_proxies(), add_known_proxies(), is_known_proxy()
|
|
||||||
- ppf.py now uses shared cache via fetch module
|
|
||||||
|
|
||||||
### [x] Config Validation
|
|
||||||
- config.py: validate() method checks config values on startup
|
|
||||||
- Validates: port ranges, timeout values, thread counts, engine names
|
|
||||||
- Warns on missing source_file, unknown engines
|
|
||||||
- Errors on unwritable database directories
|
|
||||||
- Integrated into ppf.py, proxywatchd.py, scraper.py main entry points
|
|
||||||
|
|
||||||
### [x] Profiling Support
|
|
||||||
- config.py: Added --profile CLI argument
|
|
||||||
- ppf.py: Refactored main logic into main() function
|
|
||||||
- ppf.py: cProfile wrapper with stats output to profile.stats
|
|
||||||
- Prints top 20 functions by cumulative time on exit
|
|
||||||
- Usage: `python2 ppf.py --profile`
|
|
||||||
|
|
||||||
### [x] SIGTERM Graceful Shutdown
|
|
||||||
- ppf.py: Added signal handler converting SIGTERM to KeyboardInterrupt
|
|
||||||
- Ensures profile stats are written before container exit
|
|
||||||
- Allows clean thread shutdown in containerized environments
|
|
||||||
- Podman stop now triggers proper cleanup instead of SIGKILL
|
|
||||||
|
|
||||||
### [x] Unicode Exception Handling (Python 2)
|
|
||||||
- Problem: `repr(e)` on exceptions with unicode content caused encoding errors
|
|
||||||
- Files affected: ppf.py, scraper.py (3 exception handlers)
|
|
||||||
- Solution: Check `isinstance(err_msg, unicode)` then encode with 'backslashreplace'
|
|
||||||
- Pattern applied:
|
|
||||||
```python
|
|
||||||
try:
|
|
||||||
err_msg = repr(e)
|
|
||||||
if isinstance(err_msg, unicode):
|
|
||||||
err_msg = err_msg.encode('ascii', 'backslashreplace')
|
|
||||||
except:
|
|
||||||
err_msg = type(e).__name__
|
|
||||||
```
|
|
||||||
- Handles Korean/CJK characters in search queries without crashing
|
|
||||||
|
|
||||||
### [x] Interactive World Map (/map endpoint)
|
|
||||||
- Added Leaflet.js interactive map showing proxy distribution by country
|
|
||||||
- Modern glassmorphism UI with `backdrop-filter: blur(12px)`
|
|
||||||
- CartoDB dark tiles for dark theme
|
|
||||||
- Circle markers sized proportionally to proxy count per country
|
|
||||||
- Hover effects with smooth transitions
|
|
||||||
- Stats overlay showing total countries/proxies
|
|
||||||
- Legend with proxy count scale
|
|
||||||
- Country coordinates and names lookup tables
|
|
||||||
|
|
||||||
### [x] Dashboard v3 - Electric Cyan Theme
|
|
||||||
- Translucent glass-morphism effects with `backdrop-filter: blur()`
|
|
||||||
- Electric cyan glow borders `rgba(56,189,248,...)` on all graph wrappers
|
|
||||||
- Gradient overlays using `::before` pseudo-elements
|
|
||||||
- Unified styling across: .chart-wrap, .histo-wrap, .stats-wrap, .lb-wrap, .pie-wrap
|
|
||||||
- New .tor-card wrapper for Tor Exit Nodes with hover effects
|
|
||||||
- Lighter background color scheme (#1e2738 bg, #181f2a card)
|
|
||||||
|
|
||||||
### [x] Map Endpoint Styling Update
|
|
||||||
- Converted from gold/bronze theme (#c8b48c) to electric cyan (#38bdf8)
|
|
||||||
- Glass panels with electric glow matching dashboard
|
|
||||||
- Map markers for approximate locations now cyan instead of gold
|
|
||||||
- Unified map_bg color with dashboard background (#1e2738)
|
|
||||||
- Updated Leaflet controls, popups, and legend to cyan theme
|
|
||||||
|
|
||||||
### [x] MITM Re-test Optimization
|
|
||||||
- Skip redundant SSL checks for proxies already known to be MITM
|
|
||||||
- Added `mitm_retest_skipped` counter to Stats class
|
|
||||||
- Optimization in `_try_ssl_check()` checks existing MITM flag before testing
|
|
||||||
- Avoids 6k+ unnecessary re-tests per session (based on production metrics)
|
|
||||||
|
|
||||||
### [x] Memory Profiling Endpoint
|
|
||||||
- /api/memory endpoint with comprehensive memory analysis
|
|
||||||
- objgraph integration for object type distribution
|
|
||||||
- pympler integration for memory summaries
|
|
||||||
- Memory sample history tracking (RSS over time)
|
|
||||||
- Process memory from /proc/self/status
|
|
||||||
- GC statistics and collection counts
|
|
||||||
|
|
||||||
### [x] Database Context Manager Refactoring
|
|
||||||
- Refactored all DB operations to use `_db_context()` context manager
|
|
||||||
- `prepare_jobs()`, `submit_collected()`, `_run()` now use `with self._db_context() as db:`
|
|
||||||
- `fetch_rows()` accepts db parameter for dependency injection
|
|
||||||
- Removed deprecated `_prep_db()` and `_close_db()` methods
|
|
||||||
- Connections guaranteed to close even on exceptions
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Deployment Troubleshooting Log
|
## Container Debugging Checklist
|
||||||
|
|
||||||
### [x] Container Crash on Startup (2024-12-24)
|
|
||||||
|
|
||||||
**Symptoms:**
|
|
||||||
- Container starts then immediately disappears
|
|
||||||
- `podman ps` shows no running containers
|
|
||||||
- `podman logs ppf` returns "no such container"
|
|
||||||
- Port 8081 not listening
|
|
||||||
|
|
||||||
**Debugging Process:**
|
|
||||||
|
|
||||||
1. **Initial diagnosis** - SSH to odin, checked container state:
|
|
||||||
```bash
|
|
||||||
sudo -u podman podman ps -a # Empty
|
|
||||||
sudo ss -tlnp | grep 8081 # Nothing listening
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Ran container in foreground** to capture output:
|
|
||||||
```bash
|
|
||||||
sudo -u podman bash -c 'cd /home/podman/ppf && \
|
|
||||||
timeout 25 podman run --rm --name ppf --network=host \
|
|
||||||
-v ./src:/app:ro -v ./data:/app/data \
|
|
||||||
-v ./config.ini:/app/config.ini:ro \
|
|
||||||
localhost/ppf python2 -u proxywatchd.py 2>&1'
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Found the error** in httpd thread startup:
|
|
||||||
```
|
|
||||||
error: [Errno 98] Address already in use: ('0.0.0.0', 8081)
|
|
||||||
```
|
|
||||||
Container started, httpd failed to bind, process continued but HTTP unavailable.
|
|
||||||
|
|
||||||
4. **Identified root cause** - orphaned processes from previous debug attempts:
|
|
||||||
```bash
|
|
||||||
ps aux | grep -E "[p]pf|[p]roxy"
|
|
||||||
# Found: python2 ppf.py (PID 6421) still running, holding port 8081
|
|
||||||
# Found: conmon, timeout, bash processes from stale container
|
|
||||||
```
|
|
||||||
|
|
||||||
5. **Why orphans existed:**
|
|
||||||
- Previous `timeout 15 podman run` commands timed out
|
|
||||||
- `podman rm -f` doesn't kill processes when container metadata is corrupted
|
|
||||||
- Orphaned python2 process kept running with port bound
|
|
||||||
|
|
||||||
**Root Cause:**
|
|
||||||
Stale container processes from interrupted debug sessions held port 8081.
|
|
||||||
The container started successfully but httpd thread failed to bind,
|
|
||||||
causing silent failure (no HTTP endpoints) while proxy testing continued.
|
|
||||||
|
|
||||||
**Fix Applied:**
|
|
||||||
```bash
|
|
||||||
# Force kill all orphaned processes
|
|
||||||
sudo pkill -9 -f "ppf.py"
|
|
||||||
sudo pkill -9 -f "proxywatchd.py"
|
|
||||||
sudo pkill -9 -f "conmon.*ppf"
|
|
||||||
sleep 2
|
|
||||||
|
|
||||||
# Verify port is free
|
|
||||||
sudo ss -tlnp | grep 8081 # Should show nothing
|
|
||||||
|
|
||||||
# Clean podman state
|
|
||||||
sudo -u podman podman rm -f -a
|
|
||||||
sudo -u podman podman container prune -f
|
|
||||||
|
|
||||||
# Start fresh
|
|
||||||
sudo -u podman bash -c 'cd /home/podman/ppf && \
|
|
||||||
podman run -d --rm --name ppf --network=host \
|
|
||||||
-v ./src:/app:ro -v ./data:/app/data \
|
|
||||||
-v ./config.ini:/app/config.ini:ro \
|
|
||||||
localhost/ppf python2 -u proxywatchd.py'
|
|
||||||
```
|
|
||||||
|
|
||||||
**Verification:**
|
|
||||||
```bash
|
|
||||||
curl -sf http://localhost:8081/health
|
|
||||||
# {"status": "ok", "timestamp": 1766573885}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Prevention:**
|
|
||||||
- Use `podman-compose` for reliable container management
|
|
||||||
- Use `pkill -9 -f` to kill orphaned processes before restart
|
|
||||||
- Check port availability before starting: `ss -tlnp | grep 8081`
|
|
||||||
- Run container foreground first to capture startup errors
|
|
||||||
|
|
||||||
**Correct Deployment Procedure:**
|
|
||||||
```bash
|
|
||||||
# As root or with sudo
|
|
||||||
sudo -i -u podman bash
|
|
||||||
cd /home/podman/ppf
|
|
||||||
podman-compose down
|
|
||||||
podman-compose up -d
|
|
||||||
podman ps
|
|
||||||
podman logs -f ppf
|
|
||||||
```
|
|
||||||
|
|
||||||
**docker-compose.yml (updated):**
|
|
||||||
```yaml
|
|
||||||
version: '3.8'
|
|
||||||
|
|
||||||
services:
|
|
||||||
ppf:
|
|
||||||
image: localhost/ppf:latest
|
|
||||||
container_name: ppf
|
|
||||||
network_mode: host
|
|
||||||
volumes:
|
|
||||||
- ./src:/app:ro
|
|
||||||
- ./data:/app/data
|
|
||||||
- ./config.ini:/app/config.ini:ro
|
|
||||||
command: python2 -u proxywatchd.py
|
|
||||||
restart: unless-stopped
|
|
||||||
environment:
|
|
||||||
- PYTHONUNBUFFERED=1
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [x] SSH Connection Flooding / fail2ban (2024-12-24)
|
|
||||||
|
|
||||||
**Symptoms:**
|
|
||||||
- SSH connections timing out or reset
|
|
||||||
- "Connection refused" errors
|
|
||||||
- Intermittent access to odin
|
|
||||||
|
|
||||||
**Root Cause:**
|
|
||||||
Multiple individual SSH commands triggered fail2ban rate limiting.
|
|
||||||
|
|
||||||
**Fix Applied:**
|
|
||||||
Created `~/.claude/rules/ssh-usage.md` with batching best practices.
|
|
||||||
|
|
||||||
**Key Pattern:**
|
|
||||||
```bash
|
|
||||||
# BAD: 5 separate connections
|
|
||||||
ssh host 'cmd1'
|
|
||||||
ssh host 'cmd2'
|
|
||||||
ssh host 'cmd3'
|
|
||||||
|
|
||||||
# GOOD: 1 connection, all commands
|
|
||||||
ssh host bash <<'EOF'
|
|
||||||
cmd1
|
|
||||||
cmd2
|
|
||||||
cmd3
|
|
||||||
EOF
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### [!] Podman Container Metadata Disappears (2024-12-24)
|
|
||||||
|
|
||||||
**Symptoms:**
|
|
||||||
- `podman ps -a` shows empty even though process is running
|
|
||||||
- `podman logs ppf` returns "no such container"
|
|
||||||
- Port is listening and service responds to health checks
|
|
||||||
|
|
||||||
**Observed Behavior:**
|
|
||||||
```
|
|
||||||
# Container starts
|
|
||||||
podman run -d --name ppf ...
|
|
||||||
# Returns container ID: dc55f0a218b7...
|
|
||||||
|
|
||||||
# Immediately after
|
|
||||||
podman ps -a # Empty!
|
|
||||||
ss -tlnp | grep 8081 # Shows python2 listening
|
|
||||||
curl localhost:8081/health # {"status": "ok"}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Analysis:**
|
|
||||||
- The process runs correctly inside the container namespace
|
|
||||||
- Container metadata in podman's database is lost/corrupted
|
|
||||||
- May be related to `--rm` flag interaction with detached mode
|
|
||||||
- Rootless podman with overlayfs can have state sync issues
|
|
||||||
|
|
||||||
**Workaround:**
|
|
||||||
Service works despite missing metadata. Monitor via:
|
|
||||||
- `ss -tlnp | grep 8081` - port listening
|
|
||||||
- `ps aux | grep proxywatchd` - process running
|
|
||||||
- `curl localhost:8081/health` - service responding
|
|
||||||
|
|
||||||
**Impact:** Low. Service functions correctly. Only `podman logs` unavailable.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Container Debugging Checklist
|
|
||||||
|
|
||||||
When container fails to start or crashes:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
┌───┬─────────────────────────────────────────────────────────────────────────┐
|
1. Check for orphans: ps aux | grep -E "[p]rocess_name"
|
||||||
│ 1 │ Check for orphans: ps aux | grep -E "[p]rocess_name"
|
2. Check port conflicts: ss -tlnp | grep PORT
|
||||||
│ 2 │ Check port conflicts: ss -tlnp | grep PORT
|
3. Run foreground: podman run --rm (no -d) to see output
|
||||||
│ 3 │ Run foreground: podman run --rm (no -d) to see output
|
4. Check podman state: podman ps -a
|
||||||
│ 4 │ Check podman state: podman ps -a
|
5. Clean stale: pkill -9 -f "pattern" && podman rm -f -a
|
||||||
│ 5 │ Clean stale: pkill -9 -f "pattern" && podman rm -f -a
|
6. Verify deps: config files, data dirs, volumes exist
|
||||||
│ 6 │ Verify deps: config files, data dirs, volumes exist
|
7. Check logs: podman logs container_name 2>&1 | tail -50
|
||||||
│ 7 │ Check logs: podman logs container_name 2>&1 | tail -50
|
8. Health check: curl -sf http://localhost:PORT/health
|
||||||
│ 8 │ Health check: curl -sf http://localhost:PORT/health
|
|
||||||
└───┴─────────────────────────────────────────────────────────────────────────┘
|
|
||||||
|
|
||||||
Note: If podman ps shows empty but port is listening and health check passes,
|
|
||||||
the service is running correctly despite metadata issues. See "Podman Container
|
|
||||||
Metadata Disappears" section above.
|
|
||||||
```
|
```
|
||||||
|
|||||||
Reference in New Issue
Block a user