add latency tracking and dynamic thread scaling

- dbs.py: add avg_latency, latency_samples columns with migration
- dbs.py: update_proxy_latency() with exponential moving average
- proxywatchd.py: ThreadScaler class for dynamic thread count
- proxywatchd.py: calculate/record latency for successful proxies
- proxywatchd.py: _spawn_thread(), _remove_thread(), _adjust_threads()
- scaler reports status alongside periodic stats
This commit is contained in:
Username
2025-12-21 00:08:19 +01:00
parent 1e43f50aa6
commit 79475c2bff
3 changed files with 221 additions and 79 deletions

88
TODO.md
View File

@@ -108,82 +108,28 @@ and report() methods. Integrated into main loop with configurable stats_interval
---
### [ ] 12. Dynamic Thread Scaling
### [x] 12. Dynamic Thread Scaling
**Problem:** Fixed thread count regardless of success rate or system load.
**Implementation:**
```python
# proxywatchd.py
class ThreadScaler:
"""Dynamically adjust thread count based on performance."""
def __init__(self, min_threads=5, max_threads=50):
self.min = min_threads
self.max = max_threads
self.current = min_threads
self.success_rate_window = []
def record_result(self, success):
self.success_rate_window.append(success)
if len(self.success_rate_window) > 100:
self.success_rate_window.pop(0)
def recommended_threads(self):
if len(self.success_rate_window) < 20:
return self.current
success_rate = sum(self.success_rate_window) / len(self.success_rate_window)
# High success rate -> can handle more threads
if success_rate > 0.7 and self.current < self.max:
return self.current + 5
# Low success rate -> reduce load
elif success_rate < 0.3 and self.current > self.min:
return self.current - 5
return self.current
```
**Files:** proxywatchd.py
**Effort:** Medium
**Risk:** Medium
**Completed.** Added dynamic thread scaling based on queue depth and success rate.
- ThreadScaler class in proxywatchd.py with should_scale(), status_line()
- Scales up when queue is deep (2x target) and success rate > 10%
- Scales down when queue is shallow or success rate drops
- Min/max threads derived from config.watchd.threads (1/4x to 2x)
- 30-second cooldown between scaling decisions
- _spawn_thread(), _remove_thread(), _adjust_threads() helper methods
- Scaler status reported alongside periodic stats
---
### [ ] 13. Latency Tracking
### [x] 13. Latency Tracking
**Problem:** No visibility into proxy speed. A working but slow proxy may be
less useful than a fast one.
**Implementation:**
```python
# dbs.py - add columns
# ALTER TABLE proxylist ADD COLUMN avg_latency REAL DEFAULT 0
# ALTER TABLE proxylist ADD COLUMN latency_samples INTEGER DEFAULT 0
def update_proxy_latency(proxydb, proxy, latency):
"""Update rolling average latency for proxy."""
row = proxydb.execute(
'SELECT avg_latency, latency_samples FROM proxylist WHERE proxy=?',
(proxy,)
).fetchone()
if row:
old_avg, samples = row
# Exponential moving average
new_avg = (old_avg * samples + latency) / (samples + 1)
new_samples = min(samples + 1, 100) # Cap at 100 samples
proxydb.execute(
'UPDATE proxylist SET avg_latency=?, latency_samples=? WHERE proxy=?',
(new_avg, new_samples, proxy)
)
```
**Files:** dbs.py, proxywatchd.py
**Effort:** Medium
**Risk:** Low
**Completed.** Added per-proxy latency tracking with exponential moving average.
- dbs.py: avg_latency, latency_samples columns added to proxylist schema
- dbs.py: _migrate_latency_columns() for backward-compatible migration
- dbs.py: update_proxy_latency() with EMA (alpha = 2/(samples+1))
- proxywatchd.py: ProxyTestState.last_latency_ms field
- proxywatchd.py: evaluate() calculates average latency from successful tests
- proxywatchd.py: submit_collected() records latency for passing proxies
---