add latency tracking and dynamic thread scaling
- dbs.py: add avg_latency, latency_samples columns with migration - dbs.py: update_proxy_latency() with exponential moving average - proxywatchd.py: ThreadScaler class for dynamic thread count - proxywatchd.py: calculate/record latency for successful proxies - proxywatchd.py: _spawn_thread(), _remove_thread(), _adjust_threads() - scaler reports status alongside periodic stats
This commit is contained in:
88
TODO.md
88
TODO.md
@@ -108,82 +108,28 @@ and report() methods. Integrated into main loop with configurable stats_interval
|
||||
|
||||
---
|
||||
|
||||
### [ ] 12. Dynamic Thread Scaling
|
||||
### [x] 12. Dynamic Thread Scaling
|
||||
|
||||
**Problem:** Fixed thread count regardless of success rate or system load.
|
||||
|
||||
**Implementation:**
|
||||
```python
|
||||
# proxywatchd.py
|
||||
class ThreadScaler:
|
||||
"""Dynamically adjust thread count based on performance."""
|
||||
|
||||
def __init__(self, min_threads=5, max_threads=50):
|
||||
self.min = min_threads
|
||||
self.max = max_threads
|
||||
self.current = min_threads
|
||||
self.success_rate_window = []
|
||||
|
||||
def record_result(self, success):
|
||||
self.success_rate_window.append(success)
|
||||
if len(self.success_rate_window) > 100:
|
||||
self.success_rate_window.pop(0)
|
||||
|
||||
def recommended_threads(self):
|
||||
if len(self.success_rate_window) < 20:
|
||||
return self.current
|
||||
|
||||
success_rate = sum(self.success_rate_window) / len(self.success_rate_window)
|
||||
|
||||
# High success rate -> can handle more threads
|
||||
if success_rate > 0.7 and self.current < self.max:
|
||||
return self.current + 5
|
||||
# Low success rate -> reduce load
|
||||
elif success_rate < 0.3 and self.current > self.min:
|
||||
return self.current - 5
|
||||
|
||||
return self.current
|
||||
```
|
||||
|
||||
**Files:** proxywatchd.py
|
||||
**Effort:** Medium
|
||||
**Risk:** Medium
|
||||
**Completed.** Added dynamic thread scaling based on queue depth and success rate.
|
||||
- ThreadScaler class in proxywatchd.py with should_scale(), status_line()
|
||||
- Scales up when queue is deep (2x target) and success rate > 10%
|
||||
- Scales down when queue is shallow or success rate drops
|
||||
- Min/max threads derived from config.watchd.threads (1/4x to 2x)
|
||||
- 30-second cooldown between scaling decisions
|
||||
- _spawn_thread(), _remove_thread(), _adjust_threads() helper methods
|
||||
- Scaler status reported alongside periodic stats
|
||||
|
||||
---
|
||||
|
||||
### [ ] 13. Latency Tracking
|
||||
### [x] 13. Latency Tracking
|
||||
|
||||
**Problem:** No visibility into proxy speed. A working but slow proxy may be
|
||||
less useful than a fast one.
|
||||
|
||||
**Implementation:**
|
||||
```python
|
||||
# dbs.py - add columns
|
||||
# ALTER TABLE proxylist ADD COLUMN avg_latency REAL DEFAULT 0
|
||||
# ALTER TABLE proxylist ADD COLUMN latency_samples INTEGER DEFAULT 0
|
||||
|
||||
def update_proxy_latency(proxydb, proxy, latency):
|
||||
"""Update rolling average latency for proxy."""
|
||||
row = proxydb.execute(
|
||||
'SELECT avg_latency, latency_samples FROM proxylist WHERE proxy=?',
|
||||
(proxy,)
|
||||
).fetchone()
|
||||
|
||||
if row:
|
||||
old_avg, samples = row
|
||||
# Exponential moving average
|
||||
new_avg = (old_avg * samples + latency) / (samples + 1)
|
||||
new_samples = min(samples + 1, 100) # Cap at 100 samples
|
||||
|
||||
proxydb.execute(
|
||||
'UPDATE proxylist SET avg_latency=?, latency_samples=? WHERE proxy=?',
|
||||
(new_avg, new_samples, proxy)
|
||||
)
|
||||
```
|
||||
|
||||
**Files:** dbs.py, proxywatchd.py
|
||||
**Effort:** Medium
|
||||
**Risk:** Low
|
||||
**Completed.** Added per-proxy latency tracking with exponential moving average.
|
||||
- dbs.py: avg_latency, latency_samples columns added to proxylist schema
|
||||
- dbs.py: _migrate_latency_columns() for backward-compatible migration
|
||||
- dbs.py: update_proxy_latency() with EMA (alpha = 2/(samples+1))
|
||||
- proxywatchd.py: ProxyTestState.last_latency_ms field
|
||||
- proxywatchd.py: evaluate() calculates average latency from successful tests
|
||||
- proxywatchd.py: submit_collected() records latency for passing proxies
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user