username/ppf
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
458 Commits 1 Branch 0 Tags
ff21c75a7ac6483bf5ebc31ad4072da0274bd8b3
Commit Graph

29 Commits

Author SHA1 Message Date
Username
dfcd8f0c00 add test provenance columns and worker report fields 
Add last_check/last_target columns to proxylist schema with migration.
Include checktype and target in V2 worker report payload.
 2026-02-17 21:06:21 +01:00
Username
66441f9292 dbs: add url scoring columns to uris table 
Migration functions for check_interval, working_ratio, avg_fetch_time,
last_worker, and yield_rate columns with sensible defaults.
 2026-02-17 15:19:59 +01:00
Username
da832d94b7 dbs: add last_seen column to proxylist 2026-02-17 13:41:25 +01:00
Username
c19959cda2 dbs: add 19 proxy sources from 7 new repositories 
Expand PROXY_SOURCES with proxifly, vakhov, prxchk, sunny9577,
officialputuid, hookzof, and iplocate lists. Add source_proto
and protos_working schema columns for protocol intelligence.
Remove completed proxy source expansion task from roadmap.
 2026-02-17 13:13:23 +01:00
Username
64b3629585 dbs: add CDN filtering and verification tables 
- CDN_PREFIXES: filter Cloudflare, Fastly, Akamai, CloudFront, Google
- is_cdn_ip(): check if IP belongs to known CDN ranges
- insert_proxies(): skip CDN IPs with count in log message
- verification tables: worker_results, verification_queue, worker_trust
- queue_verification(): add proxies for manager re-testing
- get_verification_stats(): queue size and trigger breakdown
- get_all_worker_trust(): trust scores for all workers
 2026-01-08 09:05:13 +01:00
Username
3b361916fa fetch, dbs: minor refactoring 2025-12-28 15:18:42 +01:00
Username
481dc514fb fetch: add IPv6, auth proxy, and confidence scoring support 2025-12-26 19:13:36 +01:00
Username
269fed55ff refactor core modules, integrate network stats 2025-12-25 11:13:20 +01:00
Username
689ea8153b dbs: add batch operations and session state persistence 
- Add batch_update_proxy_latency for bulk updates
- Add save/load_session_state for stats persistence
- Improve insert_urls to return new count
 2025-12-24 00:19:46 +01:00
Username
53f37510f3 dashboard: add system monitoring and enhanced stats 
- prominent check type badge in header (SSL/judges/http/irc)
- system monitor bar: load, memory, disk, process RSS
- anonymity breakdown: elite/anonymous/transparent counts
- database health: size, recent activity, dead proxy count
- enhanced Tor pool stats: requests, success rate, latency
- SQLite ANALYZE/VACUUM functions for query optimization
- database statistics API functions
 2025-12-23 17:47:12 +01:00
Username
e0e330301a dbs: add session persistence and stats history 
- session_state table for persisting runtime stats across restarts
- stats_history table for hourly snapshots (24h graphs)
- latency tracking with exponential moving average
- anonymity detection columns (transparent/anonymous/elite)
- curated PROXY_SOURCES list for seeding
- migration functions for existing databases
 2025-12-23 17:23:04 +01:00
Username
f382a4ab6a ppf: add content hash for duplicate proxy list detection 2025-12-22 00:03:12 +01:00
Username
77867d0b2d dbs: add latency columns and migration 2025-12-21 23:37:38 +01:00
Username
79475c2bff add latency tracking and dynamic thread scaling 
- dbs.py: add avg_latency, latency_samples columns with migration
- dbs.py: update_proxy_latency() with exponential moving average
- proxywatchd.py: ThreadScaler class for dynamic thread count
- proxywatchd.py: calculate/record latency for successful proxies
- proxywatchd.py: _spawn_thread(), _remove_thread(), _adjust_threads()
- scaler reports status alongside periodic stats
 2025-12-21 00:08:19 +01:00
Username
e24f68500c style: normalize indentation and improve code style 
- convert tabs to 4-space indentation
- add docstrings to modules and classes
- remove unused import (copy)
- use explicit object inheritance
- use 'while True' over 'while 1'
- use 'while args' over 'while len(args)'
- use '{}' over 'dict()'
- consistent string formatting
- Python 2/3 compatible Queue import
 2025-12-20 23:18:45 +01:00
Username
90a6756ade dbs: add indexes and optimize batch inserts 2025-12-20 18:25:33 +01:00
Your Name
d7db366857 split to ip/port, "cleanse" ips and ports, bugfixes 2021-08-22 20:39:50 +02:00
Your Name
a005cc109c force ssl check (mitm) every 3 consecutive success check 
needs a new entry in proxies.sqlite

sqlite3 proxies.sqlite "alter table proxylist add consecutive_success int"
sqlite3 proxies.sqlite "update proxylist set consecutive_success=0"
 2021-07-03 18:47:34 +02:00
Your Name
47fee8df4f proxylist database: add "mitm" row 
update any existing db as follow:

sqlite3 proxies.sqlite "alter table proxylist add mitm int"
sqlite3 proxies.sqlite "update proxylist set mitm=0"
 2021-07-03 18:47:08 +02:00
Your Name
d658309ee4 dbs: ensure items are unique 2021-02-06 23:22:39 +01:00
Your Name
615af656a2 fix: differentiate log message on added url/proxy 2021-02-06 12:21:16 +01:00
Your Name
78b29a1187 some changes 2021-01-24 03:52:56 +01:00
Mickaël Serneels
8900153871 set default error value to 1 for new urls 2019-05-01 17:43:28 +02:00
rofl0r
8400eab7ee insert_proxies: remove 500-at-a-time logic 
it's now done by mysqlite.py executemany.
 2019-01-18 21:50:48 +00:00
rofl0r
8be5ab1567 ppf: move insert function into dbs.py 2019-01-18 21:43:17 +00:00
rofl0r
0dad0176f3 ppf: add new field proxies_added to be able to rate sites 
sqlite3 urls.sqlite "alter table uris add proxies_added INT"
sqlite3 urls.sqlite "update uris set proxies_added=0"
 2019-01-18 15:44:09 +00:00
rofl0r
69d366f7eb ppf: add retrievals field so we know whether an url is new 
use

sqlite3 urls.sqlite "alter table uris add retrievals INT"
sqlite3 urls.sqlite "update uris set retrievals=0"
 2019-01-13 16:40:12 +00:00
rofl0r
bc41bad9de dbs.py: remove unused column hash 2019-01-13 16:40:12 +00:00
mickael
4c6a83373f split databases 2019-01-11 00:25:01 +00:00