rofl0r
ddee92d20f
watchd: introduce configurable 'outage_threshold'
2019-01-18 15:34:49 +00:00
mickael
aaac14d34e
worker: add threading lock
...
add lock to avoid same proxy to be scanned multiple time when
a small number a jobs is handed to worker
2019-01-13 16:50:54 +00:00
rofl0r
7d59404d31
watchd: add totals statistics
2019-01-11 00:52:11 +00:00
mickael
4c6a83373f
split databases
2019-01-11 00:25:01 +00:00
rofl0r
ef9158015f
proxywatchd: make checktime constants configurable
...
this requires only saving the last checked time in `tested`.
you can run the following sql statement to update the existing values
in the database:
sqlite3 proxylist.sqlite \
"update proxylist set tested=tested-(1800+(failed*3600)) where failed < 6"
2019-01-11 00:25:01 +00:00
rofl0r
befb346941
proxywatchd: preliminary support for ip caching
...
whenever we make a socks4 check, the ip of the destination server
needs to be resolved because socks4 does not support server-side
dns resolution. in order to prevent doing the same lookups over
and over, we know manually resolve the ip before first usage, and
store it in a cache.
2019-01-10 19:22:21 +00:00
rofl0r
bd1e85a883
proxywatchd: add safeguards against tor outage
2019-01-08 16:06:01 +00:00
rofl0r
fb917184b0
watchd: remove catch-it-all except statements
...
it took considerable time to figure out why it suddenly stopped
working due to a typo inside the try/except block.
2019-01-08 03:34:44 +00:00
rofl0r
f16f754b0e
implement combo config parser
...
allows all options to be overridden by command line.
e.g.
[watchd]
threads=10
debug=false
--watch.threads=50 --debug=true
2019-01-08 02:17:10 +00:00
mickael
15bdbf63fd
pyflakes: remove unused imports
2019-01-07 23:27:28 +00:00
mickael
1b3ce72efc
add and use combining class
2019-01-07 23:19:14 +00:00
rofl0r
dfa90109da
watchd: first halfbaked attempt to detect/combat tor issues
2019-01-07 21:35:36 +00:00
rofl0r
9e3c08fcf4
watchd: do not count name resolution errors as proxy fail
2019-01-07 15:57:15 +00:00
rofl0r
be99b8517b
watchd: beautify logging
2019-01-07 15:38:51 +00:00
rofl0r
d93f14ad67
watchd: distribute remaining jobs among threads if one is idle
2019-01-07 15:38:51 +00:00
rofl0r
497f8b2050
watchd: remove non-printable chars from logged response
2019-01-07 02:54:33 +00:00
rofl0r
e51dbe113b
watchd: optional debug output
2019-01-06 22:44:18 +00:00
rofl0r
312f77469e
watchd: disable some noise in the proxy-found log output
2019-01-06 22:42:16 +00:00
rofl0r
5a6c49d88b
watchd: collect stats
2019-01-06 22:42:09 +00:00
mickael
358044ccea
watchd: make use of ssl configurable
2019-01-06 20:09:27 +00:00
mickael
0ab3a6d066
connect_socket: fix proto bug
2019-01-06 19:43:05 +00:00
mickael
91654d4b9e
remove unused "is_drone_bl()" function
2019-01-06 19:12:42 +00:00
mickael
e4636df468
delay thread startup
2019-01-06 17:52:56 +00:00
rofl0r
7d8583f7cc
no need to re-raise keyboardinterrupt
2019-01-06 16:30:32 +00:00
rofl0r
14e4b832f2
proxywatchd: dont delete timestamp when printing halt msg
2019-01-06 16:29:17 +00:00
rofl0r
6c515d9335
don't continue testing offline proxies
2019-01-06 16:20:29 +00:00
rofl0r
a0dcd9fa8f
save total_duration as int into the db
...
without that, the DB saves it as float even though the column type is
INT.
requires a run of
sqlite3 proxylist.sqlite "update proxylist set success_count=0,total_duration=0"
to fix existing entries.
2019-01-06 15:23:23 +00:00
mickael
1adf78119d
take care of "failinc"
2019-01-06 14:57:09 +00:00
mickael
cff3e1304c
remove rocksockexception block
2019-01-06 14:55:31 +00:00
mickael
0a34833d3d
"prettier" thread exit message
2019-01-06 05:13:23 +00:00
mickael
6b4ca67b33
use e.get_failedproxy()
2019-01-06 05:02:02 +00:00
mickael
c661047381
connect_socket: single srv for all protos
2019-01-06 04:42:17 +00:00
mickael
e872caaa77
catch some rocksock errors
...
this allows ppf to skip checks when a theorical proxy fails in certain manners
2019-01-06 04:12:57 +00:00
mickael
aab7ef15d1
remove unnecessary global variable
2019-01-06 01:29:35 +00:00
mickael
ecc6bbce10
spaces2tab
2019-01-06 01:18:08 +00:00
mickael
922b630881
print number of defined threads
2019-01-06 00:51:20 +00:00
mickael
4d513898fb
make "submit_after" a configuratble option (config.ini)
2019-01-06 00:38:22 +00:00
mickael
1cafce67f0
no need to store 'duration' value in db
2019-01-05 23:55:40 +00:00
mickael
67cf6a18ea
get rid of self.duration
2019-01-05 23:40:46 +00:00
mickael
dd4473279a
fix: store total_duration in ms
2019-01-05 23:39:36 +00:00
mickael
0cb3525fe9
store total_duration in ms
2019-01-05 23:23:21 +00:00
mickael
aab72ad68b
Merge branch 'master' of https://gitlab.com/mserneels/ppf
2019-01-05 22:57:57 +00:00
mickael
d93f4dcaf2
introduce success_count and total_duration (proxylist.sqlit
...
run those commands to update the database:
sqlite3 proxylist.sqlite "alter table proxylist add success_count int"
sqlite3 proxylist.sqlite "alter table proxylist add total_duration int"
sqlite3 proxylist.sqlite "update proxylist set success_count=0,total_duration=0"
2019-01-05 22:24:38 +00:00
rofl0r
2bb75b6c42
proxywatchd: improve logging
2019-01-05 22:19:49 +00:00
mickael
3348435e21
remove unused variables
2019-01-05 21:56:45 +00:00
rofl0r
dacbc8177a
proxywatchd: fix memleak
2019-01-05 18:31:36 +00:00
rofl0r
74d9d965bb
proxywatchd: always use a new mysql obj
...
could prevent memleaks, also helps thread-ownership issues.
2019-01-05 18:14:52 +00:00
mickael
cb342c3818
sqlite: use executemany
...
avoid loop inserting the whole list in a single query
2019-01-05 17:42:57 +00:00
rofl0r
af8f82924f
fix logic so threads do an orderly shutdown
...
basically the issue was that the main loop received the SIGINT
and therefore broke out before reaching the parts of the code
that care about bringing down the child threads.
therefore there's now a finish() method that needs to be called
after stop().
because sqlite dbs insists to be used from the thread that created
the object, the DB cleanup operation are done from the thread
that controls it.
for standalone operation, in order to keep the main thread alive,
an additional run() method is used. this is not necessary when
used via ppf.py.
2019-01-05 17:17:27 +00:00
rofl0r
47221bae94
properly clean submitted job list after db write
2019-01-05 16:44:31 +00:00
