Commit Graph

125 Commits

Author SHA1 Message Date
mickael
c661047381 connect_socket: single srv for all protos 2019-01-06 04:42:17 +00:00
mickael
e872caaa77 catch some rocksock errors
this allows ppf to skip checks when a theorical proxy fails in certain manners
2019-01-06 04:12:57 +00:00
mickael
aab7ef15d1 remove unnecessary global variable 2019-01-06 01:29:35 +00:00
mickael
ecc6bbce10 spaces2tab 2019-01-06 01:18:08 +00:00
mickael
922b630881 print number of defined threads 2019-01-06 00:51:20 +00:00
mickael
4d513898fb make "submit_after" a configuratble option (config.ini) 2019-01-06 00:38:22 +00:00
mickael
1cafce67f0 no need to store 'duration' value in db 2019-01-05 23:55:40 +00:00
mickael
67cf6a18ea get rid of self.duration 2019-01-05 23:40:46 +00:00
mickael
dd4473279a fix: store total_duration in ms 2019-01-05 23:39:36 +00:00
mickael
0cb3525fe9 store total_duration in ms 2019-01-05 23:23:21 +00:00
mickael
aab72ad68b Merge branch 'master' of https://gitlab.com/mserneels/ppf 2019-01-05 22:57:57 +00:00
mickael
d93f4dcaf2 introduce success_count and total_duration (proxylist.sqlit
run those commands to update the database:

sqlite3 proxylist.sqlite "alter table proxylist add success_count int"
sqlite3 proxylist.sqlite "alter table proxylist add total_duration int"
sqlite3 proxylist.sqlite "update proxylist set success_count=0,total_duration=0"
2019-01-05 22:24:38 +00:00
rofl0r
2bb75b6c42 proxywatchd: improve logging 2019-01-05 22:19:49 +00:00
mickael
3348435e21 remove unused variables 2019-01-05 21:56:45 +00:00
rofl0r
dacbc8177a proxywatchd: fix memleak 2019-01-05 18:31:36 +00:00
rofl0r
74d9d965bb proxywatchd: always use a new mysql obj
could prevent memleaks, also helps thread-ownership issues.
2019-01-05 18:14:52 +00:00
mickael
cb342c3818 sqlite: use executemany
avoid loop inserting the whole list in a single query
2019-01-05 17:42:57 +00:00
rofl0r
af8f82924f fix logic so threads do an orderly shutdown
basically the issue was that the main loop received the SIGINT
and therefore broke out before reaching the parts of the code
that care about bringing down the child threads.

therefore there's now a finish() method that needs to be called
after stop().

because sqlite dbs insists to be used from the thread that created
the object, the DB cleanup operation are done from the thread
that controls it.

for standalone operation, in order to keep the main thread alive,
an additional run() method is used. this is not necessary when
used via ppf.py.
2019-01-05 17:17:27 +00:00
rofl0r
47221bae94 properly clean submitted job list after db write 2019-01-05 16:44:31 +00:00
rofl0r
42f35855ea use a variable for the min number of jobs to collect
this variable determines after how many results the db is written to.
with a low value, it is written often and could more lock issues
if 2 threads concur for the db.
2019-01-05 16:39:58 +00:00
rofl0r
f45cd1190c fix performance issue in proxywatchd 2019-01-05 06:58:46 +00:00
rofl0r
9ac3ed45d6 rewrite threading code in jobwatchd
now it distributes the tasks properly among all threads,
and it can be used as a standalone program.
there are some minor performance issues which will be fixed shortly.
2019-01-05 06:35:41 +00:00
rofl0r
ffbe450aee outsource configuration to external module 2019-01-05 03:47:03 +00:00
mickael
6df8fac96e fix: don't re-test all protos every time
all protos (http, socks4, socks5) were tested even for working proxies (whose protocol has been saved..)
also, don't save dummy 'http' protocol for non working proxies..
2019-01-05 01:04:50 +00:00
mickael
85043d8e8a move includes in root directory 2019-01-05 00:22:28 +00:00