rofl0r
24d2c08c9f
ppf: make it possible to import a file containing proxies directly
...
using --file filename.html
2019-01-11 05:45:13 +00:00
rofl0r
ecf587c8f7
ppf: set newly added sites to 0,0 (err/stale)
...
we use the tuple 0,0 later on to detect whether a site is new or not.
2019-01-11 05:23:05 +00:00
rofl0r
8b10df9c1b
ppf.py: start using stale_count
2019-01-11 05:08:32 +00:00
rofl0r
d2cb7441a8
ppf: add optional debug output
2019-01-11 05:03:40 +00:00
rofl0r
b6dba08cf0
ppf: only extract ips with port >= 10
2019-01-11 03:29:13 +00:00
rofl0r
122847d888
ppf: fix bug referencing removed db field
2019-01-11 02:53:16 +00:00
rofl0r
7d59404d31
watchd: add totals statistics
2019-01-11 00:52:11 +00:00
mickael
4c6a83373f
split databases
2019-01-11 00:25:01 +00:00
mickael
b85cb863ba
remove more dead servers
2019-01-11 00:25:01 +00:00
rofl0r
5e774b4e2a
config.py: put section name in var
...
avoids errors due to typos
2019-01-11 00:25:01 +00:00
rofl0r
ef9158015f
proxywatchd: make checktime constants configurable
...
this requires only saving the last checked time in `tested`.
you can run the following sql statement to update the existing values
in the database:
sqlite3 proxylist.sqlite \
"update proxylist set tested=tested-(1800+(failed*3600)) where failed < 6"
2019-01-11 00:25:01 +00:00
rofl0r
087559637e
ppf: improve cleanhtml() and cache compiled re's
...
now it transforms e.g. '<td>118.114.116.36</td>\n<td>1080</td>'
correctly.
(the newline was formerly preventing success)
2019-01-10 19:22:21 +00:00
rofl0r
befb346941
proxywatchd: preliminary support for ip caching
...
whenever we make a socks4 check, the ip of the destination server
needs to be resolved because socks4 does not support server-side
dns resolution. in order to prevent doing the same lookups over
and over, we know manually resolve the ip before first usage, and
store it in a cache.
2019-01-10 19:22:21 +00:00
rofl0r
7067a8199f
rocksock: bump to latest
2019-01-10 19:22:21 +00:00
rofl0r
10d6b3afd8
servers.txt: remove dead server
2019-01-10 19:22:21 +00:00
mickael
383ae6f431
fix: no uris were tested because commented"
2019-01-10 00:21:57 +00:00
mserneels
0cbc434883
Merge branch 'experiment' into 'master'
...
Experiment
See merge request mserneels/ppf!11
2019-01-09 23:40:54 +00:00
mickael
da4f228479
discard urls who fail at first test
2019-01-09 23:38:59 +00:00
mickael
15dee0cd73
add -intitle:pdf to searx query
2019-01-09 23:30:55 +00:00
mickael
e94644a60e
searx: loop for 10 pages on each searx instance
2019-01-09 22:55:55 +00:00
mickael
8993727f03
changed regex
2019-01-09 20:07:28 +00:00
mickael
33887385f0
is_usable_proxy: group the 2 firsts lines
2019-01-09 19:23:09 +00:00
mickael
9828db79d4
is_usable_proxy(): dont check twice if A < 1
2019-01-09 19:11:05 +00:00
mickael
6f0d5c1ffa
modify and rename should_i_... function
...
> remove :port from D
> check if octets are within a correct range
2019-01-09 19:01:55 +00:00
mickael
a74d6dfce8
do not save invalid IPs
2019-01-09 00:42:28 +00:00
mickael
bde24ce0e9
working but inefficient script to remove invalid proxies from db
2019-01-09 00:41:46 +00:00
mserneels
484fe325f8
Merge branch 'changes9' into 'master'
...
Changes9
See merge request mserneels/ppf!10
2019-01-08 19:55:45 +00:00
rofl0r
bd1e85a883
proxywatchd: add safeguards against tor outage
2019-01-08 16:06:01 +00:00
rofl0r
6e4c45175e
ppf: add safeguards against tor outage
2019-01-08 15:48:38 +00:00
rofl0r
b6839eea79
rocksock/http2: bump to latest
2019-01-08 15:48:38 +00:00
rofl0r
b88a8426dc
rocksock: bump to latest
2019-01-08 04:38:28 +00:00
rofl0r
b9058b0334
http2: bump to latest
2019-01-08 04:38:14 +00:00
rofl0r
1f3179de48
ppf: check for valid ports
2019-01-08 04:30:50 +00:00
rofl0r
9ccf8b7854
ppf: write dates as int
2019-01-08 04:19:09 +00:00
rofl0r
35421b980e
rocksock: bump to latest
2019-01-08 03:35:40 +00:00
rofl0r
fb917184b0
watchd: remove catch-it-all except statements
...
it took considerable time to figure out why it suddenly stopped
working due to a typo inside the try/except block.
2019-01-08 03:34:44 +00:00
rofl0r
38d89f5bd9
ppf: add option for number of http retries
2019-01-08 03:30:31 +00:00
rofl0r
115c4a56f5
ppf: honor timeout
2019-01-08 03:25:52 +00:00
rofl0r
f16f754b0e
implement combo config parser
...
allows all options to be overridden by command line.
e.g.
[watchd]
threads=10
debug=false
--watch.threads=50 --debug=true
2019-01-08 02:17:10 +00:00
rofl0r
e7b8d526c0
ppf: print url if fetching failed
2019-01-08 00:46:41 +00:00
mickael
20bc8be2fa
pretty: hide printed '^C' when ^C'ing
2019-01-07 23:36:56 +00:00
mickael
15bdbf63fd
pyflakes: remove unused imports
2019-01-07 23:27:28 +00:00
mickael
1b3ce72efc
add and use combining class
2019-01-07 23:19:14 +00:00
mickael
e31ac03f6f
avoid double "checktime" usage
2019-01-07 22:09:22 +00:00
mickael
6c8b707637
Merge branch 'master' of ssh://gitlab.com/mserneels/ppf
2019-01-07 21:41:46 +00:00
mickael
1288dca38f
fixme: change var names
2019-01-07 21:41:41 +00:00
mserneels
a86cc925be
Merge branch 'changes8' into 'master'
...
watchd: first halfbaked attempt to detect/combat tor issues
See merge request mserneels/ppf!9
2019-01-07 21:38:12 +00:00
rofl0r
dfa90109da
watchd: first halfbaked attempt to detect/combat tor issues
2019-01-07 21:35:36 +00:00
mickael
aeff09d2b3
move math function inside the sql statement
2019-01-07 21:11:08 +00:00
mserneels
f47745c8d6
Merge branch 'changes7' into 'master'
...
Changes7
See merge request mserneels/ppf!8
2019-01-07 18:56:23 +00:00
