docs: update for v1.2.1 performance changes
- USAGE.md: alert output format, background seeding, per-backend errors, concurrent fetches - CHEATSHEET.md: updated alert section - DEBUG.md: added profiling section (cProfile + tracemalloc) - ROADMAP.md: added v1.2.1 milestone Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
11
ROADMAP.md
11
ROADMAP.md
@@ -97,6 +97,17 @@
|
||||
- [x] Graceful SIGTERM shutdown
|
||||
- [x] InnerTube-based YouTube channel resolution for video URLs
|
||||
|
||||
## v1.2.1 -- Performance + Polish (done)
|
||||
|
||||
- [x] HTTP opener caching at module level (eliminates per-request construction)
|
||||
- [x] `--tracemalloc` CLI flag for memory profiling (dumps to `data/derp.malloc`)
|
||||
- [x] Background seeding on `!alert add` (instant reply, seeds asynchronously)
|
||||
- [x] Per-backend error tracking with exponential backoff
|
||||
- [x] Concurrent fetches for multi-instance backends (PeerTube, Mastodon, Lemmy, SearXNG)
|
||||
- [x] `retries` parameter for `derp.http.urlopen`
|
||||
- [x] Alert output: ACTION line (metadata/URL) + PRIVMSG (full uncropped title)
|
||||
- [x] tracemalloc writes to file instead of logger (survives podman log buffer)
|
||||
|
||||
## v2.0.0 -- Multi-Server + Integrations
|
||||
|
||||
- [ ] Multi-server support (per-server config, shared plugins)
|
||||
|
||||
@@ -384,9 +384,12 @@ Archive.org (ia), Hacker News (hn), GitHub (gh), Wikipedia (wp),
|
||||
Stack Exchange (se), GitLab (gl), npm (nm), PyPI (pp), Docker Hub (dh),
|
||||
arXiv (ax), Lobsters (lb), DEV.to (dv), Medium (md), Hugging Face (hf).
|
||||
Names: lowercase alphanumeric + hyphens, 1-20 chars. Keywords: 1-100 chars.
|
||||
Max 20 alerts/channel. Polls every 5min. Format: `[name/yt/a8k2m] Title -- URL`.
|
||||
Use `!alert info <id>` to see full details. No API credentials needed. Persists
|
||||
across restarts. History stored in `data/alert_history.db`.
|
||||
Max 20 alerts/channel. Polls every 5min. Output: ACTION with `[name/tag/id] date - URL`,
|
||||
then PRIVMSG with full title. `add` replies instantly (seeds in background).
|
||||
Per-backend error tracking (5+ errors backs off that backend only).
|
||||
Multi-instance backends (pt, ft, ly, sx) fetch concurrently.
|
||||
Use `!alert info <id>` for details. Persists across restarts.
|
||||
History in `data/alert_history.db`.
|
||||
|
||||
## SearX
|
||||
|
||||
|
||||
@@ -97,6 +97,41 @@ ERROR derp.plugin failed to load plugin: plugins/broken.py
|
||||
- In container: resolver is typically `127.0.0.11` (Podman DNS)
|
||||
- Fallback: `8.8.8.8` if no resolver found in `/etc/resolv.conf`
|
||||
|
||||
## Profiling
|
||||
|
||||
### CPU (cProfile)
|
||||
|
||||
```bash
|
||||
derp --cprofile # Dump to derp.prof on shutdown
|
||||
derp --cprofile /app/data/derp.prof # Custom path
|
||||
```
|
||||
|
||||
Analyze with:
|
||||
|
||||
```python
|
||||
import pstats
|
||||
p = pstats.Stats("data/derp.prof")
|
||||
p.sort_stats("tottime").print_stats(30)
|
||||
p.sort_stats("cumulative").print_stats("plugins/", 20)
|
||||
```
|
||||
|
||||
### Memory (tracemalloc)
|
||||
|
||||
```bash
|
||||
derp --tracemalloc # 10 frames (default)
|
||||
derp --tracemalloc 25 # 25 frames deep
|
||||
```
|
||||
|
||||
Writes top 25 allocations with full tracebacks to `data/derp.malloc`
|
||||
on clean shutdown. Both flags can be combined:
|
||||
|
||||
```bash
|
||||
derp --verbose --cprofile /app/data/derp.prof --tracemalloc
|
||||
```
|
||||
|
||||
Requires clean SIGTERM shutdown (not SIGKILL) to flush data.
|
||||
Use `podman stop -t 30 derp` to allow graceful shutdown.
|
||||
|
||||
## Testing IRC Connection
|
||||
|
||||
```bash
|
||||
|
||||
@@ -721,19 +721,24 @@ Platforms searched:
|
||||
Polling and announcements:
|
||||
|
||||
- Alerts are polled every 5 minutes by default
|
||||
- On `add`, existing results are recorded without announcing (prevents flood)
|
||||
- New results announced as `[name/<tag>/<id>] Title -- URL` where tag is one of:
|
||||
`yt`, `tw`, `sx`, `rd`, `ft`, `dg`, `gn`, `kk`, `dm`, `pt`, `bs`, `ly`, `od`, `ia`,
|
||||
`hn`, `gh`, `wp`, `se`, `gl`, `nm`, `pp`, `dh`, `ax`, `lb`, `dv`, `md`, `hf`
|
||||
and `<id>` is a short deterministic ID for use with `!alert info`
|
||||
- Titles are truncated to 80 characters
|
||||
- On `add`, the bot replies immediately; existing results are seeded in the
|
||||
background to avoid flooding
|
||||
- New results announced as two lines:
|
||||
- ACTION: `* derp [name/<tag>/<id>] date - URL`
|
||||
- PRIVMSG: full uncropped title/content
|
||||
- Tags: `yt`, `tw`, `sx`, `rd`, `ft`, `dg`, `gn`, `kk`, `dm`, `pt`, `bs`, `ly`,
|
||||
`od`, `ia`, `hn`, `gh`, `wp`, `se`, `gl`, `nm`, `pp`, `dh`, `ax`, `lb`, `dv`,
|
||||
`md`, `hf` -- `<id>` is a short deterministic ID for use with `!alert info`
|
||||
- Each platform maintains its own seen list (capped at 200 per platform)
|
||||
- 5 consecutive errors doubles the poll interval (max 1 hour)
|
||||
- Per-backend error tracking with exponential backoff (5+ errors skips
|
||||
that backend with increasing cooldown; other backends unaffected)
|
||||
- Multi-instance backends (PeerTube, Mastodon, Lemmy, SearXNG) fetch
|
||||
concurrently for faster polling
|
||||
- Subscriptions persist across bot restarts via `bot.state`
|
||||
- Matched results are stored in `data/alert_history.db` (SQLite)
|
||||
- `list` shows error status indicators next to each alert
|
||||
- `list` shows per-backend error counts next to each alert
|
||||
- `check` forces an immediate poll across all platforms
|
||||
- `history` queries stored results, most recent first
|
||||
- `history` queries stored results (titles truncated), most recent first
|
||||
|
||||
### `!jwt` -- JWT Decoder
|
||||
|
||||
|
||||
Reference in New Issue
Block a user