docs: update for v1.2.1 performance changes

- USAGE.md: alert output format, background seeding, per-backend errors,
  concurrent fetches
- CHEATSHEET.md: updated alert section
- DEBUG.md: added profiling section (cProfile + tracemalloc)
- ROADMAP.md: added v1.2.1 milestone

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
user
2026-02-17 18:09:53 +01:00
parent a2a607baa2
commit e11994f320
4 changed files with 66 additions and 12 deletions

View File

@@ -97,6 +97,17 @@
- [x] Graceful SIGTERM shutdown
- [x] InnerTube-based YouTube channel resolution for video URLs
## v1.2.1 -- Performance + Polish (done)
- [x] HTTP opener caching at module level (eliminates per-request construction)
- [x] `--tracemalloc` CLI flag for memory profiling (dumps to `data/derp.malloc`)
- [x] Background seeding on `!alert add` (instant reply, seeds asynchronously)
- [x] Per-backend error tracking with exponential backoff
- [x] Concurrent fetches for multi-instance backends (PeerTube, Mastodon, Lemmy, SearXNG)
- [x] `retries` parameter for `derp.http.urlopen`
- [x] Alert output: ACTION line (metadata/URL) + PRIVMSG (full uncropped title)
- [x] tracemalloc writes to file instead of logger (survives podman log buffer)
## v2.0.0 -- Multi-Server + Integrations
- [ ] Multi-server support (per-server config, shared plugins)

View File

@@ -384,9 +384,12 @@ Archive.org (ia), Hacker News (hn), GitHub (gh), Wikipedia (wp),
Stack Exchange (se), GitLab (gl), npm (nm), PyPI (pp), Docker Hub (dh),
arXiv (ax), Lobsters (lb), DEV.to (dv), Medium (md), Hugging Face (hf).
Names: lowercase alphanumeric + hyphens, 1-20 chars. Keywords: 1-100 chars.
Max 20 alerts/channel. Polls every 5min. Format: `[name/yt/a8k2m] Title -- URL`.
Use `!alert info <id>` to see full details. No API credentials needed. Persists
across restarts. History stored in `data/alert_history.db`.
Max 20 alerts/channel. Polls every 5min. Output: ACTION with `[name/tag/id] date - URL`,
then PRIVMSG with full title. `add` replies instantly (seeds in background).
Per-backend error tracking (5+ errors backs off that backend only).
Multi-instance backends (pt, ft, ly, sx) fetch concurrently.
Use `!alert info <id>` for details. Persists across restarts.
History in `data/alert_history.db`.
## SearX

View File

@@ -97,6 +97,41 @@ ERROR derp.plugin failed to load plugin: plugins/broken.py
- In container: resolver is typically `127.0.0.11` (Podman DNS)
- Fallback: `8.8.8.8` if no resolver found in `/etc/resolv.conf`
## Profiling
### CPU (cProfile)
```bash
derp --cprofile # Dump to derp.prof on shutdown
derp --cprofile /app/data/derp.prof # Custom path
```
Analyze with:
```python
import pstats
p = pstats.Stats("data/derp.prof")
p.sort_stats("tottime").print_stats(30)
p.sort_stats("cumulative").print_stats("plugins/", 20)
```
### Memory (tracemalloc)
```bash
derp --tracemalloc # 10 frames (default)
derp --tracemalloc 25 # 25 frames deep
```
Writes top 25 allocations with full tracebacks to `data/derp.malloc`
on clean shutdown. Both flags can be combined:
```bash
derp --verbose --cprofile /app/data/derp.prof --tracemalloc
```
Requires clean SIGTERM shutdown (not SIGKILL) to flush data.
Use `podman stop -t 30 derp` to allow graceful shutdown.
## Testing IRC Connection
```bash

View File

@@ -721,19 +721,24 @@ Platforms searched:
Polling and announcements:
- Alerts are polled every 5 minutes by default
- On `add`, existing results are recorded without announcing (prevents flood)
- New results announced as `[name/<tag>/<id>] Title -- URL` where tag is one of:
`yt`, `tw`, `sx`, `rd`, `ft`, `dg`, `gn`, `kk`, `dm`, `pt`, `bs`, `ly`, `od`, `ia`,
`hn`, `gh`, `wp`, `se`, `gl`, `nm`, `pp`, `dh`, `ax`, `lb`, `dv`, `md`, `hf`
and `<id>` is a short deterministic ID for use with `!alert info`
- Titles are truncated to 80 characters
- On `add`, the bot replies immediately; existing results are seeded in the
background to avoid flooding
- New results announced as two lines:
- ACTION: `* derp [name/<tag>/<id>] date - URL`
- PRIVMSG: full uncropped title/content
- Tags: `yt`, `tw`, `sx`, `rd`, `ft`, `dg`, `gn`, `kk`, `dm`, `pt`, `bs`, `ly`,
`od`, `ia`, `hn`, `gh`, `wp`, `se`, `gl`, `nm`, `pp`, `dh`, `ax`, `lb`, `dv`,
`md`, `hf` -- `<id>` is a short deterministic ID for use with `!alert info`
- Each platform maintains its own seen list (capped at 200 per platform)
- 5 consecutive errors doubles the poll interval (max 1 hour)
- Per-backend error tracking with exponential backoff (5+ errors skips
that backend with increasing cooldown; other backends unaffected)
- Multi-instance backends (PeerTube, Mastodon, Lemmy, SearXNG) fetch
concurrently for faster polling
- Subscriptions persist across bot restarts via `bot.state`
- Matched results are stored in `data/alert_history.db` (SQLite)
- `list` shows error status indicators next to each alert
- `list` shows per-backend error counts next to each alert
- `check` forces an immediate poll across all platforms
- `history` queries stored results, most recent first
- `history` queries stored results (titles truncated), most recent first
### `!jwt` -- JWT Decoder