derp/docs/USAGE.md

# Usage Guide

## Running

```bash
# From project directory
derp

# With options
derp --config /path/to/derp.toml --verbose
```

## CLI Flags

| Flag | Description |
|------|-------------|
| `-c, --config PATH` | Config file path |
| `-v, --verbose` | Debug logging |
| `--cprofile [PATH]` | Enable cProfile, dump to PATH [derp.prof] |
| `-V, --version` | Print version |
| `-h, --help` | Show help |

## Configuration

All settings in `config/derp.toml`:

```toml
[server]
host = "irc.libera.chat"    # IRC server hostname
port = 6697                   # Port (6697 = TLS, 6667 = plain)
tls = true                    # Enable TLS encryption
nick = "derp"                 # Bot nickname
user = "derp"                 # Username (ident)
realname = "derp IRC bot"     # Real name field
password = ""                 # Server password (optional)
sasl_user = ""                # SASL PLAIN username (optional)
sasl_pass = ""                # SASL PLAIN password (optional)
ircv3_caps = [                # IRCv3 capabilities to request
    "multi-prefix",
    "away-notify",
    "server-time",
    "cap-notify",
    "account-notify",
]

[bot]
prefix = "!"                  # Command prefix character
channels = ["#test"]          # Channels to join on connect
plugins_dir = "plugins"       # Plugin directory path
rate_limit = 2.0              # Max messages per second (default: 2.0)
rate_burst = 5                # Burst capacity (default: 5)
admins = []                   # Hostmask patterns (fnmatch), IRCOPs auto-detected
timezone = "UTC"              # Timezone for calendar reminders (IANA tz name)

[logging]
level = "info"                # Logging level: debug, info, warning, error
format = "text"               # Log format: "text" (default) or "json"
```

## Built-in Commands

| Command | Description |
|---------|-------------|
| `!ping` | Bot responds with "pong" |
| `!help` | List all available commands |
| `!help <cmd>` | Show help for a specific command |
| `!help <plugin>` | Show plugin description and its commands |
| `!version` | Show bot version |
| `!uptime` | Show how long the bot has been running |
| `!echo <text>` | Echo back text (example plugin) |
| `!cert <domain> [...]` | Lookup CT logs for up to 5 domains |
| `!whoami` | Show your hostmask and admin status |
| `!load <plugin>` | Hot-load a plugin (admin) |
| `!reload <plugin>` | Reload a plugin (admin) |
| `!unload <plugin>` | Unload a plugin (admin) |
| `!admins` | Show admin patterns and detected opers (admin) |
| `!plugins` | List loaded plugins with handler counts |
| `!state <action> <plugin> [key]` | Inspect plugin state store (admin) |
| `!kick <nick> [reason]` | Kick user from channel (admin) |
| `!ban <mask>` | Ban a hostmask in channel (admin) |
| `!unban <mask>` | Remove a ban from channel (admin) |
| `!topic [text]` | Set or query channel topic (admin) |
| `!mode <mode> [args]` | Set channel mode (admin) |
| `!dns <target> [type]` | DNS lookup (A, AAAA, MX, NS, TXT, CNAME, PTR, SOA) |
| `!tdns <target> [type] [@server]` | TCP DNS lookup via SOCKS5 proxy |
| `!encode <fmt> <text>` | Encode text (b64, hex, url, rot13) |
| `!decode <fmt> <text>` | Decode text (b64, hex, url, rot13) |
| `!hash [algo] <text>` | Generate hash digests (md5, sha1, sha256, sha512) |
| `!hashid <hash>` | Identify hash type by format |
| `!defang <ioc>` | Defang URLs/IPs/domains for safe sharing |
| `!refang <text>` | Restore defanged IOCs |
| `!revshell <type> <ip> <port>` | Generate reverse shell one-liner |
| `!cidr <network>` | Subnet info (range, hosts, mask) |
| `!cidr contains <net> <ip>` | Check if IP belongs to network |
| `!whois <domain\|ip>` | WHOIS lookup via raw TCP (port 43) |
| `!portcheck <host> [ports]` | Async TCP port scan (max 20 ports) |
| `!httpcheck <url>` | HTTP status, redirects, response time |
| `!tlscheck <host> [port]` | TLS version, cipher, cert details |
| `!blacklist <ip>` | Check IP against 10 DNSBLs |
| `!rand <mode> [args]` | Random: password, hex, uuid, bytes, int, coin, dice |
| `!timer <duration> [label]` | Set countdown timer with notification |
| `!timer list` | Show active timers |
| `!timer cancel <label>` | Cancel a running timer |
| `!remind <duration> <text>` | One-shot reminder after duration |
| `!remind every <duration> <text>` | Repeating reminder at interval |
| `!remind at <YYYY-MM-DD> [HH:MM] <text>` | Calendar reminder at specific date/time |
| `!remind yearly <MM-DD> [HH:MM] <text>` | Yearly recurring reminder |
| `!remind list` | Show all active reminders |
| `!remind cancel <id>` | Cancel a reminder by ID |
| `!geoip <ip>` | GeoIP lookup (city, country, coords, timezone) |
| `!asn <ip>` | ASN lookup (AS number, organization) |
| `!tor <ip\|update>` | Check IP against Tor exit nodes |
| `!iprep <ip\|update>` | Check IP against Firehol/ET blocklists |
| `!cve <id\|search>` | CVE lookup from local NVD mirror |
| `!opslog <add\|list\|search\|del\|clear>` | Timestamped operational log |
| `!note <set\|get\|del\|list\|clear>` | Per-channel key-value notes |
| `!subdomain <domain> [brute]` | Subdomain enumeration (crt.sh + DNS) |
| `!headers <url>` | HTTP header fingerprinting |
| `!exploitdb <search\|id\|cve\|update>` | Search local Exploit-DB mirror |
| `!payload <type> [variant]` | Web vuln payload templates |
| `!dork <category\|list> [target]` | Google dork query builder |
| `!wayback <url> [YYYYMMDD]` | Wayback Machine snapshot lookup |
| `!username <user>` | Check username across ~25 services |
| `!username <user> <service>` | Check single service |
| `!username list` | Show available services by category |
| `!searx <query>` | Search SearXNG and show top results |

### Command Shorthand

Commands can be abbreviated to any unambiguous prefix:

```
!h           -> !help (unique match)
!pi          -> !ping (unique match)
!p           -> error: ambiguous (ping, plugins)
```

Exact matches always take priority over prefix matches.

### `!cert` -- Certificate Transparency Lookup

Query [crt.sh](https://crt.sh) CT logs to enumerate SSL certificates for
domains. Reports totals (expired/valid) and flags domains still serving
expired certs.

```
!cert example.com
!cert example.com badsite.com another.org
```

Output format:

```
example.com -- 127 certs (23 expired, 104 valid)
badsite.com -- 45 certs (8 expired, 37 valid) | live cert EXPIRED
broken.test -- error: timeout
```

- Max 5 domains per invocation
- crt.sh can be slow; the bot confirms receipt before querying
- Live cert check runs only when expired CT entries exist

## Per-Channel Plugin Control

Restrict which plugins are active in specific channels. Channels without
a `[channels."<name>"]` section run all plugins. Channels with a `plugins`
list only run those plugins. The `core` plugin is always active (exempt
from filtering). Private messages are always unrestricted.

```toml
[channels."#public"]
plugins = ["core", "dns", "cidr", "encode"]

[channels."#ops"]
plugins = ["core", "revshell", "payload", "exploitdb", "opslog"]

# #unrestricted -- no section, runs everything
```

When a command is denied by channel config, it is silently ignored (no
error message). Event handlers from denied plugins are also skipped.

## Structured Logging (JSON)

Set `format = "json"` in `[logging]` to emit one JSON object per log line
(JSONL), suitable for log aggregation tools.

```toml
[logging]
level = "info"
format = "json"
```

Each line contains:

| Field | Description |
|-------|-------------|
| `ts` | Timestamp (`YYYY-MM-DDTHH:MM:SS`) |
| `level` | Log level (`debug`, `info`, `warning`, `error`) |
| `logger` | Logger name (`derp.bot`, `derp.plugin`, etc.) |
| `msg` | Log message text |
| `exc` | Exception traceback (only present on errors) |

Default format is `"text"` (human-readable, same as before).

## Admin System

Commands marked as `admin` require elevated permissions. Admin access is
granted via:

1. **IRC operator status** -- detected automatically via `WHO`
2. **Hostmask patterns** -- configured in `[bot] admins`, fnmatch-style

```toml
[bot]
admins = [
    "*!~user@trusted.host",
    "ops!*@*.ops.net",
]
```

Empty by default -- only IRC operators get admin access unless patterns
are configured.

### Oper Detection

IRC operators are detected via the `*` flag in `WHO` replies. Detection
happens at two points:

- **On connect** -- `WHO #channel` for each configured channel
- **On user JOIN** -- debounced `WHO #channel` (2-second window)

The debounce prevents flooding the server during netsplit recovery: many
rapid JOINs produce a single `WHO` per channel. Note that the first
command after joining may not yet have oper status -- the debounced WHO
fires after a 2-second delay. Users who `QUIT` are removed from the oper
set automatically.

| Command | Description |
|---------|-------------|
| `!whoami` | Show your hostmask and admin status |
| `!admins` | Show configured patterns and detected opers (admin) |

Admin-restricted commands: `!load`, `!reload`, `!unload`, `!admins`, `!state`,
`!kick`, `!ban`, `!unban`, `!topic`, `!mode`.

### Writing Admin Commands

```python
@command("dangerous", help="Admin-only action", admin=True)
async def cmd_dangerous(bot, message):
    ...
```

## IRCv3 Capability Negotiation

The bot negotiates IRCv3 capabilities using `CAP LS 302` during registration.
This enables richer features on servers that support them.

### Default Capabilities

```toml
[server]
ircv3_caps = ["multi-prefix", "away-notify", "server-time",
              "cap-notify", "account-notify"]
```

| Capability | Purpose |
|------------|---------|
| `multi-prefix` | Better IRCOP/voice detection |
| `away-notify` | Receive AWAY status changes |
| `server-time` | Accurate message timestamps |
| `cap-notify` | Dynamic capability updates |
| `account-notify` | Account login/logout notices |
| `sasl` | Auto-added when SASL credentials configured |

The bot only requests caps the server advertises. SASL is automatically
included when `sasl_user` and `sasl_pass` are configured.

### Message Tags

IRCv3 message tags (`@key=value;...`) are parsed automatically and available
on the message object as `message.tags` (a `dict[str, str]`). Values are
unescaped per the IRCv3 spec.

## Channel Management

Channel management commands require admin privileges and must be used in a
channel (not DM).

```
!kick <nick> [reason]     Kick a user from the channel
!ban <mask>               Set +b on a hostmask (e.g. *!*@bad.host)
!unban <mask>             Remove +b from a hostmask
!topic [text]             Set topic (empty = query current topic)
!mode <mode> [args]       Set raw MODE on the channel
```

The bot must have channel operator status for these commands to take effect.

### Invite Auto-Join

When an admin or IRC operator sends an `INVITE`, the bot automatically joins
the target channel. Non-admin invites are silently ignored.

## Plugin State Persistence

Plugins can persist key-value data across restarts via `bot.state`:

```python
bot.state.set("myplugin", "last_run", "2026-02-15")
value = bot.state.get("myplugin", "last_run")
bot.state.delete("myplugin", "last_run")
keys = bot.state.keys("myplugin")
bot.state.clear("myplugin")
```

Data is stored in `data/state.db` (SQLite). Each plugin gets its own
namespace so keys never collide.

### Inspection Commands (admin)

```
!state list <plugin>      List keys for a plugin
!state get <plugin> <key> Get a value
!state del <plugin> <key> Delete a key
!state clear <plugin>     Clear all state for a plugin
```

## Local Databases (Wave 3)

Several plugins rely on local data files in the `data/` directory. Use the
update script or in-bot commands to populate them.

### Data Update Script

```bash
./scripts/update-data.sh                        # Update all feeds
MAXMIND_LICENSE_KEY=xxx ./scripts/update-data.sh # Include GeoLite2
```

The script is cron-friendly (exit 0/1, quiet unless `NO_COLOR` is unset).

### In-Bot Updates

```
!tor update          # Download Tor exit node list
!iprep update        # Download Firehol/ET blocklist feeds
!cve update          # Download NVD CVE feed (slow, paginated)
```

### Data Directory Layout

```
data/
  GeoLite2-City.mmdb    # MaxMind GeoIP (requires license key)
  GeoLite2-ASN.mmdb     # MaxMind ASN (requires license key)
  tor-exit-nodes.txt    # Tor exit node IPs
  iprep/                # Firehol/ET blocklist feeds
    firehol_level1.netset
    firehol_level2.netset
    et_compromised.ipset
    ...
  nvd/                  # NVD CVE JSON files
    nvd_0000.json
    ...
```

GeoLite2 databases require a free MaxMind license key. Set
`MAXMIND_LICENSE_KEY` when running the update script.

## Plugin Management

Plugins can be loaded, unloaded, and reloaded at runtime without
restarting the bot.

```
!load crtsh          # Hot-load a new plugin from plugins/
!reload crtsh        # Reload a changed plugin
!unload crtsh        # Remove a plugin and all its handlers
!plugins             # List loaded plugins with handler counts
```

The `core` plugin cannot be unloaded (prevents losing `!load`/`!reload`),
but it can be reloaded.

## Writing Plugins

Create a `.py` file in the `plugins/` directory:

```python
from derp.plugin import command, event

@command("hello", help="Greet the user")
async def cmd_hello(bot, message):
    """Handler receives bot instance and parsed Message."""
    await bot.reply(message, f"Hello, {message.nick}!")

@event("JOIN")
async def on_join(bot, message):
    """Event handlers fire on IRC events (JOIN, PART, QUIT, etc.)."""
    if message.nick != bot.nick:
        await bot.send(message.target, f"Welcome, {message.nick}")
```

### Plugin API

The `bot` object provides:

| Method | Description |
|--------|-------------|
| `bot.send(target, text)` | Send message to channel or nick |
| `bot.reply(msg, text)` | Reply to source (channel or PM) |
| `bot.action(target, text)` | Send `/me` action |
| `bot.join(channel)` | Join a channel |
| `bot.part(channel [, reason])` | Leave a channel |
| `bot.quit([reason])` | Disconnect from server |
| `bot.kick(channel, nick [, reason])` | Kick a user |
| `bot.mode(target, mode_str, *args)` | Set a mode |
| `bot.set_topic(channel, topic)` | Set channel topic |
| `bot.state` | Plugin state store (get/set/delete/keys/clear) |

The `message` object provides:

| Attribute | Description |
|-----------|-------------|
| `message.nick` | Sender's nickname |
| `message.prefix` | Full `nick!user@host` prefix |
| `message.command` | IRC command (PRIVMSG, JOIN, etc.) |
| `message.target` | First param (channel or nick) |
| `message.text` | Trailing text content |
| `message.is_channel` | Whether target is a channel |
| `message.params` | All message parameters |
| `message.tags` | IRCv3 message tags (dict) |

## Message Truncation

Messages are automatically split at UTF-8 safe boundaries to comply with
the IRC 512-byte line limit (RFC 2812). The overhead of `PRIVMSG <target> :`
and `\r\n` is accounted for, so plugins can send arbitrarily long text
without worrying about protocol limits.

### `!remind` -- Reminders

Set one-shot, repeating, or calendar-based reminders. Duration-based reminders
are in-memory and lost on restart. Calendar reminders (`at`, `yearly`) are
persisted via `bot.state` and restored on reconnect.

```
!remind 5m check the oven               One-shot after 5 minutes
!remind 2d12h renew cert                 One-shot after 2 days 12 hours
!remind every 1h drink water             Repeat every hour
!remind at 2027-06-15 deploy release     Fire at specific date (noon default)
!remind at 2027-06-15 14:30 deploy       Fire at specific date + time
!remind yearly 02-14 valentines          Every year on Feb 14 at noon
!remind yearly 12-25 09:00 merry xmas    Every year on Dec 25 at 09:00
!remind list                             Show all active reminders
!remind cancel abc123                    Cancel by ID
```

- Default time when omitted: **12:00** (noon) in configured timezone
- Timezone: `bot.timezone` in config, default `UTC`
- Leap day (02-29): fires on Feb 28 in non-leap years
- Past dates for `at` are rejected
- Calendar reminders survive restarts; duration-based do not

## Reconnect Backoff

On connection loss, the bot reconnects with exponential backoff and jitter:

- Initial delay: 5 seconds
- Growth: doubles each attempt (5s, 10s, 20s, 40s, ...)
- Cap: 300 seconds (5 minutes)
- Jitter: +/- 25% to avoid thundering herd
- Resets to 5s after a successful connection

### `!dork` -- Google Dork Query Builder

Generate Google dork queries for a target domain. Template-based, no HTTP
requests -- just outputs the query string for manual use.

```
!dork list                    List all dork categories
!dork admin example.com       Admin/login panel dorks
!dork files example.com       Exposed document dorks
```

Categories: admin, backup, cloud, config, creds, dirs, errors, exposed,
files, login.

### `!wayback` -- Wayback Machine Lookup

Check the Wayback Machine for archived snapshots of a URL.

```
!wayback example.com                Check latest snapshot
!wayback example.com/page 20240101  Check snapshot near a date
```

Auto-prepends `https://` if no scheme is provided. Uses the Wayback Machine
availability API.

### `!username` -- Username Enumeration

Check username availability across ~25 services using HTTP probes and
public JSON APIs.  Supports GitHub, GitLab, Reddit, Docker Hub, Keybase,
Dev.to, Twitch, Steam, and more.

```
!username list                     List services by category
!username john                     Full scan (~25 services)
!username john github              Check single service
```

Output format:

```
Full scan:
  Checking "john" across 25 services...
  john -- 8 found, 14 not found, 3 errors
  Found: GitHub, GitLab, Reddit, Twitch, Steam, PyPI, Docker Hub, Medium

Single service:
  GitHub: john -> found | https://github.com/john
  GitHub: john -> not found

List:
  Dev: GitHub, GitLab, Codeberg, ...
  Social: Reddit, Twitter/X, ...
  Media: Twitch, Spotify, ...
  Other: Steam, Pastebin, ...
```

- Username must match `[a-zA-Z0-9._-]{1,39}`
- Full scan sends acknowledgment before probing
- 8 parallel workers, 20s overall timeout
- Three check methods: HTTP status, JSON API, body search

### `!rss` -- RSS/Atom Feed Subscriptions

Subscribe RSS or Atom feeds to IRC channels with automatic polling and
new-item announcements.

```
!rss add <url> [name]       Subscribe a feed to this channel (admin)
!rss del <name>             Unsubscribe a feed (admin)
!rss list                   List feeds in this channel
!rss check <name>           Force-poll a feed now
```

- `add` and `del` require admin privileges
- All subcommands must be used in a channel (not PM)
- If `name` is omitted on `add`, it is derived from the URL hostname
- Names must be lowercase alphanumeric + hyphens, 1-20 characters
- Maximum 20 feeds per channel

Polling and announcements:

- Feeds are polled every 10 minutes by default
- On `add`, existing items are recorded without announcing (prevents flood)
- New items are announced as `[name] title -- link`
- Maximum 5 items announced per poll; excess shown as `... and N more`
- Titles are truncated to 80 characters
- Supports HTTP conditional requests (`ETag`, `If-Modified-Since`)
- 5 consecutive errors doubles the poll interval (max 1 hour)
- Feed subscriptions persist across bot restarts via `bot.state`

### `!yt` -- YouTube Channel Subscriptions

Follow YouTube channels and announce new videos in IRC channels. Accepts any
YouTube URL (video, channel, handle, shorts, embed) and resolves it to the
channel's Atom feed.

```
!yt follow <url> [name]    Follow a YouTube channel (admin)
!yt unfollow <name>        Unfollow a channel (admin)
!yt list                   List followed channels
!yt check <name>           Force-poll a channel now
```

- `follow` and `unfollow` require admin privileges
- All subcommands must be used in a channel (not PM)
- If `name` is omitted on `follow`, it is derived from the channel title
- Names must be lowercase alphanumeric + hyphens, 1-20 characters
- Maximum 20 channels per IRC channel

URL resolution:

- `/channel/UCXXX` URLs extract the channel ID directly
- All other YouTube URLs (handles, videos, shorts) fetch the page to resolve
- Constructs the YouTube Atom feed URL from the resolved channel ID

Polling and announcements:

- Channels are polled every 10 minutes by default
- On `follow`, existing videos are recorded without announcing (prevents flood)
- New videos are announced as `[name] Video Title -- https://www.youtube.com/watch?v=ID`
- Maximum 5 videos announced per poll; excess shown as `... and N more`
- Titles are truncated to 80 characters
- Supports HTTP conditional requests (`ETag`, `If-Modified-Since`)
- 5 consecutive errors doubles the poll interval (max 1 hour)
- Subscriptions persist across bot restarts via `bot.state`

### `!twitch` -- Twitch Livestream Notifications

Follow Twitch streamers and get notified when they go live. Uses Twitch's
public GQL endpoint (no API credentials required).

```
!twitch follow <username> [name]   Follow a streamer (admin)
!twitch unfollow <name>            Unfollow a streamer (admin)
!twitch list                       List followed streamers
!twitch check <name>               Check status now
```

- `follow` and `unfollow` require admin privileges
- All subcommands must be used in a channel (not PM)
- If `name` is omitted on `follow`, it defaults to the Twitch login (lowercase)
- Names must be lowercase alphanumeric + hyphens, 1-20 characters
- Twitch usernames must match `[a-zA-Z0-9_]{1,25}`
- Maximum 20 streamers per IRC channel

Polling and announcements:

- Streamers are polled every 2 minutes by default
- On `follow`, the current stream state is recorded without announcing
- Announcements fire on state transitions: offline to live, or new stream ID
- Format: `[name] is live: Stream Title (Game) -- https://twitch.tv/login`
- Game is omitted if not set; titles are truncated to 80 characters
- 5 consecutive errors doubles the poll interval (max 1 hour)
- Subscriptions persist across bot restarts via `bot.state`
- `list` shows live/error status indicators next to each streamer
- `check` forces an immediate poll and reports current status

### `!searx` -- SearXNG Web Search

Search the local SearXNG instance and display top results.

```
!searx <query...>     Search SearXNG and show top results
```

- Open to all users, channel only (no PM)
- Query is everything after `!searx`
- Shows top 3 results as `Title -- URL`
- Titles truncated to 80 characters
- Query limited to 200 characters

Output format:

```
Title One -- https://example.com/page1
Title Two -- https://example.com/page2
Title Three -- https://example.com/page3
```

### `!alert` -- Keyword Alert Subscriptions

Search keywords across multiple platforms (YouTube, Twitch, SearXNG) and
announce new results. Unlike `!rss`/`!yt`/`!twitch` which follow specific
channels/feeds, `!alert` searches keywords across all supported platforms
simultaneously.

```
!alert add <name> <keyword...>   Add keyword alert (admin)
!alert del <name>                Remove alert (admin)
!alert list                      List alerts
!alert check <name>              Force-poll now
```

- `add` and `del` require admin privileges
- All subcommands must be used in a channel (not PM)
- Name is required as the first argument after `add`; everything after is the keyword
- Names must be lowercase alphanumeric + hyphens, 1-20 characters
- Keywords: 1-100 characters, free-form text
- Maximum 20 alerts per IRC channel

Platforms searched:

- **YouTube** -- InnerTube search API (no auth required)
- **Twitch** -- Public GQL endpoint: live streams and VODs (no auth required)
- **SearXNG** -- Local SearXNG instance (no auth required)

Polling and announcements:

- Alerts are polled every 5 minutes by default
- On `add`, existing results are recorded without announcing (prevents flood)
- New results announced as `[name/yt] Title -- URL`, `[name/tw] Title -- URL`,
  or `[name/sx] Title -- URL`
- Maximum 5 items announced per platform per poll; excess shown as `... and N more`
- Titles are truncated to 80 characters
- Each platform maintains its own seen list (capped at 200 per platform)
- 5 consecutive errors doubles the poll interval (max 1 hour)
- Subscriptions persist across bot restarts via `bot.state`
- `list` shows error status indicators next to each alert
- `check` forces an immediate poll across all platforms