From b47c26dd149f3c028b7c7d23d2c2bd8a294eab51 Mon Sep 17 00:00:00 2001 From: Username Date: Sat, 20 Dec 2025 21:36:09 +0100 Subject: [PATCH] docs: update for v1.4.0 features - Add anti-flood, rate limiting, scheduled cleanup to feature lists - Update version to 1.4.0, test count to 205 - Document /pastes endpoint with query parameters - Add anti-flood fields to /challenge response - Update CLI docs with new commands (list, search, export) - Add decision log entries for recent features --- PROJECT.md | 8 ++++-- README.md | 12 ++++++++ ROADMAP.md | 42 +++++++++++++++++++++------ TASKLIST.md | 7 ++++- TODO.md | 5 +++- documentation/api.md | 67 ++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 129 insertions(+), 12 deletions(-) diff --git a/PROJECT.md b/PROJECT.md index ab9f41c..09b9b40 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -119,7 +119,7 @@ A self-hosted pastebin API that: ## Current Status -**Version:** 1.2.0 +**Version:** 1.4.0 ``` ┌─────────────────────────────────┬────────────────────────────────────────────┐ @@ -134,6 +134,8 @@ A self-hosted pastebin API that: │ Paste expiry │ Complete │ Content-hash deduplication │ Complete │ Proof-of-work │ Complete +│ Anti-flood (dynamic PoW) │ Complete +│ IP-based rate limiting │ Complete │ URL prefix support │ Complete │ /client endpoint │ Complete │ E2E encryption (CLI) │ Complete @@ -145,6 +147,8 @@ A self-hosted pastebin API that: │ Container deployment │ Complete │ Security tooling │ Complete │ CI/CD pipeline │ Complete -│ Test suite │ 147 tests passing +│ Scheduled cleanup │ Complete +│ CLI paste listing/search │ Complete +│ Test suite │ 205 tests passing └─────────────────────────────────┴────────────────────────────────────────────┘ ``` diff --git a/README.md b/README.md index 05207e4..0d44812 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,8 @@ A lightweight, secure pastebin REST API built with Flask. - **Abuse prevention** - Content-hash deduplication throttles repeated identical submissions - **Entropy enforcement** - Optional minimum entropy requirement to enforce client-side encryption - **Proof-of-work** - Configurable computational puzzle prevents automated spam +- **Anti-flood** - Dynamic PoW difficulty increases under attack, decays when abuse stops +- **Rate limiting** - Per-IP request throttling with auth user multiplier - **E2E encryption** - Client-side AES-256-GCM encryption with key in URL fragment (zero-knowledge) - **Burn-after-read** - Single-access pastes that auto-delete after first retrieval - **Custom expiry** - Per-paste expiry override via X-Expiry header @@ -229,6 +231,16 @@ Configuration via environment variables: | `FLASKPASTE_POW_DIFFICULTY` | `20` | PoW difficulty (leading zero bits, 0=disabled) | | `FLASKPASTE_POW_TTL` | `300` (5 min) | PoW challenge validity period | | `FLASKPASTE_POW_SECRET` | (auto) | Secret for signing PoW challenges | +| `FLASKPASTE_ANTIFLOOD` | `1` | Enable anti-flood (dynamic PoW difficulty) | +| `FLASKPASTE_ANTIFLOOD_WINDOW` | `60` | Anti-flood measurement window (seconds) | +| `FLASKPASTE_ANTIFLOOD_THRESHOLD` | `5` | Requests per window before difficulty increase | +| `FLASKPASTE_ANTIFLOOD_STEP` | `2` | Difficulty bits added per threshold breach | +| `FLASKPASTE_ANTIFLOOD_MAX` | `28` | Maximum PoW difficulty | +| `FLASKPASTE_ANTIFLOOD_DECAY` | `60` | Seconds before difficulty decreases | +| `FLASKPASTE_RATE_LIMIT` | `1` | Enable IP-based rate limiting | +| `FLASKPASTE_RATE_WINDOW` | `60` | Rate limit window (seconds) | +| `FLASKPASTE_RATE_MAX` | `10` | Max requests per window (anon) | +| `FLASKPASTE_RATE_AUTH_MULT` | `5` | Multiplier for authenticated users | | `FLASKPASTE_URL_PREFIX` | (empty) | URL prefix for reverse proxy deployments | | `FLASKPASTE_MIN_ENTROPY` | `0` | Min entropy bits/byte (0=disabled, 6.0=require encryption) | | `FLASKPASTE_MIN_ENTROPY_SIZE` | `256` | Only check entropy for content >= this size | diff --git a/ROADMAP.md b/ROADMAP.md index 8cc2cfb..a6f3740 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -2,7 +2,7 @@ ## Current State -FlaskPaste v1.2.0 is deployed with PKI integration and comprehensive security tooling. +FlaskPaste v1.4.0 is deployed with comprehensive security hardening and abuse prevention. **Implemented:** - Full REST API (CRUD operations) @@ -11,6 +11,8 @@ FlaskPaste v1.2.0 is deployed with PKI integration and comprehensive security to - Minimal PKI (CA generation, certificate issuance, revocation) - Content-hash deduplication (abuse prevention) - Proof-of-work spam prevention +- Anti-flood system (dynamic PoW difficulty under load) +- IP-based rate limiting (configurable per-IP limits) - Entropy enforcement (require encrypted uploads) - E2E encryption in CLI (AES-256-GCM, key in URL fragment) - URL prefix support for reverse proxy deployments @@ -18,11 +20,13 @@ FlaskPaste v1.2.0 is deployed with PKI integration and comprehensive security to - Automatic paste expiry - Burn-after-read pastes - Custom expiry per paste +- Scheduled cleanup (pastes, hashes, rate limits) - Security headers and request tracing - Container deployment support - Security tooling (ruff, bandit, mypy, pip-audit) - CI/CD pipeline with lint, security, and test jobs -- Comprehensive test suite (147 tests) +- CLI with list, search, update, export commands +- Comprehensive test suite (205 tests) ## Phase 1: Hardening (Complete) @@ -38,7 +42,7 @@ Focus: Production readiness and operational excellence. │ 4 │ Proxy trust validation │ Done │ 5 │ Proof-of-work spam prevention │ Done │ 6 │ Entropy enforcement │ Done -│ 7 │ Test coverage > 90% │ Done (147 tests) +│ 7 │ Test coverage > 90% │ Done (205 tests) │ 8 │ Documentation complete │ Done └───┴─────────────────────────────────┴────────────────────────────────────┘ ``` @@ -67,14 +71,26 @@ Focus: User-requested enhancements within scope. ┌───┬─────────────────────────────────┬────────────────────────────────────┐ │ # │ Feature │ Status ├───┼─────────────────────────────────┼────────────────────────────────────┤ -│ 1 │ E2E encryption (client-side) │ Done (CLI -e flag, zero-knowledge) +│ 1 │ E2E encryption (client-side) │ Done (CLI encrypts by default) │ 2 │ URL prefix support │ Done │ 3 │ Custom expiry per paste │ Done (X-Expiry header) │ 4 │ Burn-after-read option │ Done (X-Burn-After-Read header) │ 5 │ Minimal PKI (CA + issuance) │ Done +│ 6 │ Anti-flood (dynamic PoW) │ Done (v1.4.0) +│ 7 │ IP-based rate limiting │ Done (v1.4.0) +│ 8 │ Scheduled cleanup │ Done (v1.4.0) └───┴─────────────────────────────────┴────────────────────────────────────┘ ``` +### Anti-Flood System (v1.4.0) + +Dynamic proof-of-work difficulty that increases under abuse: +- Base difficulty: 20 bits (configurable) +- Threshold: 5 requests per 60s window triggers increase +- Step: +2 bits per threshold breach +- Maximum: 28 bits +- Decay: Returns to base after 60s of normal traffic + ### PKI Features Integrated certificate authority for mTLS: @@ -103,15 +119,20 @@ Focus: Integration with external systems. ### CLI Client (Complete) -Standalone Python CLI with encryption and PKI support: -- `fpaste create file.txt` - Create paste from file -- `fpaste create -e file.txt` - Create encrypted paste (E2E) +Standalone Python CLI with encryption, PKI, and paste management: +- `fpaste file.txt` - Create encrypted paste (file path shortcut) +- `fpaste create -E file.txt` - Create unencrypted paste - `fpaste get ` - Get paste (auto-decrypts with URL fragment key) - `fpaste delete ` - Delete paste -- `fpaste info` - Show server info +- `fpaste info` - Show server info (includes PoW difficulty) +- `fpaste list` - List your pastes +- `fpaste search --type image/*` - Search pastes by type/date +- `fpaste update ` - Update paste content/metadata +- `fpaste export -o dir/` - Export all pastes to directory - `fpaste pki status` - Show PKI status - `fpaste pki issue -n "name"` - Request client certificate - `fpaste pki revoke ` - Revoke certificate +- Automatic retry on PoW failure (max 5 attempts) - Config file for server URL and cert fingerprint - Downloadable via `curl https://server/client > fpaste` @@ -144,6 +165,11 @@ These features will not be implemented: | 2024-12 | Minimal PKI | Self-contained mTLS without external CA | 2024-12 | Security tooling (ruff/bandit) | Code quality and security scanning | 2024-12 | CI/CD with job dependencies | Tests wait for lint to pass +| 2024-12 | Anti-flood dynamic PoW | Adaptive difficulty under attack +| 2024-12 | IP-based rate limiting | Per-IP request throttling +| 2024-12 | Scheduled cleanup (in-process) | No external cron needed +| 2024-12 | CLI encrypt-by-default | Security-first design +| 2024-12 | CLI retry on PoW failure | Graceful handling of stale tokens ## Review Schedule diff --git a/TASKLIST.md b/TASKLIST.md index 3039795..6209f9d 100644 --- a/TASKLIST.md +++ b/TASKLIST.md @@ -16,7 +16,6 @@ Prioritized, actionable tasks. Each task is small and completable in one session | Status | Task |--------|-------------------------------------------------------------- -| ☐ | Add paste listing for authenticated users | ☐ | Add rate limit headers (X-RateLimit-*) ## Priority 3: Quality @@ -38,6 +37,12 @@ Prioritized, actionable tasks. Each task is small and completable in one session | Date | Task |------------|-------------------------------------------------------------- +| 2024-12 | Implement anti-flood (dynamic PoW difficulty) +| 2024-12 | Implement IP-based rate limiting +| 2024-12 | Add scheduled cleanup (pastes, hashes, rate limits) +| 2024-12 | Add CLI list/search/update/export commands +| 2024-12 | Add CLI PoW retry (max 5 attempts) +| 2024-12 | Add paste listing for authenticated users | 2024-12 | Implement minimal PKI (CA, issuance, revocation) | 2024-12 | Add security tooling (ruff, bandit, mypy, pip-audit) | 2024-12 | Create Makefile with dev workflow targets diff --git a/TODO.md b/TODO.md index 0552c7c..c1777f0 100644 --- a/TODO.md +++ b/TODO.md @@ -26,10 +26,13 @@ Unstructured intake buffer for ideas, issues, and observations. Items here are r ## Questions -- Should expired paste cleanup run in-process or via external cron? - Certificate renewal: reissue with same CN or require new request? - Should revoked certs be purged after grace period? +## Resolved + +- Expired paste cleanup runs in-process via before_request hook (no cron needed) + ## Debt - Mypy has pre-existing type errors (runs with --ignore-missing-imports) diff --git a/documentation/api.md b/documentation/api.md index 91f142f..c6d9691 100644 --- a/documentation/api.md +++ b/documentation/api.md @@ -52,13 +52,30 @@ Host: localhost:5000 "enabled": true, "nonce": "a1b2c3d4e5f6a7b8a1b2c3d4e5f6a7b8", "difficulty": 20, + "base_difficulty": 20, + "elevated": false, "expires": 1700000300, "token": "a1b2c3d4...:1700000300:20:signature" } ``` +**Response (PoW enabled, anti-flood active):** +```json +{ + "enabled": true, + "nonce": "a1b2c3d4e5f6a7b8a1b2c3d4e5f6a7b8", + "difficulty": 24, + "base_difficulty": 20, + "elevated": true, + "expires": 1700000300, + "token": "a1b2c3d4...:1700000300:24:signature" +} +``` + The client must find a number `N` such that `SHA256(nonce + ":" + N)` has at least `difficulty` leading zero bits. +When `elevated` is true, the difficulty has been increased due to high request volume (anti-flood protection). The `base_difficulty` shows the normal difficulty level. + --- ### GET /health @@ -123,6 +140,55 @@ Host: localhost:5000 --- +### GET /pastes + +List pastes for the authenticated user. Requires authentication. + +**Request:** +```http +GET /pastes HTTP/1.1 +Host: localhost:5000 +X-SSL-Client-SHA1: a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2 +``` + +**Query Parameters:** +| Parameter | Type | Description | +|-----------|------|-------------| +| `limit` | int | Maximum results (default: 100, max: 1000) | +| `offset` | int | Skip first N results (default: 0) | +| `type` | string | Filter by MIME type (glob pattern, e.g., `image/*`) | +| `after` | int | Created after timestamp (Unix epoch) | +| `before` | int | Created before timestamp (Unix epoch) | + +**Response (200 OK):** +```json +{ + "pastes": [ + { + "id": "a1b2c3d4e5f6", + "size": 1234, + "mime_type": "text/plain", + "created_at": "2024-12-20T10:30:00Z", + "expires_at": "2024-12-25T10:30:00Z", + "burn_after_read": false, + "password_protected": false + } + ], + "total": 42, + "limit": 100, + "offset": 0 +} +``` + +**Response (401 Unauthorized):** +```json +{ + "error": "Authentication required" +} +``` + +--- + ### POST / Create a new paste. @@ -223,6 +289,7 @@ Password protected content | 400 | Paste too small (below minimum size) | | 413 | Paste too large | | 429 | Duplicate content rate limit exceeded | +| 429 | Rate limit exceeded (per-IP throttling) | **Size Limits:** - Minimum: disabled by default (`FLASKPASTE_MIN_SIZE`, e.g. 64 bytes for encryption enforcement)