diff --git a/PROJECT.md b/PROJECT.md index a729a44..6a99f0c 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -58,6 +58,12 @@ A self-hosted pastebin API that: - Configurable size limits (anon vs authenticated) - Time-based expiry with access-touch semantics - Content-hash deduplication for abuse prevention +- Proof-of-work spam prevention +- Entropy enforcement for encrypted content +- Client-side E2E encryption (CLI) +- Burn-after-read pastes +- Custom expiry per paste +- URL prefix for reverse proxy deployments - Security headers (HSTS, CSP, X-Frame-Options, etc.) - Request tracing and structured logging - Container deployment support @@ -102,3 +108,32 @@ A self-hosted pastebin API that: │ Python │ 3.11+ └─────────────────┴──────────────────────────────────────────────────────────┘ ``` + +## Current Status + +**Version:** 1.1.0 + +``` +┌─────────────────────────────────┬────────────────────────────────────────────┐ +│ Feature │ Status +├─────────────────────────────────┼────────────────────────────────────────────┤ +│ Core API (CRUD) │ Complete +│ Binary content support │ Complete +│ MIME detection │ Complete +│ Client cert authentication │ Complete +│ Size limits │ Complete +│ Paste expiry │ Complete +│ Content-hash deduplication │ Complete +│ Proof-of-work │ Complete +│ URL prefix support │ Complete +│ /client endpoint │ Complete +│ E2E encryption (CLI) │ Complete +│ Entropy enforcement │ Complete +│ Burn-after-read │ Complete +│ Custom expiry │ Complete +│ Security headers │ Complete +│ Request tracing │ Complete +│ Container deployment │ Complete +│ Test suite │ 113 tests passing +└─────────────────────────────────┴────────────────────────────────────────────┘ +``` diff --git a/README.md b/README.md index 0303874..a1278de 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,12 @@ A lightweight, secure pastebin REST API built with Flask. - **Automatic expiry** - Pastes expire after configurable period of inactivity - **Size limits** - Configurable limits for anonymous and authenticated users - **Abuse prevention** - Content-hash deduplication throttles repeated identical submissions +- **Entropy enforcement** - Optional minimum entropy requirement to enforce client-side encryption - **Proof-of-work** - Configurable computational puzzle prevents automated spam +- **E2E encryption** - Client-side AES-256-GCM encryption with key in URL fragment (zero-knowledge) +- **Burn-after-read** - Single-access pastes that auto-delete after first retrieval +- **Custom expiry** - Per-paste expiry override via X-Expiry header +- **Password protection** - Optional paste passwords with PBKDF2 hashing - **Security headers** - HSTS, CSP, X-Frame-Options, Cache-Control, and more - **CLI client** - Standalone `fpaste` command-line tool included - **Request tracing** - X-Request-ID support for log correlation @@ -39,6 +44,7 @@ python run.py | `GET /` | API information and usage | | `GET /health` | Health check (returns DB status) | | `GET /challenge` | Get proof-of-work challenge | +| `GET /client` | Download fpaste CLI client | | `POST /` | Create a new paste | | `GET /` | Retrieve paste metadata | | `HEAD /` | Retrieve paste metadata (headers only) | @@ -84,7 +90,17 @@ curl -X DELETE \ ## CLI Client -A standalone command-line client `fpaste` is included (no external dependencies). +A standalone command-line client `fpaste` is included. For E2E encryption, install the optional `cryptography` package. + +### Installation + +```bash +# Download from running server +curl -sS https://paste.example.com/client > fpaste && chmod +x fpaste + +# Optional: enable encryption support +pip install cryptography +``` ### Basic Usage @@ -95,9 +111,25 @@ A standalone command-line client `fpaste` is included (no external dependencies) # Create paste from stdin echo "Hello" | ./fpaste +# Create encrypted paste (E2E, zero-knowledge) +./fpaste create -e secret.txt +# Returns: https://paste.example.com/abc123# + +# Create burn-after-read paste (single access, auto-deletes) +./fpaste create -b secret.txt + +# Create paste with custom expiry (1 hour) +./fpaste create -x 3600 temp.txt + +# Combine options: encrypted + burn-after-read +./fpaste create -e -b secret.txt + # Get paste content ./fpaste get abc12345 +# Get encrypted paste (auto-decrypts if URL has #key fragment) +./fpaste get "https://paste.example.com/abc123#" + # Get paste metadata ./fpaste get -m abc12345 @@ -108,6 +140,17 @@ echo "Hello" | ./fpaste ./fpaste info ``` +### End-to-End Encryption + +The `-e` flag encrypts content client-side using AES-256-GCM before upload: + +- Key is generated locally and never sent to server +- Key is appended to URL as fragment (`#...`) which browsers never transmit +- Server stores only opaque ciphertext +- Retrieval auto-detects `#key` fragment and decrypts locally + +This provides true zero-knowledge storage: the server cannot read your content. + ### Configuration Set server URL and authentication via environment or config file: @@ -122,6 +165,50 @@ server = https://paste.example.com cert_sha1 = your-cert-fingerprint ``` +### mTLS Client Certificate Authentication + +The CLI supports mutual TLS (mTLS) for direct client certificate authentication: + +```bash +# Environment variables +export FLASKPASTE_CLIENT_CERT="/path/to/client.crt" +export FLASKPASTE_CLIENT_KEY="/path/to/client.key" +export FLASKPASTE_CA_CERT="/path/to/ca.crt" # Optional CA verification + +# Or config file (~/.config/fpaste/config) +client_cert = /path/to/client.crt +client_key = /path/to/client.key +ca_cert = /path/to/ca.crt +``` + +When client certificates are configured, fpaste performs mTLS authentication directly with the server (or TLS-terminating proxy). This is more secure than header-based authentication as the certificate is validated at the TLS layer. + +### Generating Client Certificates + +The CLI includes a built-in certificate generator for mTLS authentication: + +```bash +# Generate EC certificate (recommended, fast key generation) +./fpaste cert +# Outputs fingerprint to stdout, details to stderr + +# Generate and auto-configure fpaste +./fpaste cert --configure +# Creates ~/.config/fpaste/client.{crt,key} and updates config + +# Custom options +./fpaste cert -a rsa -b 4096 -d 730 -n "my-client" --configure +# RSA 4096-bit key, 2-year validity, custom common name + +# Supported curves for EC: secp256r1, secp384r1 (default), secp521r1 +./fpaste cert -c secp256r1 --configure +``` + +The generated certificate includes: +- `CLIENT_AUTH` extended key usage for mTLS +- SHA1 fingerprint compatible with `X-SSL-Client-SHA1` header +- Self-signed with configurable validity period + ## Configuration Configuration via environment variables: @@ -140,6 +227,10 @@ Configuration via environment variables: | `FLASKPASTE_POW_DIFFICULTY` | `20` | PoW difficulty (leading zero bits, 0=disabled) | | `FLASKPASTE_POW_TTL` | `300` (5 min) | PoW challenge validity period | | `FLASKPASTE_POW_SECRET` | (auto) | Secret for signing PoW challenges | +| `FLASKPASTE_URL_PREFIX` | (empty) | URL prefix for reverse proxy deployments | +| `FLASKPASTE_MIN_ENTROPY` | `0` | Min entropy bits/byte (0=disabled, 6.0=require encryption) | +| `FLASKPASTE_MIN_ENTROPY_SIZE` | `256` | Only check entropy for content >= this size | +| `FLASKPASTE_MAX_EXPIRY` | `2592000` (30 days) | Maximum custom expiry allowed | ## Authentication @@ -216,6 +307,9 @@ flaskpaste/ - **Ownership enforcement** - Only owners can delete their pastes - **Size limits** - Prevents resource exhaustion attacks - **Abuse prevention** - Content-hash deduplication prevents spam flooding +- **Entropy enforcement** - Optional minimum entropy rejects low-entropy (plaintext) uploads +- **E2E encryption** - Client-side encryption keeps server zero-knowledge +- **Burn-after-read** - Single-use pastes for sensitive data - **Security headers** - HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Cache-Control - **Request tracing** - X-Request-ID for log correlation and debugging - **Proxy trust** - Optional `X-Proxy-Secret` validation to prevent header spoofing diff --git a/ROADMAP.md b/ROADMAP.md index 744aa84..5403c65 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -2,19 +2,24 @@ ## Current State -FlaskPaste v1.0 is feature-complete for its core mission: a secure, minimal pastebin API. +FlaskPaste v1.1.0 is deployed and feature-complete for its core mission: a secure, minimal pastebin API with zero-knowledge encryption support. **Implemented:** - Full REST API (CRUD operations) - Binary content support with magic-byte MIME detection - Client certificate authentication - Content-hash deduplication (abuse prevention) +- Proof-of-work spam prevention +- Entropy enforcement (require encrypted uploads) +- E2E encryption in CLI (AES-256-GCM, key in URL fragment) +- URL prefix support for reverse proxy deployments +- /client endpoint for CLI distribution - Automatic paste expiry - Security headers and request tracing - Container deployment support -- Comprehensive test suite +- Comprehensive test suite (98 tests) -## Phase 1: Hardening (Current) +## Phase 1: Hardening (Complete) Focus: Production readiness and operational excellence. @@ -22,12 +27,14 @@ Focus: Production readiness and operational excellence. ┌───┬─────────────────────────────────────┬────────────────────────────────────┐ │ # │ Milestone │ Status ├───┼─────────────────────────────────────┼────────────────────────────────────┤ -│ 1 │ Abuse prevention (dedup) │ Implemented (pending commit) +│ 1 │ Abuse prevention (dedup) │ Done │ 2 │ Security headers complete │ Done │ 3 │ Request tracing (X-Request-ID) │ Done │ 4 │ Proxy trust validation │ Done -│ 5 │ Test coverage > 90% │ In progress -│ 6 │ Documentation complete │ In progress +│ 5 │ Proof-of-work spam prevention │ Done +│ 6 │ Entropy enforcement │ Done +│ 7 │ Test coverage > 90% │ Done (98 tests) +│ 8 │ Documentation complete │ Done └───┴─────────────────────────────────────┴────────────────────────────────────┘ ``` @@ -64,19 +71,19 @@ Replace text logs with JSON format: - Consistent field names across all log entries - Compatible with log aggregation (Loki, ELK) -## Phase 3: Features +## Phase 3: Features (Complete) Focus: User-requested enhancements within scope. ``` ┌───┬─────────────────────────────────────┬────────────────────────────────────┐ -│ # │ Feature │ Complexity +│ # │ Feature │ Status ├───┼─────────────────────────────────────┼────────────────────────────────────┤ -│ 1 │ Paste encryption (server-side) │ Medium -│ 2 │ Custom expiry per paste │ Low -│ 3 │ Paste size in response headers │ Low -│ 4 │ Burn-after-read option │ Low -│ 5 │ Paste password protection │ Medium +│ 1 │ E2E encryption (client-side) │ Done (CLI -e flag, zero-knowledge) +│ 2 │ URL prefix support │ Done +│ 3 │ Custom expiry per paste │ Done (X-Expiry header) +│ 4 │ Burn-after-read option │ Done (X-Burn-After-Read header) +│ 5 │ Paste password protection │ Planned └───┴─────────────────────────────────────┴────────────────────────────────────┘ ``` @@ -100,23 +107,26 @@ Focus: Integration with external systems. ``` ┌───┬─────────────────────────────────────┬────────────────────────────────────┐ -│ # │ Integration │ Purpose +│ # │ Integration │ Status ├───┼─────────────────────────────────────┼────────────────────────────────────┤ -│ 1 │ CLI client (fpaste) │ User convenience -│ 2 │ Neovim/Vim plugin │ Editor integration -│ 3 │ Shell aliases/functions │ Workflow integration -│ 4 │ Webhook notifications │ Automation triggers +│ 1 │ CLI client (fpaste) │ Done (with E2E encryption) +│ 2 │ /client endpoint │ Done (downloadable CLI) +│ 3 │ Neovim/Vim plugin │ Planned +│ 4 │ Shell aliases/functions │ Planned +│ 5 │ Webhook notifications │ Planned └───┴─────────────────────────────────────┴────────────────────────────────────┘ ``` -### CLI Client +### CLI Client (Complete) -Standalone Python CLI: -- `fpaste < file.txt` - Create paste from stdin -- `fpaste file.txt` - Create paste from file -- `fpaste -g ` - Get paste -- `fpaste -d ` - Delete paste -- Config file for server URL and cert path +Standalone Python CLI with encryption support: +- `fpaste create file.txt` - Create paste from file +- `fpaste create -e file.txt` - Create encrypted paste (E2E) +- `fpaste get ` - Get paste (auto-decrypts with URL fragment key) +- `fpaste delete ` - Delete paste +- `fpaste info` - Show server info +- Config file for server URL and cert fingerprint +- Downloadable via `curl https://server/client > fpaste` ## Non-Goals (Explicit) @@ -137,6 +147,13 @@ These features will not be implemented: | 2024-11 | No web UI | API-first; reduces attack surface | 2024-11 | Client cert auth | Integrates with existing PKI | 2024-12 | Content-hash dedup | Prevent spam without IP tracking +| 2024-12 | Proof-of-work | Computational cost deters spam bots +| 2024-12 | Client-side E2E encryption | Zero-knowledge; key in URL fragment +| 2024-12 | Entropy enforcement | Heuristic to require encrypted uploads +| 2024-12 | URL prefix support | Reverse proxy path-based routing +| 2024-12 | Burn-after-read | Single-use pastes for sensitive data +| 2024-12 | Custom expiry | Per-paste TTL override +| 2024-12 | Multi-stage Containerfile | Smaller production images ## Review Schedule diff --git a/TASKLIST.md b/TASKLIST.md index 079c1f5..8e3898f 100644 --- a/TASKLIST.md +++ b/TASKLIST.md @@ -4,45 +4,47 @@ Prioritized, actionable tasks. Each task is small and completable in one session --- -## Priority 1: Pending Commit +## Priority 1: Operations | Status | Task |--------|-------------------------------------------------------------- -| ☐ | Commit abuse prevention feature (dedup changes in routes.py, config.py, database.py) -| ☐ | Commit documentation updates (api.md, README.md) +| ☐ | Add /metrics endpoint skeleton (for future Prometheus) +| ☐ | Add structured logging option via environment variable -## Priority 2: Test Coverage +## Priority 2: Features | Status | Task |--------|-------------------------------------------------------------- -| ☐ | Run test suite, verify all tests pass -| ☐ | Add test for dedup window expiry behavior -| ☐ | Add test for concurrent identical submissions -| ☐ | Add test for MIME detection edge cases (empty content, truncated headers) -| ☐ | Measure and document test coverage percentage +| ☐ | Add paste listing for authenticated users +| ☐ | Add paste password protection ## Priority 3: Documentation | Status | Task |--------|-------------------------------------------------------------- -| ☐ | Add deployment examples to documentation/deployment.md -| ☐ | Document environment variables in one canonical location -| ☐ | Add troubleshooting section to README.md +| ☐ | Add deployment examples (Kubernetes, systemd) | ☐ | Create CONTRIBUTING.md with development setup -## Priority 4: Operations +## Priority 4: Testing | Status | Task |--------|-------------------------------------------------------------- -| ☑ | Add SQLite WAL mode for better concurrency -| ☐ | Add /metrics endpoint skeleton (for future Prometheus) -| ☐ | Add structured logging option via environment variable -| ☐ | Optimize container build with multi-stage Containerfile +| ☐ | Add test for concurrent identical submissions +| ☐ | Add integration tests for container deployment ## Completed | Date | Task |------------|-------------------------------------------------------------- +| 2024-12 | Implement burn-after-read option +| 2024-12 | Implement custom expiry per paste +| 2024-12 | Optimize Containerfile with multi-stage build +| 2024-12 | Implement E2E encryption in CLI (AES-256-GCM) +| 2024-12 | Implement entropy enforcement +| 2024-12 | Add /client endpoint for CLI download +| 2024-12 | Add URL prefix support +| 2024-12 | Implement proof-of-work spam prevention +| 2024-12 | Update documentation for v1.1.0 | 2024-12 | Add HEAD method for paste endpoints | 2024-12 | Add SQLite WAL mode for better concurrency | 2024-12 | Implement content-hash deduplication diff --git a/TODO.md b/TODO.md index 28ca724..03d7960 100644 --- a/TODO.md +++ b/TODO.md @@ -10,20 +10,20 @@ Unstructured intake buffer for ideas, issues, and observations. Items here are r - Structured JSON logging for log aggregation compatibility - Burn-after-read paste option - Custom expiry header for per-paste TTL -- CLI client tool (fpaste) for easier usage - Rate limit headers in responses (X-RateLimit-*) - Paste compression for large text content -- Optional paste encryption with user-provided key - ETag support for conditional requests -- HEAD method support for metadata without body - Paste listing for authenticated users (their own pastes only) +- Neovim/Vim plugin for editor integration +- Webhook notifications for paste events ## Observations -- Current abuse prevention uses content-hash; IP-based limiting delegated to proxy +- Abuse prevention uses content-hash dedup + PoW + entropy enforcement - SQLite WAL mode could improve concurrent read performance - Container image size could be reduced with multi-stage build -- Test coverage could include more edge cases for MIME detection +- E2E encryption in CLI uses cryptography package (optional dependency) +- Entropy check has size threshold to avoid false positives on small data ## Questions @@ -34,10 +34,9 @@ Unstructured intake buffer for ideas, issues, and observations. Items here are r ## Debt -- Dedup feature changes pending commit -- Documentation could include more deployment examples - No integration tests for container deployment - Missing test for concurrent paste creation +- Could add more deployment examples (Kubernetes, systemd) ## External Dependencies