diff --git a/PROJECT.md b/PROJECT.md index 6a99f0c..ab9f41c 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -18,8 +18,9 @@ Public pastebin services present risks: A self-hosted pastebin API that: - Stores pastes locally in SQLite - Supports client certificate authentication via reverse proxy +- Provides optional built-in PKI for certificate management - Automatically expires content based on access patterns -- Prevents abuse through content-hash deduplication +- Prevents abuse through content-hash deduplication and proof-of-work - Serves text and binary content with proper MIME detection - Runs behind any reverse proxy (nginx, HAProxy, Caddy) @@ -32,6 +33,7 @@ A self-hosted pastebin API that: │ Security │ Zero injection vulnerabilities │ │ All OWASP headers implemented │ │ Input validation on all endpoints +│ │ Security scanning in CI (bandit) ├────────────────────────────────┼────────────────────────────────────────────┤ │ Reliability │ SQLite ACID guarantees │ │ Graceful degradation on errors @@ -44,6 +46,7 @@ A self-hosted pastebin API that: │ Operability │ Container-ready (Podman/Docker) │ │ Gunicorn-compatible WSGI │ │ Request tracing via X-Request-ID +│ │ Prometheus metrics endpoint └────────────────────────────────┴────────────────────────────────────────────┘ ``` @@ -55,6 +58,7 @@ A self-hosted pastebin API that: - Text and binary content support - Magic-byte MIME type detection - Client certificate authentication (via proxy header) +- Built-in PKI (CA generation, certificate issuance, revocation) - Configurable size limits (anon vs authenticated) - Time-based expiry with access-touch semantics - Content-hash deduplication for abuse prevention @@ -68,6 +72,7 @@ A self-hosted pastebin API that: - Request tracing and structured logging - Container deployment support - SQLite storage +- CI/CD with security scanning ### Out of Scope @@ -90,7 +95,7 @@ A self-hosted pastebin API that: ## Assumptions - Deployment behind a TLS-terminating reverse proxy -- Client certificates managed externally (PKI, mTLS) +- Client certificates managed externally or via built-in PKI - Operators have container runtime (Podman/Docker) or Python venv - SQLite performance sufficient for expected load @@ -104,14 +109,17 @@ A self-hosted pastebin API that: │ Database │ SQLite 3 (built-in) │ WSGI Server │ Gunicorn (production) │ Container │ Podman / Docker -│ Testing │ pytest, pytest-cov +│ Testing │ pytest +│ Linting │ ruff, mypy +│ Security │ bandit, pip-audit +│ CI/CD │ Gitea Actions │ Python │ 3.11+ └─────────────────┴──────────────────────────────────────────────────────────┘ ``` ## Current Status -**Version:** 1.1.0 +**Version:** 1.2.0 ``` ┌─────────────────────────────────┬────────────────────────────────────────────┐ @@ -121,6 +129,7 @@ A self-hosted pastebin API that: │ Binary content support │ Complete │ MIME detection │ Complete │ Client cert authentication │ Complete +│ Built-in PKI │ Complete │ Size limits │ Complete │ Paste expiry │ Complete │ Content-hash deduplication │ Complete @@ -134,6 +143,8 @@ A self-hosted pastebin API that: │ Security headers │ Complete │ Request tracing │ Complete │ Container deployment │ Complete -│ Test suite │ 113 tests passing +│ Security tooling │ Complete +│ CI/CD pipeline │ Complete +│ Test suite │ 147 tests passing └─────────────────────────────────┴────────────────────────────────────────────┘ ``` diff --git a/ROADMAP.md b/ROADMAP.md index 5403c65..8cc2cfb 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -2,12 +2,13 @@ ## Current State -FlaskPaste v1.1.0 is deployed and feature-complete for its core mission: a secure, minimal pastebin API with zero-knowledge encryption support. +FlaskPaste v1.2.0 is deployed with PKI integration and comprehensive security tooling. **Implemented:** - Full REST API (CRUD operations) - Binary content support with magic-byte MIME detection - Client certificate authentication +- Minimal PKI (CA generation, certificate issuance, revocation) - Content-hash deduplication (abuse prevention) - Proof-of-work spam prevention - Entropy enforcement (require encrypted uploads) @@ -15,116 +16,102 @@ FlaskPaste v1.1.0 is deployed and feature-complete for its core mission: a secur - URL prefix support for reverse proxy deployments - /client endpoint for CLI distribution - Automatic paste expiry +- Burn-after-read pastes +- Custom expiry per paste - Security headers and request tracing - Container deployment support -- Comprehensive test suite (98 tests) +- Security tooling (ruff, bandit, mypy, pip-audit) +- CI/CD pipeline with lint, security, and test jobs +- Comprehensive test suite (147 tests) ## Phase 1: Hardening (Complete) Focus: Production readiness and operational excellence. ``` -┌───┬─────────────────────────────────────┬────────────────────────────────────┐ -│ # │ Milestone │ Status -├───┼─────────────────────────────────────┼────────────────────────────────────┤ -│ 1 │ Abuse prevention (dedup) │ Done -│ 2 │ Security headers complete │ Done -│ 3 │ Request tracing (X-Request-ID) │ Done -│ 4 │ Proxy trust validation │ Done -│ 5 │ Proof-of-work spam prevention │ Done -│ 6 │ Entropy enforcement │ Done -│ 7 │ Test coverage > 90% │ Done (98 tests) -│ 8 │ Documentation complete │ Done -└───┴─────────────────────────────────────┴────────────────────────────────────┘ +┌───┬─────────────────────────────────┬────────────────────────────────────┐ +│ # │ Milestone │ Status +├───┼─────────────────────────────────┼────────────────────────────────────┤ +│ 1 │ Abuse prevention (dedup) │ Done +│ 2 │ Security headers complete │ Done +│ 3 │ Request tracing (X-Request-ID) │ Done +│ 4 │ Proxy trust validation │ Done +│ 5 │ Proof-of-work spam prevention │ Done +│ 6 │ Entropy enforcement │ Done +│ 7 │ Test coverage > 90% │ Done (147 tests) +│ 8 │ Documentation complete │ Done +└───┴─────────────────────────────────┴────────────────────────────────────┘ ``` -## Phase 2: Operations +## Phase 2: Operations (Complete) Focus: Deployment, monitoring, and maintenance tooling. ``` -┌───┬─────────────────────────────────────┬────────────────────────────────────┐ -│ # │ Milestone │ Dependencies -├───┼─────────────────────────────────────┼────────────────────────────────────┤ -│ 1 │ Prometheus metrics endpoint │ None -│ 2 │ Structured JSON logging │ None -│ 3 │ Admin API (stats, cleanup) │ Auth improvements -│ 4 │ Ansible deployment role │ None -│ 5 │ CI/CD pipeline │ Container registry access -└───┴─────────────────────────────────────┴────────────────────────────────────┘ +┌───┬─────────────────────────────────┬────────────────────────────────────┐ +│ # │ Milestone │ Status +├───┼─────────────────────────────────┼────────────────────────────────────┤ +│ 1 │ Prometheus metrics endpoint │ Done (prometheus-flask-exporter) +│ 2 │ Structured JSON logging │ Done (production mode) +│ 3 │ Security tooling (lint/scan) │ Done (ruff, bandit, mypy) +│ 4 │ CI/CD pipeline │ Done (Gitea Actions) +│ 5 │ Multi-stage Containerfile │ Done +└───┴─────────────────────────────────┴────────────────────────────────────┘ ``` -### Prometheus Metrics - -Expose `/metrics` endpoint with: -- `flaskpaste_pastes_total` (counter) -- `flaskpaste_pastes_created` (counter) -- `flaskpaste_pastes_deleted` (counter) -- `flaskpaste_pastes_expired` (counter) -- `flaskpaste_storage_bytes` (gauge) -- `flaskpaste_request_duration_seconds` (histogram) - -### Structured Logging - -Replace text logs with JSON format: -- Timestamp, level, message, request_id -- Consistent field names across all log entries -- Compatible with log aggregation (Loki, ELK) - ## Phase 3: Features (Complete) Focus: User-requested enhancements within scope. ``` -┌───┬─────────────────────────────────────┬────────────────────────────────────┐ -│ # │ Feature │ Status -├───┼─────────────────────────────────────┼────────────────────────────────────┤ -│ 1 │ E2E encryption (client-side) │ Done (CLI -e flag, zero-knowledge) -│ 2 │ URL prefix support │ Done -│ 3 │ Custom expiry per paste │ Done (X-Expiry header) -│ 4 │ Burn-after-read option │ Done (X-Burn-After-Read header) -│ 5 │ Paste password protection │ Planned -└───┴─────────────────────────────────────┴────────────────────────────────────┘ +┌───┬─────────────────────────────────┬────────────────────────────────────┐ +│ # │ Feature │ Status +├───┼─────────────────────────────────┼────────────────────────────────────┤ +│ 1 │ E2E encryption (client-side) │ Done (CLI -e flag, zero-knowledge) +│ 2 │ URL prefix support │ Done +│ 3 │ Custom expiry per paste │ Done (X-Expiry header) +│ 4 │ Burn-after-read option │ Done (X-Burn-After-Read header) +│ 5 │ Minimal PKI (CA + issuance) │ Done +└───┴─────────────────────────────────┴────────────────────────────────────┘ ``` -### Burn-After-Read +### PKI Features -Single-access pastes that delete after first retrieval: -- `POST /` with `X-Burn-After-Read: true` header -- Paste deleted after first `GET //raw` -- Metadata `GET /` does not trigger burn +Integrated certificate authority for mTLS: +- `POST /pki/ca` - Generate CA (first-run bootstrap) +- `GET /pki/status` - CA status and fingerprint +- `GET /pki/ca.crt` - Download CA certificate +- `POST /pki/issue` - Issue client certificate +- `POST /pki/revoke/` - Revoke certificate +- CLI: `fpaste pki status`, `fpaste pki issue`, `fpaste pki revoke` -### Custom Expiry - -Allow per-paste expiry override: -- `POST /` with `X-Expiry: 3600` header (seconds) -- Capped at server maximum (e.g., 30 days) -- Default unchanged for pastes without header - -## Phase 4: Ecosystem +## Phase 4: Ecosystem (In Progress) Focus: Integration with external systems. ``` -┌───┬─────────────────────────────────────┬────────────────────────────────────┐ -│ # │ Integration │ Status -├───┼─────────────────────────────────────┼────────────────────────────────────┤ -│ 1 │ CLI client (fpaste) │ Done (with E2E encryption) -│ 2 │ /client endpoint │ Done (downloadable CLI) -│ 3 │ Neovim/Vim plugin │ Planned -│ 4 │ Shell aliases/functions │ Planned -│ 5 │ Webhook notifications │ Planned -└───┴─────────────────────────────────────┴────────────────────────────────────┘ +┌───┬─────────────────────────────────┬────────────────────────────────────┐ +│ # │ Integration │ Status +├───┼─────────────────────────────────┼────────────────────────────────────┤ +│ 1 │ CLI client (fpaste) │ Done (with E2E + PKI) +│ 2 │ /client endpoint │ Done (downloadable CLI) +│ 3 │ Ansible deployment role │ Planned +│ 4 │ Kubernetes manifests │ Planned +│ 5 │ Shell aliases/functions │ Planned +└───┴─────────────────────────────────┴────────────────────────────────────┘ ``` ### CLI Client (Complete) -Standalone Python CLI with encryption support: +Standalone Python CLI with encryption and PKI support: - `fpaste create file.txt` - Create paste from file - `fpaste create -e file.txt` - Create encrypted paste (E2E) - `fpaste get ` - Get paste (auto-decrypts with URL fragment key) - `fpaste delete ` - Delete paste - `fpaste info` - Show server info +- `fpaste pki status` - Show PKI status +- `fpaste pki issue -n "name"` - Request client certificate +- `fpaste pki revoke ` - Revoke certificate - Config file for server URL and cert fingerprint - Downloadable via `curl https://server/client > fpaste` @@ -154,6 +141,9 @@ These features will not be implemented: | 2024-12 | Burn-after-read | Single-use pastes for sensitive data | 2024-12 | Custom expiry | Per-paste TTL override | 2024-12 | Multi-stage Containerfile | Smaller production images +| 2024-12 | Minimal PKI | Self-contained mTLS without external CA +| 2024-12 | Security tooling (ruff/bandit) | Code quality and security scanning +| 2024-12 | CI/CD with job dependencies | Tests wait for lint to pass ## Review Schedule diff --git a/TASKLIST.md b/TASKLIST.md index 8e3898f..3039795 100644 --- a/TASKLIST.md +++ b/TASKLIST.md @@ -4,38 +4,47 @@ Prioritized, actionable tasks. Each task is small and completable in one session --- -## Priority 1: Operations +## Priority 1: Ecosystem | Status | Task |--------|-------------------------------------------------------------- -| ☐ | Add /metrics endpoint skeleton (for future Prometheus) -| ☐ | Add structured logging option via environment variable +| ☐ | Create Ansible deployment role +| ☐ | Add Kubernetes manifests (Deployment, Service, ConfigMap) +| ☐ | Add systemd service unit example ## Priority 2: Features | Status | Task |--------|-------------------------------------------------------------- | ☐ | Add paste listing for authenticated users -| ☐ | Add paste password protection +| ☐ | Add rate limit headers (X-RateLimit-*) -## Priority 3: Documentation - -| Status | Task -|--------|-------------------------------------------------------------- -| ☐ | Add deployment examples (Kubernetes, systemd) -| ☐ | Create CONTRIBUTING.md with development setup - -## Priority 4: Testing +## Priority 3: Quality | Status | Task |--------|-------------------------------------------------------------- +| ☐ | Fix mypy type errors (currently ignored) | ☐ | Add test for concurrent identical submissions | ☐ | Add integration tests for container deployment +## Priority 4: Documentation + +| Status | Task +|--------|-------------------------------------------------------------- +| ☐ | Create CONTRIBUTING.md with development setup +| ☐ | Add PKI usage examples to documentation + ## Completed | Date | Task |------------|-------------------------------------------------------------- +| 2024-12 | Implement minimal PKI (CA, issuance, revocation) +| 2024-12 | Add security tooling (ruff, bandit, mypy, pip-audit) +| 2024-12 | Create Makefile with dev workflow targets +| 2024-12 | Setup CI/CD pipeline (Gitea Actions) +| 2024-12 | Fix all ruff lint issues +| 2024-12 | Optimize CI workflow (concurrency, job deps) +| 2024-12 | Add PKI commands to CLI (status, issue, revoke) | 2024-12 | Implement burn-after-read option | 2024-12 | Implement custom expiry per paste | 2024-12 | Optimize Containerfile with multi-stage build diff --git a/TODO.md b/TODO.md index 03d7960..0552c7c 100644 --- a/TODO.md +++ b/TODO.md @@ -6,34 +6,33 @@ Unstructured intake buffer for ideas, issues, and observations. Items here are r ## Ideas -- Prometheus metrics endpoint (`/metrics`) for monitoring integration -- Structured JSON logging for log aggregation compatibility -- Burn-after-read paste option -- Custom expiry header for per-paste TTL - Rate limit headers in responses (X-RateLimit-*) - Paste compression for large text content - ETag support for conditional requests - Paste listing for authenticated users (their own pastes only) - Neovim/Vim plugin for editor integration - Webhook notifications for paste events +- Certificate renewal reminder in CLI +- Admin endpoint for CA key rotation ## Observations -- Abuse prevention uses content-hash dedup + PoW + entropy enforcement -- SQLite WAL mode could improve concurrent read performance -- Container image size could be reduced with multi-stage build -- E2E encryption in CLI uses cryptography package (optional dependency) -- Entropy check has size threshold to avoid false positives on small data +- PKI uses AES-256-GCM for CA private key encryption (PBKDF2 key derivation) +- SHA1 fingerprints are X.509 standard, not security-relevant (usedforsecurity=False) +- Revoked certificates are soft-deleted (status tracked, not removed) +- CI pipeline: lint runs parallel with security, tests wait for lint +- Ruff replaces flake8/isort/pyupgrade with single fast tool +- Bandit configured for medium+ severity only (-ll flag) ## Questions - Should expired paste cleanup run in-process or via external cron? -- Is SQLite sufficient for anticipated load, or plan for PostgreSQL? -- Should burn-after-read pastes show in metadata before burn? -- Password-protected pastes: derive key from password or store hash? +- Certificate renewal: reissue with same CN or require new request? +- Should revoked certs be purged after grace period? ## Debt +- Mypy has pre-existing type errors (runs with --ignore-missing-imports) - No integration tests for container deployment - Missing test for concurrent paste creation - Could add more deployment examples (Kubernetes, systemd) @@ -41,8 +40,7 @@ Unstructured intake buffer for ideas, issues, and observations. Items here are r ## External Dependencies - Consider adding `python-magic` for better MIME detection (currently magic bytes only) -- Evaluate `structlog` for structured logging when implemented -- Look into `prometheus-flask-exporter` for metrics +- cryptography package required for PKI features (optional otherwise) ---