ansible/flaskpaste
1
0
Fork 0
forked from claw/flaskpaste
Code Pull Requests Activity

docs: update project status for v1.2.0

Browse Source
This commit is contained in:
Username
2025-12-20 17:24:41 +01:00
parent 260d6e894d
commit 486bac1d85
4 changed files with 112 additions and 104 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
PROJECT.md
View File

@@ -18,8 +18,9 @@ Public pastebin services present risks:
A self-hosted pastebin API that: A self-hosted pastebin API that:
- Stores pastes locally in SQLite - Stores pastes locally in SQLite
- Supports client certificate authentication via reverse proxy - Supports client certificate authentication via reverse proxy
- Provides optional built-in PKI for certificate management
- Automatically expires content based on access patterns - Automatically expires content based on access patterns
- Prevents abuse through content-hash deduplication - Prevents abuse through content-hash deduplication and proof-of-work
- Serves text and binary content with proper MIME detection - Serves text and binary content with proper MIME detection
- Runs behind any reverse proxy (nginx, HAProxy, Caddy) - Runs behind any reverse proxy (nginx, HAProxy, Caddy)
@@ -32,6 +33,7 @@ A self-hosted pastebin API that:
│ Security │ Zero injection vulnerabilities │ Security │ Zero injection vulnerabilities
│ │ All OWASP headers implemented │ │ All OWASP headers implemented
│ │ Input validation on all endpoints │ │ Input validation on all endpoints
│ │ Security scanning in CI (bandit)
├────────────────────────────────┼────────────────────────────────────────────┤ ├────────────────────────────────┼────────────────────────────────────────────┤
│ Reliability │ SQLite ACID guarantees │ Reliability │ SQLite ACID guarantees
│ │ Graceful degradation on errors │ │ Graceful degradation on errors
@@ -44,6 +46,7 @@ A self-hosted pastebin API that:
│ Operability │ Container-ready (Podman/Docker) │ Operability │ Container-ready (Podman/Docker)
│ │ Gunicorn-compatible WSGI │ │ Gunicorn-compatible WSGI
│ │ Request tracing via X-Request-ID │ │ Request tracing via X-Request-ID
│ │ Prometheus metrics endpoint
└────────────────────────────────┴────────────────────────────────────────────┘ └────────────────────────────────┴────────────────────────────────────────────┘
``` ```
@@ -55,6 +58,7 @@ A self-hosted pastebin API that:
- Text and binary content support - Text and binary content support
- Magic-byte MIME type detection - Magic-byte MIME type detection
- Client certificate authentication (via proxy header) - Client certificate authentication (via proxy header)
- Built-in PKI (CA generation, certificate issuance, revocation)
- Configurable size limits (anon vs authenticated) - Configurable size limits (anon vs authenticated)
- Time-based expiry with access-touch semantics - Time-based expiry with access-touch semantics
- Content-hash deduplication for abuse prevention - Content-hash deduplication for abuse prevention
@@ -68,6 +72,7 @@ A self-hosted pastebin API that:
- Request tracing and structured logging - Request tracing and structured logging
- Container deployment support - Container deployment support
- SQLite storage - SQLite storage
- CI/CD with security scanning
### Out of Scope ### Out of Scope
@@ -90,7 +95,7 @@ A self-hosted pastebin API that:
## Assumptions ## Assumptions
- Deployment behind a TLS-terminating reverse proxy - Deployment behind a TLS-terminating reverse proxy
- Client certificates managed externally (PKI, mTLS) - Client certificates managed externally or via built-in PKI
- Operators have container runtime (Podman/Docker) or Python venv - Operators have container runtime (Podman/Docker) or Python venv
- SQLite performance sufficient for expected load - SQLite performance sufficient for expected load
@@ -104,14 +109,17 @@ A self-hosted pastebin API that:
│ Database │ SQLite 3 (built-in) │ Database │ SQLite 3 (built-in)
│ WSGI Server │ Gunicorn (production) │ WSGI Server │ Gunicorn (production)
│ Container │ Podman / Docker │ Container │ Podman / Docker
│ Testing │ pytest, pytest-cov │ Testing │ pytest
│ Linting │ ruff, mypy
│ Security │ bandit, pip-audit
│ CI/CD │ Gitea Actions
│ Python │ 3.11+ │ Python │ 3.11+
└─────────────────┴──────────────────────────────────────────────────────────┘ └─────────────────┴──────────────────────────────────────────────────────────┘
``` ```
## Current Status ## Current Status
**Version:** 1.1.0 **Version:** 1.2.0
``` ```
┌─────────────────────────────────┬────────────────────────────────────────────┐ ┌─────────────────────────────────┬────────────────────────────────────────────┐
@@ -121,6 +129,7 @@ A self-hosted pastebin API that:
│ Binary content support │ Complete │ Binary content support │ Complete
│ MIME detection │ Complete │ MIME detection │ Complete
│ Client cert authentication │ Complete │ Client cert authentication │ Complete
│ Built-in PKI │ Complete
│ Size limits │ Complete │ Size limits │ Complete
│ Paste expiry │ Complete │ Paste expiry │ Complete
│ Content-hash deduplication │ Complete │ Content-hash deduplication │ Complete
@@ -134,6 +143,8 @@ A self-hosted pastebin API that:
│ Security headers │ Complete │ Security headers │ Complete
│ Request tracing │ Complete │ Request tracing │ Complete
│ Container deployment │ Complete │ Container deployment │ Complete
│ Test suite │ 113 tests passing │ Security tooling │ Complete
│ CI/CD pipeline │ Complete
│ Test suite │ 147 tests passing
└─────────────────────────────────┴────────────────────────────────────────────┘ └─────────────────────────────────┴────────────────────────────────────────────┘
``` ```

136
ROADMAP.md
View File

@@ -2,12 +2,13 @@
## Current State ## Current State
FlaskPaste v1.1.0 is deployed and feature-complete for its core mission: a secure, minimal pastebin API with zero-knowledge encryption support. FlaskPaste v1.2.0 is deployed with PKI integration and comprehensive security tooling.
**Implemented:** **Implemented:**
- Full REST API (CRUD operations) - Full REST API (CRUD operations)
- Binary content support with magic-byte MIME detection - Binary content support with magic-byte MIME detection
- Client certificate authentication - Client certificate authentication
- Minimal PKI (CA generation, certificate issuance, revocation)
- Content-hash deduplication (abuse prevention) - Content-hash deduplication (abuse prevention)
- Proof-of-work spam prevention - Proof-of-work spam prevention
- Entropy enforcement (require encrypted uploads) - Entropy enforcement (require encrypted uploads)
@@ -15,116 +16,102 @@ FlaskPaste v1.1.0 is deployed and feature-complete for its core mission: a secur
- URL prefix support for reverse proxy deployments - URL prefix support for reverse proxy deployments
- /client endpoint for CLI distribution - /client endpoint for CLI distribution
- Automatic paste expiry - Automatic paste expiry
- Burn-after-read pastes
- Custom expiry per paste
- Security headers and request tracing - Security headers and request tracing
- Container deployment support - Container deployment support
- Comprehensive test suite (98 tests) - Security tooling (ruff, bandit, mypy, pip-audit)
- CI/CD pipeline with lint, security, and test jobs
- Comprehensive test suite (147 tests)
## Phase 1: Hardening (Complete) ## Phase 1: Hardening (Complete)
Focus: Production readiness and operational excellence. Focus: Production readiness and operational excellence.
``` ```
┌───┬─────────────────────────────────────┬────────────────────────────────────┐ ┌───┬─────────────────────────────────┬────────────────────────────────────┐
│ # │ Milestone │ Status │ # │ Milestone │ Status
├───┼─────────────────────────────────────┼────────────────────────────────────┤ ├───┼─────────────────────────────────┼────────────────────────────────────┤
│ 1 │ Abuse prevention (dedup) │ Done │ 1 │ Abuse prevention (dedup) │ Done
│ 2 │ Security headers complete │ Done │ 2 │ Security headers complete │ Done
│ 3 │ Request tracing (X-Request-ID) │ Done │ 3 │ Request tracing (X-Request-ID) │ Done
│ 4 │ Proxy trust validation │ Done │ 4 │ Proxy trust validation │ Done
│ 5 │ Proof-of-work spam prevention │ Done │ 5 │ Proof-of-work spam prevention │ Done
│ 6 │ Entropy enforcement │ Done │ 6 │ Entropy enforcement │ Done
│ 7 │ Test coverage > 90% │ Done (98 tests) │ 7 │ Test coverage > 90% │ Done (147 tests)
│ 8 │ Documentation complete │ Done │ 8 │ Documentation complete │ Done
└───┴─────────────────────────────────────┴────────────────────────────────────┘ └───┴─────────────────────────────────┴────────────────────────────────────┘
``` ```
## Phase 2: Operations ## Phase 2: Operations (Complete)
Focus: Deployment, monitoring, and maintenance tooling. Focus: Deployment, monitoring, and maintenance tooling.
``` ```
┌───┬─────────────────────────────────────┬────────────────────────────────────┐ ┌───┬─────────────────────────────────┬────────────────────────────────────┐
│ # │ Milestone │ Dependencies │ # │ Milestone │ Status
├───┼─────────────────────────────────────┼────────────────────────────────────┤ ├───┼─────────────────────────────────┼────────────────────────────────────┤
│ 1 │ Prometheus metrics endpoint │ None │ 1 │ Prometheus metrics endpoint │ Done (prometheus-flask-exporter)
│ 2 │ Structured JSON logging │ None │ 2 │ Structured JSON logging │ Done (production mode)
│ 3 │ Admin API (stats, cleanup) │ Auth improvements │ 3 │ Security tooling (lint/scan) │ Done (ruff, bandit, mypy)
│ 4 │ Ansible deployment role │ None │ 4 │ CI/CD pipeline │ Done (Gitea Actions)
│ 5 │ CI/CD pipeline │ Container registry access │ 5 │ Multi-stage Containerfile │ Done
└───┴─────────────────────────────────────┴────────────────────────────────────┘ └───┴─────────────────────────────────┴────────────────────────────────────┘
``` ```
### Prometheus Metrics
Expose `/metrics` endpoint with:
- `flaskpaste_pastes_total` (counter)
- `flaskpaste_pastes_created` (counter)
- `flaskpaste_pastes_deleted` (counter)
- `flaskpaste_pastes_expired` (counter)
- `flaskpaste_storage_bytes` (gauge)
- `flaskpaste_request_duration_seconds` (histogram)
### Structured Logging
Replace text logs with JSON format:
- Timestamp, level, message, request_id
- Consistent field names across all log entries
- Compatible with log aggregation (Loki, ELK)
## Phase 3: Features (Complete) ## Phase 3: Features (Complete)
Focus: User-requested enhancements within scope. Focus: User-requested enhancements within scope.
``` ```
┌───┬─────────────────────────────────────┬────────────────────────────────────┐ ┌───┬─────────────────────────────────┬────────────────────────────────────┐
│ # │ Feature │ Status │ # │ Feature │ Status
├───┼─────────────────────────────────────┼────────────────────────────────────┤ ├───┼─────────────────────────────────┼────────────────────────────────────┤
│ 1 │ E2E encryption (client-side) │ Done (CLI -e flag, zero-knowledge) │ 1 │ E2E encryption (client-side) │ Done (CLI -e flag, zero-knowledge)
│ 2 │ URL prefix support │ Done │ 2 │ URL prefix support │ Done
│ 3 │ Custom expiry per paste │ Done (X-Expiry header) │ 3 │ Custom expiry per paste │ Done (X-Expiry header)
│ 4 │ Burn-after-read option │ Done (X-Burn-After-Read header) │ 4 │ Burn-after-read option │ Done (X-Burn-After-Read header)
│ 5 │ Paste password protection │ Planned │ 5 │ Minimal PKI (CA + issuance) │ Done
└───┴─────────────────────────────────────┴────────────────────────────────────┘ └───┴─────────────────────────────────┴────────────────────────────────────┘
``` ```
### Burn-After-Read ### PKI Features
Single-access pastes that delete after first retrieval: Integrated certificate authority for mTLS:
- `POST /` with `X-Burn-After-Read: true` header - `POST /pki/ca` - Generate CA (first-run bootstrap)
- Paste deleted after first `GET /<id>/raw` - `GET /pki/status` - CA status and fingerprint
- Metadata `GET /<id>` does not trigger burn - `GET /pki/ca.crt` - Download CA certificate
- `POST /pki/issue` - Issue client certificate
- `POST /pki/revoke/<serial>` - Revoke certificate
- CLI: `fpaste pki status`, `fpaste pki issue`, `fpaste pki revoke`
### Custom Expiry ## Phase 4: Ecosystem (In Progress)
Allow per-paste expiry override:
- `POST /` with `X-Expiry: 3600` header (seconds)
- Capped at server maximum (e.g., 30 days)
- Default unchanged for pastes without header
## Phase 4: Ecosystem
Focus: Integration with external systems. Focus: Integration with external systems.
``` ```
┌───┬─────────────────────────────────────┬────────────────────────────────────┐ ┌───┬─────────────────────────────────┬────────────────────────────────────┐
│ # │ Integration │ Status │ # │ Integration │ Status
├───┼─────────────────────────────────────┼────────────────────────────────────┤ ├───┼─────────────────────────────────┼────────────────────────────────────┤
│ 1 │ CLI client (fpaste) │ Done (with E2E encryption) │ 1 │ CLI client (fpaste) │ Done (with E2E + PKI)
│ 2 │ /client endpoint │ Done (downloadable CLI) │ 2 │ /client endpoint │ Done (downloadable CLI)
│ 3 │ Neovim/Vim plugin │ Planned │ 3 │ Ansible deployment role │ Planned
│ 4 │ Shell aliases/functions │ Planned │ 4 │ Kubernetes manifests │ Planned
│ 5 │ Webhook notifications │ Planned │ 5 │ Shell aliases/functions │ Planned
└───┴─────────────────────────────────────┴────────────────────────────────────┘ └───┴─────────────────────────────────┴────────────────────────────────────┘
``` ```
### CLI Client (Complete) ### CLI Client (Complete)
Standalone Python CLI with encryption support: Standalone Python CLI with encryption and PKI support:
- `fpaste create file.txt` - Create paste from file - `fpaste create file.txt` - Create paste from file
- `fpaste create -e file.txt` - Create encrypted paste (E2E) - `fpaste create -e file.txt` - Create encrypted paste (E2E)
- `fpaste get <id>` - Get paste (auto-decrypts with URL fragment key) - `fpaste get <id>` - Get paste (auto-decrypts with URL fragment key)
- `fpaste delete <id>` - Delete paste - `fpaste delete <id>` - Delete paste
- `fpaste info` - Show server info - `fpaste info` - Show server info
- `fpaste pki status` - Show PKI status
- `fpaste pki issue -n "name"` - Request client certificate
- `fpaste pki revoke <serial>` - Revoke certificate
- Config file for server URL and cert fingerprint - Config file for server URL and cert fingerprint
- Downloadable via `curl https://server/client > fpaste` - Downloadable via `curl https://server/client > fpaste`
@@ -154,6 +141,9 @@ These features will not be implemented:
| 2024-12 | Burn-after-read | Single-use pastes for sensitive data | 2024-12 | Burn-after-read | Single-use pastes for sensitive data
| 2024-12 | Custom expiry | Per-paste TTL override | 2024-12 | Custom expiry | Per-paste TTL override
| 2024-12 | Multi-stage Containerfile | Smaller production images | 2024-12 | Multi-stage Containerfile | Smaller production images
| 2024-12 | Minimal PKI | Self-contained mTLS without external CA
| 2024-12 | Security tooling (ruff/bandit) | Code quality and security scanning
| 2024-12 | CI/CD with job dependencies | Tests wait for lint to pass
## Review Schedule ## Review Schedule

33
TASKLIST.md
View File

@@ -4,38 +4,47 @@ Prioritized, actionable tasks. Each task is small and completable in one session
--- ---
## Priority 1: Operations ## Priority 1: Ecosystem
| Status | Task | Status | Task
|--------|-------------------------------------------------------------- |--------|--------------------------------------------------------------
| ☐ | Add /metrics endpoint skeleton (for future Prometheus) | ☐ | Create Ansible deployment role
| ☐ | Add structured logging option via environment variable | ☐ | Add Kubernetes manifests (Deployment, Service, ConfigMap)
| ☐ | Add systemd service unit example
## Priority 2: Features ## Priority 2: Features
| Status | Task | Status | Task
|--------|-------------------------------------------------------------- |--------|--------------------------------------------------------------
| ☐ | Add paste listing for authenticated users | ☐ | Add paste listing for authenticated users
| ☐ | Add paste password protection | ☐ | Add rate limit headers (X-RateLimit-*)
## Priority 3: Documentation ## Priority 3: Quality
| Status | Task
|--------|--------------------------------------------------------------
| ☐ | Add deployment examples (Kubernetes, systemd)
| ☐ | Create CONTRIBUTING.md with development setup
## Priority 4: Testing
| Status | Task | Status | Task
|--------|-------------------------------------------------------------- |--------|--------------------------------------------------------------
| ☐ | Fix mypy type errors (currently ignored)
| ☐ | Add test for concurrent identical submissions | ☐ | Add test for concurrent identical submissions
| ☐ | Add integration tests for container deployment | ☐ | Add integration tests for container deployment
## Priority 4: Documentation
| Status | Task
|--------|--------------------------------------------------------------
| ☐ | Create CONTRIBUTING.md with development setup
| ☐ | Add PKI usage examples to documentation
## Completed ## Completed
| Date | Task | Date | Task
|------------|-------------------------------------------------------------- |------------|--------------------------------------------------------------
| 2024-12 | Implement minimal PKI (CA, issuance, revocation)
| 2024-12 | Add security tooling (ruff, bandit, mypy, pip-audit)
| 2024-12 | Create Makefile with dev workflow targets
| 2024-12 | Setup CI/CD pipeline (Gitea Actions)
| 2024-12 | Fix all ruff lint issues
| 2024-12 | Optimize CI workflow (concurrency, job deps)
| 2024-12 | Add PKI commands to CLI (status, issue, revoke)
| 2024-12 | Implement burn-after-read option | 2024-12 | Implement burn-after-read option
| 2024-12 | Implement custom expiry per paste | 2024-12 | Implement custom expiry per paste
| 2024-12 | Optimize Containerfile with multi-stage build | 2024-12 | Optimize Containerfile with multi-stage build

26
TODO.md
View File

@@ -6,34 +6,33 @@ Unstructured intake buffer for ideas, issues, and observations. Items here are r
## Ideas ## Ideas
- Prometheus metrics endpoint (`/metrics`) for monitoring integration
- Structured JSON logging for log aggregation compatibility
- Burn-after-read paste option
- Custom expiry header for per-paste TTL
- Rate limit headers in responses (X-RateLimit-*) - Rate limit headers in responses (X-RateLimit-*)
- Paste compression for large text content - Paste compression for large text content
- ETag support for conditional requests - ETag support for conditional requests
- Paste listing for authenticated users (their own pastes only) - Paste listing for authenticated users (their own pastes only)
- Neovim/Vim plugin for editor integration - Neovim/Vim plugin for editor integration
- Webhook notifications for paste events - Webhook notifications for paste events
- Certificate renewal reminder in CLI
- Admin endpoint for CA key rotation
## Observations ## Observations
- Abuse prevention uses content-hash dedup + PoW + entropy enforcement - PKI uses AES-256-GCM for CA private key encryption (PBKDF2 key derivation)
- SQLite WAL mode could improve concurrent read performance - SHA1 fingerprints are X.509 standard, not security-relevant (usedforsecurity=False)
- Container image size could be reduced with multi-stage build - Revoked certificates are soft-deleted (status tracked, not removed)
- E2E encryption in CLI uses cryptography package (optional dependency) - CI pipeline: lint runs parallel with security, tests wait for lint
- Entropy check has size threshold to avoid false positives on small data - Ruff replaces flake8/isort/pyupgrade with single fast tool
- Bandit configured for medium+ severity only (-ll flag)
## Questions ## Questions
- Should expired paste cleanup run in-process or via external cron? - Should expired paste cleanup run in-process or via external cron?
- Is SQLite sufficient for anticipated load, or plan for PostgreSQL? - Certificate renewal: reissue with same CN or require new request?
- Should burn-after-read pastes show in metadata before burn? - Should revoked certs be purged after grace period?
- Password-protected pastes: derive key from password or store hash?
## Debt ## Debt
- Mypy has pre-existing type errors (runs with --ignore-missing-imports)
- No integration tests for container deployment - No integration tests for container deployment
- Missing test for concurrent paste creation - Missing test for concurrent paste creation
- Could add more deployment examples (Kubernetes, systemd) - Could add more deployment examples (Kubernetes, systemd)
@@ -41,8 +40,7 @@ Unstructured intake buffer for ideas, issues, and observations. Items here are r
## External Dependencies ## External Dependencies
- Consider adding `python-magic` for better MIME detection (currently magic bytes only) - Consider adding `python-magic` for better MIME detection (currently magic bytes only)
- Evaluate `structlog` for structured logging when implemented - cryptography package required for PKI features (optional otherwise)
- Look into `prometheus-flask-exporter` for metrics
--- ---