From 2679bc8e69130a3d20a1284534d085576d975ee5 Mon Sep 17 00:00:00 2001 From: Username Date: Mon, 16 Feb 2026 20:56:55 +0100 Subject: [PATCH] docs: add url shortener documentation --- PROJECT.md | 6 +- README.md | 23 ++++- ROADMAP.md | 19 ++++- SECURITY.md | 27 ++++++ TASKLIST.md | 1 + documentation/api.md | 197 +++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 268 insertions(+), 5 deletions(-) diff --git a/PROJECT.md b/PROJECT.md index 0ed60fc..693791c 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -67,6 +67,7 @@ A self-hosted pastebin API that: - Client-side E2E encryption (CLI) - Burn-after-read pastes - Custom expiry per paste +- URL shortener with open redirect prevention - URL prefix for reverse proxy deployments - Security headers (HSTS, CSP, X-Frame-Options, etc.) - Request tracing and structured logging @@ -118,7 +119,7 @@ A self-hosted pastebin API that: ## Current Status -**Version:** 1.5.0 +**Version:** 1.6.0 ``` ┌─────────────────────────────────┬────────────────────────────────────────────┐ @@ -151,8 +152,9 @@ A self-hosted pastebin API that: │ Public certificate registration │ Complete │ CLI register command │ Complete │ systemd deployment │ Complete (security-hardened) -│ Test suite │ 301 tests passing +│ Test suite │ 346 tests passing │ Kubernetes deployment │ Complete (k3s, NodePort :30500) │ Harbor registry integration │ Complete (CI/CD + Trivy scanning) +│ URL shortener │ Complete (8-char base62, redirect, info) └─────────────────────────────────┴────────────────────────────────────────────┘ ``` diff --git a/README.md b/README.md index ccaefb9..237ee1e 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,7 @@ A lightweight, secure pastebin REST API built with Flask. - **Security headers** - HSTS, CSP, X-Frame-Options, X-Content-Type-Options - **CLI client** - Standalone `fpaste` tool with encryption support - **Request tracing** - X-Request-ID for log correlation +- **URL shortener** - `/s/` endpoints for creating, resolving, and managing short URLs - **Audit logging** - PKI certificate lifecycle events (issue, revoke, auth failure) - **Observability** - Request duration metrics via Prometheus histogram - **Minimal dependencies** - Flask + SQLite, optional cryptography for CLI @@ -57,6 +58,11 @@ python run.py | `GET /pastes` | List user's pastes (requires auth) | | `GET /register/challenge` | Get PoW challenge for registration | | `POST /register` | Register and get client certificate | +| `POST /s` | Create short URL (PoW required) | +| `GET /s` | List your short URLs (requires auth) | +| `GET /s/` | Redirect to target URL (302) | +| `GET /s//info` | Short URL metadata | +| `DELETE /s/` | Delete short URL (requires auth) | | `GET /pki` | PKI status and CA info | | `GET /pki/ca.crt` | Download CA certificate | @@ -102,6 +108,18 @@ curl -H "X-SSL-Client-SHA1: " \ http://localhost:5000/pastes ``` +### Create a short URL +```bash +curl -X POST -H "Content-Type: application/json" \ + -d '{"url":"https://example.com/long/path"}' \ + http://localhost:5000/s +``` + +### Follow a short URL +```bash +curl -L http://localhost:5000/s/AbCdEfGh +``` + ## CLI Client A standalone command-line client `fpaste` is included. For E2E encryption, install the optional `cryptography` package. @@ -280,6 +298,8 @@ Configuration via environment variables: | `FLASKPASTE_RATE_WINDOW` | `60` | Rate limit window (seconds) | | `FLASKPASTE_RATE_MAX` | `10` | Max requests per window (anon) | | `FLASKPASTE_RATE_AUTH_MULT` | `5` | Multiplier for authenticated users | +| `FLASKPASTE_SHORT_ID_LENGTH` | `8` | Short URL ID length (base62 characters) | +| `FLASKPASTE_SHORT_URL_MAX` | `2048` | Maximum target URL length | | `FLASKPASTE_URL_PREFIX` | (empty) | URL prefix for reverse proxy deployments | | `FLASKPASTE_MIN_ENTROPY` | `0` | Min entropy bits/byte (0=disabled, 6.0=require encryption) | | `FLASKPASTE_MIN_ENTROPY_SIZE` | `256` | Only check entropy for content >= this size | @@ -419,7 +439,7 @@ flaskpaste/ │ └── api/ │ ├── __init__.py # Blueprint setup │ └── routes.py # API endpoints -├── tests/ # Test suite (356 tests) +├── tests/ # Test suite (346+ tests) ├── data/ # SQLite database ├── run.py # Development server ├── wsgi.py # Production WSGI entry @@ -444,6 +464,7 @@ flaskpaste/ - **Security headers** - HSTS, CSP, X-Frame-Options, X-Content-Type-Options - **Proof-of-work** - Computational puzzles prevent automated spam - **Rate limiting** - Per-IP throttling with X-RateLimit-* headers +- **Open redirect prevention** - URL shortener allows only http/https schemes with valid host - **Request tracing** - X-Request-ID for log correlation - **PKI support** - Built-in CA for client certificate issuance - **Audit logging** - PKI certificate events for compliance and forensics diff --git a/ROADMAP.md b/ROADMAP.md index 448f94a..582ad85 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -29,7 +29,8 @@ FlaskPaste v1.5.1 is deployed with comprehensive security hardening and abuse pr - CLI with list, search, update, export commands - Public certificate registration (PoW-protected) - CLI register command for certificate enrollment -- Comprehensive test suite (356 tests) +- URL shortener (create, redirect, info, delete, list) +- Comprehensive test suite (346 tests) - Complete security pentest remediation (15 items) - PKI audit logging (certificate lifecycle events) - Request duration metrics (Prometheus histogram) @@ -49,7 +50,7 @@ Focus: Production readiness and operational excellence. │ 4 │ Proxy trust validation │ Done │ 5 │ Proof-of-work spam prevention │ Done │ 6 │ Entropy enforcement │ Done -│ 7 │ Test coverage > 90% │ Done (301 tests) +│ 7 │ Test coverage > 90% │ Done (346 tests) │ 8 │ Documentation complete │ Done └───┴─────────────────────────────────┴────────────────────────────────────┘ ``` @@ -86,9 +87,21 @@ Focus: User-requested enhancements within scope. │ 6 │ Anti-flood (dynamic PoW) │ Done (v1.4.0) │ 7 │ IP-based rate limiting │ Done (v1.4.0) │ 8 │ Scheduled cleanup │ Done (v1.4.0) +│ 9 │ URL shortener │ Done (v1.6.0) └───┴─────────────────────────────────┴────────────────────────────────────┘ ``` +### URL Shortener (v1.6.0) + +Short URL creation, redirect, metadata, and management: +- `POST /s` - Create short URL (PoW + rate limit) +- `GET /s` - List own short URLs (auth required) +- `GET /s/` - 302 redirect to target +- `GET /s//info` - JSON metadata (target, clicks, expiry) +- `DELETE /s/` - Delete (owner only) +- Open redirect prevention (http/https only, netloc required) +- 8-char base62 IDs (visually distinct from paste hex IDs) + ### Anti-Flood System (v1.4.0) Dynamic proof-of-work difficulty that increases under abuse: @@ -193,6 +206,8 @@ These features will not be implemented: | 2024-12 | Pentest remediation complete | 15 security hardening items from formal review | 2024-12 | Enhanced CI security | SBOM generation, dedicated security-tests job | 2025-01 | CI/CD image build/push | Auto-build on main, push to Harbor registry +| 2026-02 | URL shortener | /s/ prefix avoids paste ID collision; base62 IDs +| 2026-02 | Open redirect prevention | http/https only, netloc required, 2048 byte limit ## Review Schedule diff --git a/SECURITY.md b/SECURITY.md index 9e68872..8618d5f 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -92,6 +92,25 @@ FLASKPASTE_MIN_ENTROPY=6.0 # Bits per byte (encrypted ~7.5-8.0) FLASKPASTE_MIN_ENTROPY_SIZE=256 # Only check content >= this size ``` +### URL Shortener Security + +**Open Redirect Prevention** + +Short URL creation validates target URLs: + +- Only `http` and `https` schemes allowed (rejects `javascript:`, `data:`, `ftp:`, `file:`) +- Network location (hostname) required — rejects scheme-only URLs +- Maximum URL length: 2048 bytes +- Short IDs: 8-char base62 (`[a-zA-Z0-9]`) with `secrets.choice()` for unpredictability +- Redirect responses include `Cache-Control: no-store, no-cache` to prevent caching + +**Access Controls** + +- Creation: rate-limited + proof-of-work (same as paste creation) +- Redirect: lookup rate limiting prevents enumeration +- Deletion: owner authentication required +- Listing: authentication required, shows only own URLs + ### Security Headers All responses include: @@ -120,6 +139,12 @@ All requests receive `X-Request-ID` header for log correlation and debugging. Pa - Configurable length (default 12 characters) - Validated on all endpoints +### Short URL IDs + +- Base62 only (`[a-zA-Z0-9]+`) +- 8 characters (configurable via `FLASKPASTE_SHORT_ID_LENGTH`) +- Validated on all `/s/` endpoints + ### MIME Types - Magic byte detection for binary formats @@ -245,6 +270,7 @@ Security fixes are released as soon as possible. Subscribe to repository release - Authentication bypass - Information disclosure - Denial of service (application-level) +- Open redirect via URL shortener ### Out of Scope @@ -258,6 +284,7 @@ Security fixes are released as soon as possible. Subscribe to repository release | Version | Security Changes | |---------|------------------| +| 1.6.0 | URL shortener with open redirect prevention, scheme allowlist, target URL validation | | 1.5.0 | Pentest remediation (15 items): timing attack prevention, serial collision detection, lookup rate limiting, content hash locking, anti-flood memory limits, CLI path validation, SSL hostname verification, config permission checks | | 1.4.0 | Anti-flood dynamic PoW, IP-based rate limiting, audit logging | | 1.2.0 | Password protection with PBKDF2, code modernization | diff --git a/TASKLIST.md b/TASKLIST.md index e87e46c..eb26b88 100644 --- a/TASKLIST.md +++ b/TASKLIST.md @@ -14,6 +14,7 @@ Prioritized, actionable tasks. Each task is small and completable in one session | Date | Task |------------|-------------------------------------------------------------- +| 2026-02 | Add URL shortener (create, redirect, info, delete, list) | 2025-01 | Add CI/CD image build and push to Harbor | 2025-01 | Add Kubernetes manifests (Deployment, Service, ConfigMap) | 2024-12 | Add PKI usage examples (documentation/pki.md) diff --git a/documentation/api.md b/documentation/api.md index b97f066..4ef096d 100644 --- a/documentation/api.md +++ b/documentation/api.md @@ -459,6 +459,203 @@ X-SSL-Client-SHA1: a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2 --- +## URL Shortener + +FlaskPaste includes a built-in URL shortener under the `/s/` namespace. Short URLs use 8-character base62 IDs (`[a-zA-Z0-9]`), visually distinct from paste hex IDs. + +### POST /s + +Create a new short URL. Requires proof-of-work and respects rate limits. + +**Request (JSON):** +```http +POST /s HTTP/1.1 +Host: localhost:5000 +Content-Type: application/json +X-PoW-Token: +X-PoW-Solution: + +{"url": "https://example.com/some/long/path?query=value"} +``` + +**Request (Raw):** +```http +POST /s HTTP/1.1 +Host: localhost:5000 +Content-Type: text/plain +X-PoW-Token: +X-PoW-Solution: + +https://example.com/some/long/path +``` + +**Optional Headers:** +| Header | Description | +|--------|-------------| +| `X-Expiry` | Custom expiry in seconds | +| `X-SSL-Client-SHA1` | Client certificate fingerprint (for ownership) | + +**Response (201 Created):** +```json +{ + "id": "AbCdEfGh", + "url": "/s/AbCdEfGh", + "target_url": "https://example.com/some/long/path?query=value", + "created_at": 1700000000, + "owner": "a1b2c3d4...", + "expires_at": 1700003600 +} +``` + +**Errors:** +| Code | Description | +|------|-------------| +| 400 | No URL provided | +| 400 | Invalid URL scheme (only http/https allowed) | +| 400 | Invalid URL: missing host | +| 400 | URL too long (max 2048 bytes) | +| 400 | Proof-of-work required/failed | +| 429 | Rate limit or duplicate URL limit exceeded | + +**Security:** +- Only `http` and `https` schemes are accepted (prevents `javascript:`, `data:`, `file:` etc.) +- URLs must have a valid network location (host) +- Maximum URL length: 2048 bytes (configurable via `FLASKPASTE_SHORT_URL_MAX`) + +--- + +### GET /s + +List short URLs owned by the authenticated user. + +**Request:** +```http +GET /s HTTP/1.1 +Host: localhost:5000 +X-SSL-Client-SHA1: a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2 +``` + +**Query Parameters:** +| Parameter | Type | Description | +|-----------|------|-------------| +| `limit` | int | Maximum results (default: 50, max: 200) | +| `offset` | int | Pagination offset (default: 0) | + +**Response (200 OK):** +```json +{ + "urls": [ + { + "id": "AbCdEfGh", + "target_url": "https://example.com", + "created_at": 1700000000, + "access_count": 42, + "url": "/s/AbCdEfGh" + } + ], + "count": 1, + "total": 1, + "limit": 50, + "offset": 0 +} +``` + +**Errors:** +| Code | Description | +|------|-------------| +| 401 | Authentication required | + +--- + +### GET /s/{id} + +### HEAD /s/{id} + +Redirect to the target URL. Returns HTTP 302 with `Location` header. + +**Request:** +```http +GET /s/AbCdEfGh HTTP/1.1 +Host: localhost:5000 +``` + +**Response (302 Found):** +```http +HTTP/1.1 302 Found +Location: https://example.com/some/long/path +Cache-Control: no-store, no-cache, must-revalidate, private +``` + +Each access increments the short URL's access counter. + +**Errors:** +| Code | Description | +|------|-------------| +| 400 | Invalid short URL ID format | +| 404 | Short URL not found or expired | + +--- + +### GET /s/{id}/info + +Retrieve short URL metadata without incrementing the access counter. + +**Request:** +```http +GET /s/AbCdEfGh/info HTTP/1.1 +Host: localhost:5000 +``` + +**Response (200 OK):** +```json +{ + "id": "AbCdEfGh", + "target_url": "https://example.com/some/long/path", + "created_at": 1700000000, + "last_accessed": 1700001000, + "access_count": 42, + "url": "/s/AbCdEfGh", + "owner": "a1b2c3d4...", + "expires_at": 1700086400 +} +``` + +**Errors:** +| Code | Description | +|------|-------------| +| 400 | Invalid short URL ID format | +| 404 | Short URL not found or expired | + +--- + +### DELETE /s/{id} + +Delete a short URL. Requires authentication and ownership (or admin rights). + +**Request:** +```http +DELETE /s/AbCdEfGh HTTP/1.1 +Host: localhost:5000 +X-SSL-Client-SHA1: a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2 +``` + +**Response (200 OK):** +```json +{ + "message": "Short URL deleted" +} +``` + +**Errors:** +| Code | Description | +|------|-------------| +| 400 | Invalid short URL ID format | +| 401 | Authentication required | +| 403 | Permission denied (not owner or admin) | +| 404 | Short URL not found | + +--- + ## MIME Type Detection FlaskPaste automatically detects MIME types using: