docs: Add project documentation

- README.md: Quick start, API endpoints, ports - PROJECT.md: Architecture, components, data models - ROADMAP.md: Version milestones (v0.1.x - v1.0.0) - TASKS.md: Current sprint tasks - TODO.md: Backlog items - docs/INSTALL.md: Setup and deployment - docs/USAGE.md: API examples - docs/CHEATSHEET.md: Quick reference
2026-02-05 21:06:05 +01:00
parent a8f616970a
commit e6e7b622b2
8 changed files with 571 additions and 31 deletions
--- a/PROJECT.md
+++ b/PROJECT.md
@@ -0,0 +1,58 @@
+# ESP32-Web Project
+
+## Overview
+
+REST API backend for ESP32 CSI sensor fleet. Collects and stores sensor data (BLE, WiFi probes, alerts, events) for OPSEC/OSINT/Purple team operations.
+
+## Goals
+
+- Aggregate data from multiple ESP32 sensors
+- Provide REST API for querying devices, alerts, events
+- Track device presence and movement across zones
+- Enable fleet management (commands, OTA, calibration)
+
+## Architecture
+
+```
+ESP32 Sensors (UDP 5500) ──▶ UDP Collector ──▶ SQLite/PostgreSQL
+                                   │
+                                   ▼
+                            Flask REST API (TCP 5500)
+                                   │
+                                   ▼
+                            Clients / Dashboards
+```
+
+## Components
+
+| Component | Path | Description |
+|-----------|------|-------------|
+| App Factory | `src/esp32_web/__init__.py` | Flask application factory |
+| Config | `src/esp32_web/config.py` | Configuration classes |
+| Models | `src/esp32_web/models/` | SQLAlchemy ORM models |
+| API | `src/esp32_web/api/` | REST API blueprints |
+| Collector | `src/esp32_web/collector/` | UDP listener + parsers |
+
+## Data Models
+
+| Model | Description |
+|-------|-------------|
+| Sensor | ESP32 sensor node (hostname, IP, status) |
+| Device | Discovered BLE/WiFi device (MAC, vendor, name) |
+| Sighting | Device seen by sensor (RSSI, timestamp) |
+| Alert | Security alert (deauth, flood) |
+| Event | Sensor event (motion, presence, calibration) |
+| Probe | WiFi probe request (SSID, channel) |
+
+## Dependencies
+
+- Flask 3.x
+- SQLAlchemy 2.x
+- Flask-Migrate
+- Flask-CORS
+- gunicorn
+
+## Related Projects
+
+- [esp32-hacking](~/git/esp32-hacking/) - ESP32 firmware
+- [esp-tools](~/git/esp-tools/) - CLI tools (esp-ctl, esp-fleet)
--- a/README.md
+++ b/README.md
@@ -5,7 +5,10 @@ REST API backend for ESP32 sensor fleet (OPSEC/OSINT/Purple team).
 ## Quick Start

 ```bash
-# Install
+# Clone and install
+cd ~/git/esp32-web
+python3 -m venv .venv
+source .venv/bin/activate
 pip install -e ".[dev]"

 # Initialize database
@@ -13,11 +16,40 @@ flask --app src/esp32_web db init
 flask --app src/esp32_web db migrate -m "initial"
 flask --app src/esp32_web db upgrade

-# Run development server
-make dev
+# Start server
+make start
+make status
 ```

-## Ports
+## Server Management
+
+```bash
+make start    # Start server (0.0.0.0:5500)
+make stop     # Stop server
+make restart  # Restart server
+make status   # Show PID + uptime
+make logs     # Tail log file
+```
+
+## API Endpoints
+
+Base URL: `http://localhost:5500`
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | `/health` | Health check with uptime |
+| GET | `/api/v1/sensors` | List sensors |
+| GET | `/api/v1/sensors/<hostname>` | Get sensor |
+| POST | `/api/v1/sensors/<hostname>/command` | Send command |
+| GET | `/api/v1/devices` | List devices |
+| GET | `/api/v1/devices/<mac>` | Get device + sightings |
+| GET | `/api/v1/alerts` | List alerts |
+| GET | `/api/v1/events` | List events |
+| GET | `/api/v1/probes` | List probe requests |
+| GET | `/api/v1/probes/ssids` | List SSIDs with counts |
+| GET | `/api/v1/stats` | Aggregate statistics |
+
+## Network Ports

 | Port | Protocol | Description |
 |------|----------|-------------|
@@ -25,33 +57,12 @@ make dev
 | 5500 | UDP | Sensor data collector |
 | 5501 | UDP | Sensor commands (outbound) |

-## API Endpoints
+## Documentation

-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| GET | `/health` | Health check |
-| GET | `/api/v1/sensors` | List sensors |
-| GET | `/api/v1/sensors/<hostname>` | Get sensor |
-| POST | `/api/v1/sensors/<hostname>/command` | Send command |
-| GET | `/api/v1/devices` | List devices |
-| GET | `/api/v1/devices/<mac>` | Get device |
-| GET | `/api/v1/alerts` | List alerts |
-| GET | `/api/v1/events` | List events |
-| GET | `/api/v1/probes` | List probe requests |
-| GET | `/api/v1/probes/ssids` | List SSIDs |
-| GET | `/api/v1/stats` | Statistics |
+- [Installation](docs/INSTALL.md)
+- [Usage Guide](docs/USAGE.md)
+- [Cheatsheet](docs/CHEATSHEET.md)

-## Container
+## License

-```bash
-make build
-make run
-make logs
-make stop
-```
-
-## Testing
-
-```bash
-make test
-```
+MIT
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -0,0 +1,54 @@
+# ESP32-Web Roadmap
+
+## v0.1.0 - Project Scaffold [DONE]
+
+- [x] Flask app factory with blueprints
+- [x] SQLAlchemy models (Sensor, Device, Sighting, Alert, Event, Probe)
+- [x] UDP collector for sensor data streams
+- [x] Core API endpoints
+- [x] pytest setup
+- [x] Containerfile + Makefile
+
+## v0.1.1 - Server Management [DONE]
+
+- [x] start/stop/restart/status via Makefile
+- [x] Health endpoint with uptime
+- [x] Database migrations (Flask-Migrate)
+- [x] Listen on all interfaces
+
+## v0.2.0 - OSINT Features
+
+- [ ] MAC vendor lookup (IEEE OUI database)
+- [ ] BLE company_id to manufacturer mapping
+- [ ] Device profile enrichment
+- [ ] Export endpoints (CSV, JSON)
+
+## v0.3.0 - Fleet Management
+
+- [ ] Sensor config endpoint (GET/PUT)
+- [ ] OTA trigger endpoint
+- [ ] Calibration trigger endpoint
+- [ ] Sensor history/metrics
+
+## v0.4.0 - Zones & Presence
+
+- [ ] Zone management (assign sensors to areas)
+- [ ] Device zone tracking
+- [ ] Dwell time analysis
+- [ ] Presence history
+
+## v1.0.0 - Production Ready
+
+- [ ] Authentication (API keys or JWT)
+- [ ] Rate limiting
+- [ ] PostgreSQL support
+- [ ] Systemd service file
+- [ ] Production deployment guide
+
+## Future
+
+- WebSocket for real-time updates
+- Web dashboard (htmx + Pico CSS)
+- Home Assistant integration
+- Grafana dashboards
+- Webhook callbacks for alerts
--- a/TASKS.md
+++ b/TASKS.md
@@ -0,0 +1,46 @@
+# ESP32-Web Tasks
+
+**Last Updated:** 2026-02-05
+
+## Current Sprint: v0.2.0 — OSINT Features
+
+### P1 - High
+- [ ] Download and parse IEEE OUI database
+- [ ] MAC vendor lookup utility
+- [ ] BLE company_id mapping
+- [ ] `GET /api/v1/devices/<mac>/profile` enriched endpoint
+
+### P2 - Normal
+- [ ] Export endpoints (`/api/v1/export/devices.csv`)
+- [ ] Add vendor field population on device creation
+- [ ] Sensor heartbeat timeout detection
+
+### P3 - Low
+- [ ] Add pagination to all list endpoints
+- [ ] Add OpenAPI/Swagger spec
+- [ ] Add request logging middleware
+
+## Completed: v0.1.1 - Server Management
+
+- [x] Makefile start/stop/restart/status commands
+- [x] Health endpoint with uptime tracking
+- [x] CLI module (esp32-web command)
+- [x] Initial database migration
+- [x] Listen on all interfaces (0.0.0.0:5500)
+
+## Completed: v0.1.0 - Project Scaffold
+
+- [x] Flask app factory pattern
+- [x] SQLAlchemy 2.x models
+- [x] API blueprint with endpoints
+- [x] UDP collector with parsers
+- [x] pytest fixtures and tests
+- [x] Containerfile for podman
+- [x] Makefile for common tasks
+
+## Notes
+
+- Sensors send data to UDP 5500
+- API listens on TCP 5500
+- Commands sent to sensors on UDP 5501
+- SQLite for dev, PostgreSQL for prod
--- a/TODO.md
+++ b/TODO.md
@@ -0,0 +1,57 @@
+# ESP32-Web TODO
+
+## API
+
+- [ ] Pagination for all list endpoints
+- [ ] Filter by date range
+- [ ] Sort options
+- [ ] OpenAPI/Swagger spec generation
+- [ ] Rate limiting (flask-limiter)
+- [ ] API authentication (JWT or API keys)
+
+## OSINT
+
+- [ ] IEEE OUI database download script
+- [ ] MAC vendor lookup on device creation
+- [ ] BLE company ID database
+- [ ] Device fingerprinting by advertisement patterns
+- [ ] SSID categorization (home, corporate, mobile hotspot)
+
+## Collector
+
+- [ ] Heartbeat timeout (mark sensor offline)
+- [ ] CSI data storage (optional, high volume)
+- [ ] Data retention policy (auto-cleanup old records)
+- [ ] Metrics collection (packets/sec, errors)
+
+## Fleet Management
+
+- [ ] Sensor config read/write
+- [ ] OTA orchestration
+- [ ] Calibration management
+- [ ] Bulk commands
+
+## Deployment
+
+- [ ] Systemd service file
+- [ ] PostgreSQL configuration
+- [ ] Nginx reverse proxy config
+- [ ] TLS setup guide
+- [ ] Backup/restore scripts
+
+## Testing
+
+- [ ] API endpoint tests (all routes)
+- [ ] Collector parser tests (edge cases)
+- [ ] Integration tests with mock sensors
+- [ ] Load testing
+
+## Ideas
+
+- WebSocket for live updates
+- SSE (Server-Sent Events) alternative
+- Webhook callbacks for alerts
+- Slack/Discord notifications
+- Home Assistant MQTT integration
+- Grafana dashboard templates
+- ML device classification
--- a/docs/CHEATSHEET.md
+++ b/docs/CHEATSHEET.md
@@ -0,0 +1,97 @@
+# ESP32-Web Cheatsheet
+
+## Server Management
+
+```bash
+make start      # Start server
+make stop       # Stop server
+make restart    # Restart server
+make status     # Show PID + uptime
+make logs       # Tail logs
+make dev        # Run with debug mode
+```
+
+## Database
+
+```bash
+make migrate              # Apply migrations
+make migrate-init         # Initialize migrations
+make migrate-create msg="description"  # Create migration
+```
+
+## Testing
+
+```bash
+make test       # Run all tests
+pytest -v       # Verbose output
+pytest -k test_sensors  # Run specific tests
+```
+
+## Container
+
+```bash
+make build          # Build image
+make container-run  # Run container
+make container-stop # Stop container
+make container-logs # View logs
+```
+
+## API Quick Reference
+
+```bash
+# Health
+curl localhost:5500/health
+
+# Sensors
+curl localhost:5500/api/v1/sensors
+curl localhost:5500/api/v1/sensors/HOSTNAME
+curl -X POST localhost:5500/api/v1/sensors/HOSTNAME/command -H "Content-Type: application/json" -d '{"command":"STATUS"}'
+
+# Devices
+curl localhost:5500/api/v1/devices
+curl "localhost:5500/api/v1/devices?type=ble&limit=50"
+curl localhost:5500/api/v1/devices/MAC
+
+# Alerts
+curl localhost:5500/api/v1/alerts
+curl "localhost:5500/api/v1/alerts?type=deauth&hours=48"
+
+# Events
+curl localhost:5500/api/v1/events
+curl "localhost:5500/api/v1/events?type=presence"
+
+# Probes
+curl localhost:5500/api/v1/probes
+curl localhost:5500/api/v1/probes/ssids
+
+# Stats
+curl localhost:5500/api/v1/stats
+```
+
+## Query Parameters
+
+| Parameter | Endpoints | Description |
+|-----------|-----------|-------------|
+| type | devices, alerts, events | Filter by type |
+| hours | alerts, events, probes | Time window (default: 24) |
+| limit | devices, alerts, events, probes | Max results (default: 100) |
+| offset | devices, alerts, events, probes | Skip N results |
+| ssid | probes | Filter by SSID |
+| sensor_id | alerts, events | Filter by sensor |
+
+## Files
+
+| File | Description |
+|------|-------------|
+| `/tmp/esp32-web.pid` | Server PID file |
+| `/tmp/esp32-web.log` | Server log file |
+| `instance/esp32.db` | SQLite database |
+| `.env` | Environment config |
+
+## Ports
+
+| Port | Protocol | Description |
+|------|----------|-------------|
+| 5500 | TCP | HTTP API |
+| 5500 | UDP | Sensor data |
+| 5501 | UDP | Sensor commands |
--- a/docs/INSTALL.md
+++ b/docs/INSTALL.md
@@ -0,0 +1,76 @@
+# Installation
+
+## Requirements
+
+- Python 3.11+
+- pip
+- SQLite (included) or PostgreSQL
+
+## Development Setup
+
+```bash
+# Clone repository
+cd ~/git/esp32-web
+
+# Create virtual environment
+python3 -m venv .venv
+source .venv/bin/activate
+
+# Install with dev dependencies
+pip install -e ".[dev]"
+
+# Initialize database
+flask --app src/esp32_web db init
+flask --app src/esp32_web db migrate -m "initial"
+flask --app src/esp32_web db upgrade
+
+# Verify installation
+make test
+```
+
+## Configuration
+
+Copy `.env.example` to `.env` and adjust:
+
+```bash
+cp .env.example .env
+```
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| SECRET_KEY | dev-key | Flask secret key |
+| DATABASE_URL | sqlite:///esp32.db | Database connection |
+| UDP_PORT | 5500 | UDP collector port |
+| CMD_PORT | 5501 | Sensor command port |
+| SENSOR_TIMEOUT | 60 | Seconds before sensor marked offline |
+
+## Container Deployment
+
+```bash
+# Build image
+make build
+
+# Run container
+make container-run
+
+# View logs
+make container-logs
+
+# Stop container
+make container-stop
+```
+
+## Production
+
+For production, use gunicorn:
+
+```bash
+gunicorn -b 0.0.0.0:5500 -w 4 'esp32_web:create_app()'
+```
+
+With PostgreSQL:
+
+```bash
+export DATABASE_URL="postgresql://user:pass@localhost/esp32web"
+flask --app src/esp32_web db upgrade
+```
--- a/docs/USAGE.md
+++ b/docs/USAGE.md
@@ -0,0 +1,141 @@
+# Usage Guide
+
+## Starting the Server
+
+```bash
+# Activate virtual environment
+cd ~/git/esp32-web
+source .venv/bin/activate
+
+# Start server
+make start
+
+# Check status
+make status
+
+# View logs
+make logs
+```
+
+## API Examples
+
+### Health Check
+
+```bash
+curl http://localhost:5500/health
+```
+
+Response:
+```json
+{"status": "ok", "uptime": "5m23s", "uptime_seconds": 323}
+```
+
+### List Sensors
+
+```bash
+curl http://localhost:5500/api/v1/sensors
+```
+
+Response:
+```json
+{
+  "sensors": [
+    {"id": 1, "hostname": "hollow-acorn", "ip": "192.168.129.31", "status": "online", "last_seen": "..."},
+    {"id": 2, "hostname": "muddy-storm", "ip": "192.168.129.29", "status": "online", "last_seen": "..."}
+  ]
+}
+```
+
+### Get Sensor Details
+
+```bash
+curl http://localhost:5500/api/v1/sensors/hollow-acorn
+```
+
+### Send Command to Sensor
+
+```bash
+curl -X POST http://localhost:5500/api/v1/sensors/hollow-acorn/command \
+  -H "Content-Type: application/json" \
+  -d '{"command": "STATUS"}'
+```
+
+### List Devices
+
+```bash
+# All devices
+curl http://localhost:5500/api/v1/devices
+
+# BLE devices only
+curl "http://localhost:5500/api/v1/devices?type=ble"
+
+# With pagination
+curl "http://localhost:5500/api/v1/devices?limit=50&offset=0"
+```
+
+### Get Device Details
+
+```bash
+curl http://localhost:5500/api/v1/devices/aa:bb:cc:dd:ee:ff
+```
+
+### List Alerts
+
+```bash
+# Last 24 hours (default)
+curl http://localhost:5500/api/v1/alerts
+
+# Last 7 days
+curl "http://localhost:5500/api/v1/alerts?hours=168"
+
+# Filter by type
+curl "http://localhost:5500/api/v1/alerts?type=deauth"
+```
+
+### List Probe Requests
+
+```bash
+# All probes
+curl http://localhost:5500/api/v1/probes
+
+# Filter by SSID
+curl "http://localhost:5500/api/v1/probes?ssid=MyNetwork"
+
+# List SSIDs with counts
+curl http://localhost:5500/api/v1/probes/ssids
+```
+
+### Get Statistics
+
+```bash
+curl http://localhost:5500/api/v1/stats
+```
+
+Response:
+```json
+{
+  "sensors": {"total": 3, "online": 3},
+  "devices": {"total": 42, "ble": 35, "wifi": 7},
+  "alerts": {"count": 5, "hours": 24},
+  "events": {"count": 128, "hours": 24},
+  "probes": {"count": 89, "hours": 24}
+}
+```
+
+## Allowed Sensor Commands
+
+Commands that can be sent via the API:
+
+| Command | Description |
+|---------|-------------|
+| STATUS | Get sensor status |
+| REBOOT | Reboot sensor |
+| IDENTIFY | Flash LED for 5 seconds |
+| BLE ON/OFF | Toggle BLE scanning |
+| ADAPTIVE ON/OFF | Toggle adaptive sampling |
+| RATE <10-100> | Set CSI rate |
+| POWER <2-20> | Set TX power (dBm) |
+| CSIMODE RAW/COMPACT/HYBRID | Set CSI output mode |
+| PRESENCE ON/OFF | Toggle presence detection |
+| CALIBRATE [seconds] | Start baseline calibration |
+| CHANSCAN ON/OFF/NOW | Channel scanning |