docs: Add v2.0 Flask API backend roadmap

Plan for REST API backend (purple team/OPSEC/OSINT): - Phase 1: Project scaffold, SQLAlchemy models, podman - Phase 2: UDP collector for sensor data streams - Phase 3: Core API endpoints (sensors, devices, alerts, probes, events) - Phase 4: OSINT features (OUI lookup, BLE company ID mapping) - Phase 5: Fleet management API Port allocation: - TCP 5500: HTTP REST API - UDP 5500: Sensor data collector - UDP 5501: Sensor commands (outbound) Also documents completed v1.8 (HTTPS OTA) and v1.9 (multi-channel scanning, BLE fingerprinting) milestones.
2026-02-05 20:41:24 +01:00
parent d58b6dd814
commit f87ddec742
4 changed files with 247 additions and 37 deletions
--- a/PROJECT.md
+++ b/PROJECT.md
@@ -1,47 +1,97 @@
 # ESP32 Hacking Project

 ## Overview
-Firmware customization and experimentation for ESP32 CSI (Channel State Information) sensors used in the [wifi-sensing](~/git/wifi-sensing/) project.
+
+Firmware and tooling for ESP32 CSI (Channel State Information) sensors used for passive wireless sensing, device tracking, and security monitoring (OPSEC/OSINT/Purple team).

 ## Goals
- Document current firmware and configuration
- Add remote management commands (reboot, identify, status)
- Implement OTA firmware updates
- Explore BLE scanning for complementary presence detection
- Optimize CSI data pipeline (adaptive rate, on-device processing)

-## Current State
- 3x ESP32-DevKitC V1 deployed with stock `csi_recv_router` firmware
- Firmware sends raw CSI data via UDP at ~100 pkt/s per device
- No remote management capability (must physically access USB)
+- Custom ESP32 firmware with remote management, OTA updates, adaptive sampling
+- BLE scanning and WiFi probe/deauth detection for device intelligence
+- CSI-based presence detection without cameras
+- Flask REST API backend for data aggregation and OSINT queries
+- Fleet management for multiple sensors
+
+## Components
+
+| Component | Location | Description |
+|-----------|----------|-------------|
+| Firmware | `get-started/csi_recv_router/` | ESP32 sensor firmware (C, ESP-IDF) |
+| CLI Tools | `~/git/esp-tools/` | `esp-ctl`, `esp-fleet`, `esp-ota` |
+| Flask API | `~/git/esp32-web/` | REST API backend (Python, Flask) |
+
+## Current State (v1.9)
+
+- 3x ESP32-DevKitC V1 deployed with custom firmware
+- UDP data streams: CSI_DATA, BLE_DATA, PROBE_DATA, ALERT_DATA, EVENT
+- Remote management via UDP commands (port 5501)
+- OTA firmware updates (HTTP/HTTPS)
+- Presence detection via CSI baseline calibration
+- Multi-channel scanning for broader WiFi coverage
+- BLE fingerprinting (company_id, tx_power, flags)

 ## Hardware
+
 | Device | Chip | IP Address | Location |
 |--------|------|------------|----------|
 | muddy-storm | ESP32-WROOM-32 | 192.168.129.29 | Living Room |
 | amber-maple | ESP32-WROOM-32 | 192.168.129.30 | Office |
 | hollow-acorn | ESP32-WROOM-32 | 192.168.129.31 | Kitchen |

+## Network Ports
+
+| Port | Protocol | Direction | Description |
+|------|----------|-----------|-------------|
+| 5500 | UDP | Sensors → Flask | Sensor data streams (CSI, BLE, probes, alerts, events) |
+| 5500 | TCP | Clients → Flask | REST API (HTTP) |
+| 5501 | UDP | Flask → Sensors | Sensor commands |
+
+## Data Streams (UDP 5500)
+
+| Stream | Format | Use Case |
+|--------|--------|----------|
+| CSI_DATA | hostname,count,mac,rssi,... | Motion/presence detection |
+| BLE_DATA | hostname,mac,rssi,type,name,company_id,tx_power,flags | Device fingerprinting |
+| PROBE_DATA | hostname,mac,rssi,ssid,channel | Device tracking, SSID enumeration |
+| ALERT_DATA | hostname,type,source,target,rssi | Wireless IDS (deauth detection) |
+| EVENT | hostname,key=value pairs | Motion, presence, calibration events |
+
 ## Software Stack
+
 | Component | Version | Purpose |
 |-----------|---------|---------|
-| ESP-IDF | v5.5.2 | Development framework (`~/esp/esp-idf/`) |
-| esp-csi | latest | CSI extraction library |
-| FreeRTOS | (bundled) | RTOS kernel |
+| ESP-IDF | v5.5.2 | Firmware development framework |
+| Flask | 3.x | REST API backend |
+| SQLAlchemy | 2.x | Database ORM |
+| SQLite/PostgreSQL | - | Data storage |
+| podman | - | Container runtime |

 ## Key Paths
+
 | Path | Description |
 |------|-------------|
 | `~/git/esp32-hacking/` | This project (firmware sources, docs) |
-| `~/git/wifi-sensing/` | Server application (receives CSI data) |
+| `~/git/esp32-web/` | Flask API backend (planned) |
+| `~/git/esp-tools/` | CLI tools (esp-ctl, esp-fleet, esp-ota) |
 | `~/esp/esp-idf/` | ESP-IDF toolchain |
-| `~/esp/esp-csi/` | Original esp-csi repo (build from here) |

-## Dependencies
- ESP-IDF toolchain (`~/esp/esp-idf/`)
- USB cable for initial flash (OTA planned)
+## API Endpoints (Planned)
+
+Base URL: `http://<host>:5500/api/v1`
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | `/sensors` | List sensors with status |
+| GET | `/devices` | List discovered devices |
+| GET | `/alerts` | Alert feed with filters |
+| GET | `/probes` | Probe requests |
+| GET | `/events` | Sensor events |
+| POST | `/sensors/<id>/command` | Send command to sensor |
+| GET | `/stats` | Aggregate statistics |

 ## References
+
 - [ESP-IDF Docs](https://docs.espressif.com/projects/esp-idf/en/latest/)
 - [ESP-CSI GitHub](https://github.com/espressif/esp-csi)
- [ESP32 WiFi CSI API](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/network/esp_wifi.html)
+- [IEEE OUI Database](https://standards-oui.ieee.org/)
+- [Bluetooth Company Identifiers](https://www.bluetooth.com/specifications/assigned-numbers/company-identifiers/)
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -131,9 +131,77 @@ Note: Promiscuous mode (probe/deauth capture) disabled on original ESP32 — bre
 - [x] presence= and pr_score= fields in STATUS reply
 - [x] NVS persistence for baseline (bl_amps blob, bl_nsub) and presence config
 - [ ] Tune presence threshold per room with real-world testing
- [ ] Pi-side presence event handling in watch daemon

-## v2.0 - Hardware Upgrade (ESP32-S3/C6)
+## v1.8 - HTTPS OTA Support [DONE]
+- [x] Support HTTPS URLs for OTA updates (esp_https_ota)
+- [x] CI: Upload firmware to Gitea releases for OTA
+- [x] CI: Simplify deploy script to pure POSIX sh
+
+## v1.9 - Multi-Channel Scanning & BLE Fingerprinting [DONE]
+- [x] CHANSCAN command (ON/OFF/NOW/INTERVAL)
+- [x] Channel hopping (1-13) with 100ms dwell, pause CSI during scan
+- [x] chanscan= field in STATUS, chanscan_int NVS persistence
+- [x] BLE fingerprinting: company_id, tx_power, adv_flags in BLE_DATA
+- [x] Historical presence sessions support
+
+## v2.0 - Flask API Backend (Purple Team)
+
+REST API backend for OPSEC/OSINT/Purple team operations using ESP32 sensor fleet.
+API-first design; frontend dashboard deferred to v2.1+.
+
+- **HTTP API:** TCP 5500
+- **UDP Collector:** UDP 5500 (sensor data)
+- **Sensor Commands:** UDP 5501 (outbound)
+
+### Phase 1: Project Setup
+- [ ] Project scaffold (`~/git/esp32-web/`) with Flask + SQLAlchemy + Blueprints
+- [ ] Database schema: sensors, devices, sightings, alerts, events, probes
+- [ ] Containerfile for podman deployment
+- [ ] Makefile (build, run, dev, stop, logs)
+- [ ] pytest setup with fixtures
+
+### Phase 2: UDP Collector
+- [ ] Async UDP listener daemon (threading or asyncio)
+- [ ] Parse all sensor streams: CSI_DATA, BLE_DATA, PROBE_DATA, ALERT_DATA, EVENT
+- [ ] Store to database with timestamps
+- [ ] Sensor heartbeat tracking (online/offline status)
+- [ ] Run as background thread alongside Flask
+
+### Phase 3: Core API Endpoints
+- [ ] `GET /api/v1/sensors` — list sensors with status, uptime, last_seen
+- [ ] `GET /api/v1/sensors/<id>/status` — detailed sensor info
+- [ ] `POST /api/v1/sensors/<id>/command` — send UDP command (proxy)
+- [ ] `GET /api/v1/devices` — list all discovered devices (BLE + WiFi)
+- [ ] `GET /api/v1/devices/<mac>` — device profile (sightings, zones, vendor)
+- [ ] `GET /api/v1/alerts` — alert feed with pagination + filters
+- [ ] `GET /api/v1/probes` — probe requests with SSID enumeration
+- [ ] `GET /api/v1/events` — sensor events (motion, presence, calibration)
+
+### Phase 4: OSINT Features
+- [ ] MAC vendor lookup (IEEE OUI database)
+- [ ] BLE company_id to manufacturer mapping
+- [ ] `GET /api/v1/devices/<mac>/profile` — enriched device intel
+- [ ] `GET /api/v1/stats` — aggregate statistics (device counts, alert counts)
+- [ ] Export endpoints: `GET /api/v1/export/devices.csv`, `.json`
+
+### Phase 5: Fleet Management API
+- [ ] `GET /api/v1/sensors/<id>/config` — current sensor configuration
+- [ ] `PUT /api/v1/sensors/<id>/config` — update sensor settings
+- [ ] `POST /api/v1/sensors/<id>/ota` — trigger OTA update
+- [ ] `POST /api/v1/sensors/<id>/calibrate` — trigger baseline calibration
+- [ ] `GET /api/v1/sensors/<id>/history` — historical metrics
+
+## v2.1 - Web Dashboard (Future)
+
+Frontend dashboard using htmx + Pico CSS, built on top of v2.0 API.
+
+- [ ] Live sensor status dashboard
+- [ ] Device inventory table with search/filter
+- [ ] Alert timeline with severity badges
+- [ ] Presence heatmap per zone
+- [ ] Sensor fleet management UI
+
+## v3.0 - Hardware Upgrade (ESP32-S3/C6)

 Requires replacing current ESP32 (original) DevKitC V1 boards with ESP32-S3
 or ESP32-C6 modules. The original ESP32 lacks FTM and has CSI/promiscuous
@@ -152,10 +220,11 @@ mode conflicts.
 ## Future
 - AP+STA config portal (WIFI_MODE_APSTA, captive portal for initial setup)
 - ESP-NOW mesh (ESP32-to-ESP32 CSI)
- Multi-channel scanning (hop across WiFi channels)
- BLE device fingerprinting (identify phone models by advertisement patterns)
- Historical presence logging (who was here, when, how long)
 - External sensor support (PIR, temp/humidity via GPIO)
 - Pin mapping for ESP32-DevKitC V1
 - Compare CSI quality: passive (router) vs active (ESP-NOW)
 - Multi-sensor deployment guide (placement, zones, triangulation)
+- Home Assistant integration (MQTT discovery, entity creation)
+- Grafana dashboards for long-term analytics
+- ML-based device classification (phone vs laptop vs IoT)
+- RSSI triangulation with 3+ sensors (approximate device location)
--- a/TASKS.md
+++ b/TASKS.md
@@ -2,15 +2,41 @@

 **Last Updated:** 2026-02-05

-## Current Sprint: v1.7+ — Presence Tuning & Integration
+## Current Sprint: v2.0 — Flask API Backend

-### P2 - Normal
+### P0 - Critical (Phase 1: Project Setup)
+- [ ] Create project scaffold `~/git/esp32-web/`
+- [ ] Flask app factory pattern with Blueprints
+- [ ] HTTP API on TCP 5500, UDP collector on UDP 5500
+- [ ] SQLAlchemy models: Sensor, Device, Sighting, Alert, Event, Probe
+- [ ] Containerfile for podman
+- [ ] Makefile (build, run, dev, stop, logs)
+- [ ] Basic pytest setup
+
+### P1 - High (Phase 2: UDP Collector)
+- [ ] UDP listener thread (parse CSI_DATA, BLE_DATA, PROBE_DATA, ALERT_DATA, EVENT)
+- [ ] Store parsed data to SQLite/PostgreSQL
+- [ ] Sensor heartbeat tracking (mark online/offline)
+- [ ] Integrate collector with Flask app lifecycle
+
+### P1 - High (Phase 3: Core API)
+- [ ] `GET /api/v1/sensors` — list sensors
+- [ ] `GET /api/v1/devices` — list devices (BLE + WiFi MACs)
+- [ ] `GET /api/v1/alerts` — alert feed with pagination
+- [ ] `GET /api/v1/probes` — probe requests
+- [ ] `GET /api/v1/events` — sensor events
+- [ ] `POST /api/v1/sensors/<id>/command` — send command to sensor
+
+### P2 - Normal (Phase 4: OSINT)
+- [ ] MAC vendor lookup (OUI database)
+- [ ] BLE company_id mapping
+- [ ] `GET /api/v1/stats` — aggregate statistics
+- [ ] Export endpoints (CSV, JSON)
+
+### P2 - Normal (Backlog from v1.x)
 - [ ] Tune presence threshold per room with real-world testing
- [x] Pi-side presence event handling in watch daemon
- [x] Firmware security audit (cppcheck + manual review — no issues found)
 - [ ] Power consumption measurements using POWERTEST + external meter
 - [ ] Test OTA rollback (flash bad firmware, verify auto-revert)
- [ ] Create HA webhook automations for deauth_flood / unknown_probe

 ### P3 - Low
 - [ ] Deep sleep mode with wake-on-CSI-motion
@@ -19,6 +45,20 @@
 - [ ] Document esp-radar console features
 - [ ] Pin mapping for ESP32-DevKitC V1

+## Completed: v1.9 - Multi-Channel Scanning & BLE Fingerprinting
+
+- [x] CHANSCAN command (ON/OFF/NOW/INTERVAL)
+- [x] Channel hopping (1-13) with 100ms dwell, pause CSI during scan
+- [x] chanscan= field in STATUS, chanscan_int NVS persistence
+- [x] BLE fingerprinting: company_id, tx_power, adv_flags in BLE_DATA
+- [x] Historical presence sessions support
+
+## Completed: v1.8 - HTTPS OTA Support
+
+- [x] Support HTTPS URLs for OTA updates (esp_https_ota)
+- [x] CI: Upload firmware to Gitea releases for OTA
+- [x] CI: Simplify deploy script to pure POSIX sh
+
 ## Completed: v1.7 - Baseline Calibration & Presence Detection

 - [x] CALIBRATE command (capture N seconds of CSI, average per-subcarrier amplitudes)
@@ -172,7 +212,7 @@
 ## Notes

 - Adaptive threshold varies by environment; 0.001-0.01 is a good starting range
- NVS keys: `send_rate`, `tx_power`, `adaptive`, `threshold`, `ble_scan`, `target_ip`, `target_port`, `hostname`, `boot_count`, `csi_mode`, `hybrid_n`, `auth_secret`, `flood_thresh`, `flood_window`, `scan_rate`, `probe_rate`, `powersave`, `presence`, `pr_thresh`, `bl_nsub`, `bl_amps`
+- NVS keys (24 total): `send_rate`, `tx_power`, `adaptive`, `threshold`, `ble_scan`, `target_ip`, `target_port`, `hostname`, `boot_count`, `csi_mode`, `hybrid_n`, `auth_secret`, `flood_thresh`, `flood_window`, `scan_rate`, `probe_rate`, `powersave`, `presence`, `pr_thresh`, `bl_nsub`, `bl_amps`, `chanscan`, `chanscan_int`
 - EVENT packets include sensor hostname: `EVENT,<hostname>,motion=... rate=... wander=...`
 - ALERT_DATA format: `ALERT_DATA,<hostname>,<deauth|disassoc>,<sender_mac>,<target_mac>,<rssi>` or `ALERT_DATA,<hostname>,deauth_flood,<count>,<window_s>`
 - STATUS fields: `uptime=`, `uptime_s=`, `heap=`, `rssi=`, `channel=`, `tx_power=`, `rate=`, `csi_rate=`, `hostname=`, `version=`, `adaptive=`, `motion=`, `ble=`, `target=`, `temp=`, `csi_count=`, `boots=`, `rssi_min=`, `rssi_max=`, `csi_mode=`, `hybrid_n=`, `auth=`, `flood_thresh=`, `powersave=`, `presence=`, `pr_score=`
--- a/TODO.md
+++ b/TODO.md
@@ -1,28 +1,79 @@
 # ESP32 Hacking TODO

+## Flask API (`~/git/esp32-web/`)
+
+### Architecture
+- [ ] App factory pattern (`create_app()`)
+- [ ] Blueprints: `api`, `collector`
+- [ ] SQLAlchemy with migrations (Flask-Migrate)
+- [ ] Background UDP collector (threading or Celery)
+- [ ] Config from environment variables
+- [ ] Port 5500: HTTP API (TCP) + UDP collector (UDP) on same port number
+
+### Database Schema
+- [ ] `sensors` — id, hostname, ip, last_seen, status, config_json
+- [ ] `devices` — mac, type (ble/wifi), vendor, first_seen, last_seen
+- [ ] `sightings` — device_id, sensor_id, rssi, timestamp
+- [ ] `alerts` — sensor_id, type, source_mac, target_mac, rssi, timestamp
+- [ ] `probes` — device_id, sensor_id, ssid, rssi, channel, timestamp
+- [ ] `events` — sensor_id, event_type, payload_json, timestamp
+
+### API Endpoints
+- [ ] Sensors: list, detail, status, command, config, history
+- [ ] Devices: list, detail, profile, sightings
+- [ ] Alerts: list with filters (type, sensor, time range)
+- [ ] Probes: list, group by SSID, group by MAC
+- [ ] Events: list with filters
+- [ ] Stats: counts, activity graphs data
+- [ ] Export: CSV, JSON for devices/alerts/probes
+
+### UDP Collector
+- [ ] Parse CSI_DATA (hostname, count, mac, rssi, features)
+- [ ] Parse BLE_DATA (hostname, mac, rssi, type, name, company_id, tx_power, flags)
+- [ ] Parse PROBE_DATA (hostname, mac, rssi, ssid, channel)
+- [ ] Parse ALERT_DATA (hostname, type, source, target, rssi OR flood count)
+- [ ] Parse EVENT (hostname, key=value pairs)
+- [ ] Heartbeat timeout detection (mark sensor offline)
+
+### OSINT
+- [ ] IEEE OUI database (download + parse)
+- [ ] BLE company ID database (Bluetooth SIG)
+- [ ] Device fingerprinting by BLE advertisement patterns
+- [ ] Probe request SSID profiling (home networks, corporate, etc.)
+
 ## Firmware
- [ ] On-device CSI processing (send metrics, not raw)
+
 - [ ] Deep sleep mode with wake-on-CSI-motion
 - [ ] Battery-optimized duty cycling
+- [ ] AP+STA config portal (captive portal for initial setup)

 ## Tools (esp-ctl)
- [ ] Multi-sensor BLE correlation (zone tracking by source sensor)
+
+- [ ] Migrate OSINT database to Flask API (esp-ctl becomes thin client)
+- [ ] `esp-ctl api` subcommand (query Flask API)

 ## Testing
+
 - [ ] Benchmark: CSI callback latency
 - [ ] Benchmark: UDP throughput at different rates
 - [ ] Power consumption measurements (per-mode: idle, CSI, BLE, probe)
+- [ ] API load testing (concurrent requests)

 ## Documentation
+
+- [ ] Flask API: OpenAPI/Swagger spec
+- [ ] Deployment guide (podman, systemd)
 - [ ] Pin mapping for ESP32-DevKitC V1
 - [ ] Compare CSI quality: passive (router) vs active (ESP-NOW)
 - [ ] Multi-sensor deployment guide (placement, zones, triangulation)

 ## Ideas
+
 - ESP-NOW mesh for direct ESP32-to-ESP32 CSI
 - External PIR sensor for CSI ground truth validation
- AP+STA config portal (captive portal for initial setup)
- Multi-channel scanning (hop across WiFi channels)
- RSSI triangulation with 3 sensors (approximate device location)
- BLE device fingerprinting (identify phone models by advertisement patterns)
- Historical presence logging (who was here, when, how long)
+- RSSI triangulation with 3+ sensors (approximate device location)
+- Home Assistant MQTT discovery integration
+- Grafana dashboards for long-term analytics
+- ML-based device classification (phone vs laptop vs IoT)
+- Webhook callbacks for alerts (Slack, Discord, ntfy)
+- Rate limiting and API authentication (JWT)