From f87ddec74210ec4c0b60314fa9bd70a83a737955 Mon Sep 17 00:00:00 2001 From: user Date: Thu, 5 Feb 2026 20:41:24 +0100 Subject: [PATCH] docs: Add v2.0 Flask API backend roadmap Plan for REST API backend (purple team/OPSEC/OSINT): - Phase 1: Project scaffold, SQLAlchemy models, podman - Phase 2: UDP collector for sensor data streams - Phase 3: Core API endpoints (sensors, devices, alerts, probes, events) - Phase 4: OSINT features (OUI lookup, BLE company ID mapping) - Phase 5: Fleet management API Port allocation: - TCP 5500: HTTP REST API - UDP 5500: Sensor data collector - UDP 5501: Sensor commands (outbound) Also documents completed v1.8 (HTTPS OTA) and v1.9 (multi-channel scanning, BLE fingerprinting) milestones. --- PROJECT.md | 88 ++++++++++++++++++++++++++++++++++++++++++------------ ROADMAP.md | 79 ++++++++++++++++++++++++++++++++++++++++++++---- TASKS.md | 52 ++++++++++++++++++++++++++++---- TODO.md | 65 +++++++++++++++++++++++++++++++++++----- 4 files changed, 247 insertions(+), 37 deletions(-) diff --git a/PROJECT.md b/PROJECT.md index ee42800..80dfcf4 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -1,47 +1,97 @@ # ESP32 Hacking Project ## Overview -Firmware customization and experimentation for ESP32 CSI (Channel State Information) sensors used in the [wifi-sensing](~/git/wifi-sensing/) project. + +Firmware and tooling for ESP32 CSI (Channel State Information) sensors used for passive wireless sensing, device tracking, and security monitoring (OPSEC/OSINT/Purple team). ## Goals -- Document current firmware and configuration -- Add remote management commands (reboot, identify, status) -- Implement OTA firmware updates -- Explore BLE scanning for complementary presence detection -- Optimize CSI data pipeline (adaptive rate, on-device processing) -## Current State -- 3x ESP32-DevKitC V1 deployed with stock `csi_recv_router` firmware -- Firmware sends raw CSI data via UDP at ~100 pkt/s per device -- No remote management capability (must physically access USB) +- Custom ESP32 firmware with remote management, OTA updates, adaptive sampling +- BLE scanning and WiFi probe/deauth detection for device intelligence +- CSI-based presence detection without cameras +- Flask REST API backend for data aggregation and OSINT queries +- Fleet management for multiple sensors + +## Components + +| Component | Location | Description | +|-----------|----------|-------------| +| Firmware | `get-started/csi_recv_router/` | ESP32 sensor firmware (C, ESP-IDF) | +| CLI Tools | `~/git/esp-tools/` | `esp-ctl`, `esp-fleet`, `esp-ota` | +| Flask API | `~/git/esp32-web/` | REST API backend (Python, Flask) | + +## Current State (v1.9) + +- 3x ESP32-DevKitC V1 deployed with custom firmware +- UDP data streams: CSI_DATA, BLE_DATA, PROBE_DATA, ALERT_DATA, EVENT +- Remote management via UDP commands (port 5501) +- OTA firmware updates (HTTP/HTTPS) +- Presence detection via CSI baseline calibration +- Multi-channel scanning for broader WiFi coverage +- BLE fingerprinting (company_id, tx_power, flags) ## Hardware + | Device | Chip | IP Address | Location | |--------|------|------------|----------| | muddy-storm | ESP32-WROOM-32 | 192.168.129.29 | Living Room | | amber-maple | ESP32-WROOM-32 | 192.168.129.30 | Office | | hollow-acorn | ESP32-WROOM-32 | 192.168.129.31 | Kitchen | +## Network Ports + +| Port | Protocol | Direction | Description | +|------|----------|-----------|-------------| +| 5500 | UDP | Sensors → Flask | Sensor data streams (CSI, BLE, probes, alerts, events) | +| 5500 | TCP | Clients → Flask | REST API (HTTP) | +| 5501 | UDP | Flask → Sensors | Sensor commands | + +## Data Streams (UDP 5500) + +| Stream | Format | Use Case | +|--------|--------|----------| +| CSI_DATA | hostname,count,mac,rssi,... | Motion/presence detection | +| BLE_DATA | hostname,mac,rssi,type,name,company_id,tx_power,flags | Device fingerprinting | +| PROBE_DATA | hostname,mac,rssi,ssid,channel | Device tracking, SSID enumeration | +| ALERT_DATA | hostname,type,source,target,rssi | Wireless IDS (deauth detection) | +| EVENT | hostname,key=value pairs | Motion, presence, calibration events | + ## Software Stack + | Component | Version | Purpose | |-----------|---------|---------| -| ESP-IDF | v5.5.2 | Development framework (`~/esp/esp-idf/`) | -| esp-csi | latest | CSI extraction library | -| FreeRTOS | (bundled) | RTOS kernel | +| ESP-IDF | v5.5.2 | Firmware development framework | +| Flask | 3.x | REST API backend | +| SQLAlchemy | 2.x | Database ORM | +| SQLite/PostgreSQL | - | Data storage | +| podman | - | Container runtime | ## Key Paths + | Path | Description | |------|-------------| | `~/git/esp32-hacking/` | This project (firmware sources, docs) | -| `~/git/wifi-sensing/` | Server application (receives CSI data) | +| `~/git/esp32-web/` | Flask API backend (planned) | +| `~/git/esp-tools/` | CLI tools (esp-ctl, esp-fleet, esp-ota) | | `~/esp/esp-idf/` | ESP-IDF toolchain | -| `~/esp/esp-csi/` | Original esp-csi repo (build from here) | -## Dependencies -- ESP-IDF toolchain (`~/esp/esp-idf/`) -- USB cable for initial flash (OTA planned) +## API Endpoints (Planned) + +Base URL: `http://:5500/api/v1` + +| Method | Endpoint | Description | +|--------|----------|-------------| +| GET | `/sensors` | List sensors with status | +| GET | `/devices` | List discovered devices | +| GET | `/alerts` | Alert feed with filters | +| GET | `/probes` | Probe requests | +| GET | `/events` | Sensor events | +| POST | `/sensors//command` | Send command to sensor | +| GET | `/stats` | Aggregate statistics | ## References + - [ESP-IDF Docs](https://docs.espressif.com/projects/esp-idf/en/latest/) - [ESP-CSI GitHub](https://github.com/espressif/esp-csi) -- [ESP32 WiFi CSI API](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/network/esp_wifi.html) +- [IEEE OUI Database](https://standards-oui.ieee.org/) +- [Bluetooth Company Identifiers](https://www.bluetooth.com/specifications/assigned-numbers/company-identifiers/) diff --git a/ROADMAP.md b/ROADMAP.md index 6ddcd58..187462d 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -131,9 +131,77 @@ Note: Promiscuous mode (probe/deauth capture) disabled on original ESP32 — bre - [x] presence= and pr_score= fields in STATUS reply - [x] NVS persistence for baseline (bl_amps blob, bl_nsub) and presence config - [ ] Tune presence threshold per room with real-world testing -- [ ] Pi-side presence event handling in watch daemon -## v2.0 - Hardware Upgrade (ESP32-S3/C6) +## v1.8 - HTTPS OTA Support [DONE] +- [x] Support HTTPS URLs for OTA updates (esp_https_ota) +- [x] CI: Upload firmware to Gitea releases for OTA +- [x] CI: Simplify deploy script to pure POSIX sh + +## v1.9 - Multi-Channel Scanning & BLE Fingerprinting [DONE] +- [x] CHANSCAN command (ON/OFF/NOW/INTERVAL) +- [x] Channel hopping (1-13) with 100ms dwell, pause CSI during scan +- [x] chanscan= field in STATUS, chanscan_int NVS persistence +- [x] BLE fingerprinting: company_id, tx_power, adv_flags in BLE_DATA +- [x] Historical presence sessions support + +## v2.0 - Flask API Backend (Purple Team) + +REST API backend for OPSEC/OSINT/Purple team operations using ESP32 sensor fleet. +API-first design; frontend dashboard deferred to v2.1+. + +- **HTTP API:** TCP 5500 +- **UDP Collector:** UDP 5500 (sensor data) +- **Sensor Commands:** UDP 5501 (outbound) + +### Phase 1: Project Setup +- [ ] Project scaffold (`~/git/esp32-web/`) with Flask + SQLAlchemy + Blueprints +- [ ] Database schema: sensors, devices, sightings, alerts, events, probes +- [ ] Containerfile for podman deployment +- [ ] Makefile (build, run, dev, stop, logs) +- [ ] pytest setup with fixtures + +### Phase 2: UDP Collector +- [ ] Async UDP listener daemon (threading or asyncio) +- [ ] Parse all sensor streams: CSI_DATA, BLE_DATA, PROBE_DATA, ALERT_DATA, EVENT +- [ ] Store to database with timestamps +- [ ] Sensor heartbeat tracking (online/offline status) +- [ ] Run as background thread alongside Flask + +### Phase 3: Core API Endpoints +- [ ] `GET /api/v1/sensors` — list sensors with status, uptime, last_seen +- [ ] `GET /api/v1/sensors//status` — detailed sensor info +- [ ] `POST /api/v1/sensors//command` — send UDP command (proxy) +- [ ] `GET /api/v1/devices` — list all discovered devices (BLE + WiFi) +- [ ] `GET /api/v1/devices/` — device profile (sightings, zones, vendor) +- [ ] `GET /api/v1/alerts` — alert feed with pagination + filters +- [ ] `GET /api/v1/probes` — probe requests with SSID enumeration +- [ ] `GET /api/v1/events` — sensor events (motion, presence, calibration) + +### Phase 4: OSINT Features +- [ ] MAC vendor lookup (IEEE OUI database) +- [ ] BLE company_id to manufacturer mapping +- [ ] `GET /api/v1/devices//profile` — enriched device intel +- [ ] `GET /api/v1/stats` — aggregate statistics (device counts, alert counts) +- [ ] Export endpoints: `GET /api/v1/export/devices.csv`, `.json` + +### Phase 5: Fleet Management API +- [ ] `GET /api/v1/sensors//config` — current sensor configuration +- [ ] `PUT /api/v1/sensors//config` — update sensor settings +- [ ] `POST /api/v1/sensors//ota` — trigger OTA update +- [ ] `POST /api/v1/sensors//calibrate` — trigger baseline calibration +- [ ] `GET /api/v1/sensors//history` — historical metrics + +## v2.1 - Web Dashboard (Future) + +Frontend dashboard using htmx + Pico CSS, built on top of v2.0 API. + +- [ ] Live sensor status dashboard +- [ ] Device inventory table with search/filter +- [ ] Alert timeline with severity badges +- [ ] Presence heatmap per zone +- [ ] Sensor fleet management UI + +## v3.0 - Hardware Upgrade (ESP32-S3/C6) Requires replacing current ESP32 (original) DevKitC V1 boards with ESP32-S3 or ESP32-C6 modules. The original ESP32 lacks FTM and has CSI/promiscuous @@ -152,10 +220,11 @@ mode conflicts. ## Future - AP+STA config portal (WIFI_MODE_APSTA, captive portal for initial setup) - ESP-NOW mesh (ESP32-to-ESP32 CSI) -- Multi-channel scanning (hop across WiFi channels) -- BLE device fingerprinting (identify phone models by advertisement patterns) -- Historical presence logging (who was here, when, how long) - External sensor support (PIR, temp/humidity via GPIO) - Pin mapping for ESP32-DevKitC V1 - Compare CSI quality: passive (router) vs active (ESP-NOW) - Multi-sensor deployment guide (placement, zones, triangulation) +- Home Assistant integration (MQTT discovery, entity creation) +- Grafana dashboards for long-term analytics +- ML-based device classification (phone vs laptop vs IoT) +- RSSI triangulation with 3+ sensors (approximate device location) diff --git a/TASKS.md b/TASKS.md index 87d96f2..715e46a 100644 --- a/TASKS.md +++ b/TASKS.md @@ -2,15 +2,41 @@ **Last Updated:** 2026-02-05 -## Current Sprint: v1.7+ — Presence Tuning & Integration +## Current Sprint: v2.0 — Flask API Backend -### P2 - Normal +### P0 - Critical (Phase 1: Project Setup) +- [ ] Create project scaffold `~/git/esp32-web/` +- [ ] Flask app factory pattern with Blueprints +- [ ] HTTP API on TCP 5500, UDP collector on UDP 5500 +- [ ] SQLAlchemy models: Sensor, Device, Sighting, Alert, Event, Probe +- [ ] Containerfile for podman +- [ ] Makefile (build, run, dev, stop, logs) +- [ ] Basic pytest setup + +### P1 - High (Phase 2: UDP Collector) +- [ ] UDP listener thread (parse CSI_DATA, BLE_DATA, PROBE_DATA, ALERT_DATA, EVENT) +- [ ] Store parsed data to SQLite/PostgreSQL +- [ ] Sensor heartbeat tracking (mark online/offline) +- [ ] Integrate collector with Flask app lifecycle + +### P1 - High (Phase 3: Core API) +- [ ] `GET /api/v1/sensors` — list sensors +- [ ] `GET /api/v1/devices` — list devices (BLE + WiFi MACs) +- [ ] `GET /api/v1/alerts` — alert feed with pagination +- [ ] `GET /api/v1/probes` — probe requests +- [ ] `GET /api/v1/events` — sensor events +- [ ] `POST /api/v1/sensors//command` — send command to sensor + +### P2 - Normal (Phase 4: OSINT) +- [ ] MAC vendor lookup (OUI database) +- [ ] BLE company_id mapping +- [ ] `GET /api/v1/stats` — aggregate statistics +- [ ] Export endpoints (CSV, JSON) + +### P2 - Normal (Backlog from v1.x) - [ ] Tune presence threshold per room with real-world testing -- [x] Pi-side presence event handling in watch daemon -- [x] Firmware security audit (cppcheck + manual review — no issues found) - [ ] Power consumption measurements using POWERTEST + external meter - [ ] Test OTA rollback (flash bad firmware, verify auto-revert) -- [ ] Create HA webhook automations for deauth_flood / unknown_probe ### P3 - Low - [ ] Deep sleep mode with wake-on-CSI-motion @@ -19,6 +45,20 @@ - [ ] Document esp-radar console features - [ ] Pin mapping for ESP32-DevKitC V1 +## Completed: v1.9 - Multi-Channel Scanning & BLE Fingerprinting + +- [x] CHANSCAN command (ON/OFF/NOW/INTERVAL) +- [x] Channel hopping (1-13) with 100ms dwell, pause CSI during scan +- [x] chanscan= field in STATUS, chanscan_int NVS persistence +- [x] BLE fingerprinting: company_id, tx_power, adv_flags in BLE_DATA +- [x] Historical presence sessions support + +## Completed: v1.8 - HTTPS OTA Support + +- [x] Support HTTPS URLs for OTA updates (esp_https_ota) +- [x] CI: Upload firmware to Gitea releases for OTA +- [x] CI: Simplify deploy script to pure POSIX sh + ## Completed: v1.7 - Baseline Calibration & Presence Detection - [x] CALIBRATE command (capture N seconds of CSI, average per-subcarrier amplitudes) @@ -172,7 +212,7 @@ ## Notes - Adaptive threshold varies by environment; 0.001-0.01 is a good starting range -- NVS keys: `send_rate`, `tx_power`, `adaptive`, `threshold`, `ble_scan`, `target_ip`, `target_port`, `hostname`, `boot_count`, `csi_mode`, `hybrid_n`, `auth_secret`, `flood_thresh`, `flood_window`, `scan_rate`, `probe_rate`, `powersave`, `presence`, `pr_thresh`, `bl_nsub`, `bl_amps` +- NVS keys (24 total): `send_rate`, `tx_power`, `adaptive`, `threshold`, `ble_scan`, `target_ip`, `target_port`, `hostname`, `boot_count`, `csi_mode`, `hybrid_n`, `auth_secret`, `flood_thresh`, `flood_window`, `scan_rate`, `probe_rate`, `powersave`, `presence`, `pr_thresh`, `bl_nsub`, `bl_amps`, `chanscan`, `chanscan_int` - EVENT packets include sensor hostname: `EVENT,,motion=... rate=... wander=...` - ALERT_DATA format: `ALERT_DATA,,,,,` or `ALERT_DATA,,deauth_flood,,` - STATUS fields: `uptime=`, `uptime_s=`, `heap=`, `rssi=`, `channel=`, `tx_power=`, `rate=`, `csi_rate=`, `hostname=`, `version=`, `adaptive=`, `motion=`, `ble=`, `target=`, `temp=`, `csi_count=`, `boots=`, `rssi_min=`, `rssi_max=`, `csi_mode=`, `hybrid_n=`, `auth=`, `flood_thresh=`, `powersave=`, `presence=`, `pr_score=` diff --git a/TODO.md b/TODO.md index d3642fc..9fa8149 100644 --- a/TODO.md +++ b/TODO.md @@ -1,28 +1,79 @@ # ESP32 Hacking TODO +## Flask API (`~/git/esp32-web/`) + +### Architecture +- [ ] App factory pattern (`create_app()`) +- [ ] Blueprints: `api`, `collector` +- [ ] SQLAlchemy with migrations (Flask-Migrate) +- [ ] Background UDP collector (threading or Celery) +- [ ] Config from environment variables +- [ ] Port 5500: HTTP API (TCP) + UDP collector (UDP) on same port number + +### Database Schema +- [ ] `sensors` — id, hostname, ip, last_seen, status, config_json +- [ ] `devices` — mac, type (ble/wifi), vendor, first_seen, last_seen +- [ ] `sightings` — device_id, sensor_id, rssi, timestamp +- [ ] `alerts` — sensor_id, type, source_mac, target_mac, rssi, timestamp +- [ ] `probes` — device_id, sensor_id, ssid, rssi, channel, timestamp +- [ ] `events` — sensor_id, event_type, payload_json, timestamp + +### API Endpoints +- [ ] Sensors: list, detail, status, command, config, history +- [ ] Devices: list, detail, profile, sightings +- [ ] Alerts: list with filters (type, sensor, time range) +- [ ] Probes: list, group by SSID, group by MAC +- [ ] Events: list with filters +- [ ] Stats: counts, activity graphs data +- [ ] Export: CSV, JSON for devices/alerts/probes + +### UDP Collector +- [ ] Parse CSI_DATA (hostname, count, mac, rssi, features) +- [ ] Parse BLE_DATA (hostname, mac, rssi, type, name, company_id, tx_power, flags) +- [ ] Parse PROBE_DATA (hostname, mac, rssi, ssid, channel) +- [ ] Parse ALERT_DATA (hostname, type, source, target, rssi OR flood count) +- [ ] Parse EVENT (hostname, key=value pairs) +- [ ] Heartbeat timeout detection (mark sensor offline) + +### OSINT +- [ ] IEEE OUI database (download + parse) +- [ ] BLE company ID database (Bluetooth SIG) +- [ ] Device fingerprinting by BLE advertisement patterns +- [ ] Probe request SSID profiling (home networks, corporate, etc.) + ## Firmware -- [ ] On-device CSI processing (send metrics, not raw) + - [ ] Deep sleep mode with wake-on-CSI-motion - [ ] Battery-optimized duty cycling +- [ ] AP+STA config portal (captive portal for initial setup) ## Tools (esp-ctl) -- [ ] Multi-sensor BLE correlation (zone tracking by source sensor) + +- [ ] Migrate OSINT database to Flask API (esp-ctl becomes thin client) +- [ ] `esp-ctl api` subcommand (query Flask API) ## Testing + - [ ] Benchmark: CSI callback latency - [ ] Benchmark: UDP throughput at different rates - [ ] Power consumption measurements (per-mode: idle, CSI, BLE, probe) +- [ ] API load testing (concurrent requests) ## Documentation + +- [ ] Flask API: OpenAPI/Swagger spec +- [ ] Deployment guide (podman, systemd) - [ ] Pin mapping for ESP32-DevKitC V1 - [ ] Compare CSI quality: passive (router) vs active (ESP-NOW) - [ ] Multi-sensor deployment guide (placement, zones, triangulation) ## Ideas + - ESP-NOW mesh for direct ESP32-to-ESP32 CSI - External PIR sensor for CSI ground truth validation -- AP+STA config portal (captive portal for initial setup) -- Multi-channel scanning (hop across WiFi channels) -- RSSI triangulation with 3 sensors (approximate device location) -- BLE device fingerprinting (identify phone models by advertisement patterns) -- Historical presence logging (who was here, when, how long) +- RSSI triangulation with 3+ sensors (approximate device location) +- Home Assistant MQTT discovery integration +- Grafana dashboards for long-term analytics +- ML-based device classification (phone vs laptop vs IoT) +- Webhook callbacks for alerts (Slack, Discord, ntfy) +- Rate limiting and API authentication (JWT)