f87ddec742
Plan for REST API backend (purple team/OPSEC/OSINT): - Phase 1: Project scaffold, SQLAlchemy models, podman - Phase 2: UDP collector for sensor data streams - Phase 3: Core API endpoints (sensors, devices, alerts, probes, events) - Phase 4: OSINT features (OUI lookup, BLE company ID mapping) - Phase 5: Fleet management API Port allocation: - TCP 5500: HTTP REST API - UDP 5500: Sensor data collector - UDP 5501: Sensor commands (outbound) Also documents completed v1.8 (HTTPS OTA) and v1.9 (multi-channel scanning, BLE fingerprinting) milestones.
224 lines
11 KiB
Markdown
224 lines
11 KiB
Markdown
# ESP32 Hacking Tasks
|
|
|
|
**Last Updated:** 2026-02-05
|
|
|
|
## Current Sprint: v2.0 — Flask API Backend
|
|
|
|
### P0 - Critical (Phase 1: Project Setup)
|
|
- [ ] Create project scaffold `~/git/esp32-web/`
|
|
- [ ] Flask app factory pattern with Blueprints
|
|
- [ ] HTTP API on TCP 5500, UDP collector on UDP 5500
|
|
- [ ] SQLAlchemy models: Sensor, Device, Sighting, Alert, Event, Probe
|
|
- [ ] Containerfile for podman
|
|
- [ ] Makefile (build, run, dev, stop, logs)
|
|
- [ ] Basic pytest setup
|
|
|
|
### P1 - High (Phase 2: UDP Collector)
|
|
- [ ] UDP listener thread (parse CSI_DATA, BLE_DATA, PROBE_DATA, ALERT_DATA, EVENT)
|
|
- [ ] Store parsed data to SQLite/PostgreSQL
|
|
- [ ] Sensor heartbeat tracking (mark online/offline)
|
|
- [ ] Integrate collector with Flask app lifecycle
|
|
|
|
### P1 - High (Phase 3: Core API)
|
|
- [ ] `GET /api/v1/sensors` — list sensors
|
|
- [ ] `GET /api/v1/devices` — list devices (BLE + WiFi MACs)
|
|
- [ ] `GET /api/v1/alerts` — alert feed with pagination
|
|
- [ ] `GET /api/v1/probes` — probe requests
|
|
- [ ] `GET /api/v1/events` — sensor events
|
|
- [ ] `POST /api/v1/sensors/<id>/command` — send command to sensor
|
|
|
|
### P2 - Normal (Phase 4: OSINT)
|
|
- [ ] MAC vendor lookup (OUI database)
|
|
- [ ] BLE company_id mapping
|
|
- [ ] `GET /api/v1/stats` — aggregate statistics
|
|
- [ ] Export endpoints (CSV, JSON)
|
|
|
|
### P2 - Normal (Backlog from v1.x)
|
|
- [ ] Tune presence threshold per room with real-world testing
|
|
- [ ] Power consumption measurements using POWERTEST + external meter
|
|
- [ ] Test OTA rollback (flash bad firmware, verify auto-revert)
|
|
|
|
### P3 - Low
|
|
- [ ] Deep sleep mode with wake-on-CSI-motion
|
|
- [ ] Battery-optimized duty cycling
|
|
- [ ] Document esp-crab dual-antenna capabilities
|
|
- [ ] Document esp-radar console features
|
|
- [ ] Pin mapping for ESP32-DevKitC V1
|
|
|
|
## Completed: v1.9 - Multi-Channel Scanning & BLE Fingerprinting
|
|
|
|
- [x] CHANSCAN command (ON/OFF/NOW/INTERVAL)
|
|
- [x] Channel hopping (1-13) with 100ms dwell, pause CSI during scan
|
|
- [x] chanscan= field in STATUS, chanscan_int NVS persistence
|
|
- [x] BLE fingerprinting: company_id, tx_power, adv_flags in BLE_DATA
|
|
- [x] Historical presence sessions support
|
|
|
|
## Completed: v1.8 - HTTPS OTA Support
|
|
|
|
- [x] Support HTTPS URLs for OTA updates (esp_https_ota)
|
|
- [x] CI: Upload firmware to Gitea releases for OTA
|
|
- [x] CI: Simplify deploy script to pure POSIX sh
|
|
|
|
## Completed: v1.7 - Baseline Calibration & Presence Detection
|
|
|
|
- [x] CALIBRATE command (capture N seconds of CSI, average per-subcarrier amplitudes)
|
|
- [x] CALIBRATE STATUS / CALIBRATE CLEAR subcommands
|
|
- [x] Presence scoring (normalized Euclidean distance vs baseline, rolling window of 50)
|
|
- [x] PRESENCE ON/OFF command (NVS persisted, requires valid baseline)
|
|
- [x] PRESENCE THRESHOLD command (0.001-1.0, NVS persisted, default 0.05)
|
|
- [x] Presence events (`EVENT,<hostname>,presence=<0|1> score=<float>`) with 10s holdoff
|
|
- [x] Calibration done event (`EVENT,<hostname>,calibrate=done packets=<n> nsub=<n>`)
|
|
- [x] presence= and pr_score= fields in STATUS reply
|
|
- [x] NVS persistence: bl_amps (blob), bl_nsub (i8), presence (i8), pr_thresh (i32)
|
|
- [x] config_save_blob / config_erase_key NVS helpers
|
|
- [x] n_sub field in csi_features_t, amps_out parameter in csi_extract_features
|
|
- [x] Pi-side: parse presence/calibrate events in watch daemon
|
|
- [x] Pi-side: store in sensor_events table (`esp-ctl osint events`)
|
|
- [x] Pi-side: HA webhooks for presence_change and calibrate_done
|
|
- [x] Fleet OTA to v1.7, calibrated hollow-acorn + amber-maple
|
|
|
|
## Completed: v1.6 - Power Management
|
|
|
|
- [x] ESP-IDF power management framework (DFS 240/80 MHz + light sleep)
|
|
- [x] sdkconfig: CONFIG_PM_ENABLE, CONFIG_FREERTOS_USE_TICKLESS_IDLE
|
|
- [x] POWERSAVE command (WiFi modem sleep toggle, NVS persisted, default off)
|
|
- [x] powersave= field in STATUS reply
|
|
- [x] POWERTEST save/restore of powersave state
|
|
- [x] Deployed to fleet, NVS persistence verified
|
|
|
|
## Completed: v1.5 - Event Handling & NVS Persistence
|
|
|
|
- [x] EVENT packet parsing in watch daemon (motion, wifi_reconnect, powertest)
|
|
- [x] Sensor heartbeat tracking (offline/online detection, configurable timeout)
|
|
- [x] sensor_events table with indexed queries (`esp-ctl osint events`)
|
|
- [x] HA webhooks for sensor_offline, sensor_online, motion_change
|
|
- [x] NVS persistence for SCANRATE and PROBERATE commands
|
|
- [x] Sensor events count in `esp-ctl osint stats`
|
|
|
|
## Completed: v1.4 - Multi-Sensor & Validation
|
|
|
|
- [x] Multi-sensor BLE correlation in esp-ctl (zone tracking by source sensor)
|
|
- [x] Zone tracking with EMA RSSI (`esp-ctl osint zones`, `device_zones` table)
|
|
- [x] Per-sensor breakdown in MAC profile (`esp-ctl osint mac`)
|
|
- [x] POWERTEST command (7-phase power profiling with EVENT markers)
|
|
- [x] Parallel OTA fleet updates (`esp-fleet ota --parallel`)
|
|
|
|
## Completed: v1.3 - Security & OSINT
|
|
|
|
- [x] HMAC command authentication (firmware + esp-ctl/esp-cmd/esp-fleet/esp-ota)
|
|
- [x] AUTH command (set/query/disable secret, NVS persisted)
|
|
- [x] auth=on/off in STATUS
|
|
- [x] Deauth flood detection (ring buffer, aggregate ALERT_DATA)
|
|
- [x] FLOODTHRESH command (count + window, NVS persisted)
|
|
- [x] flood_thresh field in STATUS
|
|
- [x] MAC OUI vendor lookup (`esp-ctl oui`, IEEE CSV database)
|
|
- [x] OSINT SQLite database (probe_ssids, device_sightings tables)
|
|
- [x] Watch daemon (`esp-ctl watch` — listen + enrich + store)
|
|
- [x] OSINT query CLI (`esp-ctl osint probes/devices/mac/stats`)
|
|
- [x] Home Assistant webhook integration (deauth_flood, unknown_probe, unknown_ble)
|
|
- [x] Watch config file (`~/.config/esp-ctl/watch.yaml`)
|
|
|
|
## Completed: v1.2
|
|
|
|
- [x] On-device CSI feature extraction (amp_rms, amp_std, amp_max, amp_max_idx, energy)
|
|
- [x] CSIMODE command: RAW, COMPACT, HYBRID N (NVS persisted)
|
|
- [x] Compact payload format `"F:rms,std,max,idx,energy"` (~80% bandwidth reduction)
|
|
- [x] Hybrid mode: compact every packet, raw every Nth
|
|
- [x] STATUS fields: csi_mode, hybrid_n
|
|
- [x] Adaptive sampling reuses extracted energy (no duplicate computation in COMPACT/HYBRID)
|
|
|
|
## Completed: v1.1
|
|
|
|
- [x] Sensor ID in data packets (hostname prefix on CSI_DATA, BLE_DATA, EVENT)
|
|
- [x] Deauth/disassoc frame detection (ALERT_DATA via promiscuous mode)
|
|
- [x] Chip temperature reporting in STATUS reply
|
|
- [x] BLE alerting (`esp-ctl ble --known`, alert on unknown MACs)
|
|
- [x] BLE dwell time tracking (dwell column in `--track` summary)
|
|
- [x] Timestamped event logging (`esp-ctl listen --timestamp`)
|
|
- [x] Alert filter in esp-ctl (`listen -f alert`)
|
|
- [x] Runtime HOSTNAME command (NVS persisted, mDNS updated)
|
|
- [x] WiFi probe request capture (PROBE_DATA via promiscuous mode, 10s dedup)
|
|
- [x] mDNS service advertisement (`_esp-csi._udp`)
|
|
- [x] mDNS sensor discovery (`esp-ctl discover`)
|
|
- [x] Probe filter in esp-ctl (`listen -f probe`)
|
|
- [x] OTA fleet to same firmware (ca526ef)
|
|
- [x] CSI packet counter in STATUS (`csi_count=`)
|
|
- [x] Raw uptime in STATUS (`uptime_s=`)
|
|
- [x] WiFi reconnect EVENT emission
|
|
- [x] SCANRATE command (BLE scan interval tuning, 5-300s)
|
|
- [x] PROBERATE command (probe dedup cooldown tuning, 1-300s)
|
|
- [x] Fix: promiscuous mode disables CSI on original ESP32 — guarded with `#if`
|
|
- [x] FreeRTOS CPU runtime stats in PROFILE (trace facility enabled)
|
|
- [x] Increased cmd_task stack 4KB → 6KB (was 516 bytes free)
|
|
- [x] WiFi channel in STATUS (`channel=`)
|
|
- [x] Boot counter in NVS (`boots=`)
|
|
- [x] RSSI min/max tracking from CSI frames (`rssi_min=`, `rssi_max=`)
|
|
- [x] Actual CSI rate in STATUS (`csi_rate=`)
|
|
|
|
## Completed: v0.5 - BLE Scanning
|
|
|
|
- [x] Enable Bluetooth alongside WiFi (NimBLE, BLE ON/OFF command)
|
|
- [x] Periodic BLE advertisement scanning
|
|
- [x] Report device MAC, RSSI, name via UDP
|
|
- [x] Pi-side BLE device tracking (`esp-ctl ble --track`)
|
|
- [x] PROFILE command (heap, stack watermarks, CPU runtime stats)
|
|
- [x] TARGET command (runtime UDP destination config)
|
|
|
|
## Completed: v0.4 - Adaptive Sampling
|
|
|
|
- [x] On-device CSI wander calculation (coefficient of variation)
|
|
- [x] Adaptive rate: 10 pkt/s idle (3s holdoff) → 100 pkt/s on motion
|
|
- [x] EVENT notification to Pi on rate change
|
|
- [x] ADAPTIVE ON/OFF command (NVS persisted)
|
|
- [x] THRESHOLD command for tuning sensitivity (NVS persisted)
|
|
- [x] RATE command disables adaptive mode
|
|
- [x] adaptive/motion fields in STATUS reply
|
|
- [x] OTA deployed and verified on amber-maple
|
|
|
|
## Completed: v0.3 - OTA Updates
|
|
|
|
- [x] Dual OTA partition table (`partitions.csv`)
|
|
- [x] 4MB flash, custom partitions, rollback in sdkconfig.defaults
|
|
- [x] Firmware: OTA command, ota_task, LED_OTA, rollback validation
|
|
- [x] Firmware: version in STATUS reply
|
|
- [x] Pi-side `esp-ota` tool (HTTP server + OTA orchestration)
|
|
- [x] `esp-fleet ota` subcommand (sequential fleet update)
|
|
- [x] Build and USB-flash amber-maple (partition table change)
|
|
- [x] End-to-end OTA test verified
|
|
- [x] Regenerate sdkconfig.sample
|
|
- [x] Update CHEATSHEET.md, USAGE.md
|
|
|
|
## Completed: v0.2 - Remote Management
|
|
|
|
- [x] Firmware: UDP command listener (port 5501)
|
|
- [x] Firmware: LED status indicator (GPIO2)
|
|
- [x] Firmware: NVS config persistence (rate, tx_power)
|
|
- [x] Firmware: REBOOT, IDENTIFY, STATUS commands
|
|
- [x] Firmware: RATE command (10-100 Hz, restarts ping)
|
|
- [x] Firmware: POWER command (2-20 dBm)
|
|
- [x] Pi-side: `esp-cmd` CLI tool
|
|
- [x] Pi-side: `esp-fleet` fleet management tool
|
|
- [x] mDNS hostname, watchdog, human-readable uptime
|
|
|
|
## Completed: v0.1 - Documentation
|
|
|
|
- [x] Copy firmware sources to project
|
|
- [x] Document current firmware and settings
|
|
- [x] Document build & flash workflow
|
|
- [x] Create .gitignore for build artifacts
|
|
- [x] Test building firmware from this repo
|
|
- [x] Document CSI config options
|
|
|
|
## Notes
|
|
|
|
- Adaptive threshold varies by environment; 0.001-0.01 is a good starting range
|
|
- NVS keys (24 total): `send_rate`, `tx_power`, `adaptive`, `threshold`, `ble_scan`, `target_ip`, `target_port`, `hostname`, `boot_count`, `csi_mode`, `hybrid_n`, `auth_secret`, `flood_thresh`, `flood_window`, `scan_rate`, `probe_rate`, `powersave`, `presence`, `pr_thresh`, `bl_nsub`, `bl_amps`, `chanscan`, `chanscan_int`
|
|
- EVENT packets include sensor hostname: `EVENT,<hostname>,motion=... rate=... wander=...`
|
|
- ALERT_DATA format: `ALERT_DATA,<hostname>,<deauth|disassoc>,<sender_mac>,<target_mac>,<rssi>` or `ALERT_DATA,<hostname>,deauth_flood,<count>,<window_s>`
|
|
- STATUS fields: `uptime=`, `uptime_s=`, `heap=`, `rssi=`, `channel=`, `tx_power=`, `rate=`, `csi_rate=`, `hostname=`, `version=`, `adaptive=`, `motion=`, `ble=`, `target=`, `temp=`, `csi_count=`, `boots=`, `rssi_min=`, `rssi_max=`, `csi_mode=`, `hybrid_n=`, `auth=`, `flood_thresh=`, `powersave=`, `presence=`, `pr_score=`
|
|
- PROBE_DATA format: `PROBE_DATA,<hostname>,<mac>,<rssi>,<ssid>`
|
|
- Probe requests deduped per MAC (default 10s cooldown, tunable via PROBERATE)
|
|
- mDNS service: `_esp-csi._udp` on data port (for sensor discovery)
|
|
- HOSTNAME command: `HOSTNAME <name>` sets NVS + mDNS, `HOSTNAME` queries current
|
|
- **ESP32 limitation:** Promiscuous mode (deauth/probe detection) disabled — breaks CSI. Works on ESP32-C6+.
|