diff --git a/PROJECT.md b/PROJECT.md new file mode 100644 index 0000000..f693c05 --- /dev/null +++ b/PROJECT.md @@ -0,0 +1,58 @@ +# ESP32-Web Project + +## Overview + +REST API backend for ESP32 CSI sensor fleet. Collects and stores sensor data (BLE, WiFi probes, alerts, events) for OPSEC/OSINT/Purple team operations. + +## Goals + +- Aggregate data from multiple ESP32 sensors +- Provide REST API for querying devices, alerts, events +- Track device presence and movement across zones +- Enable fleet management (commands, OTA, calibration) + +## Architecture + +``` +ESP32 Sensors (UDP 5500) ──▶ UDP Collector ──▶ SQLite/PostgreSQL + │ + ▼ + Flask REST API (TCP 5500) + │ + ▼ + Clients / Dashboards +``` + +## Components + +| Component | Path | Description | +|-----------|------|-------------| +| App Factory | `src/esp32_web/__init__.py` | Flask application factory | +| Config | `src/esp32_web/config.py` | Configuration classes | +| Models | `src/esp32_web/models/` | SQLAlchemy ORM models | +| API | `src/esp32_web/api/` | REST API blueprints | +| Collector | `src/esp32_web/collector/` | UDP listener + parsers | + +## Data Models + +| Model | Description | +|-------|-------------| +| Sensor | ESP32 sensor node (hostname, IP, status) | +| Device | Discovered BLE/WiFi device (MAC, vendor, name) | +| Sighting | Device seen by sensor (RSSI, timestamp) | +| Alert | Security alert (deauth, flood) | +| Event | Sensor event (motion, presence, calibration) | +| Probe | WiFi probe request (SSID, channel) | + +## Dependencies + +- Flask 3.x +- SQLAlchemy 2.x +- Flask-Migrate +- Flask-CORS +- gunicorn + +## Related Projects + +- [esp32-hacking](~/git/esp32-hacking/) - ESP32 firmware +- [esp-tools](~/git/esp-tools/) - CLI tools (esp-ctl, esp-fleet) diff --git a/README.md b/README.md index d813a16..436e1e4 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,10 @@ REST API backend for ESP32 sensor fleet (OPSEC/OSINT/Purple team). ## Quick Start ```bash -# Install +# Clone and install +cd ~/git/esp32-web +python3 -m venv .venv +source .venv/bin/activate pip install -e ".[dev]" # Initialize database @@ -13,11 +16,40 @@ flask --app src/esp32_web db init flask --app src/esp32_web db migrate -m "initial" flask --app src/esp32_web db upgrade -# Run development server -make dev +# Start server +make start +make status ``` -## Ports +## Server Management + +```bash +make start # Start server (0.0.0.0:5500) +make stop # Stop server +make restart # Restart server +make status # Show PID + uptime +make logs # Tail log file +``` + +## API Endpoints + +Base URL: `http://localhost:5500` + +| Method | Endpoint | Description | +|--------|----------|-------------| +| GET | `/health` | Health check with uptime | +| GET | `/api/v1/sensors` | List sensors | +| GET | `/api/v1/sensors/` | Get sensor | +| POST | `/api/v1/sensors//command` | Send command | +| GET | `/api/v1/devices` | List devices | +| GET | `/api/v1/devices/` | Get device + sightings | +| GET | `/api/v1/alerts` | List alerts | +| GET | `/api/v1/events` | List events | +| GET | `/api/v1/probes` | List probe requests | +| GET | `/api/v1/probes/ssids` | List SSIDs with counts | +| GET | `/api/v1/stats` | Aggregate statistics | + +## Network Ports | Port | Protocol | Description | |------|----------|-------------| @@ -25,33 +57,12 @@ make dev | 5500 | UDP | Sensor data collector | | 5501 | UDP | Sensor commands (outbound) | -## API Endpoints +## Documentation -| Method | Endpoint | Description | -|--------|----------|-------------| -| GET | `/health` | Health check | -| GET | `/api/v1/sensors` | List sensors | -| GET | `/api/v1/sensors/` | Get sensor | -| POST | `/api/v1/sensors//command` | Send command | -| GET | `/api/v1/devices` | List devices | -| GET | `/api/v1/devices/` | Get device | -| GET | `/api/v1/alerts` | List alerts | -| GET | `/api/v1/events` | List events | -| GET | `/api/v1/probes` | List probe requests | -| GET | `/api/v1/probes/ssids` | List SSIDs | -| GET | `/api/v1/stats` | Statistics | +- [Installation](docs/INSTALL.md) +- [Usage Guide](docs/USAGE.md) +- [Cheatsheet](docs/CHEATSHEET.md) -## Container +## License -```bash -make build -make run -make logs -make stop -``` - -## Testing - -```bash -make test -``` +MIT diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..b609b14 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,54 @@ +# ESP32-Web Roadmap + +## v0.1.0 - Project Scaffold [DONE] + +- [x] Flask app factory with blueprints +- [x] SQLAlchemy models (Sensor, Device, Sighting, Alert, Event, Probe) +- [x] UDP collector for sensor data streams +- [x] Core API endpoints +- [x] pytest setup +- [x] Containerfile + Makefile + +## v0.1.1 - Server Management [DONE] + +- [x] start/stop/restart/status via Makefile +- [x] Health endpoint with uptime +- [x] Database migrations (Flask-Migrate) +- [x] Listen on all interfaces + +## v0.2.0 - OSINT Features + +- [ ] MAC vendor lookup (IEEE OUI database) +- [ ] BLE company_id to manufacturer mapping +- [ ] Device profile enrichment +- [ ] Export endpoints (CSV, JSON) + +## v0.3.0 - Fleet Management + +- [ ] Sensor config endpoint (GET/PUT) +- [ ] OTA trigger endpoint +- [ ] Calibration trigger endpoint +- [ ] Sensor history/metrics + +## v0.4.0 - Zones & Presence + +- [ ] Zone management (assign sensors to areas) +- [ ] Device zone tracking +- [ ] Dwell time analysis +- [ ] Presence history + +## v1.0.0 - Production Ready + +- [ ] Authentication (API keys or JWT) +- [ ] Rate limiting +- [ ] PostgreSQL support +- [ ] Systemd service file +- [ ] Production deployment guide + +## Future + +- WebSocket for real-time updates +- Web dashboard (htmx + Pico CSS) +- Home Assistant integration +- Grafana dashboards +- Webhook callbacks for alerts diff --git a/TASKS.md b/TASKS.md new file mode 100644 index 0000000..df32173 --- /dev/null +++ b/TASKS.md @@ -0,0 +1,46 @@ +# ESP32-Web Tasks + +**Last Updated:** 2026-02-05 + +## Current Sprint: v0.2.0 — OSINT Features + +### P1 - High +- [ ] Download and parse IEEE OUI database +- [ ] MAC vendor lookup utility +- [ ] BLE company_id mapping +- [ ] `GET /api/v1/devices//profile` enriched endpoint + +### P2 - Normal +- [ ] Export endpoints (`/api/v1/export/devices.csv`) +- [ ] Add vendor field population on device creation +- [ ] Sensor heartbeat timeout detection + +### P3 - Low +- [ ] Add pagination to all list endpoints +- [ ] Add OpenAPI/Swagger spec +- [ ] Add request logging middleware + +## Completed: v0.1.1 - Server Management + +- [x] Makefile start/stop/restart/status commands +- [x] Health endpoint with uptime tracking +- [x] CLI module (esp32-web command) +- [x] Initial database migration +- [x] Listen on all interfaces (0.0.0.0:5500) + +## Completed: v0.1.0 - Project Scaffold + +- [x] Flask app factory pattern +- [x] SQLAlchemy 2.x models +- [x] API blueprint with endpoints +- [x] UDP collector with parsers +- [x] pytest fixtures and tests +- [x] Containerfile for podman +- [x] Makefile for common tasks + +## Notes + +- Sensors send data to UDP 5500 +- API listens on TCP 5500 +- Commands sent to sensors on UDP 5501 +- SQLite for dev, PostgreSQL for prod diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..c1d3746 --- /dev/null +++ b/TODO.md @@ -0,0 +1,57 @@ +# ESP32-Web TODO + +## API + +- [ ] Pagination for all list endpoints +- [ ] Filter by date range +- [ ] Sort options +- [ ] OpenAPI/Swagger spec generation +- [ ] Rate limiting (flask-limiter) +- [ ] API authentication (JWT or API keys) + +## OSINT + +- [ ] IEEE OUI database download script +- [ ] MAC vendor lookup on device creation +- [ ] BLE company ID database +- [ ] Device fingerprinting by advertisement patterns +- [ ] SSID categorization (home, corporate, mobile hotspot) + +## Collector + +- [ ] Heartbeat timeout (mark sensor offline) +- [ ] CSI data storage (optional, high volume) +- [ ] Data retention policy (auto-cleanup old records) +- [ ] Metrics collection (packets/sec, errors) + +## Fleet Management + +- [ ] Sensor config read/write +- [ ] OTA orchestration +- [ ] Calibration management +- [ ] Bulk commands + +## Deployment + +- [ ] Systemd service file +- [ ] PostgreSQL configuration +- [ ] Nginx reverse proxy config +- [ ] TLS setup guide +- [ ] Backup/restore scripts + +## Testing + +- [ ] API endpoint tests (all routes) +- [ ] Collector parser tests (edge cases) +- [ ] Integration tests with mock sensors +- [ ] Load testing + +## Ideas + +- WebSocket for live updates +- SSE (Server-Sent Events) alternative +- Webhook callbacks for alerts +- Slack/Discord notifications +- Home Assistant MQTT integration +- Grafana dashboard templates +- ML device classification diff --git a/docs/CHEATSHEET.md b/docs/CHEATSHEET.md new file mode 100644 index 0000000..4764333 --- /dev/null +++ b/docs/CHEATSHEET.md @@ -0,0 +1,97 @@ +# ESP32-Web Cheatsheet + +## Server Management + +```bash +make start # Start server +make stop # Stop server +make restart # Restart server +make status # Show PID + uptime +make logs # Tail logs +make dev # Run with debug mode +``` + +## Database + +```bash +make migrate # Apply migrations +make migrate-init # Initialize migrations +make migrate-create msg="description" # Create migration +``` + +## Testing + +```bash +make test # Run all tests +pytest -v # Verbose output +pytest -k test_sensors # Run specific tests +``` + +## Container + +```bash +make build # Build image +make container-run # Run container +make container-stop # Stop container +make container-logs # View logs +``` + +## API Quick Reference + +```bash +# Health +curl localhost:5500/health + +# Sensors +curl localhost:5500/api/v1/sensors +curl localhost:5500/api/v1/sensors/HOSTNAME +curl -X POST localhost:5500/api/v1/sensors/HOSTNAME/command -H "Content-Type: application/json" -d '{"command":"STATUS"}' + +# Devices +curl localhost:5500/api/v1/devices +curl "localhost:5500/api/v1/devices?type=ble&limit=50" +curl localhost:5500/api/v1/devices/MAC + +# Alerts +curl localhost:5500/api/v1/alerts +curl "localhost:5500/api/v1/alerts?type=deauth&hours=48" + +# Events +curl localhost:5500/api/v1/events +curl "localhost:5500/api/v1/events?type=presence" + +# Probes +curl localhost:5500/api/v1/probes +curl localhost:5500/api/v1/probes/ssids + +# Stats +curl localhost:5500/api/v1/stats +``` + +## Query Parameters + +| Parameter | Endpoints | Description | +|-----------|-----------|-------------| +| type | devices, alerts, events | Filter by type | +| hours | alerts, events, probes | Time window (default: 24) | +| limit | devices, alerts, events, probes | Max results (default: 100) | +| offset | devices, alerts, events, probes | Skip N results | +| ssid | probes | Filter by SSID | +| sensor_id | alerts, events | Filter by sensor | + +## Files + +| File | Description | +|------|-------------| +| `/tmp/esp32-web.pid` | Server PID file | +| `/tmp/esp32-web.log` | Server log file | +| `instance/esp32.db` | SQLite database | +| `.env` | Environment config | + +## Ports + +| Port | Protocol | Description | +|------|----------|-------------| +| 5500 | TCP | HTTP API | +| 5500 | UDP | Sensor data | +| 5501 | UDP | Sensor commands | diff --git a/docs/INSTALL.md b/docs/INSTALL.md new file mode 100644 index 0000000..0092795 --- /dev/null +++ b/docs/INSTALL.md @@ -0,0 +1,76 @@ +# Installation + +## Requirements + +- Python 3.11+ +- pip +- SQLite (included) or PostgreSQL + +## Development Setup + +```bash +# Clone repository +cd ~/git/esp32-web + +# Create virtual environment +python3 -m venv .venv +source .venv/bin/activate + +# Install with dev dependencies +pip install -e ".[dev]" + +# Initialize database +flask --app src/esp32_web db init +flask --app src/esp32_web db migrate -m "initial" +flask --app src/esp32_web db upgrade + +# Verify installation +make test +``` + +## Configuration + +Copy `.env.example` to `.env` and adjust: + +```bash +cp .env.example .env +``` + +| Variable | Default | Description | +|----------|---------|-------------| +| SECRET_KEY | dev-key | Flask secret key | +| DATABASE_URL | sqlite:///esp32.db | Database connection | +| UDP_PORT | 5500 | UDP collector port | +| CMD_PORT | 5501 | Sensor command port | +| SENSOR_TIMEOUT | 60 | Seconds before sensor marked offline | + +## Container Deployment + +```bash +# Build image +make build + +# Run container +make container-run + +# View logs +make container-logs + +# Stop container +make container-stop +``` + +## Production + +For production, use gunicorn: + +```bash +gunicorn -b 0.0.0.0:5500 -w 4 'esp32_web:create_app()' +``` + +With PostgreSQL: + +```bash +export DATABASE_URL="postgresql://user:pass@localhost/esp32web" +flask --app src/esp32_web db upgrade +``` diff --git a/docs/USAGE.md b/docs/USAGE.md new file mode 100644 index 0000000..74a74be --- /dev/null +++ b/docs/USAGE.md @@ -0,0 +1,141 @@ +# Usage Guide + +## Starting the Server + +```bash +# Activate virtual environment +cd ~/git/esp32-web +source .venv/bin/activate + +# Start server +make start + +# Check status +make status + +# View logs +make logs +``` + +## API Examples + +### Health Check + +```bash +curl http://localhost:5500/health +``` + +Response: +```json +{"status": "ok", "uptime": "5m23s", "uptime_seconds": 323} +``` + +### List Sensors + +```bash +curl http://localhost:5500/api/v1/sensors +``` + +Response: +```json +{ + "sensors": [ + {"id": 1, "hostname": "hollow-acorn", "ip": "192.168.129.31", "status": "online", "last_seen": "..."}, + {"id": 2, "hostname": "muddy-storm", "ip": "192.168.129.29", "status": "online", "last_seen": "..."} + ] +} +``` + +### Get Sensor Details + +```bash +curl http://localhost:5500/api/v1/sensors/hollow-acorn +``` + +### Send Command to Sensor + +```bash +curl -X POST http://localhost:5500/api/v1/sensors/hollow-acorn/command \ + -H "Content-Type: application/json" \ + -d '{"command": "STATUS"}' +``` + +### List Devices + +```bash +# All devices +curl http://localhost:5500/api/v1/devices + +# BLE devices only +curl "http://localhost:5500/api/v1/devices?type=ble" + +# With pagination +curl "http://localhost:5500/api/v1/devices?limit=50&offset=0" +``` + +### Get Device Details + +```bash +curl http://localhost:5500/api/v1/devices/aa:bb:cc:dd:ee:ff +``` + +### List Alerts + +```bash +# Last 24 hours (default) +curl http://localhost:5500/api/v1/alerts + +# Last 7 days +curl "http://localhost:5500/api/v1/alerts?hours=168" + +# Filter by type +curl "http://localhost:5500/api/v1/alerts?type=deauth" +``` + +### List Probe Requests + +```bash +# All probes +curl http://localhost:5500/api/v1/probes + +# Filter by SSID +curl "http://localhost:5500/api/v1/probes?ssid=MyNetwork" + +# List SSIDs with counts +curl http://localhost:5500/api/v1/probes/ssids +``` + +### Get Statistics + +```bash +curl http://localhost:5500/api/v1/stats +``` + +Response: +```json +{ + "sensors": {"total": 3, "online": 3}, + "devices": {"total": 42, "ble": 35, "wifi": 7}, + "alerts": {"count": 5, "hours": 24}, + "events": {"count": 128, "hours": 24}, + "probes": {"count": 89, "hours": 24} +} +``` + +## Allowed Sensor Commands + +Commands that can be sent via the API: + +| Command | Description | +|---------|-------------| +| STATUS | Get sensor status | +| REBOOT | Reboot sensor | +| IDENTIFY | Flash LED for 5 seconds | +| BLE ON/OFF | Toggle BLE scanning | +| ADAPTIVE ON/OFF | Toggle adaptive sampling | +| RATE <10-100> | Set CSI rate | +| POWER <2-20> | Set TX power (dBm) | +| CSIMODE RAW/COMPACT/HYBRID | Set CSI output mode | +| PRESENCE ON/OFF | Toggle presence detection | +| CALIBRATE [seconds] | Start baseline calibration | +| CHANSCAN ON/OFF/NOW | Channel scanning |