Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ StringField explores the sensation of playing an “imaginary string” stretche
| Path | What’s inside |
| --- | --- |
| `docs/` | Design briefs, play paradigms, sensing survey, assumption ledger, roadmap, **Touch-to-Ground Tuning Kit**, **Classroom Code Tour** |
| `docs/Sensors/` | Per‑sensor field notes: wiring, RC values, calibration rituals, expected gesture behavior |
| `firmware/` | PlatformIO project(s). Teensy 4.0 by default; optional ESP32 target |
| `software/` | Processing + p5.js visualizers and bridges (now with gesture debugger) |
| `hardware/` | Prototype notes per sensing modality + example BOM stubs |
Expand Down Expand Up @@ -69,11 +70,14 @@ Calibration micro-rituals (two minutes per class):

## Sensor stack expansions

- **Time-of-flight (ToF):** drop in a VL53L0X/TMF8801 board, feed its analog/filtered output to `A2`, and compile with `-D SENSOR_TOF`. The sensor class now clamps the floor for noisy rooms and keeps the pin-only setup notes inline; swap in a real I²C driver later without touching `GestureEngine`.
- **Piezo contact mic:** bias a piezo disc with a megaohm resistor, clamp the extremes, and plug into `A3` with `-D SENSOR_PIEZO`. The class can _see_ hits via the onboard LED and adjust the `bias` smoothing if the room hum drifts.
- **PIR motion:** run the PIR gate to a digital pin (`-D SENSOR_PIR`). We purposely wait out the 30s warm-up, then smooth the binary gate into a motion envelope so students can _see_ lingering activity instead of a jittery square wave.
- **Electret mic (analog):** a bias resistor + RC envelope into `A4` (`-D SENSOR_ELECTRET`). The class follows bias slowly, rectifies the swing, and exposes a gain knob so you can narrate why the whisper floor is clamped where it is.
- **I²S/PDM mic:** for SPH0645/ICS-43434/etc., compile with `-D SENSOR_I2S_MIC`. The stub reads short bursts off I²S, averages absolute amplitude, and outputs a mellow envelope so the rest of the firmware doesn’t care which mic showed up.
Need the hands‑on wiring + calibration notes? Start at `docs/Sensors/` and then go deep per sensor:

- **MaKey‑style touch‑to‑ground:** [field notes](docs/Sensors/MakeyTouch.md) for grounding, debounce, and “touch plateau” behavior.
- **Time‑of‑flight (ToF):** [field notes](docs/Sensors/TimeOfFlight.md). Drop in a VL53L0X/TMF8801 board, feed its analog/filtered output to `A2`, and compile with `-D SENSOR_TOF`. The sensor class clamps the floor for noisy rooms and keeps the pin-only setup notes inline; swap in a real I²C driver later without touching `GestureEngine`.
- **Piezo contact mic:** [field notes](docs/Sensors/Piezo.md). Bias a piezo disc with a megaohm resistor, clamp the extremes, and plug into `A3` with `-D SENSOR_PIEZO`. The class can _see_ hits via the onboard LED and adjust the `bias` smoothing if the room hum drifts.
- **PIR motion:** [field notes](docs/Sensors/PIR.md). Run the PIR gate to a digital pin (`-D SENSOR_PIR`). We purposely wait out the 30s warm-up, then smooth the binary gate into a motion envelope so students can _see_ lingering activity instead of a jittery square wave.
- **Electret mic (analog):** [field notes](docs/Sensors/ElectretMic.md). A bias resistor + RC envelope into `A4` (`-D SENSOR_ELECTRET`). The class follows bias slowly, rectifies the swing, and exposes a gain knob so you can narrate why the whisper floor is clamped where it is.
- **I²S/PDM mic:** [field notes](docs/Sensors/I2SMic.md). For SPH0645/ICS-43434/etc., compile with `-D SENSOR_I2S_MIC`. The stub reads short bursts off I²S, averages absolute amplitude, and outputs a mellow envelope so the rest of the firmware doesn’t care which mic showed up.

## Highlights

Expand Down
11 changes: 11 additions & 0 deletions docs/SensingSurvey.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,14 @@
| **Hall / Flex (later)** | A3144 hall + magnet, flex sensor | position/pressure proxy | alternate mappings | mechanical complexity | optional expansion |

> Implementation note: All sensor paths conform to a common `Sensor` interface so gesture & mapping code doesn’t change when you swap hardware.

## Sensor field notes (fast links)

Each of these is a short, hands‑on guide with wiring notes, RC recipes, calibration rituals, and “what the gestures should feel like.”

- [MaKey‑style Touch‑to‑Ground](Sensors/MakeyTouch.md)
- [Time‑of‑Flight Proximity](Sensors/TimeOfFlight.md)
- [Piezo / Contact Mic](Sensors/Piezo.md)
- [PIR Motion](Sensors/PIR.md)
- [Electret Mic (Analog)](Sensors/ElectretMic.md)
- [I²S Digital Mic](Sensors/I2SMic.md)
40 changes: 40 additions & 0 deletions docs/Sensors/ElectretMic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Electret Mic (Analog Envelope)

Electret mics are cheap and expressive. They catch **air motion**—scrapes, breath, finger flicks, and tiny percussive taps.

## Wiring notes

- **Firmware path:** `firmware/src/electret_mic_sensor.cpp` (`SENSOR_ELECTRET`)
- **Signal pin:** `A4`
- You need a **bias resistor** and usually a **rectifier/envelope** stage.
- Keep the wires short; this is an antenna.

**Sketch wiring (minimal):**

- Electret mic → bias resistor (2.2 kΩ–10 kΩ) to 3.3 V.
- Coupling cap (1–10 µF) into a diode + RC envelope (see below) → `A4`.
- GND to GND.

## Recommended RC values

- **Bias resistor:** **2.2 kΩ–10 kΩ** (higher = lower current).
- **Envelope:** **10 kΩ + 1 µF** (≈10 ms) for quick plucks.
- For slower swells, try **100 kΩ + 1 µF** (≈100 ms).

## Calibration steps (live, out loud)

1. **Idle bias check:** the normalized value should hover near 0.0 after the bias tracker settles.
2. **Gain tuning:** increase `gain_` until quiet room noise is visible but not pegged.
3. **Floor clamp:** if the room hums, raise the `env_` floor clamp so idle reads 0.

## Expected gesture behavior

- **Pluck:** snap or tap near the mic for quick spikes.
- **Scrape:** steady rubbing makes a noisy plateau—great for texture.
- **Bow:** breath or slow movement can create soft, airy sustains.

## Common failure modes

- **DC offset drift:** missing/incorrect bias resistor causes wandering baselines.
- **Hum city:** long wires pick up 50/60 Hz; twist or shield.
- **Perma‑clip:** gain too hot or envelope too fast so everything is 1.0.
41 changes: 41 additions & 0 deletions docs/Sensors/I2SMic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# I²S / PDM Digital Mic

This is the **cleanest audio path**: a digital mic that spits out PCM. It’s less fragile than electrets and gives you repeatable envelopes for gestures.

## Wiring notes

- **Firmware path:** `firmware/src/i2s_mic_sensor.cpp` (`SENSOR_I2S_MIC`)
- **Interfaces:** I²S (WS/LRCLK, BCLK, DATA) with power/ground.
- Common parts: **SPH0645**, **ICS‑43434**, **INMP441**.

**Sketch wiring (generic):**

- Mic `WS/LRCLK` → board I²S WS pin
- Mic `BCLK` → board I²S BCLK pin
- Mic `DATA` → board I²S data in
- Mic `VDD` → 3.3 V
- Mic `GND` → GND

(Exact pin mapping depends on your board; follow the board’s I²S pinout.)

## Recommended RC values

- **No RC needed.** This is digital; all shaping happens in firmware.

## Calibration steps (live, out loud)

1. **Confirm I²S starts:** if `ready_` stays false, check wiring and clock pins.
2. **Gain tuning:** adjust `gain_` in `i2s_mic_sensor.cpp` so quiet taps are visible.
3. **Window size:** tweak `window_samples_` for faster or smoother envelopes.

## Expected gesture behavior

- **Pluck:** sharp peaks from taps or snaps.
- **Scrape:** textured noise bands; works well for “scratch bow” metaphors.
- **Bow:** smoother swells if you lengthen the envelope window.

## Common failure modes

- **No clock lock:** wrong WS/BCLK pins = total silence.
- **Word‑select swapped:** sounds like garbage noise (values jump wildly).
- **Sample‑rate mismatch:** aliasy hiss or sluggish response.
38 changes: 38 additions & 0 deletions docs/Sensors/MakeyTouch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# MaKey‑style Touch‑to‑Ground Sensor

This is the punk‑rock “clip it and play it” path: a single alligator lead, a person, and a shared ground. It’s the fastest way to get a class **playing** while you narrate signal paths out loud.

## Wiring notes

- **Firmware path:** `firmware/src/makey_sensor.cpp` (`SENSOR_MAKEY`)
- **Signal pin:** `D2` (`kMakeyPin`)
- **Ground:** share ground between the board and whatever the player is touching.
- Enable `INPUT_PULLUP` so the pin idles **HIGH** until touched.

**Sketch wiring:**

- Board `D2` → MaKey (or foil pad) signal lead.
- Board `GND` → player ground (wrist strap, foil ground plane, or another alligator).

## Recommended RC values

- **No RC needed** for the baseline demo. The firmware handles debouncing with an integrator.
- If your wire run is huge or you’re in a noisy room, add a **100 kΩ** series resistor and **10–100 nF** to ground near the input as a hardware low‑pass.

## Calibration steps (live, out loud)

1. **Confirm idle = HIGH:** open serial plotter; idle should sit near `0.0` after normalization.
2. **Tap cadence:** touch the pad; the envelope should rise to `~1.0` and fall without chatter.
3. **Debounce tuning:** if it flickers on jittery hands, slow the integrator in `makey_sensor.cpp`.

## Expected gesture behavior

- **Pluck:** clean spikes from tap‑release.
- **Bow:** sustained touch reads as a flat plateau (great for “long bow” metaphors).
- **Scrape:** quick repeated taps become rhythmic chatter—use that as a teachable sonic texture.

## Common failure modes

- **No shared ground:** always HIGH, never triggers. Ground is the instrument.
- **Long wires:** antenna behavior → phantom touches. Twist or shorten leads.
- **Static zap:** will look like a sudden pluck. Teach “touch the ground plane first.”
39 changes: 39 additions & 0 deletions docs/Sensors/PIR.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# PIR Motion Sensor

PIR is the “big‑gesture” detector: it doesn’t see fingers, it sees **body heat motion**. Great for dramatic bows, stage‑wide gesturing, and “presence” cues.

## Wiring notes

- **Firmware path:** `firmware/src/pir_sensor.cpp` (`SENSOR_PIR`)
- **Signal pin:** `D4` (`kPirPin`)
- Most PIR modules output a **digital gate** (HIGH/LOW).
- Many PIRs are **3.3 V only**—check the label before powering.

**Sketch wiring:**

- PIR OUT → `D4`
- PIR VCC → 3.3 V (or 5 V if explicitly supported)
- PIR GND → board GND

## Recommended RC values

- **No RC needed.** The PIR board has its own analog front‑end.
- If the digital gate chatters, add a **10 kΩ + 100 nF** RC on the output, or just increase the software guard time.

## Calibration steps (live, out loud)

1. **Warm‑up ritual:** wait 10–30 seconds; the code holds output at 0 until stable.
2. **Sensitivity trim:** adjust the PIR’s onboard pot until a slow walk triggers but idle does not.
3. **Guard window:** if it retriggers too fast, increase `guard_us_` in `pir_sensor.cpp`.

## Expected gesture behavior

- **Bow:** broad, slow arm sweeps become a smooth envelope.
- **Pluck:** not ideal—PIR doesn’t do fine taps.
- **Scrape:** use it for “stage presence” rather than micro‑gesture.

## Common failure modes

- **Always HIGH:** wrong voltage or sensor aimed at a hot vent/window.
- **Always LOW:** too short a warm‑up or sensitivity set too low.
- **False triggers:** moving heat sources (sunbeams, heaters) or flimsy mounts.
40 changes: 40 additions & 0 deletions docs/Sensors/Piezo.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Piezo / Contact Mic Sensor

This path is pure impact energy: taps, knocks, and scrape transients. It’s a **percussive string** when you want hits to be obvious and dramatic.

## Wiring notes

- **Firmware path:** `firmware/src/piezo_sensor.cpp` (`SENSOR_PIEZO`)
- **Signal pin:** `A3`
- **Bias:** a megaohm resistor to mid‑rail keeps the ADC sane.
- **Clamp:** diodes (or a small TVS) keep the piezo from spiking the input.

**Sketch wiring:**

- Piezo disk lead → `A3` through a **1 MΩ** resistor to 3.3 V (or 5 V if your ADC allows).
- Piezo disk other lead → GND.
- Optional: two **1N4148** diodes to 3.3 V and GND to clamp the swing.

## Recommended RC values

- **Bias resistor:** **1 MΩ** (470 kΩ–2.2 MΩ range is fine).
- **Envelope cap:** **10–47 nF** from `A3` to GND for a bit of smoothing.
- If you need slower decay (longer “bow”), try **100 nF**.

## Calibration steps (live, out loud)

1. **Idle bias check:** with no touch, the normalized value should sit around 0.0.
2. **Tap test:** a finger tap should spike near 1.0 and decay quickly.
3. **Gain control:** tune `swing` scaling in `piezo_sensor.cpp` until quiet taps are readable but don’t clip everything.

## Expected gesture behavior

- **Pluck:** crisp, high peaks—great for sharp string attacks.
- **Scrape:** repeated small spikes create a rough, noisy texture.
- **Bow:** harder to get a smooth sustain unless you add RC smoothing or mount in a resonant surface.

## Common failure modes

- **Floating input:** no bias resistor = chaos noise.
- **Over‑clipping:** missing clamps can slam the ADC rails.
- **Dead disk:** cracked piezo = silence. They’re fragile; mount with love.
38 changes: 38 additions & 0 deletions docs/Sensors/TimeOfFlight.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Time‑of‑Flight (ToF) Proximity Sensor

Use this when you want **distance** as a gestural axis: bow speed from range change, vibrato from micro‑wobbles, “hover pluck” from a quick dip in range.

## Wiring notes

- **Firmware path:** `firmware/src/time_of_flight_sensor.cpp` (`SENSOR_TOF`)
- **Demo pin:** `A2` expects an **analog envelope** for the in‑class build.
- **Real ToF parts:** VL53L0X / VL53L1X / TMF8801 are I²C. If you move to I²C, keep the output normalized to 0..1 so the gesture code stays unchanged.

**Sketch wiring (analog helper board):**

- ToF analog out → `A2`
- ToF VCC → 3.3 V (most are 2.8–3.3 V)
- ToF GND → board GND

## Recommended RC values

- **If using an analog helper:** start with **10 kΩ + 1 µF** (≈10 ms) for a gentle envelope.
- **If using I²C directly:** no RC needed; do filtering in code.

## Calibration steps (live, out loud)

1. **Set a “home” distance:** hold a hand at ~15 cm and watch the normalized value.
2. **Scale:** adjust the raw‑to‑normalized mapping in `time_of_flight_sensor.cpp` (or the helper board gain) until that pose lands at ~0.5.
3. **Floor clamp:** if idle never returns to 0.0, lift the clamp threshold slightly.

## Expected gesture behavior

- **Bow:** slow in/out motion yields a long arc in the envelope.
- **Pluck:** quick dive toward the sensor creates a sharp peak.
- **Vibrato:** tiny wobbles around a fixed distance give a delicate tremble.

## Common failure modes

- **Ambient IR saturation:** sunlight can pin the reading high.
- **Bad angle:** some ToF parts have narrow cones; side‑on hands vanish.
- **Power mismatch:** a 3.3 V sensor on 5 V can latch or die. Don’t.
7 changes: 6 additions & 1 deletion firmware/src/electret_mic_sensor.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,19 @@ class ElectretMicSensor : public Sensor {

SensorSample read() override {
uint32_t now = micros();
// Expected signal range: biased mic envelope on A4, sampled 0..1023. You want
// mid-rail idle (around 0.5 normalized) so the swing has room both ways.
float x = sample_raw();
// AC coupling: follow bias slowly, measure swing fast.
bias_ = 0.9994f * bias_ + 0.0006f * x; // tweak live if the room hums
float swing = fabs(x - bias_) * gain_;
// Normalization: clamp the rectified swing into 0..1, then smooth to a
// performance-friendly envelope.
env_ = 0.88f * env_ + 0.12f * constrain(swing, 0.0f, 1.0f);
// Small floor clamp to avoid whisper-noise. Bump lower if you need pianissimo.
if (env_ < 0.01f) env_ = 0.0f;
// Common failure modes: missing bias resistor (floating pin noise),
// long unshielded leads (50/60 Hz hum), or gain too high (perma-clipped 1.0).
return {env_, now};
}

Expand All @@ -45,4 +51,3 @@ Sensor& get_electret_sensor() {
}

#endif // SENSOR_ELECTRET

6 changes: 5 additions & 1 deletion firmware/src/i2s_mic_sensor.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ class I2SMicSensor : public Sensor {
int16_t sample = 0;
uint16_t count = 0;
while (I2S.available() && count < window_samples_) {
// Expected signal range: signed 16-bit PCM samples (about -32768..32767).
sample = static_cast<int16_t>(I2S.read());
total += abs(sample);
++count;
Expand All @@ -31,7 +32,11 @@ class I2SMicSensor : public Sensor {
float avg = (count > 0) ? (total / static_cast<float>(count)) : 0.0f;
// Normalize 16-bit PCM to 0..1 and overdrive slightly for quiet rooms.
float normalized = constrain((avg / 32768.0f) * gain_, 0.0f, 1.0f);
// Normalization: absolute-value average -> 0..1 energy envelope with a short
// smoothing filter so percussive taps still show up as peaks.
env_ = 0.9f * env_ + 0.1f * normalized;
// Common failure modes: I2S.begin() never locks (ready_ false => silence),
// word-select swapped (garbled noise), or sample_rate_ mismatch (aliasy hiss).
return {env_, now};
}

Expand All @@ -51,4 +56,3 @@ Sensor& get_i2s_mic_sensor() {
}

#endif // SENSOR_I2S_MIC

7 changes: 6 additions & 1 deletion firmware/src/makey_sensor.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,16 @@ class MakeySensorGuarded : public Sensor {
if (now - last_read_us_ < guard_us_) return last_sample_;
last_read_us_ = now;

// Expected signal range: digital gate with pullup enabled, so HIGH ~= 1 (open)
// and LOW ~= 0 (touch shorts to ground).
bool touch = digitalRead(kMakeyPin) == LOW; // MaKey shorts to ground
// Debounce using a tiny integrator so hand jitter does not look like a bow.
debounce_state_ = 0.9f * debounce_state_ + 0.1f * (touch ? 1.0f : 0.0f);
// Normalization: convert the debounced gate into a 0..1 envelope the rest of
// the gesture engine can treat like any other sensor.
float normalized = debounce_state_;
// Common failure modes: no shared ground (always HIGH), too-long leads acting
// like antennas (phantom touches), or ESD spikes that look like micro taps.

last_sample_ = {normalized, now};
return last_sample_;
Expand All @@ -35,4 +41,3 @@ Sensor& get_makey_sensor() {
}

#endif // SENSOR_MAKEY

7 changes: 6 additions & 1 deletion firmware/src/piezo_sensor.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,18 @@ class PiezoSensor : public Sensor {

SensorSample read() override {
static float bias = 0.5f; // mid-rail estimate in normalized units
// Expected signal range: biased piezo swing around mid-rail, sampled as 0..1023.
// If raw slams 0/1023, check your clamp diodes and that the piezo isn't floating.
int raw = analogRead(A3);
// Normalization: scale ADC to 0..1, subtract a slow bias, then magnify the swing
// into a 0..1 envelope that behaves like "hit energy."
float x = constrain(raw / 1023.0f, 0.0f, 1.0f);
bias = 0.999f * bias + 0.001f * x; // slow bias tracker; tweak in calibration session
float swing = fabs(x - bias) * 2.2f; // exaggerate small hits; clip later
float env = constrain(swing, 0.0f, 1.0f);
digitalWrite(13, env > 0.4f); // punk-rock peak lamp
// Common failure modes: cracked piezo disks (no response), mechanical mounting
// that damps transients, or missing bias resistor so the ADC sees a floating pin.
return {env, micros()};
}
};
Expand All @@ -29,4 +35,3 @@ Sensor& get_piezo_sensor() {
}

#endif // SENSOR_PIEZO

Loading