Case 010 Audio / DSP

Audio Analysis Toolkit

01 — Problem

Music-reactive applications need dependable beat and energy data from arbitrary tracks, but full transcription of mixed polyphonic audio is beyond what FFT peak-picking can honestly deliver. Most hobby tools blur that line and ship unreliable note data.

02 — Fix

A pipeline that pulls audio from a YouTube URL, decodes it to PCM and runs real-time-synced FFT band analysis, beat detection and approximate note/key detection — with each signal explicitly graded by reliability so downstream consumers know what to trust. A companion in-browser synthesiser covers the generation side: keyboard, theremin, drum sequencer, piano roll, loop recording and FX.

Exhibit 010-A WebSynth with its drum kit, piano, step sequencer and FX rack open at once over the theremin pad — the generation side of the toolkit.

Exhibit 010-B Draggable instrument panels — a multi-octave piano over the drum step-sequencer, with the QWERTY row below doubling as a playable keyboard.

Exhibit 010-C The audio analyzer front end: paste a YouTube URL to decode the track and extract frequency bands, beats and an approximate key.

03 — Proof

Band analysis: 6 frequency bands, 4096-sample FFT at 50% overlap (~10.7 Hz resolution)
Beat detection: rolling-average energy spikes, 1.5× threshold, 200 ms minimum gap
Key detection: chroma histogram matched against 24 major/minor keys
Honest limits: FFT note detection documented as approximate, with an ML (BasicPitch) upgrade path specified
Validation loop: detected notes resynthesised to audio for by-ear quality checks
WebSynth: 11 JS modules — keyboard, theremin, drum sequencer, piano roll, loops, FX, recording

04 — Method

.NET Blazor Server (YoutubeExplode + NAudio for the pipeline, requestAnimationFrame-synced canvas visualisers) and a Web Audio API synth, with analysis design and the limits write-up produced through agent research sessions.