~/shanegraffiti.com/research/self-aware-body

Shane Graffiti Inc. AI Research Division 2026

THE SELF
AWARE
BODY.

Movement sonification turning how you walk into real-time sound has strong experimental backing for stroke rehabilitation, but almost no clinical adoption. This framework explains why, and fixes it. The core problem is a paradox: the sounds that faithfully encode movement data are perceptually sterile and motivationally dead, while the sounds rich enough to sustain weeks of rehabilitation training introduce their own internal variation that drowns out the movement signal. Solving that paradox requires calibration, not just better code and calibration requires physiotherapists and patients in the loop from day one, not at the end.

Authors Kantan, Dahl, Spaich

Affiliation Aalborg University

Published arXiv 2606.14664 Jun 2026

Core Mechanism Noise Profile Calibration

§ 2.0 The Signal-Noise Paradox

Every sound model has its own internal variation stochastic fluctuations in a water sound, rhythmic evolution in generative music that exists regardless of what the patient's body is doing. The framework names this the "noise profile." The paradox is structurally unavoidable: injecting meaning and injecting noise are two sides of the same coin.

Noise-Minimal Pole

Sine Wave
Encoding

One movement variable mapped to one acoustic dimension pitch rises as the leg swings faster. Informationally pure: no autonomous variation. Consistently proven perceptually sterile, cognitively fatiguing, and motivationally inert in user testing.

⇄ calibrate between

Noise-Rich Pole

Generative
Music

Rich temporal/spectral variation sustains engagement over weeks of training. But the model's autonomous evolution can mask the movement-driven signal entirely during subtle movements, destroying clinical legibility.

The design task, restated viable region = expressive enough to sustain engagement AND transparent enough for movement-sound correspondence to remain perceptible // neither pole works the viable region lies between them, reached only through empirical calibration with real movement data from the actual target population

§ 3.2 Mix-N-Map: The Mixing Console Analogy

The platform borrows structural logic from professional audio mixing: a finite set of meaningful decisions, each with a dedicated control, consistent across every design session. Instead of editing code and recompiling to hear the result, a physiotherapist and researcher can adjust how a movement dimension is handled and immediately hear the effect while the patient is walking.

Sensitivity Control

How much does the sound respond to a given movement? Low sensitivity: only large vigorous swings produce audible changes. High sensitivity: subtle kinematic differences register. Adjusted live, one-handed, during a session.

Smoothing

Should rapid fluctuations be averaged out so feedback reflects overall movement quality rather than reacting to each momentary sensor reading? Corresponds to a mixer's low-pass filtering same knob, different domain.

Nonlinear Scaling

How should the relationship between movement amplitude and sound change be shaped to remain informative across the patient's full movement range? Clinically impaired movement is lower in amplitude than healthy-user testing approximates.

Attack / Decay

How quickly does the sound build and fade in response to a movement event? Controls whether the feedback feels continuous or event-based a design choice that must match the specific therapeutic goal of the session.

Reproducible State

The state of the interface at any moment constitutes a complete, reproducible design snapshot. It can be saved, shared with collaborators, and used as the basis for a written description no code inspection needed.

Real-Time Live Data

All parameters are modifiable while the sound is running on live sensor data. This converts the conventional slow loop (modify code → test offline → evaluate → repeat) into the rapid iterative cycle the FASTER development framework demands.

§ 4.0 Three-Phase Development Framework

Adapted from healthcare intervention science specifically the FASTER framework for technology-based rehabilitation interventions. Each phase feeds the next; skipping any one of them produces designs that fail in ways the prior phases could not have detected.

P1

Requirement-Gathering with clinicians + patients

Establish the non-negotiable design constraints before implementation begins. What setup time is acceptable in a clinical session? What terminology does the physiotherapist already understand? What movement patterns must the algorithm handle? No prototype exists yet this phase defines the boundaries of what counts as success.

P2

Real-User Interaction Testing validated interaction design

The sonic interaction itself which movement variables map to which sounds, with what mappings is validated with actual patients in real clinical conditions. Proxy testing by the design team or with healthy volunteers does not replicate the variability of impaired movement, attentional load, or clinical judgment about therapeutic relevance.

P3

Interface Evaluation validated practitioner interface

Shifts focus from the patient's experience to the physiotherapist's workflow. Can a clinician with no technical background operate this system independently, one-handed, while simultaneously supporting a patient who needs physical assistance? Think-aloud testing and feasibility studies reveal failure modes invisible to any prior phase.

§ 5.0 HearWalk The Full Arc

An ABF system for hemiparetic stroke patients. Wearable inertial sensors on four limb locations (left/right thigh, left/right shank) feed angular velocity data to a real-time sonification engine. The output: ecological wading sounds whose energetic properties reflect instantaneous limb swing velocity a metaphor stroke patients grasped intuitively, without instruction, by analogy with walking through water.

15

Hemiparetic patients in Phase 1 kinematic baseline

5

Specialist physio focus group for Phase 2 algorithm selection

9

Hemiparetic patients in Phase 2 feasibility

7

Hemiparetic patients in Phase 3 clinical feasibility

What worked

Ecological Wading

Patients immediately and without instruction described equalizing the sound between their two sides and timing steps to produce rhythmic patterns spontaneous motor strategies the therapy was aiming to produce. One patient's therapist observed the best walking quality they had ever witnessed from that patient.

What failed the noise profile in practice

Urination Variant

The negative-reinforcement sound shared spectral content with the wading texture it was embedded in. During low-velocity movement (exactly when it mattered most), it had no perceptually clear foreground to emerge from. Therapists couldn't detect when patients were moving poorly. Classic noise-profile failure.

§ 6.0 Where AI Helps, Where It Doesn't

The paper's final section is unusually direct about this not as a general AI-limitations disclaimer but as a stage-by-stage map of where the assistance is real and where it breaks down in clinical sonification design specifically.

Ideation Useful

LLMs can surface candidate mapping strategies and sonic metaphors faster than manual literature search. But they have no embodied experience of sound perception and may reproduce plausible-sounding assumptions about what patients with neurological impairment can actually attend to.

Sketching Useful With Caveats

LLMs can translate informal descriptions into structured mapping specifications and suggest scaling functions. But without explicit clinical grounding, AI sketches may be internally coherent but not fit for the target context.

Prototyping Useful But Risky

LLMs compress the time from concept to auditable prototype. However, 5–22% of generated code contains hallucinated packages or API calls, and in specialized real-time audio domains that rate is higher. A subtle audio processing bug could produce feedback that systematically misrepresents movement without being detectable by surface inspection.

Deployment Supportive But Subordinate

AI can assist with documentation, configuration interfaces, and code refactoring. But clinical software is subject to regulatory requirements (medical device legislation) that AI tools are not equipped to satisfy autonomously all AI-generated code must be audited by qualified developers.

HEAR YOUR
MOVEMENT.
CHANGE YOUR
MOVEMENT.

THE SELF AWARE BODY.

THE SELF
AWARE
BODY.