INPUT 01 · ALL UPLOADS
Visual evidence
The browser reads color, brightness, colored coverage, and local texture around the pointer. These values are available even when the file has no geographic metadata.
Label: visual featureListen feature redesign · 14 November 2026
See the pattern.
Hear the data.
Feel the change.
Upload an image or video, explore its visible pattern, and hear sound change as you move across it. When the visual has a known place and time, add measured ocean data. On supported phones, significant changes can also produce a short vibration; every device gets the same clear on-screen response.
Image + videoLive soundPhone haptics where supportedData provenance
REFERENCE VISUAL · USER SUPPLIED CROPPROBE → SOUND → FEEDBACK
01 / What the interface knows
Design around received information
The result card changes with the available evidence. Pixels, video motion, and API readings are separate types of information with different labels.
INPUT 02 · VIDEO
Time and motion
The current displayed frame and its change from earlier sampled frames can control a pulse or rhythm. The user sees a normal video player, without a frame strip.
Label: visual motionINPUT 03 · KNOWN PLACE + TIME
Measured marine values
A server request can add sea surface temperature and current speed or direction. Show units, UTC time, source, and the location used. The data is optional because uploads may lack those facts.
Label: provider valueScientific rule: a green line in an uploaded image is a green line. It becomes a temperature reading only when a verified legend or aligned source data supports that interpretation. The supplied crop resembles NASA SVS 3912, but its exact provenance and crop geometry remain unverified.
02 / User path
One continuous experience
API requests occur when a source and its context are chosen. Pointer movement works locally in the browser and does not spend an API query.
Upload image or video, or open a credited NASA example.
Decode the media into a small hidden canvas and calculate visual features.
For known coordinates and time, request matching marine values from the server.
Hover, drag, tap, or use keyboard arrows over the same media stage.
Sound, readout, visual pulse, and supported phone haptics respond together.
If an API response is unavailable, the visual instrument still works and the measured data area says “No matched data.”
03 / Layout directions
Choose the space for the data
These variations build on Hume AI, fal, and Felt references, adapted for media, live values, sound, and touch.
Instrument studio
Source at left, large media stage in the center, result and feedback at right. On phones, the result card follows the media immediately.
Best first build: the person can see which feature changed and which value controls the sound.
Full-width media
Media dominates the screen. A compact pointer label floats near the selected area while detailed results sit below.
Best for presentation mode and public demonstrations; detailed values take more scrolling.
Media plus timeline
The video and a synchronized feature plot sit together. Measurements and sound mappings form a separate inspection panel.
Best when the team adds deeper time-series analysis; it has more visual density.
A / DESKTOP COMPOSITION
SONIFYSPHERE / LISTENUPLOAD · ANALYZE · EXPLORE
VISUAL STAGE · MOVE TO PROBE
GREEN FLOW · SOUND 392 Hz▶ VIDEO TRANSPORT WHEN NEEDED
One image or continuous video. No extracted frame gallery.
A / PHONE COMPOSITION
LISTENSONIFYSPHERE
IMAGEVIDEONASA EXAMPLE
GREEN · 392 Hz▶ 00:12
VISUALGreen flow
MEASUREDNeeds place
INSTRUMENTGlass keys
FEEDBACKPulse
SOUND ONHAPTICS IF AVAILABLE
Touch holds the probe under the finger. The result stays visible immediately below the media.
04 / Interactive proof
Try the media probe
Use the supplied map or a local image or video. This small report demo reads pixels, lets you switch sound styles, and gives visual feedback. Its instruments use built-in browser oscillators as stand-ins for a future ElevenLabs sample library. It is not a scientific temperature reading.
MOVE · DRAG · OR USE ARROW KEYS05 / Technical path
From source to response
A single feature model drives all outputs. It includes the value, its provenance, units if any, confidence, location, and time; the screen never has to guess what it received.
| Stage | Data in | Processing | Data out / interface |
|---|---|---|---|
| Decode | Local image or video | Object URL, one media element, hidden downscaled canvas | Current pixel buffer |
| Probe | Pointer or touch position | Map through contained media bounds; average nearby pixels | Color band, coverage, brightness |
| Video time | Presented frame | requestVideoFrameCallback(); compare with prior sample | Local motion/change, media time |
| Optional marine | Known latitude, longitude, UTC time | Server request, validate JSON, cache by place/time | °C, m/s, degrees, source and time |
| Sound | Visual or measured feature | Clamp, normalize, map, smooth, schedule with Tone.js | Pitch, timbre, pulse, loudness |
| Feedback | Meaningful feature transition | Update readout and visual pulse; opt-in navigator.vibrate() when available | Same event understood on desktop and phone |
Key interaction decision: avoid a vibration on every pointer move or every video frame. Trigger a short, throttled pulse only for a meaningful transition, such as entering a strong current band. The sound can change continuously; haptics are discrete.
06 / Data and API contract
Use marine values when they match
The NASA visualization and Meteomatics API play different roles. A NASA key is for applicable api.nasa.gov endpoints; the existing GIBS imagery is a public visualization service.
NASA VISUAL SOURCE
Credited example
- Use a verified NASA SVS asset and preserve its title, credits, temporal range, and original URL.
- Use full uncropped map geometry or known crop metadata before mapping a pixel to latitude and longitude.
- Show visual features from the media. Do not label color as an exact scientific reading without a documented scale or aligned field.
SVS 3912 depicts ECCO2 model output from March 2007 to March 2008. A current 2026 API reading must be presented as a separate context, not as the animation's measured value.
METEOMATICS MARINE API
Optional matched measurement
- Ask for a location and UTC time when the source does not provide them.
- Request
t_sea_sfc:C,ocean_current_speed:ms, andocean_current_direction:dfrom a Next.js server route. - Return typed values and provenance to the browser, then reuse a cached result while the user explores.
The hackathon credentials are valid until 22 November 2026. Actual parameter and area entitlements remain unverified because this workspace could not reach the API.
GET https://api.meteomatics.com/<UTC-time>/t_sea_sfc:C,ocean_current_speed:ms,ocean_current_direction:d/<lat>,<lon>/json
Authorization: Basic <server-generated credentials>
Response shape: data[] → parameter → coordinates[] → dates[] → value
Quota design: hovering samples local pixels, never the API. A low resolution area grid could support spatial interpolation, but Meteomatics documents area requests as a separate entitlement. Until the hackathon account is confirmed to include it, use explicit point requests and caching.
07 / Sound and haptic rules
One event, several senses
These mappings are proposed interaction choices, not claims about the underlying physics. The readout must always show which feature controls the output.
| Feature | Sound response | Visual response | Haptic response |
|---|---|---|---|
| Visual color band | Bounded note or instrument color | Probe ring and labeled swatch | One short pulse when the band changes |
| Visible motion | Pulse rate or modulation | Small moving level bars | Pulse only at a meaningful threshold |
| Measured sea temperature | Quantized pitch within a fixed range | Number, °C, source and time | Optional single pulse on entering a new band |
| Measured current speed | Rhythm density, bounded for comfort | Number in m/s and direction marker | Optional sparse pulse pattern |
| No data / land / silence | Fade to quiet | “No matched value” or “quiet area” | No pulse |
PHONE
Vibration is an enhancement
navigator.vibrate() is available only in some browsers and devices. Offer an explicit haptics switch, use brief pulses, throttle them, and stop on pause or exit. The API does not promise precise intensity or continuous textures.
DESKTOP + UNSUPPORTED PHONES
Keep feedback legible
Use the same live label, probe ring, and a small pulse indicator. Respect reduced motion settings. A desktop web page generally has no device vibration, so do not imply that the screen can physically shake.
08 / Instrument and region profiles
One region, many voices
People can hear the same visual or measured pattern through different instruments. Changing the instrument changes the timbre and arrangement; the source values and their labels remain traceable.
Short luminous notes for colored flow paths. Color controls note; motion controls note spacing.
Soft continuous texture. Temperature or color drives pitch; current speed or visible motion drives movement.
Rounded taps that reveal local changes without filling every quiet area.
Low rhythmic layer for strong currents, with sparse haptic accents on supported phones.
SOUND ASSET PIPELINE
Generate variants ahead of play
Use ElevenLabs Sound Effects on the server to produce several short, loopable candidates from each instrument prompt. Let the team audition and select the best sounds. Save the chosen audio, prompt, model, and mapping version as a reusable sample pack. The API supports a loop option and 0.5–30 second durations. Generate again only when a user explicitly asks for a new variation.
The prototype selector above uses browser oscillators; it does not claim to play generated ElevenLabs audio yet.
OPTIONAL COMPOSED TRACK
Add a backing layer
ElevenLabs Music Compose can create an instrumental background from a prompt. Its Video to Music endpoint can create a track from a video, but that sends the user's video to the provider and produces a rendered track rather than a responsive hover instrument. Offer it only as an explicit, separate creation action.
Keep the live probe and scientific readout operating in the browser while a generated track plays underneath.
Region profile: store the source ID, optional region/coordinates and UTC time, visual features, measured values and provider, selected instrument, selected sample variant, sound mapping, and haptic setting. A person can save “North Atlantic · Ocean strings · Variant 2” and switch to Glass keys without changing the underlying data.
RegionSoundProfile = {
source: "NASA example or user upload", region: "North Atlantic",
evidence: { visualBand: "green", motion: "moderate", measured: null },
instrument: "ocean-strings", variant: "02",
mapping: { bandToPitch: true, motionToRhythm: true },
output: { sound: true, haptics: "opt-in" }
}
Key handling: an ElevenLabs API key belongs in a server environment variable and is sent to ElevenLabs in the xi-api-key header. Create a restricted key with a credit cap. An ElevenLabs key is configured in the local ignored environment file; its access is unverified because this workspace cannot reach the API. Never put it in browser code or this HTML file. ElevenLabs authentication
09 / Build sequence
Make the data contract real
The current Listen page uses NASA GIBS imagery and preset Tone.js notes. This redesign needs an analyzer, a selectable sound profile, and continuous controls before measured values can be meaningful in the interface.
SPRINT 01
Visual instrument
Build upload, image/video preview, hidden pixel canvas, pointer and touch probe, stable visual features, sound smoothing, visible result, and keyboard operation.
SPRINT 02
NASA example and provenance
Use a verified, credited source; identify map projection, crop, and animation time. Keep uploaded media clearly marked as user supplied.
SPRINT 03
Marine enrichment
Add server-side Meteomatics requests, validate entitlements and sample responses, cache results, then connect measured values to the same sound and feedback model.
SPRINT 04
Instrument library
Verify the locally configured ElevenLabs key and set its endpoint and credit limits, generate and audition several loopable variants per instrument, cache the selected assets, and let people save a region sound profile. Keep the browser sound engine responsive while the sample changes.
10 / Evidence
Research behind the decisions
Links are to primary provider and browser documentation, with Mobbin screens used only as interface precedents.