Audio processing and mixing

Trace asynchronous sound commands, decoder policy, spatial mixing, bus rebuilds, and resampled PCM streams.

Asterune transfers world sound state to an audio worker through commands. The worker owns the Kira audio manager, voice handles, and mixer buses. World simulation updates playback intent and spatial state; the audio backend consumes samples on its own clock.

This separation creates several distinct stages: a script changes a sound, the client synchronizes that state, the worker prepares a source, and the backend consumes its samples. A world update is not an acknowledgement that the corresponding sample has reached the output device.

State synchronization and voice generationsCopy link

The client tracks playback, pause, stop, and effect generations for each sound. A changed playback generation starts a voice. A changed effect signature also restarts a playing voice because per-voice processing can require a different prepared sample buffer. Other updates change gain, pan, and playback rate on an existing handle.

A start replaces any earlier voice with the same instance ID. Removing a sound from the live instance set sends a stop and forgets its generations. This keeps a retained audio handle from continuing after its world object disappears. The worker removes stopped handles during sound updates.

The audio worker receives commands on a channel rather than borrowing the world while the backend mixes. Startup waits for device initialization, with a 2 s acknowledgement timeout. If initialization does not produce an available output, the engine retains no command sender. The world can continue advancing sound state without an output device.

The audio scheduling clock measures elapsed monotonic time from audio-engine creation. A scheduled start becomes a positive delay relative to that clock when the worker processes the play command. Expired or nonfinite delays do not postpone playback; future delays are capped at 3,600 s. This clock differs from a simulation tick and from a count of frames delivered by the audio device.

Decoder policy and source preparationCopy link

File playback chooses its preparation path from file size. Files smaller than 512 KiB decode into an in-memory sample buffer. Files at or above that threshold retain a path and metadata for a streaming decoder. Byte-backed sources decode into memory regardless of their byte count.

The streaming path uses Symphonia to read packets and decode frames. Opening a stream scans the selected track's packet frame counts and bounds the advertised length against the observed length. The decoder then opens the file again for playback. Streaming reduces retained decoded samples; it does not imply that opening the file avoids an initial packet scan.

Seeking can retain a packet pre-roll so codecs with inter-packet state can reconstruct the requested position. The sequential fallback reopens the file, walks packet timestamps, and retains approximately 1 s before the target. It skips earlier packets without decoding the complete preceding recording.

Per-voice effects can change the preparation policy. If a file needs any sample-buffer effect other than pitch shift, the worker decodes the file into memory even when its size normally selects streaming. Pitch shift changes playback rate and can retain the streaming decoder. The source's size alone therefore does not determine playback memory use.

A streaming decoder error is collected from its handle during subsequent updates. The worker logs the error and stops that voice. Preparation and play errors similarly prevent the new voice from entering the active handle table.

Spatial gain, panning, and DopplerCopy link

Spatialization starts from the vector from listener to emitter. The dot product of its normalized direction with the listener's right vector supplies pan, clamped from -1 to 1. Gain depends on distance and the selected rolloff mode. A sound without an emitter position keeps its base gain, centered pan, and unmodified spatial rate.

Distance attenuation is 1 up to the minimum distance and 0 at or beyond the maximum distance. Between them, let t be the normalized distance through that interval. Linear rolloff uses 1 - t; squared rolloff uses (1 - t)². The inverse path uses (minimum_distance / distance)^rolloff_scale.

Base gain multiplies sound volume by the applicable group gain, each bounded from 0 to 10. A voice routed into a real bus does not also multiply that bus's volume into its per-voice gain. The bus applies its own gain after receiving its voices. Output volume and the sound service's master volume act later on the main mix.

Doppler uses emitter velocity projected toward the listener. The engine converts that velocity with the distance factor, then computes a rate from a speed of sound of 343 m per second. The resulting Doppler multiplier is bounded from 0.5 to 2 before multiplying the sound's playback rate. This calculation uses emitter velocity; it does not subtract a separate listener-velocity term.

The custom generated-stream path applies equal-power channel gains. For pan p, its angle is (p + 1) × pi / 4; left and right gains are the cosine and sine of that angle, multiplied by the square root of 2. Static and file voices pass their pan value to Kira instead.

Voice effects and live bus processingCopy link

Per-voice digital signal processing (DSP) transforms a complete decoded sample buffer before playback. Enabled effects execute in priority order. Each voice receives its own processed buffer, so a source reused by several voices can produce different processed samples.

The effects use different state and sampling operations:

Buffer processing explains why changing an effect can restart a voice instead of modifying a continuously running filter. Pitch shift changes both duration and pitch through rate; it is not a time-preserving spectral pitch shifter.

Return buses use a separate live-processing path. The return-track builder installs delay, distortion, and compression effects on the mixed signal. Those filters operate after the contributing voices have combined. The main track also owns the ambient reverb effect. Per-voice buffer processing and live bus processing therefore have different memory, update, and ordering behavior.

Mixer topology and rebuildsCopy link

Mixer groups describe parent buses, sends to return buses, and ducking sources. The worker creates send targets first, then ordinary tracks in parent-before-child order. A missing parent routes a track to the main mix. If the remaining parent chains form a cycle, the ordering pass removes their parent links and places them at the top level rather than looping during graph construction.

The worker compares topology separately from volume. Its structural signature includes IDs, parent links, sends, ducking, and effects, but normalizes volume. If only volumes change, the worker updates existing tracks. A structural change rebuilds the graph, stops voices routed into the old graph, and clears its buses. The client forgets the corresponding playback generations so playing sounds can restart against the replacement graph.

Ducking uses a source bus's peak envelope. The meter responds immediately to a higher peak and decays with a 0.25 s release constant. The destination computes target_gain = 1 - amount × min(4 × source_level, 1), reaching full ducking strength at one quarter of full scale.

Gain approaches that target using 1 - exp(-dt / time_constant), with separate attack and release constants. The minimum constant is 0.001 s. The meter shares its latest level through atomic state; ducking does not request a synchronous world update for every audio frame.

Generated PCM and resamplingCopy link

Generated audio enters as interleaved pulse-code modulation (PCM) samples. The mixer converts 1-channel or 2-channel input into stereo frames. Mono input copies its value to both channels. Sample values are bounded to the audio range, and generated streams replace nonfinite input with zero.

A generated stream has a mutex-protected frame queue between producer and consumer. The client moves pending script samples into that queue during sound synchronization. Its capacity is 10 s at the stream's sample rate. A push past the capacity drops the oldest frames, preserving the newest queued audio.

The consumer maintains a fractional source position. For each output frame, it advances by output_dt × source_sample_rate, consumes source frames whenever the phase crosses 1, and linearly interpolates between neighboring source frames. This reconciles source and output rates without requiring one simulation update per audio sample.

Each request for a source frame from an empty queue supplies zero and increments the starvation counter. Output still interpolates preceding and next source frames, so queue exhaustion does not force every output sample to zero immediately. The consumer counts successfully consumed source frames separately. Those counters describe source consumption, not exact device presentation time. The client reports them back during its next synchronization.

Scheduled streams emit silence until their delay expires without consuming the queued source. Their custom playback controls provide gain, pan, and stop; the ordinary file voice's rate and sample-buffer DSP path do not run inside this stream consumer. A pause command stops the custom stream control rather than retaining a Kira file handle in a paused state.

For the exposed operations and properties, see Sound, SoundGroup, and SoundService.