Enable compy.audio to load and play project files (as opposed to built-in sounds)

[dsent]

Users should be able to add their own assets to projects and use them in their programs.

[dsent]

A proposed solution (based on a conversation with @nagydani, made with the help of Codex so might miss some important things as I'm not that familiar with the codebase):

• Add Project:load_blob(name) in src/model/project/project.lua.

  • Reuse Project:readfile path rules.
  • Return love.filesystem.newFileData(bytes, name) or equivalent FileData.
  • Rationale: same project lookup magic as loadfile, but for non-Lua assets.

• Extend src/util/audio.lua from fixed built-in sounds to a small audio service.

  • Keep current built-in functions unchanged.
  • Add cache table keyed by project filename.
  • Add load(name, load_blob) or construct module with injected loader.
  • Rationale: audio module owns Source creation/caching; project layer owns file lookup.

• Wire project-aware audio in ConsoleController.

  • When building compy namespace, pass a closure that calls P.current:load_blob(name).
  • Expose compy.audio.load(name) and compy.audio.play(name).
  • Make play(name) equivalent to load(name)().
  • Rationale: matches existing dofile/loadfile controller mediation.

• Cache semantics.

  • First compy.audio.load("boing.ogg") loads and caches.
  • Later load("boing.ogg") reuses cached source, returning the same resource.
  • compy.audio.play("boing.ogg") works the same by extension.
  • Clear cache on project close/reset.
  • Rationale: same “load once, stop and replay” behavior as built-in sounds.

• Playback behavior.

  • Use love.audio.stop(source) before love.audio.play(source).
  • This means repeated calls restart the sound.
  • Overlapping playback of the same sound is not possible.
  • This is already true for built-in sounds, so the proposed project-sound API preserves current semantics.

• Error behavior.

  • Missing file: return/print the same style of project file error as readfile.
  • Invalid audio data: report LÖVE audio creation error through normal user error handling.

[nagydani]

I disagree with the above API proposal. The built-in sound effect library is there to allow developers that do not wish to deal with the complexity of the love.audio API add sound effects to their games with minimal effort and minimal side effects. One reason it does not have adverse side effects is that there are not too many such sound effects and their memory footprint is small.

The other option for developers to add sounds and music to their games should be to use the love.audio API directly, without abstracting any of it away. The only obstacle in doing so currently is that love.audio.newSource(filename, type) does not access the project folder and neither does love.filesystem.newFileData(filepath), at least with relative names without slashes. This needs to be either changed or equivalent functions need to be added as compy.audio.newSource(filename, type) and compy.filesystem.newFileData(filepath), though my preference would be for the API in the love namespace to work as expected.

[dsent]

@nagydani While I won't argue against your architecture decisions, I will argue for a very simple wrapper for this use case from the UX standpoint.

Adding their own assets to the games should be among the lowest-hanging fruits for children. Making their programs/games sound nice (or funny) should not be hard or complex. Creating a simple game with a few sound effects and background music should be accessible early.

While the internal implementation might be the one you laid out, I would strongly argue for compy.audio.load("file.ogg"), compy.audio.play("file.ogg"), and compy.audio.stop() with automatic caching in there.

If someone wants to do more advanced things, they can still use the full love.audio API directly. There are still quite obvious incentives to explore the full complexity, e.g. playing the same file over multiple channels with overlap, various sound mixing, reverberation and other parametric sound effects etc. (we can add an advanced example that records audio from the mic, and replays it with reverberation — just an idea).

I would also argue for adding a special case of using a second channel without full complexity: playing sounds in background in a way that is not interrupted by playing other sounds, with optional looping (N times or indefinitely). With separate volume control for both this second output and the default output (used by compy.audio.play()/compy.audio.toggle() and friends).

The obvious use case: background music in a game. More advanced use case pushing towards using full love.audio API: switching music tunes with fade-in/fade-out, more complex mixing etc. Suggested API surface: compy.audio.bgPlay("file.ogg", -1 /indefinite repeat/, 255 /volume/)/compy.audio.bgStop() — the same compy.audio.load()'s storage and cache internally, so if we prepared a file with compy.audio.load(), it can be quickly played from cache by both compy.audio.play() and compy.audio.bgPlay().

[dsent]

Mock-up code:

nice_jump = sfx.load("njump.ogg")
-- equivalent: sfx.play("njump.ogg", 75)
-- second parameter is volume, optional, max (100 or 255) by default
nice_jump.play(75)
-- standard sounds also get the volume parameter:
sfx.win(75)
sfx.ubit_giggle(75)

music = sfx.load("theme.ogg")
-- equivalent: sfx.bgPlay("theme.ogg", -1, 25)
-- second parameter is repeat; -1: indefinite, 0: play once, int N: play N times
-- third parameter is volume
music.bgPlay(-1, 25)

[dsent]

In a discussion we had with @nagydani we converged on the following.

Simplifying asset usage is a valid and important goal. However:

• We should think the API through much more thoroughly, because music and sfx is a tricky business with tons of caveats,
• With no access to the internet, kids' ability to find and add assets to their projects is limited anyway, so we would need to create an asset repository (and/or a simple way to upload assets to Compy from a phone/tablet/desktop),
• Therefore, making 3rd-party assets first-class would require substantial work on API, asset repository (probably a top-level IDE feature similar to the code editor), and integration.

This is definitely something we should strive to make soon. Possibly in time for the official big open release and sales start, but most probably not in time for the limited release in September.

[dsent]

Preliminary requirements for the high-level audio API:

• It should make it easy to use the basics: background music, sfx, chimes,
• It absolutely should not incentivize any kind of abuse (e.g. using sfx for percussion),
• It should not allow (in contrast with the low-level API) straining the limited resources such as RAM.