JavaScript Multimedia

[web 2.0] 
SoundManager 2 for JavaScript is a neat little library to fill in one of the blank areas of JavaScript/DOM development: decent sound output. It is very cool that it's there, but it's a shame that it has to use Flash to do the playback because browser sound playback is so dismal.

JavaScript is proving more and more that it can compete full force with Flash. The 2D graphics capabilities the dojo guys have built with dojo.gfx (cross browser Canvas stuff) and dojo-svg and the likes already look quite mature.

What has been missing up until now are native in browser sound support and video support. SoundManager provides sound support but its dependency on Flash is not preferrable.

We need a native Sound object which provides play, pause and seek support in arbitrary sound files.


I'm thinking about something like this:

// Need to figure out which sound formats would be supported
// Ideally a basic set of formats (wav, mid, etc.) MUST be supported
// but the object should have a backend capable of playing anything: mov, ra etc.
var s = new Sound('sounds/haha.mp3', true);

// do we play onload or do we play streaming?
// what happens on network congestion?;

// Get the length (in millis?) and allow seeking in the file (also during playback?)


Just Googled a bit and it looks like WhatWG is already speccing something along these lines in Web Applications 1.0: Sound

They deferr streaming audio to the object element which might be reasonable for the time being, but is not going to cut it in the long run.
The WHATWG spec has no mention of the audio formats that COULD, SHOULD or MUST be supported. I don't know how you would reasonably develop cross browser applications if you can't depend on a present profile.
pause and seek are not mentioned though they might be emulated using the current position from the spec. It is not clear if these properties are settable and the solution would be hackish and therefore not preferrable. Some stuff along these lines has already been suggested by Mathieu Henry.


Along these lines JavaScript should also provide a native video implementation. This could either be a browser exposed low level codec which you pass a Canvas object.

var v = new Video('video/blabla.avi', dojo.byId('player'));;

If low level codecs are not feasible an alternative could be a video server which passes compressed video data to the browser via something like comet and uses a JavaScript library to decompress and draw on a Canvas. (When am I going to have the time to write this?)

Regarding the WHATWG spec, they will probably defer video to the object element seeing as most real video is streaming but I don't think this position is tenable for the future unless you want the browser to be hollow vehicle for the Flash plugin.

Made by alper at 2007-01-30 10:30 | Place comment (3)


Re: JavaScript Multimedia
Sounds like a job for Theora, Vorbis, and FLAC. ( You can play background music in vanilla HTML already.

As such, maybe you can hack something together by loading an iframe that plays 'background music'. If the browser caches the mp3/wav/mid/ogg files you might just get away with it. I'll give this a try later today in fact, as I'm planning on resurrecting the canvas bomberman project.
Made by: Reinier Zwitserloot on January 30,2007 22:05

Re: JavaScript Multimedia
Thanks for the kind mention. I'm not a fan of relying on a plug-in for sound and agree it would be awesome were the browsers more capable (and supported a standard method) of handling sound; however, the current landscape is such that Flash is the most reliable option at this point.

It is very difficult to try scripting (after successfully testing/detecting, support of/for) a variety of plugins, ranging from Windows Media Player to Quicktime and others, hoping that you'll be able to communicate with them via script and so on. (I did a fair bit of research/experiments in past years in this area.)

While still a 3rd-party thing, Flash is a popular plug-in which provides a common base across all major browsers that is also scriptable.

I've been told HTML 5 has an audio-related spec which Opera 9 currently supports, which may be promising. Until then, Flash seems to provide a fairly stable way of making sound work cross-browser/platform.
Made by: Scott on February 04,2007 23:21

Re: JavaScript Multimedia
Don't get me wrong I saw your demoes and they were pretty awesome functionality-wise. Getting that kind of stuff to work cross-browser is very welcome.

The only theoretically evangelical platform PoV objection I can remotely muster is that it does constitute ceding ground to Flash, but as long as browser vendors keep disappointing, it looks like there is no other choice.
Made by: Alper on February 05,2007 16:20