Lily

Finally found a use for the NY Times from Bill Orcutt on Vimeo.

Ever wondered what a web site sounds like? Not the words, but the actual data on the site itself? Here's a Lily application that let's you "play" the data in a web page like an instrument.

When the patch starts, the browser enters a DOM inspection mode and mousing over a DOM element highlights the node. Clicking on a node writes the element's data (its innerHTML value if it's a text element or the binary data if it's an image) as a sound file and the file is then loaded in a quicktime player in the patch. The sounds can then be triggered using OSC messages. While a DOM element is "playing", the browser scrolls to and highlights the element with a thick black border.

In the video, I'm spazzing out on the NY Times homepage using the monome controller, but the demo should work with any OSC enabled controller. Sound conversion code based in part on a javascript port of the baudio project.

SVG Midi Sequencer from Bill Orcutt on Vimeo.

Oddball monophonic sequencer using an svg/javascript animation. The animation generates a tree of randomly distributed nodes that represent the steps in the sequencer. Press the spacebar to add a node to the tree. Toggle notes on/off by clicking a node while holding down the "x" key. A node's note value is based on its position on the screen and is output as an OSC message to a Chuck patch that translates it to midi. Dragging the nodes around the screen in the tree changes their value. The sounds in the screencast are generated using Reaktor, but the demo should work with any midi enabled synth or sampler. Based on this traer physics example.

SVG to OSC from Bill Orcutt on Vimeo.

Another svg/javascript animation to control sound. Each circle has a pitch/amplitude value based on its position. The values are sent (via the sendosc external) as OSC messages to a Chuck synthesis patch to generate the sound. Dragging the circles around modifies the harmony as the individual note values change. Running this demo requires Chuck. Based on this processing example by Sean McCullough.

SVG to Quicktime from Bill Orcutt on Vimeo.

Here's a demo that uses an svg/javascript animation to control quicktime playback. The objects in the animation are attracted to the mouse and repelled by each other. The forces generated by their interactions are scaled and output to control mp3 playback rate. Based on this traer physics example. Features the music of Dillard Chandler .

Here's a Lily demo app that runs in the background while you surf the web. When you find something you'd like more information about, select the word(s), hit the enter key and page bot does an image search (using the flicker search api) and displays thumbnails of the results. Clicking the thumbnail displays a larger version of the image. Clicking the image again returns you to the thumbnails. Clicking away from the page bot dismisses it. Check out the screencast to get a feel for what it does.

This is a good place to mention that when you open a patch using "Open" in the Lily menu, it opens the patch as a separate window. This is fine for development, but if the patch doesn't have a UI (as in this example), then you'll probably want to keep the window hidden while you're running it. The way to do this is to save the patch as an addon and check "Hide Main Window" and "Hide Debug Window" in the save as dialog.

I'm a big fan of ffffound.com's image bookmarking site. Here's a local, client-side version of that service let's you archive images for offline viewing. It works by adding a "Save this image" item to your browser's context menu. Selecting the menu item while clicking on an image saves the binary image data and its context (the page url, date, etc) in browser storage. You can then view your saved images by pointing your browser at http://localhost:8080/archiver?offset=0.

Unlike some of the other Lily demos which have a putative utility, the CLI DOM Explorer is defiantly pointless. It's a command line DOM explorer that let's you telnet (!) to your browser and traverse & interact with the HTML document using standard Unix commands. The (somewhat) supported commands are: ls, pwd, cd & rm - ls displays the child nodes of the current node, cd changes the current node to the node supplied as an argument (".." & "/" are supported), pwd displays the path back to the root node and rm deletes the supplied node. The screencast more or less documents what you can do with it. (As with all the Lily demos, keep in mind that this is an alpha application made with beta software, so don't get your hopes up or anything.)

Inspired by Marcel Salathé's processing applet, here's four svg/javascript sketches that visualize the HTML DOM in various ways. Each sketch walks the HTML document loaded in the browser and draws the structure in the patch window using the svg external's api.

Randomized, force-directed layout based on same traer physics example that seems to have been the basis for Marcel Salathé's work. Painfully slow and unstable (it seems to fly off the canvas almost as often as it produces a visible result), though the misfires are sometimes more interesting than the intended result. JS/SVG performance issues limit the usefulness of the script for larger sites- as more objects are added to the SVG DOM, the script slows until it grinds to a halt. Screenshot above generated from www.google.com.

Completely random layout inspired by the "network" illustration in Design By Numbers. Generated from sfbay.craigslist.org.

Examples of visualizing the DOM hierarchy as nested/stacked boxes. The outer/top box is the root node- child nodes are nested/stacked. Screenshots generated from www.google.com.

Drawing the DOM from Bill Orcutt on Vimeo.

Minimal web server. Incoming http requests are output through the outlet and the response is issued after the left inlet receives some input. The demo screencast shows how to use the httpd object to enable a web browser to control a patch remotely.

Feeble attempt at this processing demo. The patch uses a metro to output mouse position at regular intervals. Each time the position is output a circle object is created at that position and a reference to the circle is passed to the animate object which manipulates the object's opacity. When the opacity reaches zero, the circle's delete method is called.

Demo showing how to combine web services to perform complex operations- in this patch, the NY Times homepage RSS feed is fetched using the rss object, the descriptions are concatenated and passed to the Yahoo term extraction API. The extracted terms are then passed to the Flickr search API and the resulting photos and captions are displayed in a table.

Lily patch building screencast demonstrating reading a text file from disk (using the text object) and then displaying its contents- first as a slideshow (using marquee) and then as a series of randomly sized/positioned comments. The demo also shows how to hide objects and connections.

Two minute screencast of building a patch to fetch Flickr "interestingness" pics for a given day. Demonstrates how web APIs and 3rd party framework JS can be wrapped as Lily objects- in this case, the flickr.interestingness.getList service, the Yahoo YUI Calendar widget and Lokesh Dhakar's Lightbox script. Also shows how a Lily patch can contain other patches (note the use here of the "format-date.json" patch loaded using the patcher object) which makes it possible to encapsulate and reuse patches.

Demonstration of the help patch for the circle object. Help patches are small programs that document the available commands for an object. They're accessible by option/alt clicking on an object or via the Help menu item. Because a help patch is a functioning program, it's possible to copy working code directly from the help patch into your current program.

Demo screencast of a rudimentary beatbox created in Lily. Probably not a use for which the browser is best suited but hopefully gives a sense of what's possible. In any event it cracks me up. Patch can be downloaded here.

Lily screencast showing building a simple patch that adds two numbers. Demonstrates patch editing techniques, using arguments to pass a paramter into an object at initialization, message ordering (when multiple connections are made to one outlet, messages are output in the order the connections were made) and using feedback by passing a process's output back into its input via a delay.