The old, beautiful ARP 2600. These are selling for many thousands of dollars currently, much more than they were around the time they were being replaced by Yamaha DX-7s

It has never been a better time to be an electronic musician, at least as far as options for composition, sound design, and collaboration are concerned.  After synthesizer manufacturers moved to digital instruments, for a short period of time the older, analog instruments plummeted in price.  Enthusiasts like to brag about finding the $50 Minimoog or a $100 ARP 2600.  I’m sure that happened for some people, but once musicians grew tired of Spartan, single-slider data entry (a la DX-7) the demand for used analog instruments brought their prices back up.

By the early 1990s, very few manufacturers were still making analog synthesizers.  So the used market for the old analogs really heated up: $1000 and above for the old Roland TR-808s and TR-909s, $1500 for the TB-303, and $2000 for Prophet-5s were not uncommon sale prices.  The problem with this equation was simple: no one was making new devices to stem the demand that had risen for the old sound.

That said, what was happening at the time was the rapid growth of digital systems.  Synthesizers and samplers based entirely on digital circuits got better and better.  So did the computer systems, too.  In the late 1990s, affordable digital systems that could not only sequence and track music but also create the musical lines proliferated.  In 1997, The Propellerheads released their classic program called ReBirth (now available for iPad, it turns out), which packaged the most coveted analog classics of the time into a single, self-contained program you could buy for a fraction of the cost of just one of the old machines.

Rebirth simulated two TB-303s, a TR-808, and a TR-909 in a computer program with faithful accuracy to the originals.

Of course, the Internet too was ascending at the same time digital music instruments were becoming more mainstream, and with the rise of the Internet so too rose the communities of musicians, collectors, collaborators, technicians, and engineers working on musical electronics.  Those communities brought people interested in what were probably fringes of musical technology together for local meetups, gear exchanges, discussions, and so forth.  This ushered in today’s golden age for electronic music.

Now, there are literally hundreds (perhaps thousands) of small manufacturers across the globe developing musical electronics and software.  Moog’s famous modular synthesizer of the 1960s has been recreated several times over today.  Standardized rack formats, power supplies, and interfaces allow anyone with an idea and a soldering iron to plug in new sound generators and modulators.

Taking orders over the Internet also makes short runs of esoteric, specialized hardware such as the Monome and the x0xb0x actually feasible.  Many of these new generation of music hackers release their source code and schematics online for free.

Software is not left behind in this revolution.  There are countless developers out there cranking out code to implement new digital signal processors, synthesizers, and sequencers that plug in to each other using common interfaces, like Apple’s Audio Units API or Steinberg’s VST.  One can build a complete studio with multitrack recording, sampling, synthesis, dynamics processing, equalization, mastering, and more using nothing more than a computer and free software.  Using Open Sound Control (OSC) and tools like OSCulator or Max/MSP, one can even connect a Wii remote or an iPhone to the studio for musical expression.

The Monome 40h is a great example of a product that would not have been feasible before the Internet. These are specialized, niche interfaces but have wide visual appeal.

Most recently, we have seen new developments for musical technology in mobile devices, like the ubiquitous iPhone or now the iPad.  Gorillaz frontman Damon Albarn recorded his most recent album using only an iPad while on tour.

The number of musical instrument applications for the iOS platform alone is staggering, ranging from generative music inspired by Brian Eno to serious commercial music instruments from Moog Music and IK Multimedia.

Here at SXSWi 2011, Ge Wang of Smule demonstrated some of the invigorating applications his team developed for the iPhone and iPad.  To be honest, I was skeptical of the iPhone as an expressive musical platform.  Seeing and hearing Smule’s innovations and drive for true expressiveness in their iOS products made me a believer.

Perhaps the only drawback to all the great musical options out there is deciding just where you want to go with them.  New poly analog keyboards?  Old classics?  Piecing together a modular synth using boutique, limited modules?  Writing an album on your phone?  All options are on the table.  To me, that is a very good thing indeed.

Overlaying text on the real world. Image courtesy of HowStuffWorks.com.

This year I switched groups at Microsoft, from the Zune marketplace group to Bing Mobile’s Augmented Reality division.  At Zune, I worked on a lot of engaging technology, including recommender systems based on directed acyclic graph processing (similar to Hadoop) and a continuous playlist generator called Smart DJ.  While I valued and enjoyed the work I did at Zune, I also felt ready to tackle something more nascent in media technology: active processing of the environment on mobile devices.  Thus began my stint into augmented reality (AR).

So aside from the sci-fi term of art, what is AR?  My first exposure of what this technology is appeared in the late 1980s or early 1990s.  I remember reading about a program for the Commodore Amiga that enabled a person to trigger drum sounds by striking virtual planes in the air.  A video camera was used to capture the human figure, and the software defined targets that corresponded to drum sounds.  Whenever the computer detected one of the planes was crossed, it would trigger the drum sound.  This illustrates the canonical definition for AR: a modality that combines the real world and virtual world, is interactive and is a real-time experience, and within it the virtual world is mapped to the virtual world in 3-dimensional space.

Word Lens, an application for the iPhone, translates text automatically

Other examples of AR come from the movies.  Didn’t it seem that every scene that depicted robotic vision overlaid infographics onto a real-time video of what the robot saw?  What was fiction five or ten years ago is coming of age now, thanks to pervasive wireless Internet connectivity, powerful handheld computers (you know, those computers you can also use as a telephone), and advances in computer vision algorithms.  Now it’s possible to use a handheld computer to source and process video, register the video with location and orientation of the camera using GPS, compass, and gyroscope sensors, and communicate with vast computing resources in the cloud over high-speed networks, all in real-time.

This sounds cool, but does the technology have potential?  Some of us remember the heady euphoria of so-called “virtual reality” in the 1990s, with technologies like VRML, backpack computers, and video goggles promising to usher in a new age of human-computer interaction.  In the ’90s, computers were not fast enough or small enough for this concept to work in true real-time, so they ultimately failed.  Users of virtual reality goggles reported the processing lag making them dizzy.

Yelp's monocle feature overlays business information on video captured from the real world. The balloons appear near where the businesses are located.

Forrester Research believes there is potential for AR “to trigger disruption in the years to come and open up new opportunities.”  Some companies are already tinkering with AR layers to drive interest and commerce.  Take, for example, the recently-released application Word Lens for the iPhone.

Word Lens allows you to point your phone’s camera at a sign and translate the words into another language.  Is this AR?  Yes: it blends the virtual world with the physical, is registered in 3D, and is a real-time experience.  Is it useful?  I haven’t traveled yet with Word Lens, but my guess is it could be useful in certain situations.  I think the jury is still out on startup time, though.  On the iPhone, you have to unlock the phone, launch the app, choose languages, and then point the camera at the sign.  Those are a lot of steps to follow, though granted almost all AR apps require about the same routine before you get what you want.

One of the first AR applications for iPhone was Yelp’s application.  Yelp is a crowdsourced review engine, and its mobile app is great for finding businesses or restaurants nearby.  The monocle feature took this a step further by turning on the camera and overlaying balloons onto the physical world with Yelp’s review scores near where the businesses exist.  When you rotate your phone, the balloons rotate with it.  This is great to make sure you’re going in the right direction to that highly reviewed falafel joint.

At Bing Mobile, we introduced Augmented Reality features in the Bing for iPhone application.  Like with Goggles, you can use your phone’s camera to recognize parts of the real world and do things in the virtual world with them.  We introduced a new view in version 2.0 called Bing Vision, and the way it works follows.  You enter camera mode and point your phone’s camera at points of interest in the world.  We have a set of recognizers that process the video from your camera and try to extract information from the image.  For example, whenever we detect that there is text in the image, we flash an indicator telling you we can turn that text into a search query.  Or whenever we recognize a barcode, an indicator points to the barcode and automatically searches for the barcode in Bing’s index.  You can do this with a variety of objects.

Cover art is a great example.  Take a picture of a book, CD, poster, etc., and automatically conduct a web search:

The cover of this book is recognized by iBing

Search results for the cover art

Barcodes are another cool example.  Whenever iBing detects a barcode in the image it displays, an indicator points to the barcode and it automatically searches for the barcode in Bing’s index to find out what it is, how much it costs, where you can find it, and so forth:

The indicator points to the barcode iBing recognizes.

Bing found the Moleskine I scanned and is ready to help me buy another one.

Scanning text is interesting.  Our application finds text in the image and then parses it to allow you to conduct web searches.  In the following example, pretend I’m reading my DSP book and want to find more information on a topic therein, the theorem of convolution.  First I point the camera at the text.

The "Aa" circle you see indicates iBing recognizes text in the image.

I tap the camera button to tell iBing I am interested in searching based on some of this text, so it parses the text using optical character recognition.

iBing is processing the text on the image I selected.

When the text is processed, I can touch words to add them to my search query.

I tapped the words "theorem" and "convolution," and now they are part of my search query.

Hitting the search button looks for more information on Bing.  I like how this works, and it’s usually faster than just typing the words myself.  This is the first version of our text feature, so it is a little limited today.  But we have big plans for it, so stay tuned.

More information about my selected topic, the theorem of convolution.

I had a chance to use iBing this holiday season to actually do something useful rather than just show off my group’s work.  My wife and mother-in-law were shopping with me one day for some audio-visual equipment.  The store was out of the product we wanted, but we happened to see a box on the floor that had a similar model number to what we wanted.  We didn’t see this product in the store at all.  So I pointed iBing’s barcode reader to the box, and it told us everything we needed to know: the model had similar features to the one we originally wanted, it had a slightly more expensive price, and we saw that we could purchase the product at lots of nearby stores that day.  Win!

So does iBing 2.0 satisfy the definition of augmented reality?  It ties the physical world with the virtual world, and it is a real-time experience.  One might say the experience is registered in 3D, but that is a bit of a stretch.  It’s close, and as time goes on we want to get closer and closer to a real-world experience.  Personally, I’m glad to have made the switch into this new world professionally.  There are a lot of fun problems to solve, and I like being on the cutting edge of something new and exciting.

This 16GB Zune HD is probably my favorite music player. It's very small, has a built in HD Radio tuner, and it sounds great. But it won't play my iTunes-purchased music.

In 1987 I got my first CD player, and now the tape and 12″ album collection I had built began to diversify into this new medium.  I also started buying more music at this time, and both Depeche Mode and New Order began to dominate my collection.  Later, when some of my friends and I could drive (and had jobs), we would drive from the suburbs into the city and check out all the myriad record shops for better selections than we could find at the mall.  I also could afford to spend more on music, and that’s when I began to notice that it was getting harder to manage all the pieces of music I could play.

Of course, I didn’t stop there.  In college I began to DJ, both at the university’s radio station (WRCT at Carnegie-Mellon, shout out!) and at local parties and clubs.  At this time, DJing was still all about vinyl.  Sure, the radio station had some CD decks, but they did not have pitch control.  They definitely didn’t have any kind of tactile interface for cueing.  So while my personal listening collection of CDs grew, so did the collection of vinyl I used in my DJ toolkit.

The Diamond Rio PMP-300, my first digital music player. It had 32MB of solid-state storage for music.

The next prong of media diversification in my personal music library happened in about 1996.  The MP3 compression format from Fraunhofer enabled the storage of literally tens of thousands of songs on a server at my employer that we could play throughout the day.  I was excited by how good the compressed songs sounded and impressed at their tiny file size.  It sounds so quaint today, but this was truly a giant leap forward.  Without advances in audio compression technology, we would never have digital music players with such vast libraries that we have today.  Forget about selling digital songs over the internet unless you’re ready to suck down linear-PCM encoded files clocking in at about 10MB per minute.

So as my nascent digital library grew, I thus began to buy digital music players so I could listen to them anywhere.  Sure, like every other kid in the 1980s, I had a Sony Walkman, but these new digital players were different.  The first one I bought was the Diamond Rio PMP300 (1998), which had 32MB of storage on solid-state memory.  Solid state!  That meant I could jog around outside with the player and not have to listen to tape sag each time my foot hit the pavement.  Plus, it was very small. The drawback?  32MB of storage just didn’t seem like enough, even with the new affordances of MP3 compression.

The Special Edition of the Rio had 64MB of storage for music. But it was still not enough!

The next step up from the original Rio was the same thing, only with more memory: the Diamond Rio PMP300 Special Edition (1999).  This one had double the memory, and I also bought a 16MB card for it, bringing its storage capacity to a whopping 80MB.  I ultimately decided that it just wasn’t enough fun to have to decide which 14 songs I want to put on my player.  Sure, I could compress with a lossier bit rate, but that sacrificed the quality of the songs.  If this were truly progress, shouldn’t I be able to listen to good quality and have more than one album on my player?

Somehow I skipped the first iPods, and the first one I owned was the iPod mini (2004), which had 4GB of space.  This was finally enough, I thought.  It wasn’t solid state, but rather it used a new tiny hard drive called a microdrive.  But the microdrive was resilient enough to withstand walking or running, and the battery on the iPod lasted a pretty long time.  The only trouble was, by this time my music library had grown past the 4GB point.  I think I had about 7GB of music at the time, and it was beginning to get tiresome deciding what to put on the player and what to leave off.

My first iPod was one of the 4GB minis. Finally, 4GB seemed like enough space, but somehow the music library was growing faster than music players' capacity.

Since then, my digital music library, listening habits, and music players have all transformed significantly.  I bought a 30GB iPod Video to hold the entire music library on one device.  I began to buy music digitally from Apple’s iTunes digital music store.  And I went to work for Zune to develop music recommendation algorithms and social information processing code — of course, when I arrived at work I got a Zune there, too.  Now, I’ve owned two cellular telephones that also are music players, and I have a 1GB iPod shuffle for working out.  I still have my Zune Pass “all you can eat” music store subscription, which means I have an unlimited supply of music at my disposal.  My iTunes library clocks in at over 8400 items, representing 32 days of music at 82GB.  Oh yeah, and I also have a giant physical music collection of both vinyl and CDs.  I’m feeling overloaded.

Where do I go from here?  I know I’m not alone in this predicament.  I don’t think new music players with more capacity or smaller footprints are going to solve this problem for me.  Tagging, searching, and sorting my iTunes library helps, but the user interface is still more accounting in Excel than it is flipping through records.  To make matters worse, I now feel that my physical music library is just sitting there decaying.  I play records and CDs sometimes, but usually I am just dialing up a song on iTunes or Zune.  I read a lot of blogs, which just blast new music at me 24/7/365.  Smart DJ, Genius, and Pandora are all there to help me find music I want to listen to, and they do a decent job.  But I think what I really need is less, not more: fewer devices, programs, and sources of music — to savor the experiences music provides rather than to be such an avid consumer.  It’s hard to turn off all the voices pushing new music in my face, but I think that’s the only way to stop being a collector and to start being a listener again.

At Zune, one of the most striking pieces of analysis of our customers’ listening habits demonstrated that in general people listened to new music almost exclusively.  Building the histogram of play events against release date was telling.  I think music has been commercialized into a consumption culture for quite a long time, from the content producers to the distributors and also to the electronics vendors that create new formats, devices, and technologies.  But alas, I think now I am going to strive for simplicity.  I might just enjoy all that music sitting around here rather than find new ways to gobble up new bytes.

Even though I have some newish players with smaller capacities, overall the amount of music they hold is growing. Interestingly, though, I think people are starting to find they just don't need 100GB of music in a portable device. I'd guess that the days of the Microdrive devices with >100GB of storage capacity are numbered.

It is now safe to say that the convergence is finally happening.  But so much more has happened since those early days.  In the video world, we now have Tivo and countless other personal video recorders.  In fact, many of these devices are shipped to consumers via their cable operators.  The content owners were aghast that viewers could skip the commercials that ultimately pay them.  And toward the end of the 2000s, of course, Internet video is light years ahead of where RealNetworks’ RealVideo began.  Now, the iPod nano even records video that people can upload, edit, share, and exploit.  People can watch video on their cell phones, computers, XBoxes, Playstations, portable DVD players, and even the old venerable television set.  Televisions have changed, too, though notably the personal computer hasn’t displaced it just yet.

So what about music technology?  The past decade has exploded with new, interesting music technology, and people have completely changed the way they consume music.  At the beginning of the decade, compact discs were still the media king, and now Apple’s iTunes Music Store is the #1 music retailer.  Also at the beginning of the decade, Napster began to shape how people consume music, engendering the idea that music should simply be free (technically, Napster began service in June 1999).  Now we have the iPod, iPhone, Zune, SoundCloud, Rhapsody, Pandora, Last.fm, Spotify, RockBand, Guitar Hero, DJ Hero, and multitudes of others.

Music production radically changed, too.  The personal computer has become powerful enough to play back scores of tracks at the same time, to implement synthesizers, samplers, effects processors, and more on a laptop with a single disk and a few pieces of software.  Ableton Live simplified the notion of playing studio tracks live to an audience, giving way to the “laptop performance.”  And Serato Final Scratch and other technology enables DJs to bridge the gap between the old vinyl world and today’s digital libraries.

On a personal level, I reached some of my own goals over the past decade.  My debut album was released in 2003 on a reputable independent music label, and a second sophomore album followed a few years later.  In between, there were compilation appearances here and there and another EP release.  Professionally, I spent over half the decade in startup companies, and in one of the two I supported entrepreneurs in residence at a venture capital firm.  We went on to ship our software to millions of people starting with nothing but an idea, and that company eventually sold to Cisco in 2008.  I also realigned my professional and personal interests by returning to Microsoft to work on data-powered media experiences, like video recommendations at MSN Video and social music experiences at Zune.

So instead of postulate what the next decade may bring, instead I am content to live in today.  Never before have we seen such a rich plethora of media technology ripe for the picking, whether we are music producers, consumers, or both.  Whatever is in store for the next decade, I’m ready.  But, let me take a moment to enjoy what we all have worked to produce in the 2000s.  I’m switching on the studio as we speak.