Music Explosion

I’m finding it harder than ever before to manage all the music I have accumulated over the years.  I suppose I began buying music around age 8, though I couldn’t pay for it myself yet.  My mom or grandmother would take me and my brother to the mall, and sometimes we would get an allowance with which to entertain ourselves.  With about $10 a pop, I could usually afford a cassette or 12″ record at the local record store.  I would come home and practically wear out the music I bought on my little boom box or mom’s turntable.  Some of my first purchases were Van Halen’s 1984, Falco 3, and Kraftwerk’s Trans-Europe Express.

This 16GB Zune HD is probably my favorite music player. It's very small, has a built in HD Radio tuner, and it sounds great. But it won't play my iTunes-purchased music.

In 1987 I got my first CD player, and now the tape and 12″ album collection I had built began to diversify into this new medium.  I also started buying more music at this time, and both Depeche Mode and New Order began to dominate my collection.  Later, when some of my friends and I could drive (and had jobs), we would drive from the suburbs into the city and check out all the myriad record shops for better selections than we could find at the mall.  I also could afford to spend more on music, and that’s when I began to notice that it was getting harder to manage all the pieces of music I could play.

Of course, I didn’t stop there.  In college I began to DJ, both at the university’s radio station (WRCT at Carnegie-Mellon, shout out!) and at local parties and clubs.  At this time, DJing was still all about vinyl.  Sure, the radio station had some CD decks, but they did not have pitch control.  They definitely didn’t have any kind of tactile interface for cueing.  So while my personal listening collection of CDs grew, so did the collection of vinyl I used in my DJ toolkit.

The Diamond Rio PMP-300, my first digital music player. It had 32MB of solid-state storage for music.

The next prong of media diversification in my personal music library happened in about 1996.  The MP3 compression format from Fraunhofer enabled the storage of literally tens of thousands of songs on a server at my employer that we could play throughout the day.  I was excited by how good the compressed songs sounded and impressed at their tiny file size.  It sounds so quaint today, but this was truly a giant leap forward.  Without advances in audio compression technology, we would never have digital music players with such vast libraries that we have today.  Forget about selling digital songs over the internet unless you’re ready to suck down linear-PCM encoded files clocking in at about 10MB per minute.

So as my nascent digital library grew, I thus began to buy digital music players so I could listen to them anywhere.  Sure, like every other kid in the 1980s, I had a Sony Walkman, but these new digital players were different.  The first one I bought was the Diamond Rio PMP300 (1998), which had 32MB of storage on solid-state memory.  Solid state!  That meant I could jog around outside with the player and not have to listen to tape sag each time my foot hit the pavement.  Plus, it was very small. The drawback?  32MB of storage just didn’t seem like enough, even with the new affordances of MP3 compression.

The Special Edition of the Rio had 64MB of storage for music. But it was still not enough!

The next step up from the original Rio was the same thing, only with more memory: the Diamond Rio PMP300 Special Edition (1999).  This one had double the memory, and I also bought a 16MB card for it, bringing its storage capacity to a whopping 80MB.  I ultimately decided that it just wasn’t enough fun to have to decide which 14 songs I want to put on my player.  Sure, I could compress with a lossier bit rate, but that sacrificed the quality of the songs.  If this were truly progress, shouldn’t I be able to listen to good quality and have more than one album on my player?

Somehow I skipped the first iPods, and the first one I owned was the iPod mini (2004), which had 4GB of space.  This was finally enough, I thought.  It wasn’t solid state, but rather it used a new tiny hard drive called a microdrive.  But the microdrive was resilient enough to withstand walking or running, and the battery on the iPod lasted a pretty long time.  The only trouble was, by this time my music library had grown past the 4GB point.  I think I had about 7GB of music at the time, and it was beginning to get tiresome deciding what to put on the player and what to leave off.

My first iPod was one of the 4GB minis. Finally, 4GB seemed like enough space, but somehow the music library was growing faster than music players' capacity.

Since then, my digital music library, listening habits, and music players have all transformed significantly.  I bought a 30GB iPod Video to hold the entire music library on one device.  I began to buy music digitally from Apple’s iTunes digital music store.  And I went to work for Zune to develop music recommendation algorithms and social information processing code — of course, when I arrived at work I got a Zune there, too.  Now, I’ve owned two cellular telephones that also are music players, and I have a 1GB iPod shuffle for working out.  I still have my Zune Pass “all you can eat” music store subscription, which means I have an unlimited supply of music at my disposal.  My iTunes library clocks in at over 8400 items, representing 32 days of music at 82GB.  Oh yeah, and I also have a giant physical music collection of both vinyl and CDs.  I’m feeling overloaded.

Where do I go from here?  I know I’m not alone in this predicament.  I don’t think new music players with more capacity or smaller footprints are going to solve this problem for me.  Tagging, searching, and sorting my iTunes library helps, but the user interface is still more accounting in Excel than it is flipping through records.  To make matters worse, I now feel that my physical music library is just sitting there decaying.  I play records and CDs sometimes, but usually I am just dialing up a song on iTunes or Zune.  I read a lot of blogs, which just blast new music at me 24/7/365.  Smart DJ, Genius, and Pandora are all there to help me find music I want to listen to, and they do a decent job.  But I think what I really need is less, not more: fewer devices, programs, and sources of music — to savor the experiences music provides rather than to be such an avid consumer.  It’s hard to turn off all the voices pushing new music in my face, but I think that’s the only way to stop being a collector and to start being a listener again.

At Zune, one of the most striking pieces of analysis of our customers’ listening habits demonstrated that in general people listened to new music almost exclusively.  Building the histogram of play events against release date was telling.  I think music has been commercialized into a consumption culture for quite a long time, from the content producers to the distributors and also to the electronics vendors that create new formats, devices, and technologies.  But alas, I think now I am going to strive for simplicity.  I might just enjoy all that music sitting around here rather than find new ways to gobble up new bytes.

Even though I have some newish players with smaller capacities, overall the amount of music they hold is growing. Interestingly, though, I think people are starting to find they just don't need 100GB of music in a portable device. I'd guess that the days of the Microdrive devices with >100GB of storage capacity are numbered.

Music Technology and Me, Part III

Once I upgraded to a professional synthesizer from my Casios, I was desperate to find a machine to help me realize complete songs.  Sure, the old Juno sounded great, but I could only play one sound on it at a time.  I also wasn’t able to control it using a computer, since it had the old-style Roland DCB buss instead of MIDI.  Again, I was in high school at the time, and let’s just say my job slinging pizza to the suburban Houston masses did not yield the kind of budget I would need to build a real studio.  Plus, I was driving by then and had to pay for non-musical things like car insurance and gasoline.

Ensoniq's performance sampler, the EPS. One unique feature of the EPS is that it could load samples into memory from disk while you played it. I'm not aware of any other sampler that does this. The band OMD actually used two Emulator IIs during one of their tours so that one band member could load samples while the other still played.

Through my voracious reading of trade literature for electronic musical instruments, I began to formulate my plan.  I knew that sampling technology was getting cheaper and cheaper, and like microprocessors, state-of-the-art samplers would blow away yesterday’s machines in terms of cost effectiveness and power.  I think the Ensoniq EPS was around at the time – a real sampler with a sequencer and a disk drive.  They sounded good, too.  The only problem was there was basically no way to find a cheap, used one.  Plus, my sampling appetite had already been aroused by my $100 Casio SK-1.

Ensoniq pretty much built its business in those days around doing what the other manufacturers did — only they were much, much cheaper.  For example, Ensoniq’s first sampler, the Mirage, did everything costlier samplers did at the time at a fraction of the cost: $1700.  Ensoniq’s first synthesizer, the ESQ-1 had the same appeal.  It had 8 voices of digital controlled oscillators with analog filters, a velocity-sensitive keyboard, and a simple sequencer for less than $1000.

The Ensoniq Mirage accelerated the hardware sampler wars. The first samplers, such as the Fairlight and Synclavier could cost tens of thousands of dollars. The E-mu Emulators clocked in just under $10,000. And then the Mirage comes along at under $2,000 to bring digital sampling to the masses.

Whereas synthesizers seemed not to depreciate steeply, samplers were another story altogether.  They are basically computers inside, limited by the same parameters that limit a computer: memory, bit depth, secondary storage, processing speed, and throughput (polyphony).  This meant that the advances in computing power in the 1980s and 1990s pushed the depreciation curves of hardware samplers even steeper⁽¹⁾.  And, as you might guess, this was good for me.  A used sampler with a sequencer would be cheap and exactly what I needed.

I wound up on the mailing list of Rogue Music, a musical instrument trading nexus in New York.  Rogue mailed its newsletter of used gear inventory every month or so, and I loved analyzing what instruments were in demand and where I could find values that fit my budget.  I found my first drum machine, a cheap Roland TR-707, this way a few months prior to becoming serious about a sampler.  One day the newsletter came, I found what I was looking for: an E-mu Systems Emulator II for about $800.

Ferris Bueller used an Emulator II to help make his case to stay home from school. It's a great idea, but I never tried it with mine.

Eight hundred dollars was a huge amount of money for me at the time.  I didn’t even have all of it then.  But I had an intense desire for the machine.  After all, that one machine would solve all my recording problems for the foreseeable future!  I knew about the Emulator from the music magazines, but I read a lot more about it after recognizing it in Ferris Bueller’s Day Off.

The Emulator cost about $8000 when it was released.  It has a basic sequencer and a fancy sampling scheme that squeezes more depth and clarity out of 8 bits than a typical 8-bit sampler (they claim it sounded more like 14-bit resolution).  It also had a floppy disk drive (or two, depending on the model) to store sounds.  The voice architecture was a lot like an analog synthesizer, except instead of using oscillators to generate basic tones, the Emulator played back sampled sounds.  In fact, it used analog filters with resonance, which meant it could sound warm and fat.  It also had 8 individual outputs for its voices which could be programmed flexibly.

The E-Mu Systems Emulator II: one of the best-sounding samplers of all time. If it weren't so large, I would love to have another one someday. But it is seriously large.. and heavy!

My grandfather again came to my rescue by chipping in $200 to help me buy the Emulator, and I still remember the day it arrived.  My brother and I plugged it in to the stereo in our living room and began loading disks of samples.  Piano?  Check.  Orchestra hits?  Check.  Weird animal noises?  You bet.  I grew to know that sampler inside out and backwards, and I squeezed all I could out of it.  I built a library of hundreds of disks, sampling from movies and television to my own other keyboards … anything I could think of, really.

I don’t have the Emulator anymore, but it definitely still has a spot in my musical upbringing.  The limits of the machine and my budget made me think really hard about how to maximize those 8 voices and the limited sampling memory in my songs.  Now, my studio has a few hardware samplers (yes, I still use them!), and they completely blow the Emulator out of the water in terms of power and flexibility.  But none of them quite sound like that fat, chunky Emu.

–

⁽¹⁾ This trend continued in such a dramatic way that the demand for hardware samplers essentially evaporated. The manufacturers raced to 16-bit machines, and then to cheap machines with 96kHz sampling rates, only to find that software sampling and virtual instruments were coming of age. There is still demand for sampling drum machines, like Akai’s MPC series, but one might argue that these are more about workflow than they are about sampling per se.

Music Technology and Me, Part II

After starting my first after-school job, I was disappointed to find how hard it was to save money.  My $3.85/hour wasn’t exactly accruing the kind of cash I would need to buy my dream studio.  But I would still ride my bike half of my weekdays and usually one weekend day to the neighborhood pizza shop and work towards my first real synthesizer.

The day I could finally afford my first synth arrived in 1990.  Some of my friends could drive by then, and we made a habit of checking out music shops and pawn shops to see what gear they had around.  Even one of the mall stores then had a great selection.  I remember playing an Oberheim Matrix-12, Matrix-6, Roland Alpha Juno, JX-10, and a JX-8p all in one sitting there.  I didn’t even look at the price tags, though.

There was one pawn shop that tended to have a great selection of old gear on Westheimer in Houston.  We walked in one day, and right in front of my eyes was a Roland Juno-60.  I had never seen one before, but the price seemed somewhat reasonable at the time: $425.  I asked for some headphones and began to put the Juno through its paces.

The Roland Juno-60

I instantly fell in love with the Juno.  It is a single oscillator synthesizer with an analog self-oscillating filter, a heavenly chorus, and an arpeggiator.  Even though the oscillator was digitally controlled, to me it sounded rich and warm.  The sub-oscillator could also give the impression of a two-osc synthesizer, and to make things even better that sub-oscillator has a woody, open square wave sound to it that complements the saw and pulse waves on the regular oscillator.  Plus, that chorus!  I did not have $400 at the time, but I was getting close.  I had to find a way to make the Juno-60 mine.

I remember a Sunday afternoon after seeing the Juno, my family all went out for burgers with my grandfather who was visiting us from the Texas valley.  The hamburger restaurant happened to be very close to the pawn shop, so I begged my parents to let me see the synth just for a few minutes while we were nearby.  I think my mom told my grandfather I almost had enough money to buy it, and he offered to pay the rest of the bill right then and there.  The Juno came home with me.

Another view of the Juno-60

I wonder how many hours I spent in my bedroom with headphones on playing that Juno.  I still love the way it sounds and use it practically every time I fire up my studio.  The Juno was a little torn up and rough, but everything worked perfectly on it.  I still am amazed at how great it can sound; you just don’t find many single-oscillator synthesizers that can cut through a mix like the Juno-60 can.  I’m glad it’s still with me.

Even though my grandfather is no longer with us, the Juno is.  I still think about him every once and a while when I play it.

Music Technology and Me, Part I

As far back as I can remember, raw synthesized sounds always intrigued me.  I remember riding in the car with my parents in our red Oldsmobile listening to the radio, hearing sounds like vocoded voices and raw bass lines probably generated by a Minimoog.  Somehow, I had to be able to harness those sounds for myself.

My first electronic instrument, a Casio MT-45 preset synthesizer

For a birthday (I’m going to guess 8th) my parents bought me a Casiotone MT-45, my first electronic keyboard.  I wound up spending countless hours with it, but since it was a preset machine it never lived up to the promise of what I always wanted: a professional analog synthesizer.  But the old Casio did travel with me pretty much everywhere, and I eventually bought myself a Casio SK-1, the first cheap sampling keyboard.  I also eventually bought a Casio SK-5, which was able to store more than just one sampled sound at a time and had orange rubber drum pads on the front.

The Casio SK-1. This was about $100 when it came out and had the ability to sample with an internal microphone. It also had a rudimentary additive synthesis routine built-in.

My dad tended to frequent pawn shops around the time I was in junior high school, and one day he came home with two old cassette decks and some speakers for me.  I actually already had my own Radio Shack 4-channel mono mixer, which I was using to mix the three keyboards together.  Now, with the two cassette decks, I could overdub until my heart was content with my three cheesy keyboards!  Interestingly, the two tape decks had slightly different speeds, so unless I was careful to record and to play back from the same deck, the recordings would eventually decay in pitch over time.  If the keyboards were tunable, that wouldn’t be a problem, but of course they were not.  So there are some unintended microtonal aspects to my early recordings.

My love affair with music technology began to pick up steam in 1986.  In the 1980s, I was like a lot of other kids who lived in the suburbs in a few ways.  For example, my parents would sometimes need to do some shopping at the mall, so they would send me off to wander around while they ticked boxes off of their shopping lists.  In March, I happened to be at a bookstore and noticed the cover of Keyboard Managine: Jean-Michel Jarre in his studio, and a special flexi-disc of a song from the Zoolook sessions was inside and waiting for me!  The magazine came home with me, and I memorized every page.

Jean-Michel Jarre's Rendez-vous Houston, April 1986

One month later, Jarre came to Houston, Texas, my city, for a giant concert downtown.  He brought his vast array of studio synthesizers, the laser harp, and a gaggle of musicians to create an outdoor concert featuring the buildings of downtown as his backdrop.  There were fireworks, huge projections, lasers, and of course, synthesized music.  Jarre also had this semi-circular controller keyboard that had huge, translucent keys; the keys alit whenever he struck them.  I was 11 at the time, and this event had a huge effect on me.  I actually missed the concert that night (long story), but my great uncle gave me a copy of the VHS recording of the event.  I actually think that this was probably a better way of experiencing it, because of all the backstage footage.  Jarre making crazy sounds on an ARP2500 in an unfinished skyscraper gave me some early inspiration.  Yes, one day I will be making crazy sounds on a huge synthesizer in an unfinished skyscraper, too.  At least, that’s what I thought.

So goes my introduction to the world of electronic sounds.  I pined for a real analog synthesizer, and in fact I lied about my age to get an after-school job early.  My savings was devoted to building my studio at $3.85/hour.  It would take a while.

The 2000s

I remember working in the interactive television world in the mid to late 1990s, and the big push back then was convergence.  The idea was that computers would begin to displace televisions and other devices, and that people would begin to use personal computers in more shapes and sizes than ever before.  By the time I left Microsoft to join the heady rush toward Internet startups, the message became tired: no one wanted to trade in their TV, and no one felt like “mousing” around on their television screens to click on things to buy.  Moreover, the content industry behind television was adamantly avoiding any kind of screen overlay or feature that distracted viewers from the screen.  After all, the commercial content industry in television is all about selling advertising at the end of the day.

It is now safe to say that the convergence is finally happening.  But so much more has happened since those early days.  In the video world, we now have Tivo and countless other personal video recorders.  In fact, many of these devices are shipped to consumers via their cable operators.  The content owners were aghast that viewers could skip the commercials that ultimately pay them.  And toward the end of the 2000s, of course, Internet video is light years ahead of where RealNetworks’ RealVideo began.  Now, the iPod nano even records video that people can upload, edit, share, and exploit.  People can watch video on their cell phones, computers, XBoxes, Playstations, portable DVD players, and even the old venerable television set.  Televisions have changed, too, though notably the personal computer hasn’t displaced it just yet.

So what about music technology?  The past decade has exploded with new, interesting music technology, and people have completely changed the way they consume music.  At the beginning of the decade, compact discs were still the media king, and now Apple’s iTunes Music Store is the #1 music retailer.  Also at the beginning of the decade, Napster began to shape how people consume music, engendering the idea that music should simply be free (technically, Napster began service in June 1999).  Now we have the iPod, iPhone, Zune, SoundCloud, Rhapsody, Pandora, Last.fm, Spotify, RockBand, Guitar Hero, DJ Hero, and multitudes of others.

Music production radically changed, too.  The personal computer has become powerful enough to play back scores of tracks at the same time, to implement synthesizers, samplers, effects processors, and more on a laptop with a single disk and a few pieces of software.  Ableton Live simplified the notion of playing studio tracks live to an audience, giving way to the “laptop performance.”  And Serato Final Scratch and other technology enables DJs to bridge the gap between the old vinyl world and today’s digital libraries.

On a personal level, I reached some of my own goals over the past decade.  My debut album was released in 2003 on a reputable independent music label, and a second sophomore album followed a few years later.  In between, there were compilation appearances here and there and another EP release.  Professionally, I spent over half the decade in startup companies, and in one of the two I supported entrepreneurs in residence at a venture capital firm.  We went on to ship our software to millions of people starting with nothing but an idea, and that company eventually sold to Cisco in 2008.  I also realigned my professional and personal interests by returning to Microsoft to work on data-powered media experiences, like video recommendations at MSN Video and social music experiences at Zune.

So instead of postulate what the next decade may bring, instead I am content to live in today.  Never before have we seen such a rich plethora of media technology ripe for the picking, whether we are music producers, consumers, or both.  Whatever is in store for the next decade, I’m ready.  But, let me take a moment to enjoy what we all have worked to produce in the 2000s.  I’m switching on the studio as we speak.

Meet in Kobe

International Society for Music Information Retrieval 2009 logo

At the end of October, I traveled to Kobe, Japan for the tenth International Society for Music Information Retrieval conference.  This was my second ISMIR, and to me this conference is different than many of the other academic conferences I attend.  For one, the work is inherently multi-disciplinary, drawing from such disparate fields as machine learning, musicology, information retrieval, acoustics, composition, statistics, and sociology.  But second, I think the crowd at ISMIR is different in a friendlier way.  Perhaps I feel this way simply because ISMIR aligns well with my own interests, or perhaps the composition of the participants and audience is varied enough to capture more than just a single vertically integrated field of expertise.

This year, Paul Lamere invited me to participate on an industrial panel during the conference, along with representatives from other companies working in the music information retrieval world.  I really enjoyed the conversation; representatives from smaller companies like Barcelona Music and Audio Technologies, The Echo Nest, and Last.fm were there to discuss MIR alongside large corporations like Yahoo, NTT, and Gracenote (I represented Microsoft).

The industry panel at ISMIR 2009. From left to right, Malcolm Slaney (Yahoo), Kunio Kashino (NTT), Keiichiro Hoashi (KDDI), Norman Casagrande (Last.fm), Oscar Celma (BMAT), Peter from Gracenote, Tom Butcher (me), and Paul Lamere (The Echo Nest)

I had two main points in the panel (though I’m sure I had opinions on a lot of other topics).  First, I frequently feel at academic conferences that even though much of the work can be technically elegant and solve problems with measurable success, sometimes it is difficult for me to get excited about slight increases in prediction accuracy (or whatever) when practical problems would prevent the research from ever being used in the real world.  For example, often work in recommendation engines measures success in a very closed way, even though that method is somewhat standard and accepted in science.

Usually with IR or machine learning problems researchers carve up their data sets and perform what are known as “hold-out experiments.”  Data from one partition are used to train an algorithm, whereas data from the remaining partition are used to test the output of the algorithm.  This works well for many cases, but in the realm of media recommendations I think the process breaks down.  These experiments fail to account for the effect of users interacting with the system and the time effects of recommendations as a whole.

Harbor Town in Kobe and Port Tower

It can be notoriously difficult to measure the true performance of a recommendation engine if you do not have the luxury of trying the system out on a large enough set of users, so I understand that the hold-out experiment is sometimes the only way one can measure an algorithm given the data set.  But too often I feel that researchers focus on satisfying F-measure curves or the bounds of the experiment and miss the mark when it comes to developing a system that real users love.

Perhaps that sounds too practical, for of course there is merit in science for science’s sake alone.  But the point I wanted to make in the panel is that we still don’t have a good way of measuring recommenders’ performance, and if someone really wants to have an impact with real users, sometimes simpler solutions that perform well and return reasonable results fast will blow away the fanciest learning algorithm.  Greg Linden sometimes discusses this idea on his blog.

The second point I wanted to make is that as far as the music industry goes, the democratization of music technology and the application of MIR-specific technology into mass-market products over the past decade presents a big opportunity for the scientific community and for industry alike.

Kobe at Dawn, from the window of the Hotel Portopia

Whether you love or hate Rock Band and Guitar Hero, they have changed the way we experience and consume music.  The recording industry has had an extremely difficult time adapting to technological advances, and as a result they watched their revenues collapse.  The labels are also desperate to sell you the same content you already have again and again by inventing new formats.  Well, Harmonix was able to do what the record labels could not by transforming the music experience.  By adding a little additional metadata to recordings people already know and by creating an immersive experience, many of us are willing to buy the same songs again and to enjoy them in this new way.

Someone in the audience asked if we on the panel could point to specific examples of MIR technology being applied to products.  The game products are some, and Microsoft Songsmith is another.  Companies like Microsoft, Last.fm, Pandora, and Apple are using MIR technology to drive data-driven music experiences with Zune Smart DJ and Genius.

There have also been huge advances in music creation technology over the past few years.  Apple placed GarageBand on every new Macintosh computer so the masses can create music without having taken a lesson.  Yamaha licensed MIR technology in some of their new music production tools in Cubase and for Vocaloid.  And, there are literally hundreds of brand new iPhone apps using MIR.

There’s an abundance of opportunity for MIR, and as long as people are entertained, enlightened, and satisfied by listening to or making music, we all have a chance to embark on some very interesting, fun work.

Thanks to: Paul Lamere, Justin Donaldson, Malcolm Slaney, Òscar Celma, and Norman Casagrande.

Unleashing Smart DJ

Last month, we released the new Zune HD device along with the Zune 4.0 suite of software and services.  Smart DJ is one of the big new features for this year’s Zune lineup, and it also happens to be one of the most rewarding features I have ever worked on at Microsoft.  Too often in large software organizations, different groups are isolated from each other, which leads to the customer’s needs or desires being lost along the way.

Glamour shot of the new Zune HD

With Smart DJ, we knew we wanted to provide an experience that lets people sit back and enjoy tracks and artsits that sound great together.  What began in a meeting in early 2009 fostered a collaboration across groups that ultimately delivered a compelling experience, despite organizational boundaries and disjoint schedules.  So, how did we get there?

Today’s Smart DJ feature actually began as a few distinct efforts.  I work on the services team, and our strengths include large-scale data processing, domain expertise with music, and really knowing about our customer.  We knew we could drive interesting listening experiences with some of our new technology investments.  The other side of the team was the group responsible for the Zune PC Client software, and they also know a lot about our users.  They have a lot of expertise with flashy user experiences, and we also knew that if we could use the user’s machine to help drive computations, we could distribute the overall load in the system.

It turned out we both were approaching similar experiences with different viewpoints and different code names.  On the service side, we need to support a variety of clients.  But in on the client side, they can finely tune the feature to their experience and provide a rich user interface to go along with it.  Earlier this year, our teams conferenced in a big room together, brainstorming and trying to figure out how to align our priorities.  We began with mutual respect and a common goal: create something compelling our users would love that helps them discover new music, enjoy the music they already have, and connects them with their music as fast as possible.

An example of the Zune Client playing a Smart DJ mix

What really struck me about this collaboration is how well it worked in the end.  We had schedules that were not aligned, and each team had their own sets of prioritized features.  There was a fair amount of give and take, which is to be expected.  But this time, it just felt different to me.  We all really wanted to provide the best experience possible.

Once we had the algorithms, the transport, and the client all working together, it was incredibly rewarding to let Smart DJ be our musical guide.  We were making improvements along the way until we shipped the code, and without revealing too much about it, you can expect to see even more improvements as time goes on.  Smart DJ gets better with age.

I was really excited to hear what people had to say about our new experience.  We try our software out internally before releasing to the public, and at a company like Microsoft, there are plenty of intelligent people out there happy to share their opinions.  We did the best we could to satisfy our internal users, but overall the comments were extremely positive.

The comments are rolling into my inbox, and I must say it’s very rewarding to read how much our users like Smart DJ.  We had the right team, the right attitude, and a shared goal to create something compelling for our users.  For the software engineer in me, that is what this game is all about.

Beamforming and Speaker Arrays

Line array of loudspeakers for sound reinforcement

Loudspeaker arrays have been part of the sound reinforcement toolkit practically since electrical signals began flowing through moving coils attached to cones.  You have probably seen these public address arrays in large concerts and stadiums.  In this application, loudspeaker cabinets are mounted in close proximity.  Designing a line array typically takes the dispersion pattern of the transducer and cabinet into account while progressively angling successive cabinets in order to achieve maximum coverage with minimal interference.  Usually these types of arrays are all driven from a single signal; that is, the sound engineers attempt to uniformly spread the signal to the audience.

Interesting new applications arise when driving each transducer independently, however, under the control of a signal processor.  Imagine two speakers being driven by the same signal, radiating in the same direction.  At some points in the field, the signal from each speaker would arrive in phase, creating a higher sound pressure level.  But at some points in the field, the signals would arrive out of phase, canceling each other.  This concept can be taken even farther by adding more and more speakers.

With recent advances in signal processing technology, today’s computers can now control vast arrays of loudspeakers individually using the above technique creatively.  Using a process called beamforming, the signal processor can manipulate the input signal by shifting the phase of the signal, by adjusting the signal’s amplitude, and by introducing delays to the signal.  This allows the signal processor to harness the power of positive and negative interference to place sounds at specific points or areas in the field by adjusting phase, amplitude, and delay for each individual loudspeaker in the array.

Yamaha YSP-1100 Sound Projector, a commercial beamforming loudspeaker array

At a recent Audio Engineering Society meeting, Ivan Tashev, Jasha Droppo, and Mike Seltzer of Microsoft Research demonstrated some applications of this to the audience.  One demo was of a loudspeaker array, which consisted of a small linear array of inexpensive loudspeakers.  The demo played two songs through the array, and the signal processor used beamforming to place one song in a location to the left of the array and another song to the right.  The listener could begin listening in one area and hear only one song, then move to the other region to hear only the other song.

The second demo the team gave at AES was of a commercial application of this technology, the Yamaha Digital Sound Projector.  The Yamaha product contains 42 drivers, a signal processor, and a multichannel amplifier.  The signal processor has 5 different beamforming modes for various listening experiences.

The most straightforward mode is stereo mode, which simply mimics a typical music listening experience.  Another mode, called 5-beam mode, simulates the 5.1 surround sound listening experience by decoding the surround sound signal, encoding the 5 channels into 5 separate beams, and directing these beams to specific points in the room.  The rear channels are reflected off the sides and rear of the room.  Another interesting mode the Sound Projector has is the so-called “My Beam” mode, which projects the input to a specific listener but leaves the rest of the sound field miraculously quiet.

CNMAT's spherical array of 120 loudspeakers

These demos were actually quite impressive, even in a large meeting room.  The Yamaha product really sounded great, and I was surprised by the quality of the 5-beam surround experience.  The focused beam modes are interesting too, though I could still hear the program slightly at other places.  I would expect to see the price of these products decrease over time, and I suspect they will also improve in quality by learning room dynamics and adapting their programs to tailor for specific environments automatically (that is, if they don’t already).

Transducer array technology is an active area of research. Berkeley’s CNMAT created a spherical array of 120 loudspeakers, which sounds exciting for interactive sound installations and performances.  Microphone array technology explores identifying distinct sound sources by analyzing signals phase and delay correlations at each input point.  This technique can reduce noise for teleconferencing applications, for example.

Internal view of a Leslie speaker. The top and bottom chambers reflect the input by rotating a speaker or reflector panel.

I’d love to actually hear what the CNMAT speakerball can do.  Many years ago, I remember reading about some of the early computer music compositions created at IRCAM with their signal processing computers to place sounds in a field expressively, using many channels of audio.  For example, a cellist would play into a microphone, and the computer could rotate the sound around the field according to the amplitude of the performance.  Playing louder might result in the sound rotating faster around the room.

Thinking about that concept so long ago really broke down a barrier in my mind.  For the longest time, I wanted to buy an old Leslie speaker and replace the motors that spin the horns and reflector with robotic servo motors.  I imagined using analog function generators like envelopes and ramps to direct sounds around the room.  Now all this is possible without any motors whatsoever.

Conversations with the Wall

My group at Microsoft recently moved into a new facility called Studios West, a welcome change from our converted document retention cavern vacated by Safeco Insurnace.  One of the interesting new features of the office buildings in Studios West is a massive, interactive LED wall in the atrium of the four buildings.

We had been briefed about the installation as part of our orientation for the new building, so I knew it would be there.  But upon entering the building the first day, hearing the washy, occasionally dissonant ambient sounds that accompany the undulating lights made me think a little more about it.  How does it work?

SoundBeam, microphone, and camera

In addition to the array of LEDs hanging in the atrium, the Wall also features directed sound beams emitted by Sennheiser AudioBeam speaker arrays, a telescopic microphone, and a video camera, all mounted to the ceiling and pointed toward the entrance from the Commons.  My best guess on how it works is that it’s a Max/MSP/Jitter program, and being the inquisitive engineer and scientist types we are, there has already been interest in hacking the LED array to bend it to our will.  Honestly, I hope the system remains closed to us.  I’d much rather see what the artist had in mind on those lights than stock quotes or other random lapses of aesthetics.

So sure, the Wall is fun to play with and nice to look at, but is it frivolous?  Why does it matter?

Over time, I began to notice how many people would do the same thing I did in front of the sensors.  Hi, I’m here!  I’m waving my hands at you, Wall, and are you going to respond to me?  Groups tend to gather for lunch at the door where the sensors are, and the Wall is frequently a conversation piece.  People share how they think it works, what it makes them feel (usually one or two words rather than poems with flowery language), late night stories of how it changes when you walk from left to right but not the other way…

Atrium with the Wall in Studio C

It finally struck me that this is the point of the Wall.  We’re in the business of delighting our consumers based on the products, services, and experiences we offer, and we spend a lot of time crafting what we do.  The march of creating products is frequently long, difficult, and fast-moving.  Microsoft is also a huge company with peculiar parlance and culture, so sometimes it can be easy to get lost in the acronym soup of software jargon and process.  The Wall is there to spark those non-sequitur conversations, both inner and with our coworkers and friends.  The Wall, for me at least, places my mind in a lighter place and whets my creative appetite.  Can you really measure the impact of that?

And so it begins…

Please allow me to introduce myself.  I’m Tom Butcher, and I have kept interests in music, sounds, electronics, software, and the myriad grey areas between for as long as I can remember.  While I admit to blogging before in a past life (we all can maintain parallel histories on the Internet, it turns out), this time around I want to focus on new directions in and interesting ideas with music technology.

A bit about me: I’ve been a musician and composer for a count of years I hesitate to admit.  In 2003, the German record label Force Inc. Music Works released my debut album Style Encoding, and one can find other releases from me both before and after as other incarnations.

I began fusing music and technology together at an early age; I was fascinated by analog synthesizer sounds and tape-splicing music before I really understood what concept-oriented music was really about.  I always wanted a vocoder as a kid (check that one off the list, three times over), and some of my fondest memories of my childhood were the gratifying successes of building strange noise generators using a breadboard, some chips, and a healthy measure of curiosity.

When I entered the professional world, I developed interactive television technology for Microsoft, and since then (fast forward through late-1990s Internet startup land) I have been working on social media processing algorithms, recommendation engines, and other various projects for MSN and Zune, where I currently work.  (Plug: I think we have some really fun music features slated for the next Zune release, but I’ll save those for another post)

That said, here’s to a new beginning.  The world of music technology has exploded over the course of the past decade, and I have loved being part of the revolution.  New technology is enabling many more people to become more intimate with music than ever before.  Of course, many of us carry more music with us in our digital devices than we could ever hope to listen to in one sitting.  The abundance of so much music and so much new data presents new challenges and new opportunities.  Let’s dive in, shall we?