Music Technology and Me, Part III

Once I upgraded to a professional synthesizer from my Casios, I was desperate to find a machine to help me realize complete songs.  Sure, the old Juno sounded great, but I could only play one sound on it at a time.  I also wasn’t able to control it using a computer, since it had the old-style Roland DCB buss instead of MIDI.  Again, I was in high school at the time, and let’s just say my job slinging pizza to the suburban Houston masses did not yield the kind of budget I would need to build a real studio.  Plus, I was driving by then and had to pay for non-musical things like car insurance and gasoline.

Ensoniq's performance sampler, the EPS. One unique feature of the EPS is that it could load samples into memory from disk while you played it. I'm not aware of any other sampler that does this. The band OMD actually used two Emulator IIs during one of their tours so that one band member could load samples while the other still played.

Through my voracious reading of trade literature for electronic musical instruments, I began to formulate my plan.  I knew that sampling technology was getting cheaper and cheaper, and like microprocessors, state-of-the-art samplers would blow away yesterday’s machines in terms of cost effectiveness and power.  I think the Ensoniq EPS was around at the time – a real sampler with a sequencer and a disk drive.  They sounded good, too.  The only problem was there was basically no way to find a cheap, used one.  Plus, my sampling appetite had already been aroused by my $100 Casio SK-1.

Ensoniq pretty much built its business in those days around doing what the other manufacturers did — only they were much, much cheaper.  For example, Ensoniq’s first sampler, the Mirage, did everything costlier samplers did at the time at a fraction of the cost: $1700.  Ensoniq’s first synthesizer, the ESQ-1 had the same appeal.  It had 8 voices of digital controlled oscillators with analog filters, a velocity-sensitive keyboard, and a simple sequencer for less than $1000.

The Ensoniq Mirage accelerated the hardware sampler wars. The first samplers, such as the Fairlight and Synclavier could cost tens of thousands of dollars. The E-mu Emulators clocked in just under $10,000. And then the Mirage comes along at under $2,000 to bring digital sampling to the masses.

Whereas synthesizers seemed not to depreciate steeply, samplers were another story altogether.  They are basically computers inside, limited by the same parameters that limit a computer: memory, bit depth, secondary storage, processing speed, and throughput (polyphony).  This meant that the advances in computing power in the 1980s and 1990s pushed the depreciation curves of hardware samplers even steeper⁽¹⁾.  And, as you might guess, this was good for me.  A used sampler with a sequencer would be cheap and exactly what I needed.

I wound up on the mailing list of Rogue Music, a musical instrument trading nexus in New York.  Rogue mailed its newsletter of used gear inventory every month or so, and I loved analyzing what instruments were in demand and where I could find values that fit my budget.  I found my first drum machine, a cheap Roland TR-707, this way a few months prior to becoming serious about a sampler.  One day the newsletter came, I found what I was looking for: an E-mu Systems Emulator II for about $800.

Ferris Bueller used an Emulator II to help make his case to stay home from school. It's a great idea, but I never tried it with mine.

Eight hundred dollars was a huge amount of money for me at the time.  I didn’t even have all of it then.  But I had an intense desire for the machine.  After all, that one machine would solve all my recording problems for the foreseeable future!  I knew about the Emulator from the music magazines, but I read a lot more about it after recognizing it in Ferris Bueller’s Day Off.

The Emulator cost about $8000 when it was released.  It has a basic sequencer and a fancy sampling scheme that squeezes more depth and clarity out of 8 bits than a typical 8-bit sampler (they claim it sounded more like 14-bit resolution).  It also had a floppy disk drive (or two, depending on the model) to store sounds.  The voice architecture was a lot like an analog synthesizer, except instead of using oscillators to generate basic tones, the Emulator played back sampled sounds.  In fact, it used analog filters with resonance, which meant it could sound warm and fat.  It also had 8 individual outputs for its voices which could be programmed flexibly.

The E-Mu Systems Emulator II: one of the best-sounding samplers of all time. If it weren't so large, I would love to have another one someday. But it is seriously large.. and heavy!

My grandfather again came to my rescue by chipping in $200 to help me buy the Emulator, and I still remember the day it arrived.  My brother and I plugged it in to the stereo in our living room and began loading disks of samples.  Piano?  Check.  Orchestra hits?  Check.  Weird animal noises?  You bet.  I grew to know that sampler inside out and backwards, and I squeezed all I could out of it.  I built a library of hundreds of disks, sampling from movies and television to my own other keyboards … anything I could think of, really.

I don’t have the Emulator anymore, but it definitely still has a spot in my musical upbringing.  The limits of the machine and my budget made me think really hard about how to maximize those 8 voices and the limited sampling memory in my songs.  Now, my studio has a few hardware samplers (yes, I still use them!), and they completely blow the Emulator out of the water in terms of power and flexibility.  But none of them quite sound like that fat, chunky Emu.

–

⁽¹⁾ This trend continued in such a dramatic way that the demand for hardware samplers essentially evaporated. The manufacturers raced to 16-bit machines, and then to cheap machines with 96kHz sampling rates, only to find that software sampling and virtual instruments were coming of age. There is still demand for sampling drum machines, like Akai’s MPC series, but one might argue that these are more about workflow than they are about sampling per se.

Music Technology and Me, Part I

As far back as I can remember, raw synthesized sounds always intrigued me.  I remember riding in the car with my parents in our red Oldsmobile listening to the radio, hearing sounds like vocoded voices and raw bass lines probably generated by a Minimoog.  Somehow, I had to be able to harness those sounds for myself.

My first electronic instrument, a Casio MT-45 preset synthesizer

For a birthday (I’m going to guess 8th) my parents bought me a Casiotone MT-45, my first electronic keyboard.  I wound up spending countless hours with it, but since it was a preset machine it never lived up to the promise of what I always wanted: a professional analog synthesizer.  But the old Casio did travel with me pretty much everywhere, and I eventually bought myself a Casio SK-1, the first cheap sampling keyboard.  I also eventually bought a Casio SK-5, which was able to store more than just one sampled sound at a time and had orange rubber drum pads on the front.

The Casio SK-1. This was about $100 when it came out and had the ability to sample with an internal microphone. It also had a rudimentary additive synthesis routine built-in.

My dad tended to frequent pawn shops around the time I was in junior high school, and one day he came home with two old cassette decks and some speakers for me.  I actually already had my own Radio Shack 4-channel mono mixer, which I was using to mix the three keyboards together.  Now, with the two cassette decks, I could overdub until my heart was content with my three cheesy keyboards!  Interestingly, the two tape decks had slightly different speeds, so unless I was careful to record and to play back from the same deck, the recordings would eventually decay in pitch over time.  If the keyboards were tunable, that wouldn’t be a problem, but of course they were not.  So there are some unintended microtonal aspects to my early recordings.

My love affair with music technology began to pick up steam in 1986.  In the 1980s, I was like a lot of other kids who lived in the suburbs in a few ways.  For example, my parents would sometimes need to do some shopping at the mall, so they would send me off to wander around while they ticked boxes off of their shopping lists.  In March, I happened to be at a bookstore and noticed the cover of Keyboard Managine: Jean-Michel Jarre in his studio, and a special flexi-disc of a song from the Zoolook sessions was inside and waiting for me!  The magazine came home with me, and I memorized every page.

Jean-Michel Jarre's Rendez-vous Houston, April 1986

One month later, Jarre came to Houston, Texas, my city, for a giant concert downtown.  He brought his vast array of studio synthesizers, the laser harp, and a gaggle of musicians to create an outdoor concert featuring the buildings of downtown as his backdrop.  There were fireworks, huge projections, lasers, and of course, synthesized music.  Jarre also had this semi-circular controller keyboard that had huge, translucent keys; the keys alit whenever he struck them.  I was 11 at the time, and this event had a huge effect on me.  I actually missed the concert that night (long story), but my great uncle gave me a copy of the VHS recording of the event.  I actually think that this was probably a better way of experiencing it, because of all the backstage footage.  Jarre making crazy sounds on an ARP2500 in an unfinished skyscraper gave me some early inspiration.  Yes, one day I will be making crazy sounds on a huge synthesizer in an unfinished skyscraper, too.  At least, that’s what I thought.

So goes my introduction to the world of electronic sounds.  I pined for a real analog synthesizer, and in fact I lied about my age to get an after-school job early.  My savings was devoted to building my studio at $3.85/hour.  It would take a while.

Meet in Kobe

International Society for Music Information Retrieval 2009 logo

At the end of October, I traveled to Kobe, Japan for the tenth International Society for Music Information Retrieval conference.  This was my second ISMIR, and to me this conference is different than many of the other academic conferences I attend.  For one, the work is inherently multi-disciplinary, drawing from such disparate fields as machine learning, musicology, information retrieval, acoustics, composition, statistics, and sociology.  But second, I think the crowd at ISMIR is different in a friendlier way.  Perhaps I feel this way simply because ISMIR aligns well with my own interests, or perhaps the composition of the participants and audience is varied enough to capture more than just a single vertically integrated field of expertise.

This year, Paul Lamere invited me to participate on an industrial panel during the conference, along with representatives from other companies working in the music information retrieval world.  I really enjoyed the conversation; representatives from smaller companies like Barcelona Music and Audio Technologies, The Echo Nest, and Last.fm were there to discuss MIR alongside large corporations like Yahoo, NTT, and Gracenote (I represented Microsoft).

The industry panel at ISMIR 2009. From left to right, Malcolm Slaney (Yahoo), Kunio Kashino (NTT), Keiichiro Hoashi (KDDI), Norman Casagrande (Last.fm), Oscar Celma (BMAT), Peter from Gracenote, Tom Butcher (me), and Paul Lamere (The Echo Nest)

I had two main points in the panel (though I’m sure I had opinions on a lot of other topics).  First, I frequently feel at academic conferences that even though much of the work can be technically elegant and solve problems with measurable success, sometimes it is difficult for me to get excited about slight increases in prediction accuracy (or whatever) when practical problems would prevent the research from ever being used in the real world.  For example, often work in recommendation engines measures success in a very closed way, even though that method is somewhat standard and accepted in science.

Usually with IR or machine learning problems researchers carve up their data sets and perform what are known as “hold-out experiments.”  Data from one partition are used to train an algorithm, whereas data from the remaining partition are used to test the output of the algorithm.  This works well for many cases, but in the realm of media recommendations I think the process breaks down.  These experiments fail to account for the effect of users interacting with the system and the time effects of recommendations as a whole.

Harbor Town in Kobe and Port Tower

It can be notoriously difficult to measure the true performance of a recommendation engine if you do not have the luxury of trying the system out on a large enough set of users, so I understand that the hold-out experiment is sometimes the only way one can measure an algorithm given the data set.  But too often I feel that researchers focus on satisfying F-measure curves or the bounds of the experiment and miss the mark when it comes to developing a system that real users love.

Perhaps that sounds too practical, for of course there is merit in science for science’s sake alone.  But the point I wanted to make in the panel is that we still don’t have a good way of measuring recommenders’ performance, and if someone really wants to have an impact with real users, sometimes simpler solutions that perform well and return reasonable results fast will blow away the fanciest learning algorithm.  Greg Linden sometimes discusses this idea on his blog.

The second point I wanted to make is that as far as the music industry goes, the democratization of music technology and the application of MIR-specific technology into mass-market products over the past decade presents a big opportunity for the scientific community and for industry alike.

Kobe at Dawn, from the window of the Hotel Portopia

Whether you love or hate Rock Band and Guitar Hero, they have changed the way we experience and consume music.  The recording industry has had an extremely difficult time adapting to technological advances, and as a result they watched their revenues collapse.  The labels are also desperate to sell you the same content you already have again and again by inventing new formats.  Well, Harmonix was able to do what the record labels could not by transforming the music experience.  By adding a little additional metadata to recordings people already know and by creating an immersive experience, many of us are willing to buy the same songs again and to enjoy them in this new way.

Someone in the audience asked if we on the panel could point to specific examples of MIR technology being applied to products.  The game products are some, and Microsoft Songsmith is another.  Companies like Microsoft, Last.fm, Pandora, and Apple are using MIR technology to drive data-driven music experiences with Zune Smart DJ and Genius.

There have also been huge advances in music creation technology over the past few years.  Apple placed GarageBand on every new Macintosh computer so the masses can create music without having taken a lesson.  Yamaha licensed MIR technology in some of their new music production tools in Cubase and for Vocaloid.  And, there are literally hundreds of brand new iPhone apps using MIR.

There’s an abundance of opportunity for MIR, and as long as people are entertained, enlightened, and satisfied by listening to or making music, we all have a chance to embark on some very interesting, fun work.

Thanks to: Paul Lamere, Justin Donaldson, Malcolm Slaney, Òscar Celma, and Norman Casagrande.

Unleashing Smart DJ

Last month, we released the new Zune HD device along with the Zune 4.0 suite of software and services.  Smart DJ is one of the big new features for this year’s Zune lineup, and it also happens to be one of the most rewarding features I have ever worked on at Microsoft.  Too often in large software organizations, different groups are isolated from each other, which leads to the customer’s needs or desires being lost along the way.

Glamour shot of the new Zune HD

With Smart DJ, we knew we wanted to provide an experience that lets people sit back and enjoy tracks and artsits that sound great together.  What began in a meeting in early 2009 fostered a collaboration across groups that ultimately delivered a compelling experience, despite organizational boundaries and disjoint schedules.  So, how did we get there?

Today’s Smart DJ feature actually began as a few distinct efforts.  I work on the services team, and our strengths include large-scale data processing, domain expertise with music, and really knowing about our customer.  We knew we could drive interesting listening experiences with some of our new technology investments.  The other side of the team was the group responsible for the Zune PC Client software, and they also know a lot about our users.  They have a lot of expertise with flashy user experiences, and we also knew that if we could use the user’s machine to help drive computations, we could distribute the overall load in the system.

It turned out we both were approaching similar experiences with different viewpoints and different code names.  On the service side, we need to support a variety of clients.  But in on the client side, they can finely tune the feature to their experience and provide a rich user interface to go along with it.  Earlier this year, our teams conferenced in a big room together, brainstorming and trying to figure out how to align our priorities.  We began with mutual respect and a common goal: create something compelling our users would love that helps them discover new music, enjoy the music they already have, and connects them with their music as fast as possible.

An example of the Zune Client playing a Smart DJ mix

What really struck me about this collaboration is how well it worked in the end.  We had schedules that were not aligned, and each team had their own sets of prioritized features.  There was a fair amount of give and take, which is to be expected.  But this time, it just felt different to me.  We all really wanted to provide the best experience possible.

Once we had the algorithms, the transport, and the client all working together, it was incredibly rewarding to let Smart DJ be our musical guide.  We were making improvements along the way until we shipped the code, and without revealing too much about it, you can expect to see even more improvements as time goes on.  Smart DJ gets better with age.

I was really excited to hear what people had to say about our new experience.  We try our software out internally before releasing to the public, and at a company like Microsoft, there are plenty of intelligent people out there happy to share their opinions.  We did the best we could to satisfy our internal users, but overall the comments were extremely positive.

The comments are rolling into my inbox, and I must say it’s very rewarding to read how much our users like Smart DJ.  We had the right team, the right attitude, and a shared goal to create something compelling for our users.  For the software engineer in me, that is what this game is all about.