The proverbial clean slate. Imagine filling this guy up with notes one at a time using a #2 pencil.

I started out writing music by scribbling on manuscript paper.  The staves were there, but no notes or clefs; those were scrawled in #2 pencil after I jammed out the outline of a song.  Of course, this was before I had any synthesizers or a computer.  I wrote all my music at the piano, note by note.  I always wanted a way to scribe and edit music faster, and that desire along with my love of electronic, synthesized sounds destined me for computer-based writing and recording.

Around the time I was using a Korg M-1 to write music, an unfortunate event happened.  I had all my hand-written music in a folder in the back seat of my friend’s car along with some CDs and clothes.  We left for a few hours to go to an event, and when we came back everything was gone.  All the music, and all those countless hours of work with paper and pencil simply disappeared.  They were my only copies.

Going Digital

This device was also not the best interface for creating music. However, it was really cheap and had a disk drive. So at least I could use it to store my songs.

After losing all my hard work in one fell swoop, I essentially never went back to writing music by hand.  I had moved on to writing everything on the Korg M-1, but I still didn’t have a computer.  So that meant whenever the Korg’s memory was full, I had to erase something already stored there to make room for something new.  Eventually I bought an extremely cheap Brother sequencer that had a 3.5″ disk drive.  This allowed me to save all my data from the M-1, erase, and start over without permanently losing anything.  While the Brother unit was technically a sequencer, after trying to write music on it I swiftly decided it was better as a disk librarian.  Somehow, I think even the old Roland MC-4s and digital control voltage sequencers would have been better than that crappy sequencer from the typewriter company.

Inevitable Progression to the Computer

Later in high school, my friend Wes had an Atari ST computer which had built-in MIDI.  I don’t remember what program we used to sequence on that thing.  But it was cheap, and it worked.  Some of our best compositions were written on that Atari, sequencing our Emu samplers and Roland synths.  We sampled everything in those days, including movies and our own other synths.  That was basically the only way we would get the rich, full sound of multiple sound sources since we only had a few synthesizers and drum machines to work with.

My first non-toy computer was the Apple //c, but unfortunately its cool, sleek design eliminated the expansion board buss.  So that meant that even though there were MIDI interfaces and sequencing programs out in the market, I had to wait.  The real breakthrough in my process happened once I had a PC clone running Windows 3.1.  I managed to scrape enough money together to buy a Mark of the Unicorn MIDI Express, which was a 1-unit rackmount device with 6 ports of MIDI input and output.  Using that and a cracked copy (yes, I  will admit it) of Steinberg Cubase, I was finally in full control of my compositions in the computer.  Cubase was MIDI-only at the time, so all my music was written on MIDI tracks in Cubase, which sequenced my entire studio.

The Atari ST series had built-in MIDI and also had rock-solid timing. My friends and I used it to record our first record.

In the early to mid 1990s, I had the computer and MIDI interface for writing and sequencing, the Emu Emulator II which did most of the work in my studio (even with only 8 voices!), my old Roland Juno-60, and a Roland TR-707 for drums.  The next pieces to arrive were the TR-808 and a Korg Wavestation SR.  I didn’t really have any effects processing then, and the old Mackie CR-1604 mixer mixed everything down to stereo.  This setup was modest, but I learned a lot about workflow and digital composition using this configuration.

My process remained essentially the same from those days in the 1990s through the early 2000s.  I began working closely with my friend Mike on a variety of projects, and we both recorded the same way.  We would use the computer as a composition tool, record our takes onto DAT in stereo, and at that point the song was done.  In all honesty, I think Mike was more of a jammer than I was, so perhaps the computer’s influence was more my style.  I loved how I could sequence pretty much anything and have full control over the song’s arrangement.

The Next Step: Digital Audio Workstations

As great as sequencing entire compositions using MIDI was, that process also had its drawbacks.  I began performing my music more, and a lot of the sounds I conjured in the studio were lost forever when the patches and memories of the synths forgot them.  Many of the best sounds were happy accidents, which meant that I couldn’t retrieve them even if I wanted.  Sometimes I would also want to go back and fix certain parts of the mix after the stereo recording was done, but I didn’t have the tracks anymore.

I started using multitrack audio in earnest around the time I was composing the music for my first album.  Cubase had the ability to record and play back multitrack audio probably in the late 1990s, but I never got around to using it deeply until 2000 or so.  ProTools had been around for a while by then, but all of a sudden it was possible for inexpensive computers to do the same work custom DSP chips did in Digidesign’s hardware and the prices really came down.

The top keyboard is the PPG Wave 2.0, and the bottom is its cousin that lives in the computer, the 2.V. Notice how similar the user interface is.

Around the same time, virtual instruments began to gain momentum.  I think the first virtual instrument (or VSTi) I used was an emulation of the old PPG Wave 2.3 published by Waldorf, the Wave 2.v.  I had an actual PPG Wave 2.3 by then, so it was fun to compare and contrast the quality of the real thing against the virtual version.  The verdict?  The virtual version sounded pretty good: not quite as present as the hardware PPG, but close enough.  Plus, it cost about 10% of what a real PPG would.

Going Too Far

I spent a lot of time collecting VST instruments, samples, and software.  I did this so much that I found I was spending more time managing software than actually making music.  At this point I decided to actually buy the software I used for making music and use only what I had legitimate licenses to use.  Part of this decision was motivated by wanting to reduce mental clutter, but part of it was also motivated by the fact that I realized I made my own living writing and developing software.  It felt hypocritical for me not to buy the software I used, especially as plans for my first solo album solidified.

While reducing software clutter, I also decided that since I had a decent collection of hardware that I should use it instead of move too much to the computer.  Pointing and clicking on knobs was too far abstracted from the tactile feel of the hardware.  So as I gradually scaled back my use of virtual instruments, I finally landed on my main workflow for writing and recording music today.  I had gone too far into the computer, and now it was time to strike a balance.

Today’s Workflow

Each recording and projet is a little different, so there is not one simple formula I use again and again.  But the tools and techniques I use have congealed into a cohesive process for me.  Here’s how it works.

I typically start by playing around with synths or drums.  Occasionally I will have an idea in mind before I get started, but not always.  For example, when producing a remix of another track I will strip down everything except the vocals to start working with the main structure of the song.  But most of the time, I just start playing.

All the audio in the studio comes in to the digital mixer, and the digital mixer has a direct connection over Firewire to the computer.  And all the control routings in the studio either originate with the computer or can be recorded and sequenced by the computer.  For those synthesizers with MIDI, I use a big MIDI patchbay to manage their connections to other devices and to the computer.  For those pieces that are control voltage only, I use a Kenton Pro-4 MIDI to CV converter as well as a MIDI/CV sequencer from Doepfer called the MAQ 16/3.  Additional control voltage lines can be sequenced from the computer using the MOTU Volta instrument, which connects to the rest of the studio via a MOTU 828 Firewire 8×8 interface.  The last bit of kit that is neither control voltage nor MIDI is a pair of drum machines, the TR-606 and TR-808.  These are synchronized with the rest of the studio using an old Korg KMS-30 MIDI to Sync24 converter.

This is a screenshot of a track I recently finished in Ableton Live. Lots of tracks!

Whenever I get an idea going that I want to build upon, I fire up the computer to play it for me.  With synths, I’ll play in what I want to record to the computer and build from there.  Sometimes, when I want modulation on the filter of a part, I’ll come back later after the track is laid out and use a knob controller like the Akai MPD-8 to sequence control voltage lines that are patched to the synth I want.

Drums are a little different.  With the drum machines, I usually build layers of 2, 4, or 8-measure patterns together on the machines I’m using.  That usually includes the 808, but lately I have also been using the TR-707 again and the MPC1000 for extra punch.  I don’t have enough inputs on the mixer to patch all the drum channels in at once, so when I’m sketching I will submix these and bring them in on a stereo pair.

Honing the Track

The next step after building up the layers of the song and defining the rough structure is to capture the audio as individual tracks in the computer.  With synths, vocals, and effects this is pretty easy.  I just hit record enable on the inputs I want from the mixer, and let the virtual tape roll.  The computer records each enabled part as an audio track, so I can either switch off the synthesizer (some of them get pretty hot) or just let it sit while the computer plays back the audio.  The digital mixer I use has the concept of record versus playback channels, so whenever the audio is recorded I toggle this mode on the channel to switch the input from analog to the computer.  Recording this way is great, because tracking parts in can be done quickly, and I don’t have to worry about the mix being different if the computer outputs were to go to another channel on the mixer.

This is a screenshot of the same track as above in the mastering stage. Note the ample metering; it's good to have a variety of ways to see the music when finalizing the recording.

Tracking in drum parts is a little different than synths because they are usually mixed together.  For this, I’ll record each drum part on its own or sometimes in layers of two.  Once all this is done, I can then distribute the drums to their own channels, busses, and effects routings if I want or mix them down in the computer to just a few channels.  I like to record changes on the drum machines live too, like mixing in or out sounds using the individual part volumes on the Rolands or muting and soloing tracks on the MPC.  Bringing a variety of patterns in makes it easy for me to mark them and cut them in the computer if I make arrangement changes later.

Once all the parts are in, I typically tweak the arrangement and create the final mix.  At this point, all the audio is being played back by the computer to the mixer.  Sometimes I’ll use some processing on channels in the computer before they come to the mixer, like EQ or compression on vocals.  All the effect sends are controlled by the mixer at this point as well.  Sometimes if I want to modulate pan or a send during the mix I will automate this in the mixer, but I don’t really do that very much.  I roll the sequencer and record the output from the mixer directly back into the sequencer, and then that’s it!  I have the raw mix.

Denouement: Finalizing and Archiving

This keyboard is what I wrote a lot of my early songs on, after the original manuscript theft debacle. It was okay, and I got to know it well. But there's really no comparison between writing using this and writing using a modern sequencer workstation.

Finishing up is a matter of saving patches on the effects units and mixer so I can come back to the mix if I need to.  I’ll export the mix audio to a file on the disk in the best quality I can, usually at 48kHz and 24bits.  Then, I pull the track into Wavelab or Cubase for mastering.  I give my tracks a little extra pop by smoothing the EQ out, giving the bass a nudge and adding a little more sizzle in the highs.  Then, I run that through a mastering compressor and limiter to squeeze a little more loudness out of the track without going overboard.  The final step is to export this processed audio as WAV, then MP3, and that’s it!

Coda: Goodbye Cubase, Hello Ableton

After all these years using Cubase, I think I am finally ready to let go of it.  Around the time Ableton Live version 3 came out, I made an investment in it for my live sets.  Now that they’re at version 8.2, Live is now a very powerful, full-featured tool for composition, DJing, and performing Live.  My fingers still know Cubase, but after getting to know Live a lot better, I don’t really use Cubase anymore.  The king is dead; long live the king!

Ensoniq's performance sampler, the EPS. One unique feature of the EPS is that it could load samples into memory from disk while you played it. I'm not aware of any other sampler that does this. The band OMD actually used two Emulator IIs during one of their tours so that one band member could load samples while the other still played.

Through my voracious reading of trade literature for electronic musical instruments, I began to formulate my plan.  I knew that sampling technology was getting cheaper and cheaper, and like microprocessors, state-of-the-art samplers would blow away yesterday’s machines in terms of cost effectiveness and power.  I think the Ensoniq EPS was around at the time – a real sampler with a sequencer and a disk drive.  They sounded good, too.  The only problem was there was basically no way to find a cheap, used one.  Plus, my sampling appetite had already been aroused by my $100 Casio SK-1.

Ensoniq pretty much built its business in those days around doing what the other manufacturers did — only they were much, much cheaper.  For example, Ensoniq’s first sampler, the Mirage, did everything costlier samplers did at the time at a fraction of the cost: $1700.  Ensoniq’s first synthesizer, the ESQ-1 had the same appeal.  It had 8 voices of digital controlled oscillators with analog filters, a velocity-sensitive keyboard, and a simple sequencer for less than $1000.

The Ensoniq Mirage accelerated the hardware sampler wars. The first samplers, such as the Fairlight and Synclavier could cost tens of thousands of dollars. The E-mu Emulators clocked in just under $10,000. And then the Mirage comes along at under $2,000 to bring digital sampling to the masses.

Whereas synthesizers seemed not to depreciate steeply, samplers were another story altogether.  They are basically computers inside, limited by the same parameters that limit a computer: memory, bit depth, secondary storage, processing speed, and throughput (polyphony).  This meant that the advances in computing power in the 1980s and 1990s pushed the depreciation curves of hardware samplers even steeper⁽¹⁾.  And, as you might guess, this was good for me.  A used sampler with a sequencer would be cheap and exactly what I needed.

I wound up on the mailing list of Rogue Music, a musical instrument trading nexus in New York.  Rogue mailed its newsletter of used gear inventory every month or so, and I loved analyzing what instruments were in demand and where I could find values that fit my budget.  I found my first drum machine, a cheap Roland TR-707, this way a few months prior to becoming serious about a sampler.  One day the newsletter came, I found what I was looking for: an E-mu Systems Emulator II for about $800.

Ferris Bueller used an Emulator II to help make his case to stay home from school. It's a great idea, but I never tried it with mine.

Eight hundred dollars was a huge amount of money for me at the time.  I didn’t even have all of it then.  But I had an intense desire for the machine.  After all, that one machine would solve all my recording problems for the foreseeable future!  I knew about the Emulator from the music magazines, but I read a lot more about it after recognizing it in Ferris Bueller’s Day Off.

The Emulator cost about $8000 when it was released.  It has a basic sequencer and a fancy sampling scheme that squeezes more depth and clarity out of 8 bits than a typical 8-bit sampler (they claim it sounded more like 14-bit resolution).  It also had a floppy disk drive (or two, depending on the model) to store sounds.  The voice architecture was a lot like an analog synthesizer, except instead of using oscillators to generate basic tones, the Emulator played back sampled sounds.  In fact, it used analog filters with resonance, which meant it could sound warm and fat.  It also had 8 individual outputs for its voices which could be programmed flexibly.

The E-Mu Systems Emulator II: one of the best-sounding samplers of all time. If it weren't so large, I would love to have another one someday. But it is seriously large.. and heavy!

My grandfather again came to my rescue by chipping in $200 to help me buy the Emulator, and I still remember the day it arrived.  My brother and I plugged it in to the stereo in our living room and began loading disks of samples.  Piano?  Check.  Orchestra hits?  Check.  Weird animal noises?  You bet.  I grew to know that sampler inside out and backwards, and I squeezed all I could out of it.  I built a library of hundreds of disks, sampling from movies and television to my own other keyboards … anything I could think of, really.

I don’t have the Emulator anymore, but it definitely still has a spot in my musical upbringing.  The limits of the machine and my budget made me think really hard about how to maximize those 8 voices and the limited sampling memory in my songs.  Now, my studio has a few hardware samplers (yes, I still use them!), and they completely blow the Emulator out of the water in terms of power and flexibility.  But none of them quite sound like that fat, chunky Emu.

–

⁽¹⁾ This trend continued in such a dramatic way that the demand for hardware samplers essentially evaporated. The manufacturers raced to 16-bit machines, and then to cheap machines with 96kHz sampling rates, only to find that software sampling and virtual instruments were coming of age. There is still demand for sampling drum machines, like Akai’s MPC series, but one might argue that these are more about workflow than they are about sampling per se.

My first electronic instrument, a Casio MT-45 preset synthesizer

For a birthday (I’m going to guess 8th) my parents bought me a Casiotone MT-45, my first electronic keyboard.  I wound up spending countless hours with it, but since it was a preset machine it never lived up to the promise of what I always wanted: a professional analog synthesizer.  But the old Casio did travel with me pretty much everywhere, and I eventually bought myself a Casio SK-1, the first cheap sampling keyboard.  I also eventually bought a Casio SK-5, which was able to store more than just one sampled sound at a time and had orange rubber drum pads on the front.

The Casio SK-1. This was about $100 when it came out and had the ability to sample with an internal microphone. It also had a rudimentary additive synthesis routine built-in.

My dad tended to frequent pawn shops around the time I was in junior high school, and one day he came home with two old cassette decks and some speakers for me.  I actually already had my own Radio Shack 4-channel mono mixer, which I was using to mix the three keyboards together.  Now, with the two cassette decks, I could overdub until my heart was content with my three cheesy keyboards!  Interestingly, the two tape decks had slightly different speeds, so unless I was careful to record and to play back from the same deck, the recordings would eventually decay in pitch over time.  If the keyboards were tunable, that wouldn’t be a problem, but of course they were not.  So there are some unintended microtonal aspects to my early recordings.

My love affair with music technology began to pick up steam in 1986.  In the 1980s, I was like a lot of other kids who lived in the suburbs in a few ways.  For example, my parents would sometimes need to do some shopping at the mall, so they would send me off to wander around while they ticked boxes off of their shopping lists.  In March, I happened to be at a bookstore and noticed the cover of Keyboard Managine: Jean-Michel Jarre in his studio, and a special flexi-disc of a song from the Zoolook sessions was inside and waiting for me!  The magazine came home with me, and I memorized every page.

Jean-Michel Jarre's Rendez-vous Houston, April 1986

One month later, Jarre came to Houston, Texas, my city, for a giant concert downtown.  He brought his vast array of studio synthesizers, the laser harp, and a gaggle of musicians to create an outdoor concert featuring the buildings of downtown as his backdrop.  There were fireworks, huge projections, lasers, and of course, synthesized music.  Jarre also had this semi-circular controller keyboard that had huge, translucent keys; the keys alit whenever he struck them.  I was 11 at the time, and this event had a huge effect on me.  I actually missed the concert that night (long story), but my great uncle gave me a copy of the VHS recording of the event.  I actually think that this was probably a better way of experiencing it, because of all the backstage footage.  Jarre making crazy sounds on an ARP2500 in an unfinished skyscraper gave me some early inspiration.  Yes, one day I will be making crazy sounds on a huge synthesizer in an unfinished skyscraper, too.  At least, that’s what I thought.

So goes my introduction to the world of electronic sounds.  I pined for a real analog synthesizer, and in fact I lied about my age to get an after-school job early.  My savings was devoted to building my studio at $3.85/hour.  It would take a while.

International Society for Music Information Retrieval 2009 logo

At the end of October, I traveled to Kobe, Japan for the tenth International Society for Music Information Retrieval conference.  This was my second ISMIR, and to me this conference is different than many of the other academic conferences I attend.  For one, the work is inherently multi-disciplinary, drawing from such disparate fields as machine learning, musicology, information retrieval, acoustics, composition, statistics, and sociology.  But second, I think the crowd at ISMIR is different in a friendlier way.  Perhaps I feel this way simply because ISMIR aligns well with my own interests, or perhaps the composition of the participants and audience is varied enough to capture more than just a single vertically integrated field of expertise.

This year, Paul Lamere invited me to participate on an industrial panel during the conference, along with representatives from other companies working in the music information retrieval world.  I really enjoyed the conversation; representatives from smaller companies like Barcelona Music and Audio Technologies, The Echo Nest, and Last.fm were there to discuss MIR alongside large corporations like Yahoo, NTT, and Gracenote (I represented Microsoft).

The industry panel at ISMIR 2009. From left to right, Malcolm Slaney (Yahoo), Kunio Kashino (NTT), Keiichiro Hoashi (KDDI), Norman Casagrande (Last.fm), Oscar Celma (BMAT), Peter from Gracenote, Tom Butcher (me), and Paul Lamere (The Echo Nest)

I had two main points in the panel (though I’m sure I had opinions on a lot of other topics).  First, I frequently feel at academic conferences that even though much of the work can be technically elegant and solve problems with measurable success, sometimes it is difficult for me to get excited about slight increases in prediction accuracy (or whatever) when practical problems would prevent the research from ever being used in the real world.  For example, often work in recommendation engines measures success in a very closed way, even though that method is somewhat standard and accepted in science.

Usually with IR or machine learning problems researchers carve up their data sets and perform what are known as “hold-out experiments.”  Data from one partition are used to train an algorithm, whereas data from the remaining partition are used to test the output of the algorithm.  This works well for many cases, but in the realm of media recommendations I think the process breaks down.  These experiments fail to account for the effect of users interacting with the system and the time effects of recommendations as a whole.

Harbor Town in Kobe and Port Tower

It can be notoriously difficult to measure the true performance of a recommendation engine if you do not have the luxury of trying the system out on a large enough set of users, so I understand that the hold-out experiment is sometimes the only way one can measure an algorithm given the data set.  But too often I feel that researchers focus on satisfying F-measure curves or the bounds of the experiment and miss the mark when it comes to developing a system that real users love.

Perhaps that sounds too practical, for of course there is merit in science for science’s sake alone.  But the point I wanted to make in the panel is that we still don’t have a good way of measuring recommenders’ performance, and if someone really wants to have an impact with real users, sometimes simpler solutions that perform well and return reasonable results fast will blow away the fanciest learning algorithm.  Greg Linden sometimes discusses this idea on his blog.

The second point I wanted to make is that as far as the music industry goes, the democratization of music technology and the application of MIR-specific technology into mass-market products over the past decade presents a big opportunity for the scientific community and for industry alike.

Kobe at Dawn, from the window of the Hotel Portopia

Whether you love or hate Rock Band and Guitar Hero, they have changed the way we experience and consume music.  The recording industry has had an extremely difficult time adapting to technological advances, and as a result they watched their revenues collapse.  The labels are also desperate to sell you the same content you already have again and again by inventing new formats.  Well, Harmonix was able to do what the record labels could not by transforming the music experience.  By adding a little additional metadata to recordings people already know and by creating an immersive experience, many of us are willing to buy the same songs again and to enjoy them in this new way.

Someone in the audience asked if we on the panel could point to specific examples of MIR technology being applied to products.  The game products are some, and Microsoft Songsmith is another.  Companies like Microsoft, Last.fm, Pandora, and Apple are using MIR technology to drive data-driven music experiences with Zune Smart DJ and Genius.

There have also been huge advances in music creation technology over the past few years.  Apple placed GarageBand on every new Macintosh computer so the masses can create music without having taken a lesson.  Yamaha licensed MIR technology in some of their new music production tools in Cubase and for Vocaloid.  And, there are literally hundreds of brand new iPhone apps using MIR.

There’s an abundance of opportunity for MIR, and as long as people are entertained, enlightened, and satisfied by listening to or making music, we all have a chance to embark on some very interesting, fun work.

Thanks to: Paul Lamere, Justin Donaldson, Malcolm Slaney, Òscar Celma, and Norman Casagrande.

Glamour shot of the new Zune HD

With Smart DJ, we knew we wanted to provide an experience that lets people sit back and enjoy tracks and artsits that sound great together.  What began in a meeting in early 2009 fostered a collaboration across groups that ultimately delivered a compelling experience, despite organizational boundaries and disjoint schedules.  So, how did we get there?

Today’s Smart DJ feature actually began as a few distinct efforts.  I work on the services team, and our strengths include large-scale data processing, domain expertise with music, and really knowing about our customer.  We knew we could drive interesting listening experiences with some of our new technology investments.  The other side of the team was the group responsible for the Zune PC Client software, and they also know a lot about our users.  They have a lot of expertise with flashy user experiences, and we also knew that if we could use the user’s machine to help drive computations, we could distribute the overall load in the system.

It turned out we both were approaching similar experiences with different viewpoints and different code names.  On the service side, we need to support a variety of clients.  But in on the client side, they can finely tune the feature to their experience and provide a rich user interface to go along with it.  Earlier this year, our teams conferenced in a big room together, brainstorming and trying to figure out how to align our priorities.  We began with mutual respect and a common goal: create something compelling our users would love that helps them discover new music, enjoy the music they already have, and connects them with their music as fast as possible.

An example of the Zune Client playing a Smart DJ mix

What really struck me about this collaboration is how well it worked in the end.  We had schedules that were not aligned, and each team had their own sets of prioritized features.  There was a fair amount of give and take, which is to be expected.  But this time, it just felt different to me.  We all really wanted to provide the best experience possible.

Once we had the algorithms, the transport, and the client all working together, it was incredibly rewarding to let Smart DJ be our musical guide.  We were making improvements along the way until we shipped the code, and without revealing too much about it, you can expect to see even more improvements as time goes on.  Smart DJ gets better with age.

I was really excited to hear what people had to say about our new experience.  We try our software out internally before releasing to the public, and at a company like Microsoft, there are plenty of intelligent people out there happy to share their opinions.  We did the best we could to satisfy our internal users, but overall the comments were extremely positive.

The comments are rolling into my inbox, and I must say it’s very rewarding to read how much our users like Smart DJ.  We had the right team, the right attitude, and a shared goal to create something compelling for our users.  For the software engineer in me, that is what this game is all about.