Non-human songs: IBM’s Watson technology learns to compose music 

An Austin-based IBM team has developed an AI program that can do what humans do -- write original songs. Should musicians be worried?

Posted May 25th, 2017

Musician Richard Daskas is trying to write a new song. 

He stands over his keyboard and plunks down a few chords. Then he stops. Janani Mukundan, who is operating a Mac laptop on a nearby table, asks him if what type of music he’s in the mood for. 

“Let’s do spooky,” she suggests, choosing the word “spooky” on her screen.

Less than a minute later, a song emerges from the laptop that sounds a bit like the soundtrack to a campy horror film. 

Daskas and Mukundan, who work at technology giant IBM’s Austin operations, have spent more than a year teaching the company’s famed Watson artificial intelligence platform how to compose music. 

Called “Watson Beat,” the software program can write an original song based on a specific mood or feeling. Watson Beat can deconstruct what you are playing, learn from it, and try to layer on top of that an emotion.

Although it’s still in the development phase, the technology’s creators say they hope Watson Beat can eventually be used by both amateurs and professional composers to generate unique music. 

“We think of Watson Beat,” Mukundan said, “as a creative assistant.” 

(The song above is a demonstration of Watson Beat using the song Jingle Bells and the “spooky” theme.)

String Quarter No. 4

Using computers to write music isn’t a new concept, of course. 

“As soon as computers came on the scene, people started using them to create music,” said Roger Dannenburg, a professor at Carnegie Mellon University. 

In 1957, two University of Illinois professors programmed a computer to compose music and it spit out an orchestral piece called “Illiac Suite,” which was later changed to “String Quartet No. 4.” It is considered to be the first song made by an electronic computer. 

Over the ensuing decades, using computers to compose original songs never really took off. Technology was used in music in obvious ways, such as using auto-tune, re-mixes or as part of basic album production. But these were still human-instigated tasks. 

Dannenburg said it’s still challenging to develop software that can compose and perform music that rivals what humans are capable of producing. “We still have a long way to go to figure out how music works,” Dannenburg said.

But in the past decade, as artificial intelligence technology has matured, startups and established companies like IBM have started developing software that could harness the power of machine intelligence to write music.

The universal language

Watson Beat was initially created by Mukundan, who was hired by IBM to work on research related to memory in drone technology. IBM has substantial operations in Austin, where it has a research and development office and its design center.

She was allowed to spend 20 percent of her time on passion projects, and began experimenting with teaching Watson music composition.  

Watson is a supercomputer developed by IBM that understands human language and can answer questions. It was originally created as a research project designed to beat humans at Jeopardy, which it successfully did in 2011.

“I’ve always been interested in this boundary between artificial intelligence and the arts, and how we use AI for computational creativity,” Mukundan said. “I thought, music is a universal language and everyone understands it. I should start with that.”

As Mukundan dived into the task of teaching Watson music theory and composition, she realized that the project needed a musician. 

So IBM hired Daskas, who has a master’s degree in music from Penn State University and comfortably plays four instruments, to work on Watson Beat.

Though he didn’t have any technical skills at the time, Daskas has since acquired some programming abilities. The Watson Beat project became so time-consuming that now Mukundan and Daskas both work on it full-time. 

By mid 2016, IBM was ready to demo Watson Beat, and media organizations took notice. The CBS morning show did a feature on Watson Beat, and Rolling Stone magazine wrote about how music producer and composer Alex Da Kid used Watson technology to write a song called “Not Easy.”

(Though technically Alex Da Kid did not actually use any of the Watson Beat samples in his song, according to an IBM spokesman.)

Ralph Barrera/American-StatesmanAt IBM's Austin offices, a two-person team have been busy teaching a computer how to write music. Called Watson Beat, this technology allows anyone, whether they are a musician or not, to compose a song. 


But IBM isn’t the only company working on an AI music composition program. There’s an explosion of software tools being marketed to composers and musicians, with varying degrees of reliance on artificial intelligence. 

Entire albums have even been released by computers. Luxembourg-based Avia Technologies said its AI program, “in collaboration with human artists,” put out an album called “Genesis” that can be used for commercials, films or video games

A France-based company called Hexachords is also touting an artificial intelligence music composition program, which the company plans to release in December. Founder Richard Portelli says one thing that what makes his software different is that it targets professional composers. 

“It’s not meant to be used by people who don’t know anything about music,” Portelli said.

Mukundan said Watson Beat differs from competitors because they used an “unsupervised” machine-learning technique that doesn’t require a large database. And their technology understands emotions, and can craft songs centered around a feeling.

Unlike Hexachords, the Watson Beat creators say their software can be used by professional composers, but also by anyone, giving music creation the democratizing effect that putting cameras on smartphones had for photography. 

Mukundan envisions a future when Watson Beat is integrated into a music streaming service like Spotify and a click of the Watson Beat button produces a newly composed song just for you based off your listening preferences. 

Ralph Barrera/American-StatesmanAt IBM's Austin offices, a  two-person team consisting of Janani Mukundan, left, and Richard Daskas have been busy teaching a computer how to write music. 

Can machines be creative?

Robots and computer automation technology have long been thought of as replacements for low-skilled jobs. We’re already seeing this in the form of self check-out lanes at grocery stores and airport check-in kiosks. 

But artificial intelligence, with machines capable of learning as they perform tasks, has broader applications, from betting on stocks to performing surgery. More interestingly, artificial intelligence is being deployed  in even the most creative human-centric tasks.

There are machines that are producing artwriting books and even crafting recipes. IBM produced an entire cookbook based off Watson-developed recipes.

But if humans are so proficient at creativity and get so much joy from it, why would they want to turn that over to machines?

Dannenburg said humans have shown an impulse to use technology to create music well before computers were even developed, noting that in 1200 AD people were trying to find ways to mechanize the ringing of church bells. 

“Humans have just an an almost innate fascination with automation and robots and machines,” he said. “The answer is because it’s there.”

Many of the artificial intelligence experts interviewed for this article insist that music composition software won’t replace humans. 

“I don’t see artificial intelligence crowding out or competing with human creativity,” said Elad Liebman, a doctoral student within the computer science program at the University of Texas who studies artificial intelligence and music.

Daskas, one of the Watson Beat creators, insists that their software should be viewed as an “enhancement.” 

“Being creative is using whatever tools and means out there at your disposal and adding your own spin on it,” he said. “If it’s a machine that generates a bunch of ideas that you may or may not like, then I’m all for it.”