>> David Plylar: Good afternoon everybody. Thank you so much for being here. My name is David Plylar with the music division at the Library of Congress, and this is our final event of our augmented realities videogame music mini-fest. We've had a great weekend, and we're so happy that so many people were able to come to these events. Today, we're going to finish out with composer and author Winnifred Phillips who will be speaking about her music and also just the whole process of composing videogame music and how that works. She will also have -- our bookshop has her book on composing game music and just that intersection there that will be on sale in the back, and she'll also be signing afterwards for those who are interested. So please join me in welcoming Winnifred Phillips. [ Applause ] >> Winifred Phillips: Hi everyone. Thanks for attending my lecture. I'm honored to have been invited to speak at the Library of Congress. As David mentioned, I'll be taking questions at the end of this lecture and if anyone would like to speak further the Library of Congress shop has kindly arranged for a book signing that's going to be taking place right over there. So please feel free to join me then. Is everybody ready? Let's get started. As David mentioned, my name is Winnifred Phillips, and I'm a composer of music for videogames. You might also know me if you've read my book, "A Composer's Guide to Game Music" published by the Massachusetts Institute of Technology Press. I've composed music for lots of games on lots of different gaming platforms ranging from phone games and simple hand-helds all the way up to state-of-the-art virtual reality systems. My best-known work includes music for games in five of the biggest franchises. Assassin's Creed, Total War, Little Big Planet, The Sims, and God of War. I'm here to talk with you about the interface between music composition and videogame design. Now this is the pinnacle of sophistication in terms of music implementation and games. So let's take a look at an example. In the video you are about to see, notice how the music changes as the main character moves from the road at the bottom of the screen all the way up to the top where he crosses a rushing river. [ Music ] So that was Frogger from Konami circa 1981 music by Takio [inaudible]. Let's take a look at the interactive music system. The game mentioned begins with a modo tempo musical idea that accompanies our little amphibian while he's crossing the road, and then it switches to a more energetic motif where [inaudible] across the river. Now it's simple but it was a major innovation in its time, and the basic interactive music technique in Frogger is still in use in modern games today including one of my own projects, Assassin's Creed Liberation developed by Ubisoft and just remastered and released for PC's, the PlayStation four, and the Xbox One. Let's take a quick look at what going from the road to the river sounds like in Assassin's Creed Liberation. [ Music ] So we've looked at a dynamic game music system applied to vastly different musical scores. With Frogger, we've seen a [inaudible] tune score from the early days of videogaming and then we've gone to the high-production values of this orchestral game music that I composed for Assassin's Creed Liberation which will be performed as a part of the Assassin's Creed Symphony World Tour. Just to give you an idea of how far game music has come, the Assassin's Creed Symphony will be having its premier performance at the Dolby Theater in LA in June this year with an 80-piece orchestra and choir. And you have to design construction for this music has remained remarkably similar to the early days of game design. It's a simple music system and it's made up of components that are assigned to specific tasks, a track for exploring the environment, and another for rowing down the river. It's the triggering of these components that makes the music interactive and the triggers depend on the actions of the player. If the player just decides to turn around and keep on exploring then the exploration music will continue and the rowing music may never begin. It's all up to the player. That's what interactive music is about. Of course, this was a simple example. From here things get a lot more interesting. As composers working in videogames, we spend a lot of time concerned with a few important questions. How do we structure this music? How many components? How many triggers? How do we mix it, and how can it best serve the action of the game? Over the course of this talk I'm going to be giving you an overview of the main forms of musical interactivity that have shaped the scores of modern games and vintage ones. I'll be showing you examples from some of my own projects where I can share with you what I've learned about interactive music while on the job, and we're also going to be looking at some examples from some other games that have touched upon interactive music as well so that we can get a more broad overview of the topic. Along the way we'll be discussing the functionality of each dynamic game music model and the way in which the player gets to interact with the musical content. But first, let's get a preview of what we're going to be covering today. Interactive music is currently split into two defined categories. The recorded music approach entails the use of audio files like wav, OGG, MP3, and other similar file formats. This category is currently sub-divided into two specific techniques, horizontal resequencing and vertical layering. And then there is the music data method in which the music isn't fixed into an audio file format at all but instead it's saved as performance data in the MIDI protocol or less often the MOD file format or it's worked into a generative based music system. I'll be spending the most time today on the recorded music approach but since that's used most heavily in modern games but also going to be doing an overview of the music data methods as well. Now let's take a step back and look at another simple but effective way of making music interactive by tasking the player with following into its sources. Let's listen to a track I composed for the videogame God of War from Sony Interactive Entertainment and watch one of the ways in which the player can interact with the music in a very direct way. >> There is safe passage through the deadly sands but only those who hear and follow the siren's song will discover it. You must find the siren's great host. Only they can guide you to Chronos, the titan. [ Music ] >> Winifred Phillips So in that gameplay scenario from God of War, the player follows the music which adjusts its spatial position then becomes either louder or fainter depending upon the player's proximity to the source. Kind of like a sophisticated version of Marco Polo. Let's take a look at another example of this technique from one of my more recent projects, the Dragon Front virtual reality game developed by High Voltage Software and available now for the Oculus Rift and the Samsung Gear VR. Dragon Front is a traditional high-fantasy adventure in virtual reality. Each game session is a self-contained match on a playing field loaded with fantasy monsters and machines. Now with all this in mind, the music for Dragon Front had to convey a suitably bold and dramatic style and this was especially important for the game's epic main theme. So I composed big Victoria's Anthem that played in full encompassing stereo as the opening logo sequence starts up. Now the theme music was designed to follow the player into the hub area, but bombastic music in the hub area could have been distracting. So at this point the music moves from a direct channel to the player and it takes up a position in the environment as though it were issuing from the speakers somewhere. Let's see how that worked. [ Music ] This becomes more fun in VR when players realize that they can actually turn around and locate the source of the music based on its position in the environment. Watch how players can look around in the Dragon Front hub area, locate the source of the music and actually turn it off if they want to. [ Music ] [ Music ] Putting music into the environment and allowing players to interact with it, that can be very powerful. Let's look at another example. Bebylon Battle Royale due to be released on multiple virtual reality platforms by developers Kite and Lightening. Bebylon is an outrageous and wacky bumper car style game with a really unique visual aesthetic. The whole premise of Bebylon hinges on immortal babies in over-the-top arena contests in a futuristic setting. Music during gameplay is represented by a group of in-game baby musicians. So the music originates from that source, and we're able to look over at the band and see the babies rocking out and playing their instruments in the VR world. Here is a video showing one of the games baby bands performing one of the tracks I composed for Bebylon Battle Royale. >> We are the grass knuckle heads. [ Music ] So as you can see from the baby bands of Bebylon, it's possible to have a game in which all of the music originates from a visible in-game source. In this case, an in-game rock band. Now what I've noticed is that players really enjoy this sort of direct tangible relationships with the music of a game. And the fun factor increases as the potential for player interactions with the musical score becomes more complex. As gamers, we like to feel as though we're participating in the way in which the musical score is composed and presented. We like the illusion that we're contributing to the creation of a piece of music and we can actually trace that enjoyment of interactive music all the way back to the 18th century. Mozart and Haydn may have been the first composers to create interactive pieces of music. They did this with a very clever game. It consisted of a large assortment of individual measures of music all numbered. Players rolled dice and depending on the numbers they rolled, the measures of music would be put into a particular order. Now the final result was a piece of finished music that seems to be completely original to whoever had rolled the dice. These musical dice games were enormously popular. Even in the 18th century people loved the sensation of power over a piece of music, of being able to interact with it, to change it into something new. Now this brings us to the first model of musical interactivity we're going to be talking about today. It's called horizontal resequencing. Now in a lot of ways, horizontal resequencing is a lot like that musical dice game that we were just talking about. The music for a horizontal resequencing model is recorded into an audio file format and then it's constructed in segments or chunks so that it can be arranged and rearranged into different orders. Essentially it can be resequenced. Now we call it horizontal resequencing because we're picturing time as a horizontal phenomenon. Flowing from left to right kind of like the notes on a staff of music. When we switch the order of musical chunks we're altering their position in time which we perceive to be horizontal. So now let's examine a fairly straightforward horizontal resequencing model as it was used in one of my projects, the Speed Racer videogame from Warner Brothers Interactive. In Speed Racer, players get to drive futuristic cars in over 350 miles per hour down hot wheels style tracks that twist into loop de loops in impossible curves. Every now and then players enter something that is called a zone mode during which they're suddenly invincible and they're traveling much, much faster than normal. The game used horizontal resequencing to enable music interactivity during the zone mode. Let's look at the component music files and some music production software so that we can see how those interactive chunks fit together. Every piece of racing music in the game had to be able to transition into zone mode. You can see here the main piece of racing music crossfaded into the zone mode track which was always 15 seconds long. Once the 15 seconds had elapsed, the main racing music would pick up again seamlessly. Let's listen to the music first by itself. [ Music ] Now let's see how that worked in the game. [ Music ] You can see that when the player exits the main racing gameplay and enters the zone mode, the music smoothly transitions into the zone mode track from the racing music. The two pieces of music were designed to be synced so that their tempos and their beats would align well throughout those transitions. Now how can these sorts of transitions be accomplished? One common way is in the placing of digital markers within the music files dropping them in locations that would make for good switching points from one interactive music file to another. We might want to drop a digital marker on every single beat making it possible to switch anywhere in a piece of music as long as the tempos and the measures were well-aligned, or we might drop markers at the start of every measure so that the music doesn't switch until the beginning of the next measure. We might even choose specific moments for the dropping of those digital markers and the music wouldn't switch until those designated points had been reached. Now the Speed Racer example included two chunks of music for dynamic switching but well this is a fairly streamlined example of horizontal resequencing. Let's take a look at an example of this method that has more files involved. This is Tron 2.0, a futuristic game set in the world of the Jeff Bridge's film about living computer programs in a digitized virtual world. One of the most interesting aspects of this videogame is its music system developed by Buena Vista Interactive and composer Nathan Grigg. In Tron 2.0, every music composition consists of a group of short chunks most lasting from 10 to 15 seconds and each encompassing about two to four measures of music. The segments could be triggered in different orders or sequences and each segment included an ambient reverb tale that would overlap nicely with whatever musical content would follow it. Let's take a look at how that worked. [ Music ] One of the advantages of formatting game music this way is the ability to alter the order in which musical events take place. Let's listen to those chunks now in a different order. [ Music ] Rearranging the order of musical content makes music feel less predictable which in turn makes it seem less repetitious. Repetition fatigue is a big problem in the field of videogame music and it stems largely from the nature of games themselves. Games are much longer experiences than those we might have in other forms of entertainment media like film and television programs. Players could spend a lot of time in any game location. So the music associated with that location has the potential to be repeated indefinitely. With a horizontal resequencing model like the one employed in Tron 2.0, the music can fluidly juggle its ideas into different sequential arrangements. And this has the potential to keep the music feeling fresh for much longer. Let's see how that system works during actual gameplay in Tron 2.0. [ Music ] When we're preparing music for an interactive system such as this horizontal resequencing model, we have to pay close attention to the volume levels of all of the interactive components of the system making sure that the overall perceived volume remains consistent and that the transitions feel smooth and natural. Any sudden jumps in oral intensity can jar the player and alert the player to the artificial nature of the music system so that's going to be something we need to avoid. Let's look at another situation in which horizontal resequencing is used, and this is one of the trickier situations for a game audio professional. Can a licensed track be adapted into this system? Now clearly if a composer is creating custom music for horizontal resequencing then the resulting composition is going to be structured in an ideal way. So is it possible to make a traditional linear music track work within a horizontal resequencing model? In the game Luminous from Q Entertainment, the music system integrated licensed music into an interactive framework that employed horizontal resequencing. Let's see how the audio team made that work. In Luminous, players manipulate falling blocks. The mechanic is somewhat reminiscent of Tetris. Line up the blocks to form the solid colors and they disappear awarding the player with a point bonus. Now the audio team at Q Entertainment made the music respond to those point bonuses in a really interesting way. Let's take as an example the track Shining by Mondo Grosso. It's an electronica track and it's built around lots of repeating patterns which actually makes it pretty ideal for this interactive music system. In the game the music would often loop a short section of the track repeatedly until the player had progressed sufficiently to trigger the music to move forward. We're about to listen to the track here. Let's pay particular attention to the lyrics. The first phrase is world of silence creep in sightless time. And the second phrase is [inaudible] of fatness, sleep in flightless mind. So let's hear how that sounds. [ Music ] Now during gameplay, the player would be manipulating those falling blocks and the music system would be playing this song and then suddenly it would start looping a short section of the song over and over and over again until the player had progressed sufficiently to trigger the music to move forward. Now remember that first phrase world of silence, creep in sightless time. That's the phrase that's about to repeat. [ Music ] Now let's see that interacted music system in action in the game. [ Music ] Using a licensed track, this horizontal resequencing music system gave players the opportunity to feel a burst of accomplishment when the music moved forward and progressed into something new. So now we've discussed the first model of music interactivity we're going to cover today. Horizontal resequencing. So let's move onto the second, vertical layering. What is it? Well, let's start with a comparison from the world of audio engineering, stem mixing. The recording of a musical performance that embodies some isolated part of the overall composition. Let's say the lead vocal in a pop track or the French horn section in an orchestral piece that by recording these elements separately we get to have enhanced control over those elements when we're mixing the final recording. Now taken to the extreme, we might separately stem every single instrument in an ensemble or conversely we might group the instruments into subsets that represent a certain percentage of the whole. The vertical layering technique shares a lot in common with the stem mixing process. In vertical layering a music composition is recorded into an audio file format but the whole performance isn't captured in a single audio file. Instead it's sectioned out into separate audio recordings that together embody the whole composition, and we call it vertical because we're picturing the simultaneous components stacked on top of each other in a vertical tower. Well like the notes of a chord on a staff of music. The music is structured so that the game's audio engine can essentially act like a virtual mixing engineer adjusting the volume levels of every single stem turning some on, some off. Now this is great because then the audio team can create triggering points in the game where layers get adjusted, activated or deactivated according to what's happening during the course of play. Because this music is subject to these kind of radical changes, it has to be structured so that its isolated components are satisfying to listen to by themselves. Now that's the main difference that separates vertical layering from stem mixing. When we stem in the traditional recording environment, we're intending to create a final track in which all of the stems are playing together, but that's not necessarily the case with vertical layering. Let's take a look at an example. When I was hired to join the music composition team for Little Big Planet two, the developers at Media Molecule had already created a pretty complex six-layer system for vertical layering. So it's a great example to show off the power of the vertical layering approach. Let's listen to a brief excerpt of one of the tracks I composed for Little Big Planet two. This is Victoria's lab and here all six layers are playing at once. [ Music ] Well you can hear there's a lot going on here. So let's break it down. Six layers each with distinct content. There is the drum layer. [ Music ] There is a layer with a small ensemble of cute instruments including Calliope, accordion, and beat-boxing. [ Music ] There's a layer with a small women's ensemble singing in counterpoint. [ Music ] There's a layer for an orchestral string section. [ Music ] There's a layer for a Soprano diva and a gothic organ. [ Music ] And finally, there's a layer for rock guitars and [inaudible]. [ Music ] What makes this track so flexible for interactive implementation is the broad range of contrasts between the content in each layer, and when you play them separately they each project a distinct emotional atmosphere. But that changes as the game engine begins to manipulate their volume levels activating and deactivating the layers and combining them into different subsets according to what's going on during gameplay at any given time. Let's listen to this track in action. Here are a few excerpts of gameplay from the Victoria's lab level of Little Big Planet two. [ Music ] With vertical layers, the music can grow more complex or it can simplify down into its most basic elements and still remain consistently true to a single musical idea. Now we're aware that we're listening to the same track but it seems to be magically morphing in sync with our actions. Here's another example. The videogame Portal Two from Valve. Composer Mike Marasce used vertical layers to sync the music with the player's actions. For instance, a low and slow music layer for when the player is navigating a series of platforms. [ Music ] And a high fast layer for whenever the player jumps. [ Music ] Now here's some gameplay from Portal Two. Notice how the slow first layer is joined by the fast second layer whenever the player jumps. [ Music ] There are two approaches to the construction and implementation of a vertical layering system, additive and interchange. In an additive model of vertical layering the layers are structured so that even if the game engine plays every single layer simultaneously and at full volume, the final result is still going to be pleasing to listen to. Now the Victoria's lab example from Little Big Planet Two was an example in the additive model, but it was a fairly complex design. So let's take a look at something that's a little bit simpler. When I was brought onto compose the music for The Maw videogame, the developers at Twisted Pixel Games wanted an interactive music system and they asked me to design it. So I put together an additive vertical layering system with three layers. Now since The Maw was a game about a one-eyed purple alien I kept the music broadly comedic. For instance, in the Luffer Lands level there is a layer for slapstick situations. [ Music ] There is a layer for just wandering around and exploring. [ Music ] And finally, there's a layer for when the Maw gets superpowers and starts firing lasers right out of his eyes. [ Music ] Since this is an additive composition, all three layers needed to be complimentary when they were played together. So let's watch some gameplay excerpts to see how this worked during the Maw videogame. [ Music ] Now the opposite of an additive model in vertical layering is a composition in which not all of the layers are designed to coexist. Instead they replace each other like interchangeable parts; hence the name the interchange method. Interchange construction is not as commonplace as additive because it's not as flexible. If some of the layers can't be played together that does reduce the number of interactive combination possibilities. However, we do encounter the interchange model occasionally and in one of my own projects, Spore Hero from Electronic Arts, I created an interchange dynamic composition in vertical layers. The Spore Hero videogame is a science fiction adventure and it begins with a menu system including three gameplay modes. The main menu, the battle mode, and the spore [inaudible]. Now for this menu system, I composed an interactive track in vertical layers according to the interchange model. Here is the main menu layer. [ Music ] Now here is the battle mode layer. [ Music ] And finally, here is the [inaudible] layer. [ Music ] The interactive music system was designed so that the layers could play together creating a consistent atmosphere that avoided choppiness. For instance, when going from the main menu to the battle mode, the music of the main menu would continue to play softly underneath the battle mode layer; however, the [inaudible] layer never coexisted with the music of the main menu. Those two layers were interchangeable but they were never simultaneous. Let's see how that worked in the game. [ Music ] So now that we've explored vertical layering, let's take a look at an issue that can be pretty problematic for our game audio pro. Can a linear non-interactive track be adapted into a vertical layering music system? Well we know that custom music works but what if the track wasn't written to be interactive? What would it take to make such a track work within an interactive system such as vertical layering? We can actually look to Hollywood for some inspiration here. Sometimes when a music supervisor on a film selects a pop track for licensing the supervisor will ask the artist to supply the music in its original stems. This is so that a custom mix can be created that's going to sit well within the final film. This is also a good strategy for a game audio designer trying to inject a little bit of vertical interactivity into a linear track. The Developer Queasy Games used this approach to integrate linear tracks into a vertical layering music system for their Sound Shapes game. So let's see how the audio team made that work. First, they connected with the artist Beck obtaining three unreleased tracks from him which then they used as interactive music for three stages of their game. Here's a short sample of one of those tracks. This one is called Cities. [ Music ] The audio team connected specific layers in the music with specific stages so that when the player progressed from one area of the game to the next, associated layers of the music would activate or deactivate. So let's take a quick look at how that functioned. [ Music ] So players of the Sound Shapes game got the chance to feel the burst of accomplishment as they progressed and uncovered different aspects of the music by virtue of its vertical layering music system. Now we've covered the two big methods that fall under the recorded music approach. Horizontal resequencing and vertical layering. Most games that aspire to incorporate interactive music usually opt for one of these two methods or even a combination of both. For instance, in Spore Hero I created tracks in both the horizontal resequencing and vertical layering models. However, there is a second category that we haven't covered just yet and it has both a long history in game design and an interesting future. Music data. When we're talking about music data we're specifically addressing music that hasn't been fixed into a recorded audio format, hasn't been rendered into a file like WAV, OGG, MP3 and so on. Instead, the music exists as a file containing a digital record of a musical performance and an accompanying sound bank consisting of instruments that will be triggered by that performance data. Now after hearing that description most composers and game audio pros are going to instantly think of MIDI, otherwise known as Musical Instrument Digital Interface. It's a data format in heavy use in music production and composition. It's been a mainstay in recording studios since the early 80s, but to understand MIDI better let's look at a comparison from the year 1917. [ Music ] Igor Stravinsky is well known for his masterworks for orchestra such as the Rite of Spring and the Fire Bird, but he also liked the idea of creating music for an instrument that didn't require musicians at all. Instead, preserving a record of a musical performance as data. We were just listening to a player piano perform Stravinsky's Etude Pour Pianola. Notice the piano roll, the perforations embody the instructions for note events in the musical composition. Today our modern MIDI software applications take their visual look directly from piano rolls. The note events are recorded as data and they trigger musical instruments or sound banks to emit their sound according to the instructions they're receiving. Now, here's where things get interesting. If the music exists as a set of instructions and a digital sound library then it's possible for a videogame's programming to manipulate those instructions directly fiddling with every single instrument and manipulating their performances in lots of different ways. Videogames are using MIDI extensively in the early 90s, and one of the most iconic examples of this is an early PC game called Monkey Island Two, LeChuck's Revenge. Developed by Lucas Arts, the game was released in 1991 and it featured the Lucas Arts patented music system known as iMuse or interactive music streaming engine which was used by the music composition team at Lucas Arts. The iMuse system could add melodies or subtract them, change which instruments were playing which parts, adjust the tempo and the key or even change on the dime from one track to another using seamless transitions. Let's take a look at this system in action as it switches from a general track for outdoor exploration to the music for the inside of a bar. [ Music ] This videogame soundtrack is now over two decades old; and yes, it certainly does show its age. Yes, the music system was revolutionary and it led to a lot of the interactive music innovations that shaped the scores of modern games. Ideas such as vertical layering and horizontal resequencing were first tried using MIDI, and the format itself hasn't completely faded away although its use has certainly dwindled over time. In the early 2000s, platforms such as the Nintendo DS made heavy use of MIDI and that led me to my first MIDI project, Shrek the Third for the Nintendo DS. Activision had already hired me to compose the music for the console version of the game, so when they also asked me to create an all new musical score for the Nintendo DS version I got the chance to learn a lot about creating a MIDI game score. For Shrek the Third DS, I was allotted only 1.5 megs of memory for the game's score. That was both for the MIDI data and the accompanying sound library. That's a very small memory budget. So it wasn't going to be possible to create a lushly atmospheric score with that. Instead I concentrated on a quirky mix of retro synth elements that could cope with the low-resolution sound files and the restrictions of the Nintendo DS hardware. For each track I composed a backing groove and three interchangeable melodies that would correspond with whatever character happened to be active at the time. Either Shrek, Puss in Boots, or Arthur. So here's an example of that system. [ Music ] Since the release of the Nintendo 3DS, MIDI scores have largely been supplanted by audio scores for Nintendo's handhelds, but MIDI hasn't entirely fled the scene. For instance some major Nintendo console games in the modern era still feature MIDI scores and this includes games from some of Nintendo's biggest franchises. Here's an interactive MIDI based score from the music team of the Legend of Zelda Twilight Princess. [ Music ] Also in recent years, some games have included tools that allow players to create music for their own levels. These games typically feature MIDI sequences which are MIDI based applications that record note events as data and then play them back. Players use these sequences to create music for their own levels and the original composers from the game's music composition teams would also use these same sequences to create music for the games. Usually characterized by an electronic sound pallet, this music can take advantage of the extensive adaptive possibilities that are inherent in the music data format. Let's see an example of this sequencer-based game music. This is from Sound Shapes. The music is Extraterrestrium by the artist known as I Am Robot and Proud. Notice how the music reacts to the gameplay. [ Music ] Now let's take a quick look at another example of this sequencer based interactive music data composition. This time from Little Big Planet Two. This is Wood Ringer by the artist Byan. Watch how the music system triggers the baseline to begin just as the player is swinging across the chasm. [ Music ] Now let's talk about another music data format that was tremendously popular in the early days of game development and it's still in use in modern games today. In early videogames, the mode or module format was popular for a lot of reasons. It has a lot of similarities to MIDI. Both file formats consist of music data designed to trigger a sound bank. However, the mod format allows the sound bank to be compressed into extremely small file sizes. Smaller than are possible with traditional MIDI. Plus, the sound bank can be incorporated directly into the Mod file and doesn't have to exist separately. And finally, Mod files can store their music data as chunks of numbered musical patterns and these can be reorganized and shuffled around as the audio team sees fit. One of the earliest examples of this can be found in the simple music interactivity of the videogame Pinball Fantasies created by [inaudible] and the team at Digital Illusions. Let's watch a little gameplay from Pinball Fantasies. Notice how the music changes quickly in response to the Pinball's trajectory and speed. [ Music ] The Mod format is still being used in modern videogame scores particularly in the casual games sector. A good example of this is Bejeweled three from Popcap Games. Composers Alexander Brand and then Peter Hodgebug used the mod format to incorporate layers and tempo changes into the music they composed for this game. In the game's lightning mode, the tempo of the music accelerated very slowly over time. So let's look at a video that shows a few slices of gameplay so you can see how the tempo changed. [ Music ] So now we move on to the final category in our exploration of music data in games. Let's return for a moment to those musical dice games we were talking about before. You remember, the rules allowed for the creation of the interactive music from a set of musical fragments that were shuffled according to the roll of dice. We've discussed a bunch of interactive music techniques today and most of them could be compared to those musical fragments from that 18th century dice game. But generative music is different. It actually shares more in common with the dice. Generative music is founded on the concept of indeterminacy. The insertion of chance into the way in which music unfolds and the ability to randomize attracts musical components in order to keep the listening experience constantly fresh and new. And to better understand this idea, let's start by examining one of the very earliest examples of generative music. In 1985, composer Peter Langston invented a system he called riffology for the opening title screen of the videogame Ball Blazer from Activision. Now riffology is a nice demonstration of the core principles of generative music in action. Mathematical probability resides at the very heart of the riffology system, a set of conditional statements that sort of resemble your classic if then. So let me give you a theoretical example of such an if then statement. Let's say we have a piece of music and it's in G major. The melody just sounded in F sharp, so what comes next? Well, that depends on how the probability rules have been defined in the system. So let's say if the melody has just sounded in F sharp then a G natural will follow it 80% of the time. The next note will be a D natural for 15% of the time, and an E natural will only sound 5% of the time, and there you have the core idea behind generative music. And also the reason why it's sometimes called algorithmic compositions since it's essentially a procedure for the calculation of musical variables. For Peter Langston's riffology, the system would choose from a set of 32 melody fragments and he would calculate which fragment was going to come next depending upon the conditional statements that Peter Langston had built into his system. Now in this way, the melody would spin out very unpredictably. So let's hear how that sounded during the opening title screen of the videogame Ball Blazer. Notice the foreground melody in the upper register carried by a high-pitched tone. [ Music ] Well that fast-paced melody in the high-pitched tone mode, it wasn't particularly striking but it was unpredictable and that's really the point. Generative music doesn't offer us a promise of memorable tunes but it does create melodies that avoid repeating themselves, and that addresses the issue of repetitions to take what we were talking about earlier. Now where generative music becomes interactive is in the involvement of the player in changing the variables that the algorithm uses to calculate musical probability. So let's look at a more modern game that includes a degenerative score. In one of the levels of the Electroplankton game from Nintendo, the player is tasked with changing the angle of the leaves, altering the trajectory of little red fruits that emit bell-like tones as they bounce off the foliage. Let's see how the music team made that system work. [ Music ] Electroplankton. Now as you could see, the result is a little reminiscent of windchimes which are essentially generative musical instruments themselves since their tones are generated by probability factors and chance. But let's take another look at a modern game that incorporated the generative score. In the Spore videogame from EA Maxis, players start as single-celled organisms in a process of simulated evolution. In this gameplay video you'll hear that the music team at EA Maxis created a foreground melody with a randomized quality a little reminiscent of Ballblazer, and you'll also notice how the music grows in complexity as the player progresses. [ Music ] So we started with Frogger, perky amphibian, and we're concluding with Spore, another lively aquatic creature. So essentially we've ended right where we began. You'll remember we began this talk to explore the interface between music composition and videogame design. We've looked at a lot of different systems. We've explored their structure and their uses. I've shared some of my personal experience as a composer of interactive music for videogames and we've also checked out a lot of other interactive game music systems from both the past and the present. We've looked at the recorded music approach checking out both horizontal resequencing and vertical layering, and we've also explored the music data method expressed via either MIDI, mod, or by virtue of a generative music system. Now all of these various forms of interactive music don't exist in isolation. Traditional linear music also has its place and there are times in which linear music really is the only viable solution particularly during narrative story-driven moments of a videogame. Linear and interactive music can work together. Since the 18th century, people have enjoyed playing games that give them the illusion of control over the structure of a piece of music. Players like the feeling of direct interaction with a musical composition. So when game composers are constructing an interactive music system, our goal is to increase that sensation of power and fun. To make the music feel reactive to the player's every move. Now when it's all said and done, our first responsibility is to meet the music needs of the game and entertain the people who are playing it. We strive towards musical excellence using whatever tools we have at our disposal. Now I hope you've enjoyed learning about the interface between music composition and videogame design. Thanks very much. [ Applause ] Thank you. Well now we've come to the Q and A portion, so if anyone has any questions please remember to share your name and speak into the microphone. It's going to be coming around. Also, if anyone would like to speak further after this, the Library of Congress shop has arranged for a book signing that's going to take place right over there so please feel free to join me then. So if anyone has a first question? >> So I was wondering, I know copyright's a big thing. But like you were saying that you usually have to take from other games even in the past. So I was wondering how do you approach that to where you can change that so that you can avoid having the copyright issues at least from your experience? >> Winifred Phillips Oh, well, copyright is a very interesting and important aspect of our work. I mean a lot of games are structured specifically around the idea of using licensed music, and I'm sure you can think of a lot electronic arts sports games in which music is introduced that for the first time you find a band that you really love by playing one of those sports games. That's been a really big avenue for young artists to make their start and it's been great for the music community at large. But that's something that especially a company like Electronic Arts is really serious about to obtain the rights and to compensate artists in a way that makes sense and is good for their careers. So the copyright is something that is very important. No game developer wants to find out that they've used a piece of music and then they haven't secured the rights appropriately so that they can't move forward with it. Particularly if you've fallen in love with a track and you've incorporated it into your game and oh God forbid, you've actually structured your gameplay around it and then you find out you can't use it. So yes, copyright is super important, and it's great that the Library of Congress has served the artistic community for so long in making sure that artists are protected and that this kind of creativity can continue in a way that satisfies both our audiences and also the artists making the music. Thank you. >> Hi, my name is Ed. What challenges do you face or conventions do you follow when you are mixing down say like a dynamic piece of music for a retail CD release or promotional material? How do you make that stay interesting to a listener if it's not as dynamic as it once was? >> Winifred Phillips Ed, that's a fantastic question. Thank you so much. You know, that's a really interesting process because a lot of the times when you're composing music for a game, you're essentially composing a lot of different bits and pieces. And you know that during gameplay they're going to be triggered by the progress of the player. So it's essentially a flexible fluid story that might happen. But you have all of these bits and pieces and afterwards when I'm creating a soundtrack and I've done that several times now for various games, I try to think about the most impactful course that the player might have taken through that level, through that piece of music and all of the bits and pieces that are associated with it, and then I will construct in my music production software, kind of like an ideal course. An ideal way to go through it, and I'll mix it so that it's a memory of the experience of playing that game. A lot of the people who buy these soundtracks are people who have played the games and they want to own the music because they want to relive their experience, and I take that seriously as when I'm doing this kind of work creating a soundtrack. I want to create music that's going to light up their memory and give them that kind of warm and happy feeling of having played that game. So that's what I'm thinking about when I'm pulling all of the interactive elements together. I want to create a sort of ideal listening experience. You're welcome. Does anyone have any other questions? >> Hi, my name is Vincent Vasana. My colleagues are -- we're from the Peabody Institute in Baltimore studying film and videogame scoring. So a couple of questions if you don't mind. A lot of us, I'm sure, have had the question, how did you kind of get into the industry, kind of break into the industry? What was your start? How did you go about all that? >> Winifred Phillips That's actually a really interesting question, Vince. My experience was a little unique. Before I got into the videogame industry I was working on a series called Radio Tales for National Public Radio. So that was very different. It was like dramatizations of classic stories for literature and mythology for the radio and that was a lot of fun. So that was my first job as a musician, as a composer. I was creating over 100 episodes of that series but I'd always been a gamer. I'd been a gamer ever since I was a kid, and it was a really important thing for me. I remember the one day when I was playing Tomb Raider and it just -- the lightbulb went off in my head about the idea of actually creating music for games. And so, I kind of got obsessed with it and I started working on trying to pursue it. Finding out who the people are at game studios and trying to reach out to them, letting them know the kind of work I had done before. I had no game credits. Of course that's the thing. It was an uphill battle, but my timing actually was very good in one instance. I reached out to a music supervisor at Sony Interactive Entertainment right when he was trying to build a music team for God of War. And I had a skill I could offer because I'm also a classically trained vocalist. So he was looking for someone who could create choral tracks for God of War. Any of you who have played it, you know that those big choral moments are such an important part of the God of War experience. So that was something that I could offer; and so, he asked me to meet him at E3 and we had our first meeting there. And at the same time when I was going to E3 to meet with him, I also met with a whole bunch of other people because I thought oh if I'm going to be going to E3, is everyone familiar? Electronic Entertainment Expo. It's a big convention in Los Angeles that's held every year for the videogame industry. So I thought if I'm going to this big convention I'd better also try to see if I can make some more work happen; and so, I was meeting with a lot of developers. And I also met with the developers of the game called Charlie and the Chocolate Factory at the same time. It was a tie into the Tim Burton film. And so, after that meeting ended and I came back, I did some demo music for both projects for both God of War and for Charlie and the Chocolate Factory, and I was hired for both at the same time. Yes. So I mean you can think that's a hugely divergent thing. One is like light and whimsical and full of imagination, and the other one is dark and brooding and full of angst. But it was kind of cool for me though because then I got to start my career with two very different tracks. And from then on, that's basically been the way my career has been. I have swung back and forth from let's say kind of a grand and dark orchestral project like Dragon Front or Assassin's Creed, and then I'll swing back to something light and airy like Spore Hero or The Maw. And I've gotten to do that ever since. It's been actually a lot of fun. But my experience has been unique. I can't think of a lot of other composers who started out like that. It's partly just how fortunate I was in my timing. And to answer your question Vince, it really does need you to just keep with it and to study the industry, to try to understand what's happening, to go to conventions and conferences like the Game Developer's Conference, the Electronic Entertainment Expo, and to just try to make yourself known and to reach out, and eventually your timing is going to be good and you're going to make that connection with somebody who's looking for what you can provide, and then you can just kick off your career from there. >> I have one more question if you don't mind. >> Winifred Phillips Sure. >> How often do you use virtual libraries in comparison to a real orchestra? Do you use virtual libraries more for mockups or do you use it more for scoring a game or something like that? >> Winifred Phillips Yes. That's a great question. Orchestral sample libraries are something I use fairly frequently and it depends very much on each project. Now there have been projects where I had used it just in the mockup stage at the beginning and then the project has gone on to record with a live orchestra. So that's always fun. But then there are other projects where the budget is just not going to accommodate that, and one of the things that's important to me as an artist is the option of being able to work with both large studios and also indy teams that don't have the same kind of budgets. It allows me to do a wide range of projects. They're exciting and creative. You get to work with young developers who are really hungry, and they have pie in the sky dreams. But then you can also work with studios where there's a large infrastructure and a lot of resources. So I swing back and forth with that too. I've worked with both the orchestral sample libraries and the live musicians. I find that both our difficult disciplines but very different in terms of their challenges to make an orchestral sample library sound satisfying and realistic. It requires -- and it's a minute attention to details and a really good understanding of how sample libraries work and how live musicians play so that you can approximate that sound in a way that's going to feel satisfying for listeners. And on the other hand when you're dealing with a live orchestra, you really want to be able to take advantage of the strengths of that medium and appeal to the expressiveness that a live orchestra or live soloists can bring. >> Thank you. >> Winifred Phillips You're welcome. Oh, do we have any other questions? Oh, over here. >> As someone who's played musical instruments but has never composed, if I were to try to make indy game or something and I wanted to make my own music, what kind of recommendations would you have? >> Winifred Phillips You know, that is actually something that's happened quite a bit, and I've seen development teams with the main developer also creating the music, and there have been some really interesting games that have been created that way. One of the things that's cool is if you are a musician that's creating a game you're kind of creating the game with the sense of how the music, it fits into the mechanics of the game since you're doing both. I think that's actually really neat. So that's something that I've seen done, but I do think that if you haven't composed before it might make sense to try to start doing some of that first before you try to bring those two elements together. I mean they're very different disciplines and you want to have a basic skillset that you're comfortable with before you start taking something that's hard in and of itself, music composition and then adding into it something else that's hard by itself, game design. You don't want to get overwhelmed. So the opportunity to just create music on its own or if you are familiar with any student teams or teams that are being involved in game jams, oh, for anyone who isn't familiar, a game jam is one of these events in which all of these game developers come together to create games on the fly really quick which is actually kind of a lot of fun because it becomes this very festive kind of party atmosphere and they all create these very, very quickly, and it's a way to be very creative and solve problems right on the spot. But it's also a fantastic opportunity to jump into a team right away as the composer and get an opportunity to just think about that part of the game, to create the music within the structure of a team. And I think you'd learn a lot about what goes into music composition for games by doing that first. Before you know you've put up like a thousand-ton weight on yourself of doing both at the same time. >> Thanks. >> Winifred Phillips You're welcome. >> Question. My name is Ira. I'm interested to know given the gaming consoles other platforms have grown so much in terms of their capacity. What limits do you feel are placed now or still on your budget and what's your response then in terms of strategies that you've [inaudible] to increase your expressiveness and what resources are available while stile remaining within the resources allowed? >> Winifred Phillips Great question. Yes, that's something that definitely becomes an issue from project to project. You know I honestly can't think of a single project I've worked on where budget doesn't become a factor even when the budgets are very large because there's always the opportunity to just have your ambitions explode beyond the boundaries that any game budget can accommodate. So we're always thinking about the logistics of it but since I work on a lot of different kinds of projects, I work with big teams and small teams, I'm constantly coming across the idea that the music budget isn't going to accommodate a more ambitious musical score. So at that point you sort -- you triage the situation. You look at it and you say, "How can I maximize the potential of a smaller amount of music so that it can provide coverage for a game and not become too repetitive, too annoying?" You don't want your music to become a negative. You always want it to be something people love. So there are lots of ways to do that. Sometimes the music needs to be broken apart. When I talked about vertical layering, the idea of music broken apart into its individual layers and then they can be used separately. So that makes one piece of music more flexible to cover a larger amount of time. It can morph and change and become more adaptive to what's going on and that way you can use that same piece of music for longer and it can still be satisfying. It doesn't feel repetitive. On the other hand, there is also the quite valid consideration of when music should settle back into silence to give the player room to absorb the moment when it should come in. Spotting a game and trying to determine where the music should be is also a way in which a small budget can accommodate the music resources. If you strategically place music in those positions where it's going to have maximum impact, where it's going to be meaningful, then you've used music smart, and you can still have a satisfying musical experience in a game without needing an enormous budget in order to accommodate it. So those are two projects -- two approaches that can address a problem but it's the continuing problem. Every game development studio wrestles with it at one time or another. I mean, we've all got the hope and the yearning to do something really special with music, and a lot of the times we can. And sometimes the restrictions can make us be creative in ways we couldn't have predicted. And then that is a way to grow as a development team or as an artist, as a composer. So these things, these challenges can serve to make us grow and become better. So if anyone has any further questions or if not, then I guess that we might wrap up here and thank you so much for coming to this talk. I really have appreciated. You've been really kind to me and thank you very much. [ Applause ]