I’m a software engineer now, which means I’m a programmer, and I write code in C++ mostly. My specialty is audio for games. Most recently launched Planetside 2 at SOE, and previously at Rockstar Games, I worked on Red Dead Redemption and Midnight Club. In 1997 I started with an interest in audio engineering, so I attended Full Sail, graduated with an AS in Recording Arts, worked in Jacksonville at a crappy recording studio for a year, then freelance recording local bands, at the same time went back to Full Sail to teach for 4 years. I didn’t make enough money to live on, so in 2004, I made a big gamble, and took out a huge loan to go back to school for game development, Full Sail again. Still paying that off, but it’s worked out very nicely. I enjoy being an audio code specialist in the gaming industry, which is rare.
How does programming come into play with sound engineering? Are you programming actions into the game?
Well, programming is needed wherever there are computers or digital audio devices in order to make them function at all. They get audio numbers in, do some math, and spit audio numbers out. But for games, it’s a little more abstract. There’s definitely a lot of finding action hooks, like where there’s something obvious, like an explosion, or a footstep, you find in code where that happens spatially, and you throw in some function to play a sound there, and you have the audio artists actually control what sound gets played, but you have to provide them with tools to do that. But, you need a lot more than that of course. Audio files at a simple level have embedded in them a bunch of numbers that when plotted on a graph make a sound wave. I to a limited extent write software to decode the audio files into a usable wave, though there’s a lot of “decoders” already prewritten for me to do that. So then you have a bunch of sounds that play at the same time, and they all need to be mixed together in real time…
And in some cases, they need to be processed individually, like to make some louder or softer depending on distance etc, or to make some sound muffled if they are behind a wall, etc. I guess I do all of that at some point. Though, a lot of what it is, is writing functions that do all the really complicated stuff, then writing less complicated functions that use those functions, and in the end making really simple interfaces that makes all the complicated audio code happen, to the point where you have some function that looks like PlaySound(“soundName”, position), or something like that, so at the highest level, you call PlaySound, give it the name of the sound to play, and where to play it in 3d space.
What software do you use to do the programming?
Visual Studio mostly. It simply has the best tools to edit code in the form of text, so it’s an easy choice. Integrated debugging is another advantage, as well as being able to use custom compilers for other platforms, like the PS3.
What software does your programming support?
The code written more often than not becomes directly part of the software that is the video game itself. The rest of the time, when not writing code for the game, I’m writing code for a tool application that helps produce, build, or edit the data that the game uses, in a simple example, there’s a tool that gives sound designers parameters to control for each sound that the game can play and how they behave. Parameters like volume, pitch, distance attenuation, etc. And behaviors, like, when this sound plays, does that sound play its child sounds, for example, there are different types of sounds. You can have a multitrask sound that plays all of its child sounds at once, or a randomized sound that chooses one of its children at random and plays one each time it’s called to play. Each child sound can also be a parent of more children as well, so using simple tools they can make pretty complicated behaviors without needing a programmer to adjust everything for them.
How many lines of code do you write each day?
Many thousands have been written for sure. How much per day varies between 0 and a thousand maybe. That said, quantity of lines of code isn’t a good metric for measuring productivity. In C++, if you want to, you can make a whole application fit into a single “line” of code simply by replacing all the return characters with spaces. Also, it’s usually good form to write the bare minimum of code. This has advantages for legibility, portability, maintainability, and sometimes efficiency (less work for the CPU).
You said you write in C++, isn’t that considered a sort of dated or bloated programming language these days? I’m not a programmer so I don’t know these things.
Where did you hear that? C / C++ is the opposite of bloated. For the most part, the language produces only the machine instructions you would expect to perform something you intended, and it compiles native code, which means it directly produces machine code that the computer can read, while other languages like Java or C# produce an intermediate code that’s interpreted by another layer of software, as well as implying it does a lot of things behind the scenes, essentially making it slower to run. Core games (meaning not casual games) still need to run as fast as possible, and that level of performance just isn’t possible in C# or Java (yet). The language has been around for a while, but that doesn’t mean much other than it’s bound to have a lot of people who are fluent in writing it. Also, the International Organization for Standardization (ISO) defines its standards, and more recently it’s been updated with newer conventions and paradigms to use that don’t sacrifice its ability to produce fast instructions. That said, C# and Java have their place for sure. It’s much easier to write Windows tool applications in C# for example, because it gives you a lot of tools that control the standard parts of Windows app, like scrollbars, and draggable things. And it’s not necessary that the tools run really fast because nobody plays a tool.
What do 3D artists use to make these games? What software? I hear many studios use their own custom programs.
Everything really, as far as I know. Some studios do indeed have nice custom tools. But most use 3DS Max or Maya with custom exporting tools attached. I would think this makes it easier for a newly hired artist to get up to speed and be productive quicker if they already know the software.
Another thing, I’m interested in the workflow. I’ve worked in ad agencies where there is a definitely hierarchy in how stuff gets put together. Usually there is a Creative Director and he gets a creative brief. He then sends that brief to a team or two of art director/copywriter teams. They brainstorm. Come up with ideas for that part of the campaign. Then it is presented to the technical team. The technical team then works with folks who don’t know anything about stuff works. I’m on that team usually, for some reason, we then work alone or as a team to make it come together. We did have a video/sound guy who did stuff as needed but he usually worked on commercials. He could do flash sound and some flash programming if needed as well. I was the interactive designer in this process. What is the team environment like? Do you work side by side the “creative” team or are you like the technical team in the interactive department or like the video guy?
It often seems chaotic where I’ve worked. There are designers that shape the specific aspects of a game conceptually. There’s a bit of hierarchy there I think. Though, usually the highest level concept for what game to make and what it’s like comes straight from the top, from upper management, presidents, studio directors, executive producers and the like. There’s an art team that makes all the visual assets. An art team often has an art director, and a few concept artists that draw what things should look like in 2D, in any medium. The 3D artists, I imagine, take it from there. Then there are the animators that make the models move and articulate. Then there are the programming teams. Often there are multiple teams. Where I’ve worked, it’s beneficial to have a group of programming generalists and specialists dedicated to a single game on a single team. Then there’s a (usually smaller) group of generalists and specialists that work on central technology that can be used by all games in production (present or future) by the studio. I’m currently in that group, specializing in audio. Last but not least, there’s quality assurance. I’m not sure if they are divided up into teams, as I think they are just allocated where they are needed. Typically in video games, QA is the least skilled workforce, not necessarily requiring much more education or experience than a high school diploma and an enthusiasm for games. These people play the games all day, some of them performing the same task in a game over and over and over again trying to make the game crash or exhibit some documented strange unwanted behavior. They look for flaws and report them. They report crashes when they occur, ideally with as much detail as possible, like what they were doing when it happened, how long were they playing, and can they reproduce the crash with a step by step process. QA, often under-appreciated, is absolutely necessary for development of a good game.
How did you get this training? I know you went to Full Sail, but when did you realize there was a niche in this type of work and how did you know Full Sail would be able to get you up to speed to jump in and have your niche carved.
After high school, I had little more than a very keen interest in audio production and video games. Choosing one or the other at the time, I chose audio production with the intent on making sound for film. My mom actually found Full Sail for me after we had a long talk about what I was going to do with my life after I essentially dropped out of technical college with no clear direction. I’m not sure how she found it, but she did, though now it would have been as easy as a Google search. The school of recording arts provided me with everything I needed to get into the recording industry. Critics may point out that the program was / is too easy, and I’d agree in that they don’t fail enough apathetic students from the recording arts program as they should, but if you’re driven to succeed, it’s got everything you need from a school if you just take advantage of it. So, I never realized making sound for film, and ended up falling into music production, which wasn’t a bad thing for me, as my interest quickly shifted to the music side after being further exposed. But, after years of low income, and realizing that there isn’t such a thing anymore as a full time job in the recording industry, I started entertaining the idea that maybe I should move into games, most of which sounded horrible before the mid 2000’s, through no fault of the sound designers really. I thought maybe they need programmers with audio experience. People who actually care about what the game sounds like, and people who actually understand the language of a sound designer. At the time, I think that was more or less, a guess. I’d never seen a game job opening posted for “audio programmer”. And when I went into the game development program at Full Sail, I was definitely the odd one out. The only one with any audio experience at all, and the only one I knew that had a clear goal in mind as a specialist. Being a bit more mature at the time, I did my best to absorb everything I could about the program, even though almost none of it was audio related at all. You just never know when you’re going to need to use that tool you learned about, even if it’s graphics. Suddenly after graduating, I started seeing job openings with the title “audio programmer”. Turns out I was right on time. How I knew Full Sail could get me up to speed in that sense, is that I started networking with some of the game development staff while I was teaching for recording arts, well before I even considered going back in as a student. I got a feel for what the program was like, and learned that it was much more challenging and rigorous than recording arts. Though I had never seen a line of C++ before, I knew I’d be able to figure it out if I wanted it badly enough, and I did.
So, you are sort of a pioneer in your field it seems. Is that correct? Do you have any advice for someone who’d like to get into gaming for a living be it design or coding?
Well, there are certainly others who have gone before me with equal interest for audio, but not many do it straight out of school. I’ve definitely learned a lot from other audio programmers since entering the field. Full Sail is great, but if I were to do it now, I’d consider just reading books.
I can’t offer advice very far outside the game industry, but… If you are a gamer, and have an interest in being in the industry, there are definitely a fair amount of user interface artists that mostly know Flash, etc. Flash is actually still used a lot in game UI. So they make content and use ActionScript, and make all the cool special effects on the screen when you click on things, etc. Some of my favorite UI work is in the DeadSpace series, the 3rd iteration comes out in a few days I think. Here’s a little youtube example: http://www.youtube.com/watch?v=xLF0DjFF9yo
UI is an extremely important part of a game, as menus and pages are still a reality when there’s so much to do in a typical game.
Midnight Club and Red Dead Redemption live only on Xbox 360 and PS3. But, Planetside 2 lives only on a Windows PC, and it’s free to download and play. Yes, totally free: http://www.planetside2.com/
My wife used www.Lynda.com for video tutorials on software a lot before also. It was great to get her started.
How did you get from point A to B?
I think I got from point A to B out of a combination of determination and being a spoiled lazy brat. I guess I saw that people indeed have cool jobs that I knew I could also do. So I did whatever I had to do, since I’m just a horrible, useless employee to anybody if I’m not happy with my day to day work. My work has to mean something to me, where I feel like I can look back on it and enjoy it as an observer.
How would a designer get into UI design?
I think users are driving UI’s most. And since UI’s are everywhere now, games, phones, everything really, smart futurists can somewhat project what they may look like for movies and games, in addition to wowing audiences.
What are the top projects you have worked on. Can you describe the work you did in each. For instance if it is a game could you describe the sounds or interactions that a user would make in order to hear your work. As a mostly visual person that might help me understand examples of what you do. I know what you do in theory but to know, if you do this and then that you will hear this and that because of this and that. Or maybe it too complex to describe in that fashion?
As far as my top projects, I really only have 3 that I contributed to significantly: Rockstar Games’ Midnight Club: Los Angeles and Red Dead Redemption, and Sony Online’s Planetside 2. I made some minor contributions to Rockstar’s core audio tech that got incorporated into Grand Theft Auto 4, Max Payne 3, and might still be there for the upcoming GTA5. If you play any of those first 3 games, and you hear sound at all, I was more than likely directly responsible for it. Not in a way that I design and create sounds, that’s for the sound designers. But I did make a great deal of the code that makes the sound play. There might have been a couple things in Red Dead Redemption I didn’t work on directly, but that was because I had a teammate at the time, which was great given that there is so much varied types of audio in that game it’s staggering.
I guess as an example. In Red Dead, you knock a glass bottle on a table, it falls over onto its side, rolls off the table, hits the floor and breaks. There is a lot of great object physics simulation code in that game that I had almost nothing to do with, but it doesn’t play sound, it just controls how the bottle moves and reacts to surfaces and other forces so that it looks realistic. The simulation is smart enough to know when an object collides with a surface by doing a bunch of math each frame to detect it. When it detects a collision, it handles it and controls some kind of reaction, usually dramatically changing the object’s velocity among other things. Well right about there is also a good place to play a sound. I would find places like these in the code that was already written by others (I call them hooks), and I insert a little bit of my own code to play some appropriate sound. For a collision hook like that, I gather all the data I can about that collision to figure out if a sound is appropriate, and if so, what kind of sound should play, and how loud should it play. By comparing the object’s velocity with a surface normal, I can determine if it should be an impact or a scrape or a rolling sound. If it should be an impact, how hard was it? Hard enough to play a sound? A soft sound, or hard sound? How loud? I would also hook in data driven thresholds for this stuff, so a sound designer can take control over them to a degree, and define how much force is required to play a soft or hard sound for each object. And when the physics code decides that the object just broke into pieces, there’s a hook for that, and I tell the object’s break sound to play. Of course, all kinds of other things happen. Guns fire, people talk, trains make all kinds of crazy sounds, creatures vocalize, horses walk and gallop, lightning strikes, stagecoaches go over bumps and bounce around, wind blows trees… it goes on and on. It all has sound, and I had to find all those hooks to make it happen, and I had to give the sound designers the ability to control what sounds play on what trees, etc.
What do you see is the future of the industry and where you see yourself going in the next five to ten years? Do you feel programmers who are interested in gaming and sound could make this type of transition? Do you think universities with standard Computer Science programs will develop more programs for gaming and sound as I don’t see those programs currently or do you think those programs will stay in specialty schools like Full Sail?
Where the industry and myself go in ten years’ time? I am not sure. I’m not usually very good at guessing those things, which is why I’m an engineer, and not an economist or something. I can at least say that computers will get faster, contain more memory, so graphics will look better, and sound quality will also improve. But, I am not sure there’s any lack of universities that have gaming focus. Game focus is a great thing to get students into an industry quickly when they know that’s exactly what they want to do (like me). But I don’t think they can take the place of more generalized computer science. If computer science curricula takes the game industry into account a bit, then I’m sure it’s done its job. But, really, in games, you just need to know how to write code that runs fast and efficiently, which should already be a major focus for any computer science degree.
The Physics of Sound
In games you knock a glass bottle down
It falls over onto its side, it rolls around,
It hits the floor and shatters
All three dimensional modeled matter
Created by object physics simulation code
All the sounds we hear
All the movement we steer
All the forces of realism we seek to mold
The simulation is smart
It knows when an object collides
It detects the path it should ride
It handles and controls all reactions
Is this a good place to play a sound
We find places like these in the code all around
Hints left by others to hook into
Clues left by others to pursue
Gathering all the data about collisions
What kind of sound should it make
How loud should it be when it breaks
By comparing the object’s velocity
Should it be an impact, a scrape or a rolling sound
Should it abound
Of course, all kinds of other things happen
Guns fire, people talk, trains breathe, creatures snort,
Horses walk, lightning strikes, stagecoaches roll
As the wind blows… it goes on and on.
Dedicated to Corey Shay Much respect….