Teaching Video Game Sound

Balancing Technical Know-How with Sonic Creativity

Tate Carson
Assistant Professor of Digital Sound Design
Dakota State University

Introduction

  • Trace the evolution of game audio education
  • Discuss best practices in game audio education
  • Describe DSU's course and share project-based learning models
  • Encourage course adoption
  • Show fit within a music tech program

Advice from the literature

Curriculum Standards

  • No established curriculum or standards for game audio education (Onen, Stevens, and Collins 2011; Bragança 2015; Cunningham and Smith 2015; Sarisky 2011; Hug 2007)
  • Programs vary considerably between institutions
  • More work needed on dedicated programs meeting industry needs (Onen, Stevens, and Collins 2011)

Interdisciplinary

Onen, Stevens, and Collins 2011

  • Intersection of arts and computer science.
  • Requires diverse academic skills.

Bragança, 2015

  • Interdisciplinary demands.
  • Need for academic collaboration.

Interactive Audio Special Interest Group (IASIG) 2011

  • Fusion of separate disciplines.
  • Multiple roles in the field.

Sarisky and Pennycook, 2009

  • Proposed interdisciplinary degree.
  • Industry needs for diverse skills.

Core principles

  • Onen, Stevens, and Collins 2011
    • Programs should teach core principles, not specific tools
    • Game industry is driven by technological innovation

Soft skills

  • Industry emphasizes technical skills and "soft skills" (Onen, Stevens, and Collins 2011; Bragança 2015)
  • Recommend recreating industry pressures to nurture soft skills (Bragança 2015)

Curricula

  • Core Courses
    • Core courses develop skills like recording, editing, programming (Onen, Stevens, and Collins 2011; Sarisky and Pennycook 2009)
    • Common first year allows understanding other areas (Onen, Stevens, and Collins 2011)
  • Specialized tracks
    • Tracks in areas like composition, sound design, audio programming (Sarisky and Pennycook 2009)
  • Curriculum should foster teamwork on collaborative projects (Onen, Stevens, and Collins 2011)

Project-based learning

  • Experiential, project-based learning encouraged (Onen, Stevens, and Collins 2011; Hug 2007)
  • Team projects allow reflecting on roles, developing capabilities (Onen, Stevens, and Collins 2011)
  • Ongoing journaling and portfolio development suggested (Onen, Stevens, and Collins 2011)

Industry Feedback

  • Onen, Stevens, and Collins 2011
    • Ongoing feedback from industry and updating curricula is important as game audio practices evolve
    • More pedagogical materials need to be developed to support teaching

DSU's Sound Design for Games

Course Structure

  • History and evolution of game sound
  • Sound design specifics for interactive games
  • Audio implementation techniques
  • Group project with Game Design

Game Audio History: Weeks 1 -3

Week 1 Week 2 Week 3
Adaptive, reactive audio in emergent gameplay Innovations: sampled sound, MIDI, streaming music, 3D audio Types of sounds, editing techniques, team roles
Evolution from Flash to HTML5 web games Game audio history with advances in graphics and computing Workflow for game sound designers
Link game design concepts like interactivity to audio needs Iconic games and their innovative audio tech usage Audio file formats and sound libraries

Exercise 1: Untitled Cat Game

  • Source or create audio files based on specifications in an assets list
  • Follows lessons on workflows, asset lists, and editing techniques.

Exercise 2: Horror UI Sound Effects in Reaper

  • Linear sound design project to get students comfortable with creating assets for a game
  • Edit sounds to match visuals, use advanced editing techniques.

Voice Over: Week 4

Key Points on Voice-Overs

  • Importance: Human voice enhances game immersion.
  • Evolution: From synthesis to recorded dialogue.
  • Functions: Diegetic and non-diegetic roles.
  • Roles: Involves producer, writer, and actors.

Exercise 3: Voice Over

  • Recording: Capture high-quality vocals.
  • Talent: Get a good performance.
  • Editing assets: Prepare individual sound files.

Practical sound design

Project 1: Game scene sound design

  • Objective: Redesign sound for a 30-second game cutscene.
  • Focus: Layered sound effects, no background music.
  • Techniques: Use class-taught sound design methods: layering, pitch shifting, effects etc.
  • Narrative: Shape emotional experience through sound variation.

Week 6: Interactive Music Composition

Key Points

  • Not a Composition Class: Focus on adaptive techniques: Vertical remixing, horizontal sequencing.
  • Music Team Roles: Director, producer, composer, engineers.
  • Music's Role: Sets mood, establishes setting, adapts to gameplay.

Exercise 4: Music Exporting

  • Objective: Hands-on with exporting music assets.
  • Tools Used: Reaper's export capabilities.
  • Techniques Applied: Vertical re-orchestration, linear cues.
  • Workflow Skills: Batch exporting, naming conventions.
  • Real-World Application: Skills for assembling interactive scores.

Weeks 7 - 12: Audio Implementation and Wwise

  • Middleware Choices: Wwise or FMOD
  • Initial Setup: Unity and Wwise Integration
  • Advanced Techniques: Sound Positioning & Triggering

Project 2: Unity and Wwise Integration

  • Objective: Equip students with ability to complete Wwise-Unity integration and implement spatial audio.
    • Unity Steps: Create 3D project, add cube, and set up Wwise components.
    • Wwise Setup: Import a loop sound, create an Event, enable 3D positioning.
  • Learning Goals: Master Unity/Wwise setup, sound import, Event triggering, and basic 3D audio.

Final Project Description

  • Roles: Sound Designer, Audio Implementer, Composer
  • Game Choice: Game Design Class, Open-source, Unity Demos
  • Proposal: Serves as Grading Rubric
  • Class: Last 3-4 weeks open lab

Final Project Challenges

  • Overview: challenges from group dynamics to technical difficulties, that need to be addressed for successful completion.
  • Time Constraints: Students often feel the pinch of time, both for mastering Wwise and C# basics and for delivering a polished, sound-rich game.
  • Game Selection: Finding the right game for students to work on is tricky.
  • Debugging Difficulties: With students working on different games, it becomes a complicated task for the professor to troubleshoot and debug each project, given the unfamiliarity with the individual games.

Possible curriculum updates

  • Reduce time on game audio history
  • Focus more on game music composition
  • Streamline audio implementation setup
  • Refine final project guidelines

Course Adoption?

  • Curriculum enrichment
    • Expands curriculum & skills
    • Career prep in gaming
    • Interdisciplinary learning
  • Universal techniques
    • Layering, Sound Palettes
    • Narrative Flow, Documentation

Other Applications

  • Film & TV: Dynamic Mixing, Spatial Audio
  • Podcasting: Interactive Storytelling, Layering
  • Music: Algorithmic Composition, Virtual Concerts
  • Advertising: Adaptive Sound
  • VR/AR: Spatial Audio
  • Theater: Real-Time Sound Design
  • Museums: Interactive Installations
  • Automotive: In-Car Entertainment

References

Bragança, Ricardo. 2015. “Modding Game Audio for Education.” Journal of The Audio Engineering Society, May.
Brown, Andrew R. 2016. “Game Technology in the Music Classroom: A Platform for the Design of Music and Sound,” January, 122–33.
Cunningham, Stuart, and Richard J. L Smith. 2015. “Interdisciplinary Experiences of Teaching Audio for Games.” In Proceedings of the 26th AES Conference: Audio Education.
Donnelly, Cormac. 2015. “A Proposal for Specific Industry Accreditation for Game Audio Education.” In Proceedings of the 26th AES Conference: Audio Education.
Interactive Audio Special Interest Group (IASIG). 2011. “Game Audio Curriculum Guideline.”

References

Hamilton, Robert. 2015. “Designing Next-Gen Academic Curricula for Game-Centric Procedural Audio and Music.” In Audio Engineering Society Conference: 56th International Conference: Audio for Games.
Hug, Daniel. 2007. “Game Sound Education at the Interaction Design Department of the Zurich University of the Arts - Between Research Laboratory and Experimental Education.” In Proceedings of the 2nd Audio Mostly International Conference. Germany.
Lankford, Elsa. 2020. “Breaking the Sound Barrier: The Importance of Interdisciplinary Audio Curricula.” In Audio Education: Theory, Culture, and Practice. New York: Routledge.
Onen, U., Richard Stevens, and K. Collins. 2011. “Designing an International Curriculum Guideline for Game Audio: Problems and Solutions.” Journal of Game Design and Development Education 1 (January): 38–47.

References

Sarisky, Mark J. 2011. “Integrating ‘Audio for Games’ Into the Modern Audio Production Curriculum.” In Audio Engineering Society Conference: 41st International Conference: Audio for Games.
Sarisky, Mark, and Bruce Pennycook. 2009. “A Curriculum for a Degree in Audio Design for Game Development.” In Proceedings of the 35th AES Conference. London, UK.
Summers, Chanel. 2015. “Teaching the Aesthetics of Game Audio Design.” Presented at the Game Development Conference. https://gdcvault.com/play/1022281/Teaching-the-Aesthetics-of-Game.
Wang, Richert, and Vincent Olivieri. 2018. “Sound Design for Video Games: An Interdisciplinary Course for Computer Science and Art Students.” Technical Symposium on Computer Science Education, February, 981–86. https://doi.org/10.1145/3159450.3159577.

Questions?

Download Slides

https://tatecarson.github.io/ATMI-Presentation-Marp/

43 seconds per slide!

- The purpose of this talk: - To present a history of game education and how it is beginning to form into a discipline - To exchange ideas and best practices for game audio education - To describe the course as it's taught at DSU. - To share a model for an effective project-based learning and collaboration with a game design program - To inspire other institutions to adopt similar courses or components - To demonstrate how game audio education can fit into a music technology program

- There is currently no established curriculum or set of standards for comprehensive game audio education programs. - Different academic institutions vary considerably in their game audio offerings. This can lead to gaps and inconsistencies in students' knowledge. - More work is needed to develop dedicated programs meeting industry needs.

Second Column

Onen, Stevens, and Collins (2011): - Game audio creators fall between arts/humanities and computer science programs in most schools. - The field encompasses many academic departments while requiring specialized knowledge like programming. - There are challenges incorporating knowledge from so many disciplines. Game Audio Curriculum Guideline (2011), a document written by Interactive Audio Special Interest Group - Adds that practitioners wear many hats, blurring lines between roles. Bragança (2015): - Also emphasizes game audio's interdisciplinary nature demands awareness of diverse topics. - Calls for collaboration between academic departments. Sarisky and Pennycook (2009): - Proposed an interdisciplinary degree program. - Addresses the diverse skill-set needed for composers and audio engineers in industry.

- Programs should teach core principles, not specific tools - This is because technology changes rapidly in game audio field - Curriculum should focus on adaptability and autonomous learning

- Industry emphasizes the need not only for technical skills, but also for "soft skills" like communication and teamwork. - These soft skills are often overlooked in academia but critical for industry with its needs of collaboration. - Braganca recommend recreating industry pressures in class to nurture soft skills

The literature offers suggestions for building a multi year game audio curriculum with common courses and specialized tracks. At DSU we offer core courses that are geared generally to music technology and we offer only one game audio course, but this concept is still helpful. - Onen, Stevens, and Collins suggest core courses to build essential skills like recording and programming. Sarisky and Pennycook follow this by suggesting specialized courses offer depth in areas like composition and sound design, aligning with career goals. All agree that curriculum should emphasize teamwork through collaborative projects, preparing students for the interdisciplinary nature of the field.

- Onen and Hug encourage experiential, project-based learning - Onen also recommends using simulation, role play within team projects, with time left for reflecting on roles. - Ongoing journaling and portfolio development suggested as a counterpart.

- First, my course covers the history of game sound and its evolution - Then next unit extends students existing skills in sound design and music composition for linear media to the needs of interactive game audio. - We cover the basics of audio implementation techniques - The final project is a collaborative group project with Computer Game Design program to apply skills to a real game audio project - This project allows students to utilize skills learned in class on a hands-on game development experience

## Key Takeaway: - Lectures cover evolution, innovations, and workflows in game audio history.

- Exercises provide hands-on practice and reinforce lecture concepts, such as sourcing assets, sound editing, and understanding audio's role in gameplay.

- I explain the importance of VO to games, how human voice enhances game immersion through cues, storytelling, and character interaction. - We progress from early voice synthesis to recorded dialogue. - We also discuss the roles in voice over such as producer, writer, voice actor, and engineers. ### Exercise 3: Voice Over - **Recording Engineer**: Responsibility for high-quality recordings, reflecting lesson on studio setup. - **Voice Talent & Directing**: Aligns with lesson content on role of voice actors and directors. - **Editing**: Task of preparing individual sound files, connecting to the role of the editing engineer. - **Asset Management**: Updating the asset spreadsheet relates to localization and programming.

- **Project Objective**: The main aim of this project is to take a 30-second cutscene or trailer from a game create a sound re-design. - **Content Focus**: Unlike typical sound design projects that may include background music, this project specifically asks students to focus on the sound effects that accompany on-screen actions. - **Sound Design Techniques**: Students apply various sound design techniques that we've covered in class. This includes everything from pitch modulation to volume adjustments and even applying specific filters to make each sound effect unique and multilayered. - **Narrative Flow**: One of the unique aspects of this project is the requirement to think narratively. Students plan how your sound effects can introduce rises, falls, and impactful moments that amplify the emotional experience of the scene.

- Though this class doesn't focus on composing music, it's important for students to understand how adaptive composing techniques like layering and sequencing are used to create interactive game soundtracks. - **Exercise 4**: This exercise gives students hands-on experience with the complex process of exporting music assets for games. It teaches them to use Reaper's (a digital audio workstation) powerful export capabilities to generate stems that can be integrated into game audio middleware. This process includes techniques like vertical re-orchestration, linear cinematic cues, and batch exporting, among others. - **Real-World Skills**: The exercise is designed to impart real-world skills needed to deliver production-ready music assets for a hypothetical game project, including understanding the need for documentation and technical export tasks.

- Middleware is the software that allows sound designers to place sounds in a game. Its like the DAW of the game audio world. - We use Audiokinetic Wwise as our middleware, but FMOD is another popular choice. The depth of audio implementation covered in class should depend on students' technical ability and interest. - Week 7 focuses on setting up Unity (a common game engine) and Wwise for game audio. It covers the installation process, user interface exploration, and a foundational project that involves simple spatial audio techniques. - As we progress through weeks 11-12, we introduce more complex topics like sound positioning, triggering sounds through player actions, and creating a footstep system using random containers and switches.

- **Objective**: The aim of this project is to equip students with the technical know-how to integrate Wwise with Unity, focusing specifically on spatial audio. - We don't start this process with a real game, but with more of a demo example, an animated cube. While this isn't a very engaging example, it helps to simplify the process while students are getting comfortable learning the new softwares. - **Learning Outcomes**: By the end of this project, students will have a foundational understanding of how to set up an Unity with Wwise, import sounds into Wwise, create and trigger Positioned Events in Unity, and understand the basics of 3D sound positioning.

### Final Project Speaker Notes - **Team Structure**: Students form teams of 3-4, each with distinct roles such as Sound Designer, Audio Implementer, and sometimes Composer. - **Game Selection**: Choices include games from the game design class, open-source games, or built-in Unity demos. - **Proposal**: About a month before semester's end, teams submit a proposal detailing the game they'll work on and an assets list. This proposal acts as the project's grading rubric. - **Class Format**: The final 3-4 weeks are open lab sessions. No lectures, but mandatory attendance for hands-on work and troubleshooting.

### Final Project Challenges - **Overview**: The final project presents several challenges, from group dynamics to technical difficulties, that need to be addressed for successful completion. - **Time Constraints**: Students often feel the pinch of time, both for mastering Wwise and C# basics and for delivering a polished, sound-rich game. - **Game Selection**: Finding the right game for students to work on is tricky. Games from the game design class can have architectural and bug issues, while finding suitable games online is time-consuming and has a low success rate. - **Debugging Difficulties**: With students working on different games, it becomes a complicated task for the professor to troubleshoot and debug each project, given the unfamiliarity with the individual games.

- I've found that spending three weeks on the history of game audio is excessive and takes away from practical application. In future iterations, I'll cut this down to allocate more time for composing and implementing game music. - Another area I intend to improve is the audio implementation setup. I've noticed that many students face roadblocks here, mostly due to unclear instructions for installing the necessary software. To address this, I plan to provide more explicit and simplified guidelines to ensure that we spend more time learning about game audio implementation and less time troubleshooting. - Finally, I'm looking to make the final project more effective as a learning experience. I aim to make the project description less open-ended and provide a curated list of games for sound implementation. This will help students focus on delivering higher quality work.

### Speaker Notes 1. **Why Adopt the Course:** - Adopting a game audio course enriches the music tech curriculum by integrating both creative and technical skills. It offers students new career pathways in the burgeoning gaming industry and fosters interdisciplinary learning through potential collaborations with game design programs. - The course teaches universally applicable audio production techniques. These include layering sounds for complexity, creating sound palettes for cohesiveness, and understanding narrative flow for effective storytelling across various media like film and podcasts. 2. **Applications Beyond Games:** - By adding this course, one not only diversifies curriculum but also prepares students for a wide range of audio-related careers.