introduction to 360 sound

1 Introduction

Sound in a 360 video envrionment is considered an essential element, yet, it’s very hard to deal with sensibly. A reason for its complexity is down to the multilayered nature of film soundtrack design in general and what we have come to expect film to be.

Film soundtracks always play with notions of diegesis (sonic elements that are part of the film world itself and beyond it). Often, there is a narrator or voice which if it moves around too much will cause a destabilising lack of clarity¹. Music, unless explicitly mixed with the idea that you can listen around the world in which it is being played can also cause really strange feelings in the listener when instruments move or are distributed around the head. It’s hard for the brain to resolve a drum moving in space as drums are heavy and usually on stands attached to the ground.

Film is a front-centric art where viewers are usually in a seat, looking forward and sound mixing has developed specifically with this affordance in mind. 360 video disrupts this, demanding an active and engaged viewer, interested in potentially anything and everything that’s around them. For this reason, the art itself is still developing, not enough people have done enough interesting things with this in order to know how long a VR film should be, whether or not messages can be more powerful and how to sustain interest beyond the gimmick and current novelty of being surrounded by potential information.

It is very likely therefore that VR films will need to be made in such a way that they lead the viewer along from place to place but that at any moment, one is free to look back and imagine into the future. This has an effect on the form and structure and the way narrative plays out and it is in this respect that sound will continue to be the glue that sticks this potentially discombobulating form together into a coherent experience.

While sound has been regarded as a significant and essential element in VR/360 film, dealing with the inherent complexities in concept, design, implementation and delivery is non-trivial and presents current practitioners in the industry and students considerable hurdles. These are not simplified because the tools marketplace is also complex and competitive with Google/Youtube, Facebook, Occulus, Premiere all offering different, but similar ways to manage the task.

2 quick summary of what we should already know

2.1 two-channel sound

What does stereo mean?
what is the recommended integrated loudness level for the EBUR-128 in LUFS?
What do LUFS integrated loudness to Netflix require?
How about for youTube?
What does binaural mean?

2.2 surround sound

what deos the .1 of 5.1 stand for?
what is the .1 usually used for in film sound mixing?
what is the centre channel used for?
what is the channel ordering for standard 5.1 ITU spec.
how about for SMPTE?
how is 5.1 usually mixed in film, as a front-centric art or as something immersive and enveloping throughout?
is it better to work in surround and mix down to stereo, or to work in stereo and move to surround?

2.3 ambisonics

what is aFormat?
what is bFormat?
what is the difference between ambiX and FuMa?
why is knowing the format of your bFormat important?

3 using ambisonics in your mixes

This is well covered already in the following notes: digital.eca.ed.ac.uk/sounddesignmedia/2018/11/13/lecture-09-space-and-time-b/ digital.eca.ed.ac.uk/sounddesignmedia/2018/11/20/ambisonic-guidance/ digital.eca.ed.ac.uk/sounddesignmedia/2018/11/19/ambisonic-workflows-with-reaper/

However, it’s worth just rehearsing that the 5.1 and stereo mixes, and indeed binaural mixes can happen simultaneously in most circumstances. By using different master busses to host the stereo, binaural and 5.1 mixes, you can quickly audition different versions of your mixes and change between them in one session. Depending upon how heavy you go with your processes, the spatial mixing might need to be done after more rudimentary work has been done to your audio. For example, you may be better off doing dialogue editing in a different session, de-noising things etc without having concern for how things will be spatialised. However, if your dialogue is all set inside a room and the recording was done with an ambisonic mic, then you will definitely want to keep whatever edits you’ve done in bFormat so that the spatial signature of the original recording is kept and can be balanced with other spatial information in the main mix for your project.

In Reaper, I like to have a masterfader connected to the hardware outputs of my sound system and to send a 5.1, stereo and binaural bus to that place, muting two of the three tracks depending upon what I want to monitor. These ‘master’ busses are the ones that I bounce, so it’s very important that the bus has exactly the number of channels you want your file to have, e.g. stereo and binaural, 2 channel, 5.1 6 channels, 7.1 8 channels etc.

Demonstration.

4 Starting with audio mixing for 360 video

4.1 Get a video

When mixing for a 360 video, you need a 360 video to practice on. This means either renting 360 camera, making a 360 film by exporting something from Unity, or downloading a 360 video from youtube.

To do this, you’ll need the commandline tool youtube-dl, accessibel to homebrew users on OSX with brew install youtube-dl or to chocolatey users on windows with this choco install youtube-dl. If you already have youtube-dl installed you may wish to update it: brew update youtube-dl.

Youtube hosts the 360 videos in lots of different formats within the container on their server so you can’t just download a VR video, you need to specify which of the tracks in the container you want to get. This post explains the solution I followed to get the following command sorted: github.com/ytdl-org/youtube-dl/issues/15267:

youtube-dl https://www.youtube.com/watch?v=xKNTZd9pOz0 --user-agent '' -f 266 --merge-output-format mkv

To include the audio in the above, you need to add the audio stream, best quality one is usually 251.

youtube-dl --user-agent "" https://www.youtube.com/watch?v=XPhmpfiWEEw -f 266+251 --merge-output-format mkv

4.2 Get some spatialisation tools sorted

For 360 video, you need 360 sound. This inevitably means you can hide from ambisonics no longer. Well, actually you can, you can just deliver a nice stereo or binaural mix that is headlocked and put it with the video, most likely nobody will notice that the sound doesn’t move when you shift your head around, however, the effect of immersion and involvement in the material, not the message, can be greatly enhanced with sounds that appear to move around you. Actually, it’s perhaps fairer to say that the video’s spatial phenomena will be easier to read if there are sounds that correspond to movement in the visuals.

4.3 Facebook360 spatial workstation

Basically, read the guide and you’re away.

Table of Contents