01 – Introduction to the course and your starter patch
Introducing programming languages for sound, performance and composition
We know that the work of the sound designer or composer is not bound to the manipulation of pre-recorded sound, or the synthesis of new sound. The art and craft of sound design and composition is not limited to fixed media contexts (such as film or electroacoustic composition) either. Contemporary computer games offer a popular genre where the sound and music structures aspire to be dynamic, context sensitive, malleable and adaptable1. There are many other contexts where sound is (or could be) designed to adjust itself depending on when, where and how it is heard or needed. Live performance, electronic composition and free improvisation, multimedia situations, online, Human Computer Interfaces (HCI), sound art, museum exhibits, GPS-enabled podcasts, and a range of industrial audio; in car, alarms, mobile phones, sound masking, sound reinforcement, sound therapy, and auditory interfaces of almost any kind constitute the short list.
Professional sound designers and composers can contribute significantly to these areas because such people are not only technically skilled in fundamentals of digital audio, but also understand how sound can carry information to potential listeners. Importantly, they’re also sensitive to what sounds good.
All of the above suggests that the sounds themselves are only part of the design process. The system that generates, manipulates, adjusts and presents these sounds is a significant concern for the sound designer and composer too. Thanks to software tools like MaxMSP, Pure Data and Supercollider, and the increasing ubiquity of computing power, sonic artists can rapidly prototype and test the behaviours and principles of reactive and interactive audio as well as the sounds themselves. Audio Design is now about imagining how sound might adapt to unimagined contexts and conceiving the shifting parameters that control the sounding result.
Given this (extra?) responsibility for the system that generates and manipulates sound to fit its context, sonic artists must ask of their work, “what is the sound supposed to be doing now?”, then evaluate “is it doing that?” and if not, persuade the system to change its parameters in a well designed and carefully thought out way such that the sonic consequences of these changes are meaningful.
Next steps
In order to make the most of this ‘new’ frontier, we need to develop skills in audio programming. Of course, a highly desireable way to do this is to learn how devise C++ DSP code for audio applications that can be ported and embedded into any piece of software. However, most artists on this course are not audio programmers – yet – but there are tools/languages out there that can help you learn what DSP is and rapidly prototype sound processing ideas without needing to learn how to do this in C++. If you’re sufficiently excited by this, you can eventually learn how to manage C++ but before then MaxMSP and Pure Data will be sufficient for learning the key concepts behind computer-based sound and devising the behaviours your sound should have and even testing them out in real-world contexts very easily.
What’s key to learning a new language is having some work to actually do. Of course working through the tutorials for MaxMSP is going to be very helpful in terms of learning how the language works and what it can do but these tutorials will also need to be studied in tandem with the development of your project work. Knowing what you really want to do and what you don’t need to do is key here.
MaxMSP
Max/MSP is a real-time, graphical, object-oriented programming language.
Max was named after Max Matthews, the computer music pioneer whose Music programme was written at Bell Labs in 1957; Music III (1960) to Music V (1968) established the “unit generator” paradigm that Max still embodies.
Although it can be used for many purposes, Max/MSP is a music programming language.
As the name would suggest, there are two parts to Max/MSP: Max is the MIDI programming part and MSP is the Digital Signal Processing (DSP) part (Max Signal Processing).
Max was conceived and written by Miller Puckette in the mid 1980s whilst he was working at IRCAM.
It was first taken up by IRCAM but then commercially released by Opcode (the makers of the OMS system) in 1990.
Since 1999 it has been owned by Cycling ’74 (www.cycling74.com), a company started by David Zicarelli, one of the original developers of Max.
In addition, Max/MSP can be extended by writing “external objects” in the programming language C (or even Java and Javascript) so it is very flexible indeed.
This, along with its user-friendliness (i.e. ease of programming) has established Max as possibly the most successful music programming language of all time.
Max is pretty much universal, i.e. it is used in many, many universities and studios, by musicians of every genre from all over the world.
Pure Data
Like MAX, PD is a data flow programming language.
PD was developed by Miller Puckette around the time when Opcode took Max over.
Importantly PD is broadly open-source and therefore there is a wide developer base, numerous developers have contributed to furthering its development.
The key differences between MaxMSP and PD relate to how the interfaces look. PD was extended to include graphics libraries and other toys but it’s no longer supported.
Its open-source nature means that it has been ported to the likes of Android and iOS via libPD and the Heavy Compiler can take uploaded PD patches and provide code for WWISE, HTMLF, Unity, Javascript and other other systems.
Faust
Is another extraordinary technical achievement where the aim is to support the development of complex DSP processing and implement it as highly optimised code that can be exceuted almost anywhere on any platform.
It is developed by an excellent team of programmers and is open-source. The language is somewhat convoluted, but you can very quickly make effective sound software and preview its effect within a browser: faust.grame.fr/.
Midi
The MIDI standard was first proposed by Dave Smith in 1981; the MIDI Specification 1.0 was released in 1983. The most important distinction between MIDI and other sound-related formats is that MIDI code does not contain sample data; rather, it is a control language that specifies at its most basic level when, how loud, and for how long musical notes happen. How these notes then sound depends on the hardware used to create sonic output.
The acronym stands for “Musical Instrument Digital Interface”
-consists of – hardware connection – agreed computer codes
tries to be comprehensive
– not all manufacturers implement everything – documentation usually contains a MIDI implementation chart so that you know what controller and note values are mapped to hardware functions.
MIDI Terminology
- MIDI Interface
The physical connection - MIDI Device
Anything that sends or receives MIDI data - MIDI Controller
A device which only SENDS data - MIDI Sound Source
A synthesiser, sampler, or other audio device controlled by MIDI - MIDI Channel
Data sent on up to 16 separate channels\ - Note On
A code that starts (or stops) a MIDI event - MIDI Pitch
The tempered scale numbered from 0—127 where middle C = 60 - Velocity
The speed with which a controller (e.g. key on a keyboard) is pressed; usually translated into loudness
NB A note on message with velocity 0 is effectively a note off - Continuous Controller
A continuous stream of MIDI information from e.g. Pitch Bend Wheel - Program Change Event
Code which tells a MIDI device to change its internal setup e.g. synth voice bank
Introduction to Digital Signal Processing
Digital Signal Processing (DSP) refers to processing (but not necessarily changing!) a signal through digital means.
The origins of DSP are in electrical engineering.
The signal in analogue circuits was a continuous electrical signal. A digital signal is discrete i.e. non-continuous: a stream of numbers, in the case of digital audio a stream of sample values, as we are probably already familiar with from working with ProTools and other such hard-disk recording systems.
DSP of musical signals was originally carried out in non real-time as the first computer systems weren’t fast enough.
Today however real-time DSP is possible for all but the most demanding applications.
The main uses of real-time audio processing in software are convenient audio processing in the studio i.e. as replacements for older hardware (digital or analogue) e.g. Cubase, Logic, Nuendo, Pro-Tools, Reaktor etc.live performance e.g. Ableton Live, MAX/MSP, SuperCollider etc.
MSP
The DSP extension to Max is called MSP (“Max Signal Processing” or “Miller S. Puckette”?)
It developed out of projects from IRCAM (ISPW) and from Miller Puckette (PD).
All MSP objects have the ~ (tilde) extension to their name, e.g. cycle~
Objects which have signal inputs or outputs are connected using striped patch cords so it’s easy to see what takes a signal and what doesn’t.
Some Basic Operations with MSP: record and playback soundfiles real-time sampling variable speed/position sample playback MIDI (or other) control of audio synthesis (additive, subtractive, FM etc.) and sample playback other signal processing including filtering, fft analysis (+resynthesis), granular synthesis etc.
Some real-time interfaces that could control MSP and PD: MIDI keyboards and controllers The Lemur Multitouch Control Surface Wacom graphics tablet touch- and pressure-sensitive pads games controllers e.g. joysticks pitch, timbre, and volume sensors (perhaps themselves real-time audio MSP programmes)
Technical Terms
We are probably already familiar with most of these but let us revisit them briefly:
Sampling Rate: During recording, the number of times per second (frequency) at which the Analogue-to-Digital Convertor (ADC) measures the amplitude of the sound wave and generates one sample in general, the higher the sampling rate, the more accurately the waveform is represented CD quality is 44100Hz (44.1kHz) but the latest systems can sample up to 384kHz.
Quantisation / Sample Size / Word Size / Bit Depth: The number of bits allocated to the storage of one sample the more bits, the higher the sound quality: the number of discrete steps available to measure the amplitude increases as more bits become available CD quality is 16 bit (65 536 steps), the latest technology offers 24 bit integer (16 777 216 steps) or even 32 bit floating point (4 294 967 296 steps). Compare this to the 8-bit (uncompressed) files found (or once found) on the Internet (256 steps). Quantisation directly affects the dynamic range of a digital system with approx. 6 dB per bit (= 96dB for 16bit, 144dB for 24bit systems) Quantisation noise: signal distortion that creeps into digital sound playback of low-amplitude signals due to the signal constantly being represented in the lowest few bits of the samples
Nyquist Frequency: From Harold Nyquist, Bell Labs researcher, 1928:
“For any given deformation of the received signal, the transmitted frequency range must be increased in direct proportion to the signaling speed… The conclusion is that the frequency band is directly proportional to the speed.”
The Nyquist Frequency is the theoretical frequency limit of a digital audio system, usually defined as half the sampling rate, but in practice a little less than half. Hence the CD sampling rate of 44.1kHz which represents frequencies up to about 20kHz, the average upper limit of human hearing.
Read page 2 of this for a perfectly clear explanation: www.rctn.org/bruno/npb261/aliasing.pdf
Buffers and Latency
The term real-time is quite misleading: nothing happens instantly, audio has to be buffered. Running at 44.1 KHz , the buffer latency for I/O or processing is as follows
1024 samples = 23.22ms
512 samples = 11.61ms
256 samples = 5.8ms
128 samples = 2.9ms
64 samples = 1.45ms
As well as buffering latency there is usually a separate control rate for DSP. In MSP the usual scenario is:
audio rate = 44.1KHz
buffer latency = 128-512 samples for real-time sound processing
control rate = 1KHz (DSP scheduling rate or tick)
samples/tick = 44.1
Some of the above is borrowed from notes written by Michael Edwards.