Abstract

Is it possible that everybody has the ability to compose music? Could a unique game provide a way to test this theory? In this study, music technology in the video game industry and the theory of synesthesia are used to explore an intuitional approach to improvisation. The paper discusses patterns of rhythm, melody and various solutions for looping music clips illustrated through examples of several musical indie games.

I. Introduction

Music is always a crucial element in video games. Even though it goes unnoticed during gameplay most of the time, it stands out immediately if there is any discordance within the game world. It's very difficult to create appropriate background music as a musician, not to mention as a beginner with no training. Could a game encourage players to take part in this process of creating background music and doing it well at the same time?

This subject will be explored using methods of music improvisation within actual video game examples focusing on four categories: rhythm, melody, synesthesia and the technical direction of video game’s music systems. The first and second parts discuss and define the organic structure of a piece of music; synesthesia introduces the relationship between background music, story and visual atmosphere; finally, the technical part will concentrate on looping and layering technologies for the music in video games. Hence we can have a clear end point in view of what the game will be.

II. Historical, Theoretical, Cultural Context

Rhythm

Rhythm is an indispensable element of music, carried by low frequency instruments like the bass or tuba^[1].The time signature always looks like this in 4/4 meter:

However, if this regular rhythm is sustained, it will be too monotonous to listen to. We can arrive at a much more novel rhythm by placing a rest (or syncopation) between beats, e.g.

For refreshing our ears further, we can intentionally position notes as syncopated accents on the "off" beat, in other words, making it into a backbeat. R&B and Jazz widely utilize this effect to increase elasticity:

Moreover, rhythm should be dynamic and keep changing between different period structures; this gives the whole piece a “groove”. That's how we deal with rhythm when we improvise music.

Melody

The key to improvising melody is rearranging the order of different scales. For instance, this is a C Major scale with a simple rhythm:

As a natural scale, it sounds a little tedious because human beings are already so used to it intuitively and culturally, but if we change the order of notes:

Things get more lively. It's similar to a principle called isorhythm, defined by Friedrich Ludwig, which was widely used by John Cage, Oliver Messiaen, Alban Berg, and also found in Indian music and polyphonic motets. It combines an order of rhythms with fixed pattern of different pitches. Using this principle, we can actually reproduce almost infinite bars within the same tunes, significantly lengthening our piece of music. Using it can greatly accelerate learning and skill levels for the beginner. On the other hand, tuning systems and chords also have a huge effect on the color of music.

Synesthesia

The definition of synesthesia is controversial. Synesthetes in art history, including Wassily Kandinsky, Paul Klee, Olivier Messiaen, Charles Pierre Baudelaire and Adeline Virginia Woolf, all understood synesthesia in different ways; some of them even related numbers to colors. For example, in 1704 Sir Isaac Newton observed and defined the color spectrum after he passed a beam of sunlight through a prism.[2].He correlated colors to the seven notes of an octave based on those discrete hues; C (do) Red, D (re) orange, E(mi) yellow, F(fa) green, G(so) cyan, A(la) blue, B(si) purple[3] [4].Russian composer Alexander Scriabin constructed a circle of fifths to reveal the relationship between spectral light and frequency - C is red, D flat is purple, D is yellow and so on. Henri Matisse described warm colors as Majors and cold colors as Minors. Kandinsky conveyed his view by comparing the sounds of various instruments in Concerning the Spiritual in Art[Kandinsky, Wassily. Concerning the Spiritual in Art[5]. New York: Dover Publications, 1977. Print.], saying deep blue is a cello, green is a violin.

The conventional analysis of synesthesia is that the brightness of color represents the frequency of sound; dark colors are low frequency and vice versa^[6].And reverb wet sounds (called soundscapes or pads in virtual instruments), is an effect that sustains and decays sounds after they’re generated. It refers to blank visual space due to acoustic principles.

The relationship between color and sound depends on personal and cultural experience, i.e., human beings often connect their impressions of sound and color to their young adult period. When we play a piece of music for Spring Festival, people from China will think red, but Americans will think pink or yellow. Case in point, a person born blind named Tommy Edison shared his feelings about color in a series of videos. He couldn't understand the warm meaning of red or the cold meaning of blue.

Paintings can be understood as two-dimensional compressed music, and music as painting with a timeline; musical notes are strokes of a painting. Homophony is the dominant hue, chords are the saturation, a tuning system is the transition between colors, and dynamic performance is contrast.

The example 1 presents a saturation reduction, the two chords are tuning from C Major to C Minor.

As an advanced version of example 1, example 2 is a demonstration of color transition; reducing C Major's saturation at first (chord1 – chord2) and then jumpingto G Major and increasing the saturation again (chord3 – chord4), likewise the transition of contrast color in color theory. What happens when we put those theories into practice to have visual parts interact with customized music?

Technical Direction

The simplest technique for playing video game background music (BGM) is looping. Composers intentionally make the start and end point of a music clip the same to create a seamless continuation, a loop. Additionally, a "parallelogram" loop is engaged to add more layers of music; the previous clip's end overlaps with the next clip's beginning and then we can have a short opening for BGM when the game starts.

When the environment of a game changes, how can the mood of the music shift at the same time? Sudden stops and switches (or fade outs and fade ins) of a new BGM is the normal solution, but it maybe a little bit rough on our ears. There is another way that uses clip switch points. It divides the looping clip into a series of shorter clips. As long as the music plays through one cut-point, the game system will check back with the environment variables - if the visual content changes, each point can lead the melody of the current clip to another different looping clip smoothly; if not, the current one keeps playing. That is useful in the context of a traditional game narrative whose actions involve scenes of enemies ambushing or being ambushed and the protagonist moving from space to space.

A game called Crypt of The NecroDancer has background pad music with layers of clips having the same rhythm and unisonant melody (strong beats, regular tempo, and being alike in pitch);the layers accumulate. Those clips represent the melodic voices of monsters and merchants in the game; muted by default, they will be triggered by proximity-the distance of the player. The interactivity is apparently increased in this project.

The advanced way to get the point is by using reverb. As above, it will create space for us. Moreover, it mixes the gaps between clips and eliminates the disharmony of pitches. As a result, conflicting melodies can be merged in a same space reverb. For example, Circuits is a puzzle game in which players can rearrange several premade clips to recover the original sound track that they heard at the beginning of each level. Utilizing the reverb sounds, clips become euphonic with whatever order the player wants.

A solo instrument and single notes may also be an easier approach. In Panoramical, a game about experimental music and visual art, players trigger sounds and effects by randomly pressing buttons. BGM in the project is more like ambient sounds for creating atmosphere; the developer intentionally removed metrical and temporal structures - there isn't a significant rhythm at all. Within the atmosphere music, the players' click produces one or two single notes with a random pitch each time. Our brains will accept the solo music track as a kind of motif because of the absence of rhythm.

There is another example, Ape Out, which is an indie game about stealth and assassins. During gameplay, an improvisational, stylized jazz percussion plays throughout. By throwing a soldier onto the wall and blowing him up, the furious gorilla (protagonist) sets off the sound of a crashing cymbal. It's a combination of solo instruments and clips overlapping. And due to the nature of percussion, it doesn’t require a huge effort from the composer to adjust the chords' harmony further.

III. Conclusion

With the theories and techniques discussed so far, we have gotten a quick glimpse at how the music of video games can be used to foster a new kind of teaching, appreciation and acceleration of musical skills. Knowing the fundamentals of improvisation, players will have the ability to participate in their own interactive projects and conceptualize their own ideas about music. This approach has enormous potential in the field of interactive design, especially when integrated with sensors such as HMD (Helmet-Mounted Displays) and Leap Motion (a hand-controlled sensor, that seems like a light-weight version of Kinect). As game designers, it’s always good to be clear about our intentions and objectives - there is a long way to go, but we've already seen our destination.

IV.Note

• [1]: Jones, Josh. "The Neuroscience of Bass: New Study Explains Why Bass Instruments Are Fundamental to Music."Open Culture. Open Culture LLC.,23Oct. 2015. Web. 11 Nov. 2015. http://www.openculture.com/2015/10/the-neuroscience-of-bass-new-study-explains-why-bass-instruments-are-fundamental-to-music.html/
• [2]: Newton, Isaac. Hydrostatics, Optics, Sound and Heat. England, c. 1670 - c. 1710.Collection of Cambridge University Library. Manuscript.
• [3]: Peacock, Kenneth. "Instruments to Perform Color-Music: Two Centuries of Technological Experimentation."Leonardo 21.4 (1988): 397. Web.
• [4]: Campen, Cretien Van. "The Hidden Sense: Synesthesia in Art and Science". MIT Press, Cambridge, MA, U.S.A., 2010. Print.
• [5]: Kandinsky, Wassily. Concerning the Spiritual in Art. New York: Dover Publications, 1977. Print.Ward, J., B. Huckstep, and E. Tsakanikos. "Sound-Color Synesthesia: To What Extent Does It Use Cross-Modal Mechanisms Common to Us All?"Cortex 42.2 (2006): 264-80. Web.
• [6]: Ward, J., B. Huckstep, and E. Tsakanikos. "Sound-Color Synesthesia: To What Extent Does It Use Cross-Modal Mechanisms Common to Us All?"Cortex 42.2 (2006): 264-80. Web.

Murakami, Haruki. Hard-boiled Wonderland and the End of the World: A Novel. Tokyo: Kodansha International, 1991. Print.

V.Bibliography

• [1]: Bradbury, Ray. Fahrenheit 451. New York: Simon and Schuster, 1967. Print.
• [2]: Fullerton, Tracy, Christopher Swain, and Steven Hoffman. Game Design Workshop: A Playcentric Approach to Creating Innovative Games. 3rd ed. Amsterdam: Elsevier Morgan Kaufmann, 2008. Print.
• [3]: Koster, Ralph. A Theory of Fun for Game Design. Scottsdale, AZ: Paraglyph, 2005. Print.
• [4]: Leap Motion Inc. "Introduction to Motion Control."Leap Motion Developers, 12 June. 2015. Web. 30 Sept. 2015. <https://developer.leapmotion.com/assets/Leap%20Motion%20VR%20Best%20Practices%20Guidelines.pdf/>
• [5]: Liu, Cixin, and Joel Martinsen. The Dark Forest. New York: Tor Books., 2015. Print.
• [6]: Liu, Cixin, and Ken Liu. The Three-body Problem. New York: Tor Books., 2014. Print.
• [8]: Orwell, George. 1984. Secker & Warburg, 1949. Print.
• [9]: Plato, Alexander Nehamas, and Paul Woodruff. Symposium. Indianapolis: Hackett Pub., 1989. Print.
• [10]: Plemmons, Daniel, and Paul Mandel. "Designing Intuitive Applications "Leap Motion Developers. Leap Motion Inc., Nov. 2014. Web. 30 Sept. 2015. <https://developer.leapmotion.com/articles/designing-intuitive-applications/>
• [11]: Plemmons, Daniel, and Paul Mandel. "Introduction to Motion Control."Leap Motion Developers. Leap Motion Inc., Nov. 2014. Web. 30 Sept. 2015. <https://developer.leapmotion.com/articles/intro-to-motion-control/>
• [12]: Rand, Ayn. Atlas Shrugged. New York: Random House, 1957. Print.
• [13]: Stockburger, Axel. The Rendered Arena Modalities of Space in Video and Computer Games. London: U of the Arts London, 2006. Print.
• [14]: Tekinbaş, Katie Salen, and Eric Zimmerman. Rules of Play: Game Design Fundamentals. Cambridge, MA: MIT, 2003. Print.
• [15]: Ward, J., B. Huckstep, and E. Tsakanikos. "Sound-Color Synesthesia: To What Extent Does It Use Cross-Modal Mechanisms Common to Us All?"Cortex 42.2 (2006): 264-80. Web.

Games

• Ape Out. Gabe Cuzzillo. 2015. Game.
• Circuits. Digital Tentacle, 2014. Game.
• Crypt of the NecroDancer. Brace Yourself Games. Klei Entertainment, 2015. Game.
• Panoramical. Fernando Ramallo, and Dacid Kanaga. Finji, 2015. Game.