Tetris Sound: Using Music to Affect Gameplay - By: Taylor Vaughn Galen Schmidt Jeremy Hunt
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Tetris Sound: Using Music to Affect
Gameplay
By:
Taylor Vaughn
Galen Schmidt
Jeremy HuntTetris Sound: Using Music to Affect
Gameplay
By:
Taylor Vaughn
Galen Schmidt
Jeremy Hunt
Online:
< http://cnx.org/content/col11609/1.2/ >
CONNEXIONS
Rice University, Houston, TexasThis selection and arrangement of content as a collection is copyrighted by Taylor Vaughn, Galen Schmidt, Jeremy Hunt. It is licensed under the Creative Commons Attribution License 3.0 (http://creativecommons.org/licenses/by/3.0/). Collection structure revised: December 20, 2013 PDF generated: December 20, 2013 For copyright and attribution information for the modules contained in this collection, see p. 17.
Table of Contents
Background and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1 Audio Analysis
1.1 Amplitude Analysis for Drop Speed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Beat Analysis for Game Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Frequency Analysis for Piece Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2 Implementing a Pipelined Sound Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3 Implementing the Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4 The Final Game, Conclusions, and Poster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
5 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Attributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17iv
Available for free at Connexions1
Background and Motivation
Motivation
Video games are a common pastime for many people. For example, Bioshock In nite, a relatively popular
game, has sold over 4 million copies to date. However, one problem which many video games suer from is
a lack of replay value. After the player has beaten the game, there is little point in replaying it because any
twists are already known, and any puzzles have already been solved. One way to avoid this stagnation is by
randomizing some aspects of the game. This way, the game will be dierent each time it is played and will
retain playability value much longer.
The goal of our project was to build a framework which would allow the user to load in any song le and
have it generate events to alter gameplay. The game we chose to aect was Tetris. An overview of tetris can
be found here
2 .
Implementation
Tetris was chosen for a couple of reasons:
1. It is a relatively simple game to implement. This allowed for more time to be spent on the signal
analysis section of the program, and less on the game implementation.
2. Despite its simplicity, Tetris has a surprising number of variables which can be altered to aect game
play. This gave some freedom in how to implement the events generated by the song.
We chose to aect the game in three primary ways. The rst of these is the fall speed of the pieces, which
was correlated with the volume of the sound le. This makes it so that louder sections of the song cause
the pieces to fall faster. The second aspect we chose to aect was which piece the player would get next,
which is correlated with the dominant frequency of the music at that particular time. Finally, we added a
visualizer to the game, which pulses any time a strong beat occurs in the music.
These variables were chosen because they are clearly visible to the player of the game, making it simple
to verify that the signal analysis is working correctly.
1 This content is available online at .
2 http://www.cs.cornell.edu/boom/1999sp/projects/tetris/whatis.html
Available for free at Connexions
12
Available for free at ConnexionsChapter 1
Audio Analysis
1
1.1 Amplitude Analysis for Drop Speed
We performed an analysis on the volume of the song to determine how quickly the pieces in tetris should fall.
The thinking behind this is that songs generally grow louder when they are growing more intense. Therefore,
the game should become more intense when the song does.
1.1.1 Envelope Generation
We compute a constant based on the relative volume of the song such that the current drop speed can
be determined as a fraction of the maximum fall speed. We determine the relative volume from the songs
envelope. The envelope is computed by rst taking the normalized magnitude of the signal and then setting
each second of the song to the max value in that segment of the song.
S = Number of Samples in Signal (1.1)
Pτ +S
Envelope[τ ] = n=τ abs(signal[n]) (1.2)
S · max (abs(signal))
1 This content is available online at .
Available for free at Connexions
34 CHAPTER 1. AUDIO ANALYSIS
Envelope from Raw Signal
Figure 1.1: The normalized envelope is generated from the raw signal by take max values from the
absolute value of the signal
1.1.2 Speed Calculation
Because the signal is normalized, the value of the envelope at time t is equivalent to the fractional value of
the maximum speed that the current speed is to be set at.
Speed[τ ] = Envelope[τ ] * Max Speed (1.3)
Available for free at Connexions5
2
1.2 Beat Analysis for Game Visualization
One of the algorithms we chose to implement was a beat detections algorithm. In the game this maps to a
pulsing visualization.
1.2.1 Background
When a drum is hit, a spike in energy occurs across all frequencies. When a bass guitar is plucked, there is
a spike in energy in the lower frequires. As these two instruments are commonly used in modern music to
give a song rhythm or a beat, we decided to look at energy in the lower end of the spectrum. Theoretically,
this should automatically lter out other spikes of energy created by a vocalist or electric guitar.
Beat Detection on Metric's "Gimme Sympathy"
Figure 1.2: Beats outlined by blue bars
2 This content is available online at .
Available for free at Connexions6 CHAPTER 1. AUDIO ANALYSIS
Beat Detection on Flogging Molly's "Don't Shut 'Em Down"
Figure 1.3: Beats outlined by blue bars
1.2.2 Using Energy Comparisons to Detect Beats
Energy Calculation
Because energy spikes when a beat occurs, it is fairly straight forward to implement an algorithm which
compares an average local energy to an instantaneous energy. Because we only want to look at the low
frequencies, we need to compute the FFT of the signal. Then, we compute the average local energy by taking
the integral of the square of the FFT of a second in time over the number of samples. The instantaneous
energy is calculated by taking the integral of the square of the FFT of a single sample.
Z Fmax
2
< Elocal > = (Length of Instant Signal in Time) ∗ |FFT(Slocal )| (1.4)
0
Z Fmax
2
Einstant = |FFT(Sinstant )| (1.5)
0
Constant Calculation
Probabilistically speaking, if the instantaneous energy is greater than the local energy by a large enough
quantity, we can say with a high likelihood that a beat has occured. The quantity is realized through the
variance of the instantaneous energy to the average local energy. A constant is then calculated from the
Available for free at Connexions7
variance and we can see that the instantaneous energy needs to be a factor of the constant larger than the
average local energy.
Z
2
σ 2 = (Length of Instant Signal in Time) ∗ (Einstant − < Elocal > ) (1.6)
C = (−0.0025714σ 2 ) + 1.5142857 (1.7)
< Elocal > ≥ C ∗ Einstant (1.8)
1.2.3 Filtering Extra Beats
If a beat is detected, its sample number is recorded in a vector. If a beat spikes for multiple samples, multiple
beats would be recorded which leads to an overestimation of the current tempo. To account for this, we
merely look at the samples, detect a leading edge where the rst beat is recorded, and ignore all subsequent
sequential beats.
3
1.3 Frequency Analysis for Piece Selection
1.3.1 Interval Generation for Ten Second Sections
Energy Computation
First, for a period of ten seconds the frequency spectrum is broken into bands of equal energy. The FFT
of the signal is computed for the entire ten seconds and then bandlimited to the lower 10 KHz. The total
energy across all frequencies is calculated. Because tetris has seven pieces, we will need seven frequency
bands. By dividing the total energy by seven, we know have a target energy for each sections
Z 10kHz
2
Etotal = (FFT(Signal)) (1.9)
0
Etotal
Etarget = (1.10)
7
Interval Selection
A binary search algorithm is implemented to nd the bounds of each frequency band such that the energy of
each section will equal the target energy within a reasonable error. Ideally, we could compute exactly equal
energies but we are limited by the number of samples in our seconds interval.
R I[1] R I[1]
0
(FFT(Signal))2 ± error = I[0] (FFT(Signal))
2
± error = ··· = (1.11)
R 10 kHz 2
I[7] (FFT(Signal)) ± error = Etarget
Each of these bands is then assigned a dierent game piece.
3 This content is available online at .
Available for free at Connexions8 CHAPTER 1. AUDIO ANALYSIS
Piece Selection Through Frequency Analysis
Figure 1.4: The spectrum is divided into seven equal energy sections for a ten second interval
1.3.2 Piece Selection for Half Second Sections
Once the frequency bands are computed for the section of the song, we compute the pieces. We take the
FFT of a half-second samples and calculate the energy across each frequency band and nd the maximum
energy. The next piece is assigned accordingly.
I[i+1]
Z !
2
Piece = argmax i ∈ Peices, (FFT(Ssection )) (1.12)
I[i]
This process is repeated for each half second of the song. In theory, sections of the song with vocals will
see more instances of the higher frequency pieces, while areas with bass guitar solos will be dominated by
pieces from the lower frequencies.
Available for free at ConnexionsChapter 2
1
Implementing a Pipelined Sound Engine
In order facilitate on the y sound analysis requiring demanding DSP algorithms, we implemented a parallel
three-stage pipelined sound engine that allows sound events to be time correlated to in game events. This
pipelined implementation allows the analysis of the song to be performed on chunks of the song simultaneously
along with loading new chunks from the input device and playing already analyzed chunks. This method
gives the analyzer ample time to perform the analysis, as long as that analysis takes less time to perform
than the chunk it is analyzing takes to play. Because each stage of the analysis (the Loader, the Analyzer,
and the Player) each run in their own threads, this system can take advantage of multiprocessor systems
such as the multi-core processors found in most of today's computers.
The Pipelined Sound Engine
Figure 2.1: The full sound engine pipeline system.
Stage 1 - Loader: This stage translates the input song le from its source format (e.g. MP3 or WAV)
to an internal uncompressed format and buers the result into buer chunks that will be set to other
to other pipeline stages.
Stage 2 - Analyzer: This stage takes the uncompressed buer chunk and performs various bits of time
domain and frequency domain analysis, generating a chunk of time-correlated events and the buer of
song that goes with those events.
1 This content is available online at .
Available for free at Connexions
910 CHAPTER 2. IMPLEMENTING A PIPELINED SOUND ENGINE
Stage 3 Player: This stage takes the sound and event buers from the Analyzer and plays them,
correlating any events to what is currently being played.
Each stage communicates with the next via an adapter with a loadBuer method.This method blocks the
the current stage of the pipeline until the next stage of the pipeline is ready to receive the buer. In this
way, the pipeline will ll up as much as possible while the player stage is processing the buer. The player
also controls the speed of the pipeline indirectly through this loading method. Great care was taken to
minimize the time the transfer of each buer takes. Rather than copying the buer from each stage to the
next, each stage instead just passes on control of its buer to the next stage, allocating a new buer entirely.
This allocation process is much faster than copying the data, though if the allocator is bad, it can use more
memory than is necessary. Finally, each time a stage of the pipeline blocks to wait until the next stage is
done processing or blocks waiting for new data, it relinquishes control of the processor resources, allowing
the other stages to run at full speed.
Available for free at ConnexionsChapter 3
1
Implementing the Game
The game was built using Oracle's Java programming language
2 . Java was chosen because it is designed to
allow for easy Object Oriented Programming, which allows each component piece of the program to be built
independently from the rest. In addition Java allowed for easy cross-platform development.
3.1 Program Design
Each section of the program is represented by one of the rectangles in diagram, with the arrows indicating
data ow. By separating each component of the program into its own section, they could be developed
simultaneously without aecting one another.
External Image
Please see:
http://i.imgur.com/R7VgzkN.png
Figure 3.1: The program consisted of a program interface, backend component, song loader, song
analyzer, song player.
Program Interface and Backend Component
For the program interface, a GUI (Graphical User Interface) was built in Java Swing. Although Swing is
not the most e cient platform for building a GUI in, it is one of the fastest to develop in, and is designed to
integrate with the rest of the program well. The backend component handled communication between the
song analysis sections of the program and the program interface.
Song Loader, Analyzer, and Player
The song loader was responsible for loading the data from the MP3 le, and converting it into a format that
the program could use. Once it was in this format, the loader was responsible for passing the data to the
song analyzer, which was responsible for creating events based on the contents of the song. Finally, both
the event data as well as the song buer were fed into the song player, which was responsible for playing
the le, as well as notifying the backend component about events so that it could forward them to the GUI.
More information about the structure of this sound back end can be found in the Implementing a Pipelined
Sound Engine section.
1 This content is available online at .
2 http://java.com/en/about/
Available for free at Connexions
1112 CHAPTER 3. IMPLEMENTING THE GAME
Available for free at ConnexionsChapter 4
The Final Game, Conclusions, and
1
Poster
4.1 The Game
The game was successfully built, and a copy of it can be downloaded here: Sound Tetris (All OS's)
2
In addition, all of the source code is available
3 here4 .
1 This content is available online at .
2 See the le at
3 https://github.com/chief-tyrol/301game/
4 https://github.com/chief-tyrol/301game/
Available for free at Connexions
1314 CHAPTER 4. THE FINAL GAME, CONCLUSIONS, AND POSTER
4.2 Conclusions
The analyzer works, louder portions of the song do make pieces fall faster, the visualizer does beat with the
music, and the next piece will consistently change with diering frequency.
Bugs
There are some bugs with each of the features, notably
• The beat detection does not work properly on songs without a clear beat
• The amplitude analysis has trouble nding dierences, likely because most modern songs are mastered
to be at a constant volume
• The frequency analysis does not make it clear which frequencies map to which pieces
Despite these issues, the game is playable, and shows that it is possible to use audio to aect gameplay.
Using audio to aect gameplay is a valid, and often entertaining, way of adding randomness to a game.
It can increase the diversity of the game, and thereby make it more likely that the user will continue to enjoy
the game. In addition, the framework created is generic enough that it could be applied to many games
without requiring large code rewrites, meaning that it could potentially be applied to other games.
4.3 Poster
Final Poster
5
5 See the le at
Available for free at ConnexionsChapter 5
1
Future Work
The most obvious area for future work is improving the algorithms. As mentioned in the results, all three
features could use some ne tuning to improve their results.
Analysis Improvement
New, more powerful analysis algorithms could be used to both nd beats in the song and to separate
the dominant instrument. We would like to perform instrument source separation to regain versions of the
individual instrument signals in the song, then perform amplitude analysis on each of those separated signals
to produce the next piece. Such a method for blind source separation is described here.
2
In order to improve our beat detection algorithm, we would have liked to separate the signal into many
frequency bands, then run the energy peak analysis on each of those bands to produce a possible beat in each
band. These beats could then be analyzed to determine the most likely beat of the song. Such an algorithm
is described in the more advanced sections of the resource we used to create our beat detection algorithm.
That reference can be found here.
3
Development of Analysis Library
The next major area for future work would be converting the sound analysis section of the program into a
standalone library, which developers could build into their games to improve gameplay through audio driven
events. Such a library would also contain a more abstracted and scalable version of the audio pipeline that
would allow for more stages to be placed in the audio pipeline. These stages could allow for even more
processing to be completed, but they also allow for e cient implementations of new features, such as the
ability to have the next piece in a Tetris game be generated such that not only is it based on the dominant
instrument, but that it also changes with the beat of the song.
1 This content is available online at .
2 http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6069287
3 http://archive.gamedev.net/archive/reference/programming/features/beatdetection/index.html
Available for free at Connexions
1516 INDEX
Index of Keywords and Terms
Keywords are listed by the section with that keyword (page numbers are in parentheses). Keywords
do not necessarily appear in the text of the page. They are merely associated with that section. Ex.
apples, 1.1 (1) Terms are referenced by the page they appear on. Ex. apples, 1
E elec 301, 1.1(3), 4(13)
Available for free at ConnexionsATTRIBUTIONS 17
Attributions
Collection: Tetris Sound: Using Music to Aect Gameplay
Edited by: Taylor Vaughn, Galen Schmidt, Jeremy Hunt
URL: http://cnx.org/content/col11609/1.2/
License: http://creativecommons.org/licenses/by/3.0/
Module: "Background and Motivation"
By: Galen Schmidt, Taylor Vaughn, Jeremy Hunt
URL: http://cnx.org/content/m48536/1.1/
Page: 1
Copyright: Galen Schmidt, Taylor Vaughn, Jeremy Hunt
License: http://creativecommons.org/licenses/by/3.0/
Module: "Amplitude Analysis for Drop Speed"
By: Taylor Vaughn, Galen Schmidt, Jeremy Hunt
URL: http://cnx.org/content/m48513/1.3/
Pages: 3-4
Copyright: Taylor Vaughn
License: http://creativecommons.org/licenses/by/3.0/
Module: "Beat Analysis for Game Visualization"
By: Taylor Vaughn, Jeremy Hunt, Galen Schmidt
URL: http://cnx.org/content/m48520/1.1/
Pages: 5-7
Copyright: Taylor Vaughn, Jeremy Hunt, Galen Schmidt
License: http://creativecommons.org/licenses/by/3.0/
Module: "Frequency Analysis for Piece Selection"
By: Taylor Vaughn, Jeremy Hunt, Galen Schmidt
URL: http://cnx.org/content/m48541/1.1/
Pages: 7-8
Copyright: Taylor Vaughn, Jeremy Hunt, Galen Schmidt
License: http://creativecommons.org/licenses/by/3.0/
Module: "Implementing a Pipelined Sound Engine"
By: Jeremy Hunt, Galen Schmidt, Taylor Vaughn
URL: http://cnx.org/content/m48525/1.1/
Pages: 9-10
Copyright: Jeremy Hunt, Galen Schmidt, Taylor Vaughn
License: http://creativecommons.org/licenses/by/3.0/
Module: "Implementing the Game"
By: Galen Schmidt, Jeremy Hunt, Taylor Vaughn
URL: http://cnx.org/content/m48545/1.1/
Page: 11
Copyright: Galen Schmidt, Jeremy Hunt, Taylor Vaughn
License: http://creativecommons.org/licenses/by/3.0/
Available for free at Connexions18 ATTRIBUTIONS
Module: "The Final Game, Conclusions, and Poster"
By: Galen Schmidt, Jeremy Hunt, Taylor Vaughn
URL: http://cnx.org/content/m48539/1.3/
Pages: 13-14
Copyright: Galen Schmidt, Jeremy Hunt, Taylor Vaughn
License: http://creativecommons.org/licenses/by/3.0/
Module: "Future Work"
By: Jeremy Hunt, Galen Schmidt, Taylor Vaughn
URL: http://cnx.org/content/m48542/1.1/
Page: 15
Copyright: Jeremy Hunt, Galen Schmidt, Taylor Vaughn
License: http://creativecommons.org/licenses/by/3.0/
Available for free at ConnexionsTetris Sound: Using Music to Aect Gameplay Overview of nal project for ELEC 301 About Connexions Since 1999, Connexions has been pioneering a global system where anyone can create course materials and make them fully accessible and easily reusable free of charge. We are a Web-based authoring, teaching and learning environment open to anyone interested in education, including students, teachers, professors and lifelong learners. We connect ideas and facilitate educational communities. Connexions's modular, interactive courses are in use worldwide by universities, community colleges, K-12 schools, distance learners, and lifelong learners. Connexions materials are in many languages, including English, Spanish, Chinese, Japanese, Italian, Vietnamese, French, Portuguese, and Thai. Connexions is part of an exciting new information distribution system that allows for Print on Demand Books. Connexions has partnered with innovative on-demand publisher QOOP to accelerate the delivery of printed course materials and textbooks into classrooms worldwide at lower prices than traditional academic publishers.
You can also read
