Provisioning On-line Games: A Traffic Analysis of a Busy Counter-Strike Server
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Provisioning On-line Games: A Traffic
Analysis of a Busy Counter-Strike Server
Wu-chang Feng Francis Chang Wu-chi Feng Jonathan Walpole
OGI School of Science and Engineering at OHSU
Abstract— This paper describes the results of a 500 million of what on-line games of the future are moving towards:
packet trace of a popular on-line, multi-player, game server. The large-scale, highly interactive, virtual worlds [3]. By na-
results show that the traffic behavior of this heavily loaded game
ture, traffic generated in support of this type of applica-
server is highly predictable and can be attributed to the fact that
current game designs target the saturation of the narrowest, last-mile tion is completely different than web or TCP-based traffic
link. Specifically, in order to maximize the interactivity of the game which has received most of the attention in the network re-
and to provide relatively uniform experiences between all players, search community [4], [5], [6], [7]. In particular, on-line
on-line games typically fix their usage requirements in such a way gaming requires low-latency point-to-point communica-
as to saturate the network link of their lowest speed players. While
tion as well as directed broadcast channels to facilitate
the traffic observed is highly predictable, the trace also indicates
that these on-line games provide significant challenges to current its real-time game logic. In addition, such traffic tends
network infrastructure. Due to synchronous game logic requiring to employ small, highly periodic, UDP packets. Pack-
an extreme amount of interactivity, a close look at the trace re- ets are small since the application requires extremely low
veals the presence of large, highly periodic, bursts of small pack- latencies which makes message aggregation impractical.
ets. With such stringent demands on interactivity, routers must be
designed with enough capacity to quickly route such bursts with-
Packets are highly periodic as a result of the game’s dy-
out delay. As current routers are designed for bulk data transfers namic requirement of frequent, predictable state updates
with larger packets, a significant, concentrated deployment of on- amongst clients and servers. Finally, packets are sent via
line game servers will have the potential for overwhelming current UDP since clients typically send packets at an interval that
networking equipment. is much shorter than the time it would take to retransmit
I. I NTRODUCTION lost packets. In addition, the latency induced via socket
buffers [8] and delayed acknowledgements is often too
Due to the recent global explosion of on-line multi- large to support meaningful interactivity. In this paper,
player gaming, it is becoming more important to under- we take a closer look at a class of applications that look to
stand its network behavior and usage in order to provision become a major component in the traffic mix in the future.
and design future network infrastructure. With the up-
coming launches of Microsoft’s Xbox and Sony’s Playsta- II. BACKGROUND
tion 2 on-line game networks and with the emergence of
massively multi-player on-line games [1], it is clear that In order to understand the characteristics of Internet
a large increase in gaming traffic is imminent. While not gaming, we examined the behavior of an extremely popu-
indicative of all on-line games, the class of games known lar Counter-Strike server [3]. Counter-Strike is a modifi-
as “first-person shooters” has dominated much of today’s cation to the popular Half-Life game and has become one
gaming traffic [2]. With a large list of popular games such of the most popular and most network-intensive games
as Doom, Quake, Half-Life, Counter-Strike, Unreal Tour- played over the Internet as of this writing. Counter-Strike
nament, Day of Defeat, Medal of Honor, Command & is a part of a large class of multi-player, on-line, first-
Conquer Renegade, etc., these games are representative person shooters that has dominated network gaming traf-
fic over the last several years. As of May 2002, there
This work is supported by the National Science Foundation under were more than 20,000 Counter-Strike servers active [3].
Grant EIA-0130344 and the generous donations of Intel Corporation.
Any opinions, findings, or recommendations expressed are those of the The game is architected as a client-server application with
author(s) and do not necessarily reflect the views of NSF or Intel. multiple clients communicating and coordinating with a
central server that keeps track of the global game state.
Traffic generated by the game can be attributed to a num-
ber of sources. The most dominant source is the real-time
action and coordinate information sent back and forth be-
tween clients and server. This information is periodically
sent from all of the clients to the server. The server then
takes this information and performs a periodic broadcastStart Time Thu Apr 11 08:55:04 2002 tional Internet connectivity, the speed of the server used,
Stop Time Thu Apr 18 14:56:21 2002 and the superiority of our game configuration (which in-
Total Time 7 d, 6 h, 1 m, 17.03 s cludes modules that eliminate cheating and deter team
killing [12]), this server quickly became heavily utilized
Maps Played 339
with connections arriving from all parts of the world ir-
Established Connections 16030 respective of the time of day. The server itself was con-
Unique Clients Established 5886 figured with a maximum capacity of 22 players. After a
Attempted Connections 24004 brief warm-up period, we recorded the traffic to and from
Total Packets 500,000,000 the server over the course of a week (April 11-18). The
Total Packets In 273,846,081 trace collected consisted of a half billion packets. Note
Total Packets Out 226,153,919 that while we are able to effectively analyze this single
Total Bytes 64.42 GB server, the results in this study do not directly apply to
Total Bytes In 24.92 GB overall aggregate load behavior of the entire collection of
Total Bytes Out 39.49 GB Counter-Strike servers. In particular, it is expected that
active user populations will not, in general, exhibit the
Mean Packet Load 798.11 pps
predictability of the server studied in this paper and that
Mean Packet Load In 437.12 pps
the global usage pattern itself may exhibit a high degree
Mean Packet Load Out 360.99 pps
of self-similarity [2], [13], [14].
Mean Bandwidth 883 kbs
Table III-A summarizes the trace itself. The trace cov-
Mean Bandwidth In 341 kbs
ers over a week of continuous operation of the game
Mean Bandwidth Out 542 kbs
server. Over 300 maps were played during this time frame
Mean Packet Size 80.33 bytes
and more than 16000 user sessions were established. Due
Mean Packet Size In 39.72 bytes to the popularity of the server, more than 8000 connec-
Mean Packet Size Out 129.51 bytes tions were refused due to the lack of open slots on the
TABLE I
server and each player was connected to the game an av-
erage of approximately 15 minutes. In addition, over the
T RACE INFORMATION
500 million packet trace, more than 60 GB of data were
to each client, effectively distributing the global state of sent including both network headers and application data.
the game. In addition to game physics datum, the game Overall, the bandwidth consumed approached 1M bs for
engine allows for broadcast text-messaging and broad- the duration of the trace. The table also shows that even
cast voice communication amongst players all through the though the application received more packets than it sent,
centralized server. The game server also supports the up- its outgoing bandwidth exceeded its incoming bandwidth.
load and download of customized logos that can be seen This was because the mean size of outgoing application
by everyone on a per-client basis. Each client is able to data packets was more than three times the size of incom-
customize a texture map that may be placed on the sur- ing application data packets.
faces of the current map. These images are uploaded and
To understand the dynamics of the trace, Figure 1 plots
downloaded when users join the game and when a new
the per-minute average bandwidth and packet load ob-
map starts so that each client can properly display the cus-
served at the server. As the figure shows, while there is a
tom decals of the other users. Finally, the game server
lot of short-term variation in the trace, the trace exhibits
supports downloads of entire maps, which may consist
fairly predictable behavior over the long-term. Aggregate
of sounds, texture libraries and a compiled Binary Space
bandwidth consumed by the server hovers around 900
Partitioning tree [9]. In order to prevent the server from
kilobits per second (kbs) while the server sees a packet
becoming overwhelmed by concurrent downloads, these
rate of around 800 packets per second (pps). In addition,
downloads are rate-limited at the server.
throughout the experiment, the number of active players
III. E VALUATION also remained at or near the capacity of 22. The trace also
encompasses several brief network outages that did not
A. Trace summary significantly impact the analysis of the results [12].
To properly evaluate the traffic generated by this rep-
resentative application, we hosted a shared Counter- B. Periodicity and predictability
Strike (version 1.3) server for two of the most popu- While visually it appears that the server’s network load
lar on-line gaming communities in the Northwest region: is relatively stable, the true measure of variability is its
olygamer.com and mshmro.com [10], [11]. Because of associated Hurst parameter [6], [15]. In order to measure
the large followings of these communities, our excep- variability across multiple time scales, we use the stan-1300 1200
1200
1100 1000
1000
Packet load (packets/sec)
Bandwidth (kilobits/sec)
900
800
800
700
600
600
500
400
400
300
200 200
100
4/13 4/14 4/16 4/18 4/13 4/14 4/16 4/18
4/12
4/15
4/17
4/12
4/15
4/17
4/11
4/11
0 0
0 2000 4000 6000 8000 10000 0 2000 4000 6000 8000 10000
Time (min) Time (min)
(a) Total bandwidth (b) Packet load
Fig. 1. Per-minute network load of server for entire trace
the number of frames per block. The Hurst parameter (H)
0.0 can be estimated by taking the magnitude of the slope of
−1.0
the best-fit line through the data points (β) and calculating
H via the relation H = 1 − β2 . The Hurst parameter thus
−2.0
ranges between 12 and 1. A short-range or no dependence
log (normalized variance)
−3.0 sequence will have a slope of −1 which corresponds to an
−4.0
H of 12 while a long-range dependent sequence will have
an H closer to 1.
−5.0
Figure 2 shows the variance-time plot of the trace. For
−6.0 this plot, the interval size that is plotted is the normalized
50ms 30min interval size using a base interval of m = 10ms. The
−7.0 H=1/2
Server trace
plot shows three distinct regions of behavior. For small m
−8.0 (m < 50ms), there is a high degree of increased smooth-
0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0
log (interval size) ness as the interval size is increased, as shown by the
slope of the variance plot as H drops below 12 . For larger
Fig. 2. Variance time plot for total server packet load interval sizes (50ms < m < 30min), significant vari-
ability and burstiness remains even as the interval size is
increased. Finally, for large interval sizes (m > 30min),
dard aggregated variance method to estimate the Hurst pa-
the plot shows typical behavior for a short-range depen-
rameter (H) of the trace. In this method, the sequence is
dent sequence as larger interval sizes consistently produce
divided into multiple, consecutive, equally-sized, blocks.
reduced variability with an estimated H of around 12 .
The values within the block are averaged and the vari-
ance of the sequence of averages is calculated. For a To fully understand the behavior at varying time-scales,
short-range dependent process, as the block size (m) is Figure 3 plots the total packet load observed at the server
increased, the variance of the resulting sequence consis- averaged over a range of time intervals. Figure 3(a) plots
tently decreases. In contrast, for long-range dependent the first 200 10ms intervals of the trace. The figure ex-
sequences, the sequence maintains high variability across hibits an extremely bursty, highly periodic pattern which
block sizes and time scales. To determine the degree of is the result of the synchronous operation of the game
long-range dependence, the log of the normalized vari- server logic itself which is written to deterministically
ance is plotted against the log of the block size. The nor- flood its clients with state updates about every 50ms [12].
malized variance is calculated as the variance of the ag- Given this behavior, Figure 3(b) shows the plot of the first
gregated sequence divided by the variance of the initial, 200 50ms intervals of the trace. As expected, aggregat-
unaggregated sequence. The block size, in this case, is ing over this interval smooths out the packet load con-2500 1500
2000
Packet load (packets/sec)
Packet load (packets/sec)
1000
1500
1000
500
500
0 0
0 50 100 150
200 0 50 100 150
200
Interval number Interval number
(a) m = 10ms (b) m = 50ms
1500 1500
Packet load (packets/sec)
1000 Packet load (packets/sec) 1000
500 500
0 0
0 3600 7200 10800
14400
18000 0 50 100 150
200
Interval number Interval number
(c) m = 1sec (d) m = 30min
Fig. 3. Total packet load over a range of interval sizes
siderably. Between m = 50ms and m = 30min, how- variability. Figure 3(d) plots the first 200 30min intervals
ever, the variance-time plot exhibits a high degree of vari- of the trace. As the figure shows, the variability has been
ability. This variability can be attributed to the network eliminated.
disruptions caused by the 30min map time of the server. The predictability of the aggregate leads us to exam-
As the server loads a new map every half-hour, the net- ine how predictable the resource consumption of each in-
work traffic dips significantly for a short period of time. dividual flow is in the trace. Perhaps the most interest-
Because most of the clients will have the maps stored lo- ing observation is that when the mean bandwidth of the
cally, this down time is due completely to the server doing server is divided by the total number of players allowed
local tasks to perform the map change over. Figure 3(c) by the game server itself (22), the bandwidth consumed
shows this behavior with a plot of the first 18000 1sec per player is on average 40kbps. This is no coincidence
intervals. Noticeable dips appear every 1800 (30min) in- as the game is meant to be played uniformly across a wide
tervals. Because map changing is a configuration-specific range of network speeds, down to and including the ubiq-
feature, this behavior is not a generic characteristic and uitous 56kbps modem. As typical performance of 56kbps
can be affected by game administrators changing maps modems range from 40 − 50kbs [16], it is clear that this
directly, players voting to change maps or extending the particular game was designed to saturate the narrowest
current map, or a different map time limit setting. For last-mile link. Going back to the trace itself, we mea-
this trace and server configuration, increasing the interval sured the mean bandwidth consumed by each flow at the
size beyond the default map time of 30min removes theure shows, almost all of the incoming packets are smaller
100 than 60 bytes while a large fraction of outgoing packets
have sizes spread between 0 and 300 bytes. This is sig-
80 nificantly different than aggregate traffic seen within In-
ternet exchange points [2] in which the mean packet size
observed was above 400 bytes.
Number of players
60
IV. I MPLICATIONS ON ROUTING I NFRASTRUCTURE
40 Perhaps the most significant aspect of the trace is the
observation that the game traffic itself consists of large,
20
periodic bursts of short packets. While the trace is of only
a single game over a single week, we believe that this is
a characteristic that will be fundamental in all sufficiently
0
0 50000 100000 150000 loaded, highly interactive, on-line games due to the nature
Bandwidth (bits/sec) of the application and the underlying game logic. Short
packets are required for low latency while highly peri-
Fig. 4. Client bandwidth histogram odic traffic allows the game to provide uniform interac-
tivity amongst all of its clients. Some evidence of this ex-
server in order to get a picture of the bandwidth distri-
ists in aggregate measures of other game applications [2].
bution across clients. Assuming minimal packet loss and
Unfortunately for games, routers are not necessarily de-
a negligible difference in link-layer header overhead be-
signed for this type of traffic. With the explosion of the
tween the last-mile link and the server’s link, the band-
web and peer-to-peer networks, the majority of traffic be-
width measured at the server will be quite close to what
ing carried in today’s networks involve bulk data trans-
is sent across the last hop. Figure 4 shows a histogram
fers using larger-sized TCP segments. Router designers
of bandwidths across all sessions in the trace that lasted
and vendors often make packet size assumptions when
longer than 30sec. The figure shows that the overwhelm-
building their gear, often expecting average sizes in be-
ing majority of flows are pegged at modem rates or be-
tween 1000 and 2000 bits (125-250 bytes) [18]. Thus, a
low even though connections arrived via diverse network
significant shift in packet size from the deployment of on-
mediums. The figure also shows that some flows do, in
line games will make the route lookup function the bot-
fact, exceed the 56kbps barrier. This is due to the fact
tleneck versus the link speed [19]. Routing devices that
that the client can be specially configured to crank up the
are not designed to handle small packets will then see sig-
update rate to and from the server in order to improve the
nificant packet-loss or even worse consistent packet delay
interactivity of gameplay even futher. As shown by the
and delay jitter when handling game traffic [20]. Anec-
histogram, only a handful of elite (”l337”) players con-
todal evidence using a commerical-off-the-shelf NAT de-
necting via high speed links have taken advantage of the
vice corroborates this [12]. Without the routing capacity
setting.
in place, hosting a successful game server behind such
C. Tiny packets a device is simply not feasible. Extending this to on-
While the good news in the trace is that for a fixed set line ventures from Microsoft and Sony and to massively
of players, the traffic generated is highly stable and pre- multi-player games, it is apparent that even mid-range
dictable, the bad news is that the traffic itself is made up of routers or firewalls within several hops of large hosted on-
large, periodic bursts of very small packets. Figure 5(a) line game servers will need to be carefully provisioned to
shows the PDF of both incoming and outgoing packets. minimize both the loss and delay induced by routing ex-
While incoming packets have an extremely narrow dis- tremely small packets. This will almost certainly require
tribution centered around the mean size of 40 bytes, out- increasing the peak route lookup capacity of intermediate
going packets have a much wider distribution around a routers as adding buffers will add an unacceptable level
significantly larger mean. This causes the outgoing band- of delay.
width to exceed the incoming bandwidth even though the The good news about the trace is that the predictabil-
rate of incoming packets exceeds that of outgoing pack- ity in resource requirements makes the modeling, simula-
ets. This is not surprising as the game server itself is log- tion, and provisioning on-line gaming traffic a relatively
ically playing the role of a broadcaster: taking state in- simple task as the traffic does not exhibit fractal behav-
formation from each client and broadcasting it out to all ior when the number of active players is relatively fixed.
other clients [17]. Figure 5(b) shows the cumulative dis- As a result of this predictability, the traffic from an ag-
tribution function (CDF) of the packet sizes. As the fig- gregation of all on-line Counter-Strike players is effec-0.10 1.0
Inbound
0.09 Outbound 0.9
0.08 0.8
0.07 0.7
0.06 0.6
Probability
Probability
0.05 0.5
0.04 0.4
0.03 0.3
0.02 0.2 Inbound
Outbound
0.01 0.1
Total
0.00 0.0
0 100 200 300 400 500 0.0 100.0 200.0 300.0 400.0 500.0
Packet size (bytes) Packet size (bytes)
(a) PDF (b) CDF
Fig. 5. Packet size distributions of trace
tively linear to the number of active players. While this [7] V. Paxson and S. Floyd, “Wide-Area Traffic: The Failure of Pois-
is the case, the actual number of players on-line over time son Modeling,” in Proceedings of ACM SIGCOMM, August 1994,
pp. 257–268.
may, in fact, exhibit a high degree of variability and self- [8] A. Goel, C. Krasic, K. Li, and J. Walpole, “Supporting Low La-
similarity. Self-similarity in aggregate game traffic in this tency TCP-Based Media Streams,” in Proceedings of IWQoS, May
case will be directly dependent on the self-similarity of 2002.
user populations [13], [14]. Since the trace itself can be [9] H. Fuchs, Z. M. Kedem, and B. F. Naylor, “On Visible Sur-
face Generation by a Priori Tree Structures,” Computer Graph-
used to more accurately develop source models for sim- ics(SIGGRAPH ’80 Proceedings), vol. 14, pp. 124–133, July
ulation [21], we hope to make the trace and associated 1980.
game log file publicly available [22]. The other silver lin- [10] olygamer.com, ,” http://www.olygamer.com/.
ing in this trace is that while the small packets of on-line [11] mshmro.com, ,” http://www.mshmro.com/.
[12] W. Feng, F. Chang, W. Feng, and J. Walpole, “Provisioning On-
games have the potential to wreak havoc on routing infras- line Games: A Traffic Analysis of a Busy Counter-Strike Server,”
tructure, the periodicity and predictability of packet sizes in OHSU TR CSE-02-005, May 2002.
allows for meaningful performance optimizations within [13] T. Henderson and S. Bhatti, “Modelling User Behavior in Net-
worked Games,” in ACM Multimedia, 2001, pp. 212–220.
routers. For example, preferential route caching strate-
[14] T. Henderson, “Latency and User Behaviour on a Multiplayer
gies based on packet size or packet frequency may pro- Game Server,” in Networked Group Communication, 2001, pp.
vide significant improvements in packet throughput. We 1–13.
hope to explore these issues further on a network proces- [15] H.E. Hurst, “Long-Term Storage Capacity of Reservoirs,” Proc.
American Society of Civil Engineers, vol. 76, no. 11, 1950.
sor testbed [23]. [16] J. Kristoff, “Understanding 56Kbps Modems,” http:
R EFERENCES //homepage.interaccess.com/˜jkristof/
56kmodem.html, 1997.
[1] GameSpy.com, “What’s This World Coming To? The Future [17] D. LaPointe and J. Winslow, “Analyzing and Simulating Network
of Massively Multiplayer Games,” http://www.gamespy. Game Traffic,” http://www.cs.wpi.edu/˜claypool/
com/gdc2002/mmog. mqp/net-game/game.pdf, December 2001.
[2] S. McCreary and k. claffy, “Trends in Wide Area IP Traffic Pat- [18] C. Partridge et. al, “A 50-Gb/s IP Router,” IEEE/ACM Transac-
terns: A View from Ames Internet Exchange,” in Proceedings of tions on Networking, vol. 6, no. 3, pp. 237–245, June 1998.
13th ITC Specialist Seminar on Measurement and Modeling of IP [19] Super Computer Gaming, “Super Computer Gaming: Jupiter
Traffic, September 2000, pp. 1–11. Cluster,” http://www.supercomputergaming.com/,
[3] Counter-Strike, “Counter-Strike: A Counter-Terrorism Half-Life 2002.
Modification,” http://www.counter-strike.net/. [20] I. S. MacKenzie and S. Ware, “Lag as a Determinant of Human
[4] P. Barford and M. Crovella, “Generating Representative Web Performance in Interactive Systems,” Proceedings of the ACM
Workloads for Network and Server Performance Evaluation,” in Conference on Human Factors in Computing Systems - INTER-
Proceedings of ACM SIGMETRICS, July 1998, pp. 151–160. CHI, pp. 488–493, 1993.
[5] A. Feldmann, A. Gilbert, and W. Wilinger, “Data Networks as [21] M. Borella, “Source Models of Network Game Traffic,” Computer
Cascades: Investigating the Multifractal Nature of Internet WAN Communications, vol. 23, no. 4, pp. 403–410, February 2000.
Traffic,” in Proceedings of ACM SIGCOMM, September 1998. [22] W. Feng, “cs.mshmro.com Counter-Strike Server Traffic Trace
[6] W. Leland, M. Taqqu, W. Willinger, and D. Wilson, “On 4/11-4/18,” 40GB trace available upon request, April 2002.
the Self-Similar Nature of Ethernet Traffic (Extended Version),” [23] E. Johnson and A. Kunze, “IXP1200 Programming,” 2002.
IEEE/ACM Transactions on Networking, vol. 2, no. 1, February
1994.You can also read
