Networking Best Practices
By Mark Seminatore, Development Lead
XNA Developer Connection (XDC)
Many common networking techniques that apply to Windows game development also apply to writing multiplayer games for Xbox 360. However, when it comes to networking, Xbox 360 provides both advantages and unique challenges. This white paper summarizes networking best practices to consider when creating Xbox LIVE games.
Prefer Xbox 360-to-Xbox 360 Networking
Many Windows multiplayer games follow a client-server model, where a Windows computer acts as the game client and a dedicated game server authoritatively manages the game simulation. Although custom game servers are supported for Xbox LIVE games, a number of factors make an Xbox-to-Xbox model appealing. These factors include the following:
Engineering, certifying, and supporting custom game servers in Microsoft-approved data centers are complex and costly operations. An Xbox-to-Xbox topology avoids significant development issues and ongoing support costs.
Peer-to-peer games must deal with Network Address Translator(NAT) traversal, a technically difficult problem. The Xbox LIVE architecture and the Xbox 360 Secure Network Library (SNL) automatically handle NAT traversal. This means that most Xbox 360 consoles can communicate with other Xbox 360 consoles, even if one or both consoles are behind NATs.
The Xbox LIVE Matchmaking system can help ensure that when players join games, they join games that are logically close to each other. This implies better connections than could be obtained with servers in a remote data center.
It can be costly in terms of server processing and bandwidth to route voice chat through a central server.
The built-in security of the Xbox 360 network layer means games don't have to rely on an external server-based authority.
Note that the Xbox-to-Xbox model does not imply that you can't have a client-server style topology. For example, one Xbox 360 console could act as the server?either a dedicated server, or a server that also functions as a client.
Avoid Small Payloads
All network packets require some level of overhead, typically in the form of packet headers, which allow routers and receivers to interpret data. Under Windows, the minimum overhead for a UDP packet is 28 bytes. However, the secure Xbox 360 protocol has a minimum of 44 bytes of overhead per UDP packet. This means that if an Xbox 360 title calls UDP send() with a single byte of data, 98 percent (44 out of 45 bytes sent) of the network bandwidth is devoted to packet overhead, while only 2 percent is devoted to actual game traffic. You can minimize this penalty by sending larger payloads less frequently. One way to do this is to coalesce smaller sends into a larger packet before calling send(). Don't send multiple small payloads if one larger payload can work just as well.
Minimize Bandwidth Usage
Developing for broadband has fantastic advantages compared to narrowband. But broadband does have its limits. Speeds quoted by broadband providers are theoretical maximums. Don't assume that 256 Kbps upstream bandwidth is guaranteed. Real available bandwidth is dependent upon a number of factors. Actual performance between any two Xbox 360 consoles will be slower than what broadband providers advertise.
To ensure that your game can be played across all broadband connections, avoid sending too much data. Be aware of how many packets are sent over a given time interval. Also be aware of how much game data is carried by a typical packet.
Ensure that the information that is sent is absolutely necessary. Examine game payloads, and then evaluate whether these payloads really need to be sent, or if the receiver can calculate, extrapolate, interpolate, or predict the data. For example, don't send data for objects that are out of view, occluded, inactive, recently destroyed, about to be removed, or that otherwise don't affect the client in the current or near future game context. Don't assume that all game objects are important to all clients.
Prioritize your game data based on context. For example, objects that are far away from the viewpoint may not require orientation or animation state information to be sent. Don't assume that all game object attributes must be up to date all the time.
If you determine that data does need to be sent, compress the data any way you can. Examples of different ways to compress data include the following:
If you're sending C/C++ structures, be sure these structures are single-byte aligned. (By default, structures are 8-byte aligned for speed, which introduces unused packing bytes into the structure.) Use the #pragma pack specifier to pack structures selectively. The following code provides an example:
#pragma pack(push) // save current alignment setting
#pragma pack(1) // specify single-byte alignment
}; // sizeof(NetMsg) is now 5 instead of 8
#pragma pack(pop) // return to original alignment setting
Take advantage of fixed number ranges and limits. For example, if a particular data item has a nonzero minimum value, you can subtract the minimum before transmission to reduce the bits that are required to represent the value. The minimum value can be restored on the receiving side.
If possible, quantize and scale data to reduce the number of bits required. For integers, send 8-bit numbers instead of 16-bit numbers, or 16-bit numbers instead of 32-bit numbers. Better yet, consider packing information into bit fields. For instance, a number in the 0-7 range can be packed into 3 bits (log base 2 of 8 is 3). For floating-point values, use FLOAT instead of DOUBLE, or send fixed point values.
Pack multiple Boolean values into bytes as bit flags.
Use custom encoding schemes or transformations. For instance, send polar coordinates (angle, distance) rather than 3D position (x, y, z).
Send deltas rather than the entire state.
Send random number seeds rather than an entire simulation state.
Avoid sending strings. Send identifiers or tokens instead.
Prioritize data. On slow/small connections, send critical data only. Send additional data on fast/large connections.
Send data for far objects less frequently than data for near objects. Send data for unimportant objects less often than data for important objects.
Consider other forms of data compression, especially if you're sending large chunks of data.
Limit Packet Frequency
Many Xbox 360 game players choose the least expensive broadband package that is available. Your game should be playable on these connections. Therefore, avoid sending packets whenever possible. Don't tie the packet send rate to the frame rate of your game. For example, if your game runs at 60 frames per second, it does not necessarily mean that it should send packets 60 times per second.
Fast action client-server games on Windows computers typically receive packets from game servers at a rate of 10 to 20 packets per second. Clients typically send packets to the server at a lower rate. By queuing data and sending it at regular intervals, rather than on an ad hoc basis, you reduce bandwidth. Keep in mind that not all games require packets to be sent on a regular basis. If you can get away with sending only three packets during the first second, ten packets during the next second, and one packet during the third second, then by all means do so.
Preference: Limit sends to 10-20 packets per second.
Regrettably, the speed of light is not fast enough to make multiplayer game development easy. A game player in Los Angeles with a theoretically ideal connection to a player in New York always has at least a 28-millisecond roundtrip latency. The same player has a 58-millisecond latency with a player in Tokyo or London. (Light travels at 186.3 miles per millisecond in a complete vacuum. Roundtrip latency induced by the speed of light in milliseconds equals miles × 2 / 186.3.) Additional latency is contributed by the following factors:
Fiber optic cable and copper wiring. The speed of light in fiber and copper is as low as 60 percent of the speed of light in a vacuum.
Modems, routers, and network stacks. DSL and cable modems have latencies in the neighborhood of 10 milliseconds. Routers have varying latencies, ranging from 1-5 milliseconds. However, a congested router that has a massive queue can exhibit latencies up to 50 milliseconds. The more routers a packet traverses, the more latency is induced.
Games themselves. Games that limit packet frequency (a preferred practice) introduce latency. If packets are sent and received ten times per second, then an average additional roundtrip latency of 100 milliseconds, and a maximum roundtrip latency of 200 milliseconds, is incurred. Frame rate also introduces latency. Even if a 60 frame per second game receives a new object position, the game doesn't display the new position until the next frame, which can be up to 16.7 milliseconds later.
Because lag exists even for two players who are next door to each other, it's essential that you account for lag in your game, and handle it appropriately. In a turn-based game, lag can be hidden by the turn. In an action game, lag must be compensated for in other ways. These methods include using approximation, dead reckoning, cubic splines, and loose synchronization.
Dozens of good resources exist that describe methods for lag compensation. Consult the Internet, simulation textbooks, and game programming literature (see the Resources section of this white paper). Be aware that some of the best solutions involve game design trade-offs, and not technical solutions.
Prefer VDP over UDP
VDP is the Xbox 360 Voice and Data Protocol, which is a custom protocol that allows games to send both voice data and game data in the same packet. If your game uses voice, you must use VDP. Not only can the VDP protocol send both voice and game data in the same packet, it can also be used to send voice or game data separately. The game data portion of the payload is automatically encrypted. The voice data portion is not encrypted.
SOCKET s = socket(AF_INET, SOCK_DGRAM, IPPROTO_VDP);
Messages that are sent on a VDP socket must be in the following format:
If the data sent on a VDP socket includes only game data (so that message payload size equals cbGameData), the network libraries can optimize cbGameData for transmission by eliminating it. This makes VDP as efficient as UDP.
Because of packet overhead, avoid using one socket for voice data and another socket for game data. Whenever possible, use a single VDP socket. You can send game data if there is only game data to send. You can send voice data if there is only voice data to send. And you can send both types of data in a single packet if you need to send both. VDP is the best general purpose socket protocol on Xbox 360.
Prefer UDP over TCP
If you're not sending voice, you can use any socket type?VDP, UDP or TCP. VDP and UDP exhibit the best performance characteristics in terms of network traffic and CPU usage. But because both VDP and UDP are connectionless protocols, using these protocols implies that games need to handle both dropped and out-of-order packets. TCP has guaranteed in-order delivery, but requires additional bandwidth because of larger headers and TCP handshaking.
In addition, if TCP needs to resend data because of dropped packets, latency can skyrocket. There are some cases where TCP may be the best option?for synchronizing state, player handshaking, connection management, and exchanging critical game state, for example. However, typical game data is best sent on VDP or UDP sockets. Avoid using TCP for normal gameplay data.
Minimize Voice Traffic
Don't send voice traffic if the receiver is not going to use it. For instance, muted players should not send voice data to players who mute them. Similarly, it might not make sense to hear from a player who is invisible to the receiver.
Consider interesting game design techniques for limiting voice, such as microphone pick-ups, walkie-talkies, voice time limits, speaker podiums, player proximity, player visibility, prioritization (friends first and proximity second, for example), and voice channels (four players per channel, for example). Keep in mind that in typical conversion, a person can distinguish only three or four simultaneous voice streams.
Send Voice Data Peer-to-Peer
Aggregating voice data through a single server can be expensive in terms of both CPU performance and bandwidth. Send voice data from Xbox console to Xbox console even if you use a client-server model. This may limit the ability to piggy-back voice and data on a single VDP packet, but the trade-off is worthwhile.
To minimize packet overhead, send and receive data on a single port if possible. Rather than using port numbers, use identifiers in the payload to indicate different types of data. Sending a merged payload on a single port (rather than on two ports) saves a minimum of 44 bytes of network traffic per packet.
Use Port 1000
The Xbox 360 Secure Network Library (SNL) has a special optimization for port 1000. To minimize packet overhead, you should send and receive all data on port 1000 for both source and destination. This can save up to four bytes per packet (two 2-byte port numbers). If you can't send all data on port 1000, we recommend that you send and receive on port numbers in the 1001-1255 range. This can save two bytes per packet (two 1-byte port numbers are sent). If the source port and the destination port are in different port categories, then both ports must use the larger of the two port description sizes. Take advantage of these optimizations, and don't underestimate the cumulative cost of extra packet overhead.
Source Port Dest Port Packet Overhead (bytes) NetMon Packet Types
1000 1000 0 TCP0, UDP0, VDP0, VDPVO0, VDPDO0
1001-1255 1001-1255 2 TCP1, UDP1, VDP1, VDPVO1, VDPDO1
1000 1001-1255 2 TCP1, UDP1, VDP1, VDPVO1, VDPDO1
1001-1255 1000 2 TCP1, UDP1, VDP1, VDPVO1, VDPDO1
1-65535 1-65535 4 TCP2, UDP2, VDP2, VDPVO2, VDPDO2
Use Payloads Divisible by Eight
The Xbox 360 Secure Network Layer pads payloads for encryption purposes. To minimize packet overhead, send payloads whose sizes are evenly divisible by eight. This can save up to 7 bytes per packet. In particular, avoid payloads that are 1-2 bytes larger than a size that is divisible by eight. For example, a payload of 33 bytes requires 7 bytes of padding, whereas a payload of 31 bytes requires only 1 byte of padding. If you can't avoid the extra space, use it to your advantage to send data that would otherwise be sent in the next packet.
The appropriate choice of ports, as well as an understanding of padding rules, can have a significant effect on packet overhead.
Packet Header Size
Protocol Windows Optimal Xbox 360 Worst-Case Xbox 360
TCP 40 56 60
UDP 28 44 51
VDP Data Only ? 44 51
VDP Voice Only ? 44 51
VDP Data and Voice ? 46 52
Don't Fling Packets Until the Connection is Established
The first time that two online Xbox 360 consoles communicate with each other, the two endpoints enter into a network handshaking period called key exchange. If either Xbox 360 console is behind a NAT, the handshaking may require a few seconds to complete?up to ten seconds if packets are dropped. During this time, although the game can send data to the other endpoint, the data is simply queued up on the sender until handshaking is complete. If too much data is queued up, the sender's connection is likely to drop packets. In this case, the receiver either doesn't get all the data it expected, or it gets the data later than expected, or both.
The best way to handle the initial communication is for the sender to wait for a response before it begins sending a full message stream.
Give Key Exchange Time to Complete
In Windows games, it's common to time out an initial connection after just a few seconds. Beyond this period of time, the likelihood of a good connection is relatively small. On the Xbox 360 console, the key exchange handshake may require a number of seconds to punch through NATs. If the initial connection requires a couple of seconds, the connection is not necessarily bad. It is more likely that the delay indicates that NAT traversal is being established, and the resulting connection will be just fine.
Wait at least three seconds for a response, and up to ten seconds, before giving up on the connection. While a connection is being established, calls to XNetGetConnectStatus() return XNET_CONNECT_STATUS_PENDING. Consider using the QoS functions, discussed below, to measure connectivity.
Use Host Migration
If the session host connection is lost, the gameplay experience can suffer. Fortunately, Xbox LIVE provides the ability to migrate the session host, by allowing gameplay to continue for one or more subsets of the original session. The steps involved in host migration are:
The game recognizes the need for migration.
The game identifies a new host.
The new host calls XSessionMigrateHost().
The new host receives new XSESSION_INFO and distributes it to other session members.
Session members call XSessionMigrateHost() with the new XSESSION_INFO.
All session members call XSessionLeaveRemote() for players no longer in the session.
The game updates any session-aware game features.
An ordered list of migration hosts can be pre-determined at matchmaking time and shared with all session participants. In order to avoid warnings about slot count mismatch, do not remove the old host prior to migration. Remove the old host after migration.
Arbitrated sessions can also be migrated. If migration occurs before XSessionStart() is called, the game should call XSessionArbitrationRegister() again after migration is completed. After XSessionStart() is called, the session can still be migrated, but re-registration is not possible and statistics are written to the original session.
Use XRNM or QNet
If you are developing exclusively for the Xbox 360 system, take advantage of the XRNM and QNet networking libraries. XRNM is fast and efficient, and it provides a complete networking library with a reliable VDP/UDP messaging layer. It also supports a flexible set of options, including packet sequencing, ordered delivery, and data aggregation.
QNet is targeted at Xbox LIVE Arcade titles. QNet allows Xbox 360 developers to quickly and easily add LIVE or system link functionality to games. QNet is built on top of XRNM, and provides a higher-level abstraction of LIVE and networking functionality.
Compute Ping Time Using the Quality of Service Functions
The XNetQoS (Quality of Service) family of functions computes ping times between two or more Xbox 360 consoles, and also between Xbox 360 consoles and servers. The QoS functions allow games to probe multiple connections simultaneously, with very little development effort. Using other techniques for computing ping times is discouraged, because other methods are less accurate, and also because they can't take Xbox 360 NAT traversal into account. For instance, when either the source or the destination Xbox 360 console is behind a NAT, computing ping times by measuring the time it takes to first establish a connection produces extremely inaccurate results. Always use the QoS functions to determine ping times.
The bandwidth estimates that are provided by the QoS functions are not completely accurate for a number of reasons. The reported bandwidth may be overstated or understated, or it may be accurate. Avoid setting data send rates based on bandwidth, and never display bandwidth estimates. Bandwidth measurements may be appropriate for making comparisons. Use ping times instead of bandwidth estimates for hosting or player count decisions.
Scale Based on Quality of Service
Rather than designing a game to handle a fixed maximum number of players, scale the game limitations based on the measured quality of service. This recommendation applies not only to player maximums, but to other data as well. For instance, a game could send additional data to Xbox 360 consoles that have high-quality broadband connections (voice and other non-critical data, for example), while sending only normal game data to Xbox 360 consoles that have lower-quality broadband connections.
Although XNetQosLookup() provides useful data, it may not be practical to use during gameplay. Consider developing a set of title specific quality metrics that take into account factors that are important to game play. This could be as simple as a roundtrip time counter that is added to certain packets. More sophisticated metrics could track per-connection packet loss, queue lengths, bandwidth usage, latency that includes game loop processing, and so on. XrnmQueryInfo() provides access to numerous metrics that can be queried and tuned to improve network performance. Use quality of service metrics periodically during the game to scale connections over time.
Recover From Dropped Packets
VDP and UDP sockets do not guarantee packet delivery, nor do they guarantee that packets are delivered in the same order that they are sent. Under normal Internet conditions, some packets are permanently lost, while other packets arrive late?sometimes really late. Write your network layer so that it can handle both dropped and late packets. This does not necessarily mean that you need to write your own guaranteed delivery protocol. If you really need guaranteed delivery, consider using XRNM or QNet, or perhaps even TCP.
Be aware that if a packet is dropped, it's likely that the next few packets will also be dropped. Plan accordingly. The following techniques can help the receiver recover from dropped packets.
Send redundant information, so that if one or more packets are dropped, the receiver still eventually gets the data. Yes, this goes against the recommendation to avoid sending too much data, but as with all things involved in networking, you must carefully weigh the tradeoffs. Send redundant data if the packet size is otherwise very small, or contains extra bytes of padding.
Design your game so that if a few packets are dropped, it doesn't matter. For instance, rather than just a single blow, require multiple sword blows to kill the goblin. This way, it's OK if an individual sword attack message is dropped?a player just needs to slash a couple more times.
Include sequence numbers or timestamps in packets, so that the receiver can detect out-of-order and dropped packets.
Be cautious when sending delta values by using VDP/UDP. Although deltas can reduce packet size, they are meaningless if the context is lost because of dropped packets. Periodically synchronize the absolute state of a value to guarantee consistency.
Avoid sending either too much data or too many packets at once. Transmissions that are sent in bunches are likely to be queued up on modems or routers. These devices typically have limited buffer space. Consequently, a bunched transmission is much more likely to be dropped altogether.
At some point, your game just needs to give up if an Xbox 360 console hasn't heard from another Xbox 360 console in a long time. Plan for this condition and deal with it appropriately.
Disassociate Network Communication from Frame Rate
=Keeping networking separate from gameplay is one of the first steps for avoiding latency. Don't tie network communication to frame rate or controller input?you never want the game to freeze simply because a packet hasn't arrived. Don't stall gameplay because the game has to wait for information from other Xbox 360 consoles. And don't stall gameplay by using blocking sockets. Instead, use non-blocking sockets or a separate network thread.
Handle Network and Online API Errors
The Internet is not a reliable medium. It is exactly the opposite?guaranteed to be unreliable and unpredictable. Although all Xbox 360 networking and online functions return error codes to indicate failure conditions, ensure that your game detects and gracefully handles these failures. You can use the following methods:
Display an informational message and return the player to a safe state if connectivity is lost.
Detect and manage socket error conditions.
Don't Roll Your Own Network Security
All Xbox 360 packets are automatically encrypted and authenticated by using cryptographically secure algorithms. Don't attempt to add additional levels of security by encrypting, hashing, or calculating checksums for game payloads?you'll simply reduce the performance of your game.
It is good if you are still concerned about hackers. The Xbox 360 team is concerned, too. Measures have been taken to detect and ignore pirated online games and modified Xbox 360 consoles. Microsoft has a team of engineers that is dedicated to researching and resolving security issues, and we will continue to improve the resiliency of the hardware and software.
Network Security Does Not Equal Game Security
Although Xbox 360 network security is robust, no form of security can promise 100 percent effectiveness. One way that hackers can attack an Xbox 360 game is to block packets, or selectively block them. For instance, if an attacker can guess that certain size packets that are sent on a specific port indicate a particular type of data, your game could be vulnerable. Therefore, consider the following security recommendations.
Send and receive all data on a single port (port 1000) by using VDP.
Consider methods for randomly adjusting packet size.
Add code to check for impossible or improbable situations (for example, too much health, too many lives, too many headshots, movement too fast) and log or report them to users.
Practice defense in depth. Validate all incoming network messages to avoid access to invalid memory, buffer overruns, and so forth. Never trust the network.
Review Network and LIVE Certification Requirements
Familiarize yourself with all the Xbox 360 Technical Certification Requirements. The latest TCRs are available on Xbox 360 Central. Many of the TCRs will shape the choices you make regarding game design. Pay particular attention to the TCRs that address game bandwidth, latency tolerance, and usage of LIVE functionality.
NetMon is a very useful tool that can be used to monitor network traffic during gameplay. Version 2.0 of NetMon is installed in the XDK redist folder, and is not compatible with Windows Vista. Version 3.0 is fully compatible with Vista, and is available for download at http://connect.microsoft.com.
Use NetMon throughout development to spot-check your network performance. Follow these steps:
Put your development computer and development kits on a hub, not on a switch. Switches optimize network traffic, which prevents NetMon from seeing all traffic.
For Version 2.0, copy %XEDK%\redist\netmon\xlinkparse.dll to the NetMon\parsers directory.
For Version 3.0, copy %XEDK%\redist\netmon\xsp.npl to the NetMon\NPL directory. Add the line include "xsp.npl" to sparse.npl.
Filter out all network traffic from computers other than the development kits that are monitored.
Monitor the frequency of packets sent.
Examine the size of packets.
Watch for unexpected packets during certain portions of the game.
Watch for packets on ports other than 1000.
Look for packets that use protocols other than VDP. This should be the exception. Because VDP cannot be used for broadcasts, UDP can be used for discovery. TCP is used for some communication to the LIVE servers.
Timestamps can be used to synchronize capture data to game logs.
If your game has deterministic replay features, compare captures over time.
Save and checkpoint captures for future reference and comparison.
Test Real World Conditions
Test your game on something other than a LAN. Take your Xbox 360 development kits home and test gameplay on the Internet. Consider using custom tools and code that simulate the scenarios that your game will encounter when two players face off across a continent. For instance, add code to automatically drop or delay packets in the game. Include a real-time network condition display alongside the frame rate and memory use displays.
Design your Xbox LIVE game so that it works well under the following real world Internet conditions:
Minimum upstream and downstream bandwidth: 64 kilobits per second.
Typical upstream and downstream bandwidth: 64-128 kilobits per second.
Typical roundtrip latency: 100-300 milliseconds.
Worst case roundtrip latency: 1-5 seconds.
Typical packet loss: two percent.
Worst case packet loss: ten percent.
If you design your title to handle these types of Internet conditions smoothly, your game stands a much better chance of success. If your game exceeds these guidelines (for example, if your game can handle 50 percent packet loss), that's great. You should also cope with cases where Internet conditions go beyond the tolerance of your game. For example, if your game is unplayable at latencies above 250 milliseconds, it should detect these conditions and inform the player. Better yet, if your game detects sessions that have latencies exceeding 250 milliseconds, your game should not even allow the player to see or join these sessions.
There are many networking programming resources available. The following list provides some useful references. Note that inclusion does not imply endorsement by Microsoft.
Abrash, Michael. Quake's Game Engine: The Big Picture
Aronson, Jesse. Dead Reckoning: Latency Hiding for Networked Games
Aronson, Jesse. Using Groupings for Networked Gaming
Barron, Todd. Multiplayer Game Programming. Portland, OR: Premier Press, 2001.
Bernier, Yahn. Latency Compensating Methods in Client/Server In-game Protocol Design and Optimization
Bernier, Yahn. Half-Life and Team Fortress Networking: Closing the Loop on Scalable Network Gaming Backend Services
Bernier, Yahn. Leveling the Playing Field: Implementing Lag Compensation to Improve the Online Multiplayer Experience
Bettner, Paul and Mark Terrano. GDC 2001: 1500 Archers on a 28.8: Network Programming in Age of Empires and Beyond
Caldwell, Nick. Defeating Lag with Cubic Splines
Fitch, Crosbie. Cyberspace in the 21st Century: Part Five, Scalability with a Big 'S'
Frohnmayer, Mark and Tim Gift. The TRIBES Engine Networking Model
Haag, Chris. Targeting: A Variation of Dead Reckoning
Howland, Geoff. What is Lag?
Lambright, Rick. Distributing Object State for Networked Games Using Object Views
Lincroft, Peter. The Internet Sucks: What I Learned Coding X-Wing vs. TIE Fighter
Ng, Yu-Shen. Designing Fast-Action Games for the Internet
Ng, Yu-Shen. Internet Game Design
O'Brien, Larry. Multiplayer Math
Peer-to-Peer Working Group
Royer, Dan. Network Game Programming
Simpson, Jake. Networking for Games 101
Sweeney, Tim. Unreal Networking Architecture
Treglia, Dante, ed. Game Programming Gems III. Boston: Charles River Media, 2002.
Kirmse, Andrew, ed. "Network and Multiplayer." In Treglia, Dante, ed. Game Programming Gems III. Boston: Charles River Media, 2002.
Built on 2007年7月19日 6:39.
c 2007 Microsoft Corporation. All rights reserved.