6IT

UDP: Unreliable Transfers

[[This is Chapter 13(b) from "beta" Volume IV of the upcoming book "Development&Deployment of Multiplayer Online Games", which is currently being beta-tested. Beta-testing is intended to improve the quality of the book, and provides free e-copy of the "release" book to those who help with improving; for further details see "Book Beta Testing". All the content published during Beta Testing, is subject to change before the book is published.

To navigate through the book, you may want to use Development&Deployment of MOG: Table of Contents.]]

[[Also note that Chapter XI(a) has been skipped from publishing on the site; it is too much similar to a recent post of mine Once again on TCP vs UDP, and you can get most of the related discussion there]]

As it was mentioned above, UDP is all about datagrams¹, and UDP datagram sits right on top of IP packet. Typical UDP datagram has the following structure:

IP Header (20-60 bytes for IPv4, 40 bytes + Extension Headers for IPv6)

UDP Header (8 bytes)

UDP Payload (see discussion about sizes below)

More importantly, for UDP there is one-to-one correspondence of UDP datagrams with IP packets.

you may think of UDP as of an analog of good old C: you can do pretty much everything, but it is not because the language helps you – it is rather because it doesn’t stand in the way 🙂

It means that working with UDP is logically very much the same as working with IP directly. In other words, you have full control but need to do everything yourself. With UDP, you don’t have any built-in reliability (though you can implement it yourself), no flow control (though you can implement it yourself), and so on, and so forth. Just to give you some feeling about "how on-your-own you are with UDP": for UDP, even checksum is optional.²

If making parallels with programming languages, you may think of UDP as of an analog of good old C: you can do pretty much everything, but it is not because the language helps you – it is rather because it doesn’t stand in the way 🙂 .

¹ Which is just a fancy name for “packet”

² that is, in IPv4

On UDP payload size

With UDP (unlike TCP) you do have control over the datagrams/packets which you're sending. On the flip side of it, you DO need to care about maximum datagram size. For answering the question "what maximum size of UDP datagram is acceptable for the public Internet", there are several different schools of thought.

Conservative school.

MTU
maximum transmission unit (MTU) of a communications protocol of a layer is the size (in bytes or octets) of the largest protocol data unit that the layer can pass onwards
— Wikipedia —

Conservative school of thought with regards to UDP datagram size says that we need to fit the whole packet into 576 bytes (which is often considered minimum practical MTU for the Internet³), which effectively means that we should limit UDP payload to 512 bytes or so (number 548=576-20-8 is a bit too optimistic due to IPv4 header options).

Moderate school.

Moderate school says that we should limit ourselves to UDP payload sizes of around 1400-something. This is based on a very practical observation that a typical Ethernet MTU is 1500; then, allowing for various encapsulations (such as those in PPPoE etc.) and header sizes, we'll have around 1400 bytes left for our UDP payload.

Close results will be obtained if we take IPv6 minimum MTU of 1280 bytes as a baseline (which will lead us to around-1200 bytes for our UDP payload).

Optimistic school.

Optimistic school says "hey, we have a standard which says that we can have UDP payload up to 65467 bytes" (65467=65535-(60+8)). And this is technically true, and the sky probably won't fall. However, we need to keep in mind that such large datagrams are almost guaranteed to be fragmented, which (combined with a fact that even if one single fragment is lost, the whole datagram is lost) - it leads to increased datagram loss rate🙁.Also, if we're using UDP for our network-tick packets, then we need to keep in mind that sending 60K bytes payload 20 times per second would lead to a traffic of 60000*8*20 bit/s/player = 10MBit/s/player, which is usually WAY too much for an MOG.

Theoretically Correct School.

Theoretically correct school of thought says that we should try to perform so-called PMTUD, to find out minimum MTU on the path from Server to Client

Theoretically correct school of thought (coming from multiple RFCs) says that we should try to perform so-called "Path MTU Discovery" (a.k.a. PMTUD), to find out minimum MTU on the path from Server to Client, and to optimise our packet size based on it. However, for fast-paced games I would usually advise against doing it.

The idea of PMTU in a nutshell looks as follows.⁴ At some point, we're sending a packet with a so-called DF=Don't Fragment IP bit. If this packet doesn't fit into next MTU, the router in question will drop the packet, sending a special ICMP packet (with type=3, code=4) back to the sender. So, the sender can learn whether the last packet sent is larger or smaller than minimum MTU along the path (so-called "path MTU"). If the sender has learned that the packet does fit MTU - it usually increases size-of-packets-it-sends, if it learns that the packet doesn't fit MTU - it can either use last-packet-size-which-has-fit as a future MTU, or attempt to find MTU more precisely.

For last 15+ years, PMTUD is routinely performed for TCP connections (and works pretty well for last 10+ years), and usually, the rationale for doing PMTUD is that as every packet is small, and has a significant overhead, so when we're transferring large files, we will save quite a bit of bandwidth if we're using MTU of 1500 bytes over those "guaranteed" 576 bytes (for TCP, the gain from changing MTU from 576 bytes to 1500 bytes is around 6-7%).

Most of the game packets are usually pretty small, and division between packets is rarely driven by MTU (but is usually driven by timing restrictions).

However, for games, this logic doesn't usually apply. Most of the game packets are usually pretty small, and division between packets is rarely driven by MTU (but is usually driven by timing restrictions). Under these conditions, personally I usually feel that PMTUD is not really necessary. Moreover, PMTUD can even be mildly detrimental for some players at least in two distinct scenarios:

for those players sitting behind broken firewalls which drop ICMP packets. While this was quite a problem 15+ years ago, these days it is quite uncommon.
for those players whose path has changed in the middle of connection (in particular, it happens rather frequently for mobile devices). While a fairly common scenario, it rarely causes Big Problems though.

Neither of these two issues is really Big Enough in practice (that is, if you have TCP fallback), but they do serve as an argument against PMTUD (and as there aren't too many really compelling pro-PMTUD arguments in MOG context, it usually becomes a kind of a stalemate).

My Personal Take

My personal take on the question of "how big UDP datagram to use" depends on quite a few specifics, but in some "average case" it is usually along the following lines:

be moderate and go for 1200-1400 UDP payload
have TCP fallback as a "last resort" even if you think that your game is only "barely playable" over TCP
as for PMTUD - you MAY want to try it for "slow-paced updates" (see definition below) if your "Reliable UDP" library supports it. However, YMMV, and often you may be better without PMTUD, so if your library doesn't support PMTUD - I would certainly be perfectly ok with it.

³ technically, this is wrong, and 576 is not a minimum MTU, but a “minimum packet size all hosts should be able to accept”, but finding a link with MTU < 576 bytes, while possible, is quite a challenge

⁴ the procedure described below is for IPv4; for IPv6, the procedure is a bit different, but ideology is still pretty much the same

On UDP broadcast/multicast – Don’t Hold Your Breath

Indeed, it would be Really Great to send each update only once... However, don’t hold your breath over it.

UDP as such is known for supporting broadcast and multicast, which MIGHT look as “just the ticket” for your game. Indeed, it would be Really Great to send each update only once, and let the Internet infrastructure populate the packet to all the hundreds of thousands of your players. However, don’t hold your breath over it.

First of all, let's note that as UDP sits right on top of IP, to implement broadcast or multicast, it relies on IP broadcast/multicast. Now let's look at these two beasts.

IP broadcasts works only within LAN (and is very popular for LAN-based P2P-architecture games for peer discovery). IP broadcast doesn’t work beyond so-called “broadcast domain”, which is usually limited to a single LAN or “subnet”. Broadcast never works over the Internet (and neither over Intranets). In IPv6, broadcast was abolished entirely (and effectively replaced with a special type of multicast).

IP multicast is a much more interesting beast, and your admins MAY make it working over the global Intranet with millions of users. However, while it is a standard, and you can find a lot of information how beautifully the standard is designed (which it is) and how multicast is supposed to work, there is much less obvious and much less-known among developers fact:

Multicast does not work over public Internet

Information about IP multicast being de-facto unroutable over the public Internet is scarce, and the best reference I was able to find about it, is a discussion on ServerFault. However, I assure you that at the very least for IPv4, it is a fact of life 🙁.

As most of the routers (including pretty much every backbone router out there) are not configured to support multicast, it makes multicast over the public Internet hopeless

The story goes as follows. First of all, to have multicast working, you need all the routers between your server and all of your clients to support multicast. And as most of the ISPs (including pretty much every backbone ISPs out there) are not doing it, it makes multicast over the public Internet hopeless 🙁 .⁵

A bit of side discussion: traditionally, for IPv4 multicast, there were two Big Problems preventing backbone ISPs from enabling it. The first Big Problem for supporting IP multicast over the public Internet was about multicast address allocation: with only about 64M multicast addresses available in IPv4, any schema of allocating them (except as on case-by-case basis by IANA as in “let’s use 224.0.1.1 for NTP”) will lead to a quick exhaustion of the multicast address space. With IPv6 and its unicast-prefix-based multicast addresses, it seems that this first problem has been addressed; however, there is still another Huge Obstacle. The second Big Problem for supporting IP multicast over the public Internet is that with unicast-prefix-based multicast addresses, everybody and his dog will be able to create multicast groups, and all these groups will need to be supported by backbone routers at least to a certain extent. As backbone routers are already loaded to the brim, asking them to store uncontrollable number of mappings between multicast IP addresses and their respective destinations is probably too much 🙁🙁. As a result, my rather-poorly-educated guess is that the transition to IPv6 is not likely to enable multicast over public Internet; however, I would be happy to be wrong (as noted above, for many things out there, multicast would be a Wonderful Thing™).

Still, at least at the moment, the bottom line about UDP multicast and broadcast goes as follows:

When using UDP over the public Internet, you are limited to unicast UDP packets 🙁

⁵ And for our purposes of multiplayer gaming, it doesn’t help much that it is possible to set up an across-the-globe Intranet to handle multicast.

UDP: Addressing Lack of Reliability

One thing to remember about when working with UDP, is that UDP is NOT a reliable protocol in any sense. Each and every packet on the Internet can be lost. And as UDP datagram is only an extremely thin layer on top of IP packet – each and every UDP datagram can be lost too 🙁. This is not fatal (after all, “reliable” TCP is built on top of unreliable IP packets), but it does mean lots of additional work on top of UDP.⁶

In game context, approaches to ensuring reliability over UDP tend to depend significantly on the nature of the communication. As it was noted in Chapter VII, for Server-to-Server communications we're likely to use TCP anyway, which leaves UDP to communications between Client and Server, and these communications are further divided into (a) publishing of the World State, and (b) point-to-point communications between Client and Server.

⁶ or you MIGHT be able to delegate this work to a “Reliable UDP” library

Publishing World States over UDP

First, let’s consider publishing of your world state to all your clients. As it was noted in Chapter III, you DO need to spend quite a bit of time to split your World State into Client-Side State, Server State, and Publishable State (with the following inequation usually standing: size-of(Client-Side State) << size-of(Server-Side State) << size-of(Publishable State)).

we’ll assume not only that you’re not going to transfer all those mesh triangles over the network, but also that you’ve implemented your Interest Management and worked on optimizing your Publishable State

For the purposes of this Chapter, we’ll assume that you’ve already done your homework in this regard, splitting and optimizing your Game World States. In other words, we’ll assume not only that you’re not going to transfer all those mesh triangles over the network, but also that you’ve implemented your Interest Management and worked on optimizing your Publishable State (including, but not limited to, switching to fixed-point in-transit representations where applicable). All these traffic optimizations mentioned in Chapter III, with the notable exclusion of Compression, can (and generally SHOULD) be done in exactly the same manner regardless of you using UDP or TCP (this will also help to have that fallback to TCP, which we’ve spoken about above).

Fast-paced Updates vs Slow-Paced Ones

Now let's take even a closer look to these World State updates. In different games, there are different types of updates; sometimes they even co-exist within one single game.

The first type of updates is about things-which-change-on-every-network-tick; coordinates of your characters is one good example of such fast-paced updates. The second type of updates is all about things-which-change-much-more-rarely-than-that; one nice illustration of these things is player chat (and a bit worse example is an inventory of the chest which is about to be inspected). Of course, the division line is not that clear (what about health of your character? Does it qualify as a fast-paced change or as a slow-paced one?),⁷ but (as it was noted in Chapter III) generally I find it useful to make such a split, designating each of the parts of the Publishable State into one of these two categories.

⁷ actually, the answer usually depends on your model of representing health within your Publishable Game World State. If health is represented as “health now” (and assuming that it recovers rather fast) – it will probably need to be a part of fast-paced stuff. However, if you represent health as “health was exactly 184.23 at the network tick #19345”, and then let the Client calculate the current health based on well-known health recovery algorithms – it may qualify for a slow-paced part.

Fast-paced Updates: Compression without built-in reliability

[[TODO: remove from Chapter XI (it has been moved to Chapter III)]]

When speaking about those fast-paced updates (like coordinates), the picture is usually rather obvious. We have all those updates to coordinates etc. calculated by the Server-Side at each network tick, and we need to deliver them to the Client-Side. And of course, we want to use all those Compression techniques (including Delta Compression – both whole-field and numerical flavours, and of course, Dead Reckoning) – as described in Chapter III.

Now let’s see how these Compression techniques will interplay with the inherently unreliable nature of UDP.

it may easily happen that previous packet is not available to the Client when our current packet arrives

As UDP does not guarantee us delivery of each and every packet, it may easily happen that previous packet is not available to the Client when our current packet arrives; as a result, we cannot use any kind of reference to the previous packet when using UDP. Bummer. On this basis, it might seem that we’re not able to use both Delta Compression and Dead Reckoning (which would increase traffic enormously). However, in reality it is not that bad.

The most popular way of dealing with this kind of things is the following. First of all, we note that in addition to the packets coming from Server to Client (the ones which bring World State data) there are also packets coming in the opposite direction - coming from Client to Server (normally containing inputs). Second, let’s say that each of the packets coming from Server to Client, contains the number of the “network tick” which it corresponds to (we need this pretty much anyway for other reasons too). Third, let’s say that each packet coming from Client to Server, also contains a number of last-network-tick-received-by-this-Client (!).

As soon as we have all these things in place, our Server happens to “know” that such and such Client already has network-tick #X. Then (assuming that Server keeps a few last states)⁸ Server can issue the next packet #Y, effectively saying that “this is packet #Y, WHICH IS BASED ON PACKET #X, and using all the Delta Compression and Dead Reckoning COMPARED TO THAT PACKET #X”. As we know for sure that Client already has that exact PACKET #X – we can be reasonably sure that on receiving this PACKET #Y it will be able to reconstruct the whole Publishable Game World State correctly.

Fig XI.1 shows one possible interaction between Server and Client while using the algorithm described above.
One note: the approach described above means that PACKET #Y can be easily different for different Clients (in extreme case – it can be different for each Client). However, as we’re bound to use unicast anyway, this doesn’t cause too much problems in reality.

Another note is that with this algorithm, we do NOT guarantee delivery of each-and-every packet (and that’s a Good Thing™, as otherwise we’d waste lots of time and bandwidth). Instead, what we’re doing is guaranteeing data synchronization even when some packets are lost.

⁸ Exact number depends on the question “whether we want to keep historical data per-Client or once for the whole Server”, but in any case keeping over 500 ms of history doesn’t make much sense, so if you have 20 network ticks/second, we’re speaking only about last 10 states.

Slow-Paced Updates: State Sync

As noted in Chapter III, slow-paced updates to the Publishable Game World State usually have three significant differences from fast-paced ones:

They’re slow
They MAY be large
They MAY contain non-numerical stuff (mostly strings, but MAY contain even images etc.)

In any case, for slow-paced updates I suggest to use a reliable stream, with differential updates on top of it

Whether you Really Need to implement full-scale sync of abstract data trees – or just need to deal with chat-as-a-part-of-Publishable-State (which is MUCH simpler),⁹ it depends on specifics of your game. In any case, for slow-paced updates I suggest to use a reliable stream, with differential updates on top of it (usually sent only when the change happens, and NOT on every network tick); whether you want to add a differential sync algorithm along the lines described in Chapter III (or just to resend the whole state in case of resync) – it heavily depends on your specific needs.

From UDP perspective, it means that we can use pretty much the same reliable stream which we’d use for Point-to-Point Client-to-Server communications (described right below in "Point-to-Point Communications over UDP" section).

⁹ See Chapter III for arguments that chat should be implemented as a part of Publishable State; while the sky doesn’t fall if you implement chat via multicast messages, I (both as a player and as developer) prefer to have my chat as a Publishable State

Point-to-Point Communications over UDP

To implement reliable point-to-point communications over UDP, we’ll need two very basic components:

Acknowledgement packet
Retransmit on timeout

However, if we take a tiny bit deeper look at it, we’ll see three rather different models with regards to flow control.

All the different implementations of reliable UDP use these two things one way or another. However, if we take a tiny bit deeper look at it, we’ll see three rather different models with regards to flow control.

Model 1. No Flow Control

The most obvious approach when it comes to reliable UDP, is the following:

Send the packet
Wait for acknowledgement (ideally – measure RTT with the same client and wait for like 2*typical_RTT)
If there is no acknowledgement after the wait timeout expires – re-send the packet
Rinse and repeat

Even though this approach is very simple, it does work; however, it SHOULD be restricted to one-off relatively-small commands. If we’ll try to use such a simple model for long transfers (such as file transfer) – we’ll overload target Client with packets (and in extreme cases can even overload our own Server too).

Model 2. Partial Flow Control: Limiting In-Transit Data

To ensure that we’re not trying to send that 100-megabyte-file all-at-once, the best way is to keep an eye for acknowledgements coming from the other side, to calculate number of "bytes in transit" (collective size of those packets which are not acknowledged yet), and to ensure that at any given moment there is no more than certain number of bytes “in transit”. These “bytes in transit” are quite easy to calculate on the sending side (assuming that receiving side faithfully acks at least some of the packets sent); and as soon as current number of “bytes in transit” is exceeded – sending side stops sending further packets until it gets an ack from the receiving side.¹⁰

In particular, such partial flow control seems to be the one used by ENet library.

¹⁰of course, retransmit timeouts still apply, so retransmit of the old packet MAY happen even when the number of “bytes in transit” is exceeded. Also – retransmits SHOULD NOT count against “bytes in transit” limit

Model 3. End-to-End Flow Control: Advertised Receive Window

Model 2 described above, will work reasonably well, but only as long as your clients and servers are NOT overloaded. In case if network is fast, but receiving side is slow, partial flow control as in Model 2, MAY cause too much data to be transferred – and as a result, some packets MAY be lost not because Internet has lost them, but because there wasn’t enough space on the receiving side to store these packets before processing them.

To deal with it, protocols such as TCP use end-to-end flow control with receiving side advertising how much space it has in its receiving buffers

To deal with it, protocols such as TCP use end-to-end flow control with receiving side advertising how much space it has in its receiving buffers (for TCP it is known as “Advertised Receive Window”). Then, we only need to send this “remaining space” in each ACK packet coming from the receiving side, and then use this “remaining space” as a threshold for stopping sending on the “sending side”.¹¹

In practice, however, if we can guarantee the size of this buffer on the receiving side to be not-less-than-X, we can safely skip “advertising” available window – that is, as long as we never allow to have more-than-X “in-transit bytes”. While this approach (essentially Model 2 plus an understanding of receiving side limitations) won’t work for TCP (which needs to operate for an extremely wide range of environments), it MAY (and often DOES) fly for games.

¹¹ in practice, for modern TCP implementations it is more complicated than that, with a so-called “congestion window” coming into play too, but for our purposes it doesn’t matter too much

Bottom line about Flow Control

If implementing (or choosing) reliable UDP, you DO need to take into account flow control. Simplistic Model 1 without flow control won’t work well for anything but singular packets. On the other hand, Model 2 (“partial flow control”), while still imperfect, is known to work reasonably well for games (taking into account some limitations described above). Model 3 is clearly the best one here, though in practice it is not used for games too often (most likely, because of additional complexity, though I don't see this complexity as significant enough).

Retransmission Policies

Another question which arises when implementing Reliable UDP, is “how often to retransmit those lost packets”. Original TCP uses something like “2*RTT” for the first retransmit, and then it starts to double retransmit timeout for each subsequent retransmit. This is known as “exponential backoff”, and is intended to prevent hosts from causing Internet-wide (or at least ISP-wide) network congestions. Discussion whether this exponential backoff is really necessary to avoid network congestion, is far beyond the scope of this book (though there is a research out there which says that it is not MondalEtAl). On the other hand, what is perfectly clear is that such exponential backoff does hurt interactivity (and does it badly too).

as described above in “Fast-paced Updates” section, most of the Really time-critical stuff can (and SHOULD) be transferred not over guaranteed-delivery channels, but rather as “guaranteed-synchronization” channels, making a question of “how often to retransmit” much less critical.

On the third hand 😉 , as described above in “Fast-paced Updates” section, most of the Really time-critical stuff can (and SHOULD) be transferred not over guaranteed-delivery channels, but rather as “guaranteed-synchronization” channels, making a question of “how often to retransmit” much less critical.

On the fourth hand (yes, I am half-way towards the octopus now), if you still DO want to play with retransmission times, my personal suggestion would look along the following lines (yes, I know that I will be hit badly by Internet congestion zealots, but well – I don’t care that much about them).

From what I’ve seen, it should be ok to drop exponential backoff, but only for a limited time, and with per-server congestion control. The idea here is to find a reasonable compromise between being Internet-friendly and solving the problem we have at hand. In this regard, I suggest that:

For game, "reliable" connections have two different modes of operation: “time-critical” and “connection-seeking”
I tend to say that in the “time-critical” mode, it is ok to send retransmits more aggressively than exponential backoff would dictate (for example, re-sending at constant intervals instead of exponential back-off)
- However, time length this “time-critical” mode SHOULD be very limited by design. In particular, it usually doesn’t make any sense to keep re-sending stuff at small intervals for more than a minute or so (MUCH less for faster-paced games). If the player wasn’t able to communicate for over a minute – chances that additional exponential delay would make any significant difference for him, become extremely slim.
- As soon as the length of this “time-critical” mode is over, we are actually giving up all hope to recover connection for the current game event (such as combat or whatever-else) – and are actually switching to “connection-seeking” mode
In the “connection-seeking” mode, I suggest to follow exponential-backoff; it won’t do much difference for interactivity (we've already given up all the hope to reconnect within current game event), but is significantly more Internet-friendly
In addition, I suggest to keep an eye on number-of-packets-lost (per second) server-wide (actually, any large number of connections will do), and to back off retransmits if you see a sudden increase in number of lost packets over all the channels combined (which indicates congestion either on your server, or with your ISP, so it is better to back off)

UDP Connections and KeepAlives: Simplistic Protocol

As we all know, UDP as such has nothing to do with connections – UDP is just a datagram to be delivered to certain address. However, to have some kind of reliable delivery, it IS necessary to have a concept of "connection". In particular, all the popular “reliable UDP” libraries I know, are doing it. And as soon as we have a concept of "connect", we need to find out about "disconnect" too (which, for game purposes, is usually done via keep-alive and disconnect-on-timeout, as described below).

Surprisingly, it is not that difficult; one specific schema which tends to work reasonably well:

you need some kind of handshake (resulting in some kind of connection identifier). Connection handshakes are tricky because of DDoS attacks, so some kind of "SYN cookie" (or equivalent) is necessary to handle it. For example:
- we have a secret key (let's name it "CookieKey") on the Server side; this key is not known to anybody besides our server, and SHOULD be regenerated on each Server restart.
- on the first packet (connection request) coming from the Client, we generate "cookie", which is a tuple of (Client-IP,Server-time,at-least-64-bits-of-crypto-random-data,may-be-something-else), with an additional MAC (such as CMAC or HMAC, see more on MACs on DDoS in section [[TODO]] below); MAC should be generated with a "CookieKey", effectively authenticating this whole tuple with a "CookieKey".
- we send this "cookie" back to the Client-IP, and do NOT store anything on the Server side (so there is no potential for resource exhaustion due to the attack).
- Client simply repeats its connection request, but with cookie (the one received from the Server) in it this time
- on receiving connection-request-with-cookie, Server:
- - validates that Client-IP in the cookie is the same as the source of the packet
  - validates that MAC indeed authenticates the tuple in it
  - validates that Server-time is within reasonable limits compared to current Server time
  - proceeds with connection
- such "cookies" are aimed to prevent those DDoS attacks which are substantially similar to
- this whole thing is very similar to the cookies used in SCTP and DTLS protocols, and are actually a tad better than TCP's SYN cookies designed for the same purpose
- When using an encryption protocol such as DTLS (which you SHOULD, see [[TODO]] section below), it will implement similar "cookies" itself, so in such a case you won't need to do cookies yourself (but you still need to implement protection from crypto-level DDoS, see [[TODO]] section below for details)
you need to pass this connection identifier as a part of the UDP datargrams belonging to this connection
you need to drop the connection on timeout (on both sides)
you need to have some kind of Keep-Alive packets if nothing happens, to make sure that the connection is not dropped on timeout when it is actually still alive.

Note that while this schema is MUCH simpler than TCP, it does work

Note that while this schema is MUCH simpler than TCP (in particular, it has no 4-way termination handshake), it does work (in fact, it looks pretty much as a very simplistic Bluetooth-LowEnergy protocol, which tends to work very well even with HUGE packet loss).

[[TODO!: sudden IP change; refer to TCP too]]

UDP Connections: QUIC

Recently, a new reliable-stream-over-UDP protocol emerged: QUIC. I didn't try it myself, but from what I see from their design docs, I tend to like it A LOT (that is, if you don't need to code it yourself). QUIC has several nice goodies compared to both TCP and homegrown solutions (such as Simplistic Protocol described above).

Compared to TCP, QUIC explicitly tries to reduce latencies (while I don't have data on the QUIC latencies in gaming context, as we're speaking about "slow-paced updates", it is not that important here, and any improvement over TCP counts). In addition, it integrates security into transport layer, which allows to prevent some of the attacks. In addition, QUIC behaves significantly better that TCP in case of IP change (which is very common for mobile networks).

Compared to homegrown simplistic protocol described above, QUIC has quite a few built-in goodies, including built-in packet pacing (which reduces packet loss QUIC), better connection avoidance, and forward error correction (which reduces delays in certain contexts).

Overall, IMHO (no warranties of any kind) QUIC looks very promising for these slow-but-large-updates

Overall, IMHO (no warranties of any kind) QUIC looks very promising for these slow-but-large-updates. If you're at the early stages of your development, I would certainly suggest to try QUIC (more specifically, libquic library), and to see how it works for you. It is more complicated than the rest of the Reliable UDP libraries out there, but it does more too (and its tight integration with crypto is a Very Good Thing in general).

UDP and Firewalls

In general, UDP is less friendly to firewalls than TCP. Or, looking at it from a different angle, there are less UDP-friendly network devices out there, than TCP-friendly ones. In other words, there are quite a few people on the Internet who will be able to connect to your server via TCP, but won’t be able to connect via UDP. QUIC estimates number of such people at about 6-9%.

From my experience, most of the time such things happen over not-so-common connections (such as hotel connections, work connections, etc.), so if you’re targeting ONLY players-playing-from-home, it might be not that bad. On the other hand, as soon as you’re adding mobile support, things will become significantly worse in this regard, and it MAY become a yet another argument to provide TCP fallback for those players who currently cannot connect via UDP.

UDP Hole Punching

NAT
Network address translation (NAT) is a methodology of remapping one IP address space into another by modifying network address information in Internet Protocol (IP) datagram packet headers while they are in transit across a traffic routing device.
— Wikipedia —

One very common issue which game developers are asking about with regards to UDP, is so-called UDP hole punching. Such hole punching is absolutely necessary when both your clients are sitting behind so-called NATs (and with NATs being extremely common these days for home users, it means that hole punching is absolutely necessary for peer-to-peer over-the-Internet games). However, as long as we’re restricting ourselves to server-centric architectures (see Chapter II for reasoning behind), we’ll have generally no need to implement punchthrough. This is quite a relief, as in some cases (specifically with some overzealous NAT implementations, as well as “symmetric NAT” implementations) it becomes a Really Big Headache.

As a result of hole punching being unnecessary for server-centric games, I do NOT want to delve into a detailed discussion of "how does hole punching work" (which requires another rather long discussion “what this whole NAT thing is about” as a prerequisite). However, for those of you who are trying to create a P2P game running over-the-Internet, I will still give a few pointers.

[[P2P-specific]]: From what I’ve seen and heard, proper UDP hole punching is a kind of Black Magic spell which has three different levels:

Novice-level spell. Just implement STUN protocol, PLUS make sure that there is a keep-alive in your UDP implementation, so that that the punched hole is NOT closed by NAT devices (if it closes, you’ll need to punch it again). From what I’ve heard, “at least one UDP packet every 15 seconds” should be good enough (though it is very much depends on implementations of the NAT devices involved, and YMMV). This Novice-level spell will usually work for most of your players.
TTL
Time to live (TTL) or hop limit is a mechanism that limits the lifespan or lifetime of data in a computer or network. TTL may be implemented as a counter or timestamp attached to or embedded in the data.
— Wikipedia —
Apprentice-level spell. This one is intended to address those overzealous NAT devices which blacklist IPs when a packet-from-IP-unknown-to-NAT-device, arrives. To deal with it, on top of novice-level stuff, you’ll need to send first your packet with a Really Small TTL (such as 2), and then gradually increase TTL for subsequent packets. This technique more-or-less guarantees that there is a packet which reaches your NAT device, but does NOT reach your peer’s NAT device, so that by the moment when your packet reaches her NAT device, her packet has already been there, and there is already a hole in her NAT, so that your packet doesn't cause your IP to be banned (phew).
Expert-level spell. From hole punching point of view, the most annoying NATs are so-called Symmetric NATs; these beasts tend to change ports between different connections of the same source to different targets, so usual STUN doesn’t work with them. That’s the point where REAL Black Magic begins. In practice, while the ports are different, existing implementations usually simply increment ports, which MIGHT be (ab)used to establish a punch-through connection. More on it in Takeda (and supposedly in RakNet sources too).
With this in mind, there MIGHT also exist a Master-level spell (the one which solves All The Punchthrough Problems), but I haven’t encountered it (yet) in my quest for Holy Connectivity.

Let's note that the whole thing described above sits on top of STUN; in theory, there are also TURN and ICE protocols (leveraging STUN). However, with TURN requiring a relay server (!) - it is rarely an option for P2P games, so the process described above is probably your best bet.

Comparison of well-known Reliable UDP implementations

[[TODO: add libyohimbo]]

[[TODO: analyse support for blocking/nonblocking]]

There are several different "Reliable UDP" libraries out there; from our perspective, all of them are essentially targeted towards those "slow-paced updates" over UDP. Below is a table which I've managed to collect (as everything else, take it with a big pinch of salt):

Library	ENet	UDT	RakNet	libquic
License	Permissive	Permissive	Permissive¹²	Permissive
Last Commit ¹³	2 months ago	3 years ago	1 year ago	2 weeks ago
Reliability ¹⁴	Optional	Optional	Optional	Mandatory
Streams per connection	Single	Single	Single	Multiple
Flow Control	No(?)	Yes	Yes	Yes
Congestion Control	No	Yes	Yes	Yes
Path MTU Discovery (PMTUD)	No	Optional(?)	Optional	No
Integrated Crypto	No	No	No	Yes
Integrated DDoS Protection	No	No	No	Yes¹⁵
Integrated Punchthrough	No	No	No	Yes

As you can see, each of the libraries has its own advantages and disadvantages, so you'll need to pick your poison yourself. As noted above, I find QUIC protocol to be very promising, though didn't try it (or libquic) myself yet.

¹² since 2014

¹³ as of Apr’16

¹⁴ having Reliability “optional” means that you can implement your own UDP-based stuff (such as fast-paced updates) on top of the same library

¹⁵ well, to certain extent, more on DDoS protection in [[TODO]] section below

[[To Be Continued...

This concludes beta Chapter 13(b) from the upcoming book "Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)". Stay tuned for beta Chapter 13(c), dedicated to encrypting and otherwise protecting UDP in game and game-like environments.]]