“Multi-Coring" and “Non-Blocking“ instead of "Multi-Threading" - with a Script

The following are slides and a script of my talk on ACCU2018 in Bristol, UK, on Apr 14th. Note that "script" ~= "how I planned to speak about it", which is slightly different from "transcript" ~= "how it turned out" <wink />

Good afternoon everybody. Thanks for coming, and I hope that during this talk I will be able to tell a thing or three which might be of interest.

As it says on the tin, I am going to argue that usual multi-threading patterns (especially those using mutexes at app-level) are not the only way to design multi-core non-blocking systems. Moreover, I am going to argue that (Re)Actor-based systems is a BETTER alternative for a REALLY wide range of real-world use cases.

Before we start, let me provide a very brief outline for the talk.

In Part I of the talk, I’ll argue that multithreading is NOT really an ultimate goal which is worth pursuing no-matter-what, but rather a TOOL to implement two related-but-distinct concepts:

#1 is multi-coring (which is very closely related to scalability)

and

#2 is non-blocking (more precisely - guarantees against response being blocked by some long-lasting I/O operation).

More importantly - user DOES NOT care HOW EXACTLY we achieve these things, which opens the door to different implementations.

In the second part of the talk, we’ll move towards a discussion of my personal favorite subject - (Re)Actors. We won’t have much time to go deeply into details - but will still discuss such things as how certain aspects of (Re)Actors can (and I think should) be implemented, how (Re)Actors fare against Shared-Memory Architectures, and how they achieve our Goals specified above

Armed with a concept of one single (Re)Actor, we’ll be able to move to architectures which use NOTHING BUT (Re)Actors - that is, at app-level.

Among other things, we’ll see examples of such architectures in an AAA first-person shooter on the Client-Side, and for the Server-Side - we’ll mention some real-world systems which handle billions-messages-per-day and write tens-of-billions-DB-transactions-per-year (and purely accidentally happen to make billions-of-dollars in the process <wink />).

Last but certainly not least, we’ll briefly discuss applicability limits of message-passing systems in general, and (Re)Actors in particular.

Giving away a spoiler, I have to say that according to my experience with architecting serious systems, MOST of the heavily-loaded distributed interactive systems out there will benefit from being implemented as a (Re)Actor-fest.

With all this in mind, we’ll come to an inevitable conclusion:

What are you waiting for? Architect your next system as a (Re)Actor-fest!

Before we can really start with the talk, I’d like to make an important announcement. I have to confess that it is not really MY presentation.

Rather - it is a talk prepared by (da-dum)… this guy (and you can also see him on my t-shirt too).

It means that if there is anything good with this talk - it should be attributed to me, and if there is anything bad - it is all HIS fault.

BTW, if you think that my accent is bad - you should be grateful that it is not him speaking; his accent is MUCH worse than mine.

Preliminaries aside, we can proceed with the substance of this talk.

At the first part of the talk, we want to take a deep breath and ask ourselves: “Does multithreading qualify as an almighty business requirement - or it is merely a puny implementation detail?”

As for “what qualifies as a requirement”, I happen to be a big fan of the following criteria laid out by one of my former managers:

“How many customers we will lose/gain if we implement this feature?” As the guy is currently worth well over a billion, he should have been doing something right.

One of the consequences of this “how many customers we will gain” approach, IF certain thing doesn’t exist in the customer’s space - it CANNOT be a business requirement, plain and simple.

This BTW, is very consistent with the well-known requirement-for-requirements that good requirement SHOULD BE implementation-free. Indeed, specifying too much in the requirements restricts our abilities to build an optimal system, for example:

• if we have to have serious security, but we’re saying “we have to use TLS-over-TCP”, we’re restricting our ability to improve latencies (using UDP-with-DTLS).
• if we want our app to run in-browser, but write it down as “we have to use JavaScript as our programming language”- we’re preventing ourselves from using C++ via emscripten (and C# via IL2CPP+emscripten)
• and so on, and so forth.

Now, with this separation between good-requirements and bad-requirements in mind, we can ask ourselves whether “we have to use multithreading” qualifies as a good requirement?

And the answer (going contrary to intuitive feelings of LOTS of developers out there) is that

As a Big Fat Rule of Thumb - no, ‘use multithreading’ does NOT qualify as a good requirement, for a simple reason that multithreading as such is not directly observable in the end-user space.

In other words - (unless our product IS an OS or a library-to-be-used-by-other-developers) we WON’T get any customers due to writing our app as multi-threaded.

OTOH, we still have business requirements which CAN be satisfied by using multi-threading.

Thinking about it a bit further, we can observe that if NOT for a business-level requirement to provide a response within the certain time, we wouldn’t care about multithreading AT ALL.

Indeed, we don’t have to use multithreading to do things correctly - it comes into play ONLY to provide reply faster. This stands for ALL the use cases of multithreading - from games (where “responsiveness” is measured in terms of milliseconds), to HPC (where “in 30 days” can qualify as “responsive enough”, especially compared to“you’ll have to wait until next century”).

More specifically, multithreading is usually summoned to solve one of two related-but-still-separate problems:

• The first one is “Doing things faster” (using multiple CPU cores). This one is fairly obvious - there are cases out there when one single CPU core is not sufficient to do whatever-we-need-to-do, within the allotted time. And to get the job done - we do have to find a way to use multiple cores.
• • A close cousin of this requirement is Scalability; in a sense - it can be seen as an ability to scale our job to as-many-cores-as-we-might-need.
  • The second problem we’re often trying to solve with multithreading is keeping our app from being blocked by long external operations. Indeed, a scenario when my desktop app “hangs” just because it is waiting for a DNS server to reply while the Internet happens to be down - is a Bad Thing(tm), there is no argument about it (though whether multithreading is a good way to satisfy this requirement - is a completely different story).

That’s it - these two simple things cover vast majority (if not all) use cases for multithreading.

To summarise Part I of the talk:

What we REALLY want is not multithreading as such; instead, we have two separate and distinct business requirements. The first one is multi-coring (and related Scalability), and the second one is being Non-Blocking.

Note that we’re STILL not at the point of saying that “multithreading is bad” (though we’ll certainly come to it later <wink />), but what we have here is that most of the time, multithreading as such is NOT a firm business requirement, which opens us a door for looking for alternative implementations.

Now, we can get to a discussion of (Re)Actors (and more generally, Message Passing) being one of the ways to implement our “Multi-Coring” and “Non-Blocking” requirements.

The concept behind Message Passing is known at least for 30 years (at least since the days of occam and Erlang, but I think that the most succinct expression of it has been given relatively recently, and reads as

Do not communicate by sharing memory;  instead, share memory by communicating.

In other words, under message passing paradigm , if we have to share something between two different CPU cores (which can be represented either by threads, or by processes, or even by different boxes) - we don’t use mutexes or atomics to share a state between them; instead - we send a message with all the necessary information.

This contrasts Message-Passing approach with Shared-Memory multi-threading approaches (which almost-inevitably require synchronization such as mutexes or atomics).

Now, we can proceed to my favorite incarnation of Message-Passing - (Re)Actors. (Re)Actors are known at least for 40 years, and are known under a half a dozen of different names, in particular Actors, Reactors, ad-hoc Finite State Machines, and Event-Driven Programs.

As we’ll be speaking about (Re)Actors quite a bit - let’s establish some basic terminology which we’ll use.

Let’s name Generic Reactor a base class for all our (Re)Actors; the only thing it has is a virtual function react().

Let’s name Infrastructure Code a piece of code which calls Generic Reactor’s react(). Quite often - this call will be within so-called “event loop” as shown on the slide.

As we can see - there is one single thread, so there is absolutely no need for thread synchronization within react(); this is very important for several reasons (including making our coding straightforward and less error-prone).

Let’s also note that get_event() function can obtain events from wherever-we-want-to - from select() (which is quite typical for Servers) to libraries such as libuv (which are common for Clients).

What is REALLY important for us now - is that when writing our (Re)Actor, we don’t really care which Infrastructure Code will call it. As the interface between Infrastructure Code and (Re)Actor is very narrow (and is defined by the list of the events processed by (Re)Actor) - it provides very good decoupling. In real-world, I’ve seen (Re)Actors which were successfully run by FIVE very different implementations of Infrastructure Code.

Let’s also note that while ALL the code within (Re)Actor::react() HAS to be multithreading-agnostic,  Infrastructure Code MAY use threading (including mutexes, thread pools, etc.) within. After all, the whole point here is that it is not THAT big deal to rewrite Infrastructure Code entirely, and well-behaving (Re)Actor should run within new Infrastructure seamlessly.

And finally, let’s name any specific derivative from Generic Reactor (the one which actually implements our react() function) - a Specific Reactor.

I need to note that a virtual-function-based implementation above is not the only one possible - for example, the same thing can be done using templates instead of virtualization - but for the purposes of our examples, we’ll stick to this one (and any changes to alternative implementations should be pretty straightforward).

As I already mentioned, (Re)Actor is one of the possible incarnations of the Message Passing. As such, it means that:

• first, (Re)Actor state (which is represented by data members of SpecificReactor) is exclusive to the (Re)Actor, and moreover - only one member function of (Re)Actor can run at the same time. This, in turn, means that we don’t need to bother with those extremely-error-prone inter-thread synchronization mechanisms such as mutexes <phew />.
• second, ALL the communication with other (Re)Actors goes via messages. To enable it, Infrastructure Code has to provide a function such as postMessage(). Here, ReactorAddress can be pretty much anything which is relevant to your (Re)Actor (for distributed systems, personally I am usually arguing for having it as a meaningful string, so translation into IP:port format can be done separately and transparently from the app level).
• • For receiving (Re)Actor, this message is translated into a special Event type (EventMessageFromOtherReactor or something).

In addition, as we have observed, (Re)Actors have a VERY clean and a VERY simplistic separation between Infrastructure Code and (Re)Actor code. And as a result of this VERY clean separation, it becomes VERY easy to deploy exactly the same (Re)Actor code into very different Infrastructures. In particular, I’ve seen (Re)Actors to be deployed in the following configurations:

• Thread for each (Re)Actor. This is the most obvious - but certainly not the only one possible - deployment model. Sometimes it is restricted to have one thread per process (which is useful to add an additional protection against inter-(Re)Actor memory corruption or heap fragmentation).
• Also, (Re)Actor can be used as a building block of Another (Re)Actor. (Re)Actors are composable (actually, they’re more flexible than any other building block I know about), so (Re)Actor can be used as a part of another (Re)Actor quite easily.
• Multiple (Re)Actors per thread. This is possible due to non-blocking nature of (Re)Actors, and with not-so-loaded (Re)Actors can be useful to save on thread maintenance costs, which in turn improves overall performance. In real-world production environments, I’ve seen up to 32 (Re)Actors per thread. Oh, and BTW - while usually, (Re)Actors within the same thread are of the same type - it is not a strict requirement and I’ve seen systems with different (Re)Actor types running within the same thread.
• And to demonstrate a more complex deployment scenario - it is possible to build a system which would store states for all (Re)Actors in a centralized manner in a cache such as memcached, Redis, or even a database. Then, when a request to process something in one of (Re)Actors arrives, (Re)Actor’s state can be retrieved from Redis, deserialized (effectively constructing (Re)Actor object from the serialised state), then (Re)Actor::react() function can be called, then the (potentially modified) state can be serialized, and written back to Redis. To avoid races, we’ll need to have some locking mechanism (preferably optimistic one) - but the whole thing is perfectly viable, AND for certain classes of (Re)Actors it can provide an easy way to ensure Fault Tolerance.

We don’t have time to go into further variations of possible deployment options, but what is most important for us now, is that all these deployments can be made with our (Re)Actor code (more specifically - (Re)Actor::react() function) BEING EXACTLY THE SAME. It means that the choice of the surrounding Infrastructure becomes a DEPLOYMENT-TIME option, and can be changed “on the fly” with EXACTLY ZERO work at app-level.

This has been observed to provide significant benefits for real-world deployments, as such flexibility allows to design optimal configurations - AND without rewriting app-level code which is almost-universally a non-starter.

When writing (Re)Actor code, there are two not-so-obvious features to be kept in mind. The first one is what I prefer to call (mostly-)non-blocking processing.

The non-blocking code is long criticised for being unnecessarily convoluted, but I think that this perception stems from two misunderstandings. One misunderstanding assumes that to write non-blocking code, you have to resort to the stuff such as lambda pyramids or (Djikstra forbid) “callback hell”. While it was indeed the case 10 years ago, these days several major programming languages (AT LEAST C++, C#, and JavaScript) have introduced a concept of “await”. We don’t have time to discuss how it is working internally, BUT - we’ll use it in our examples to demonstrate how it can be used at the app-level.

The second misunderstanding is that non-blocking approach is all-or-nothing, so there is a MISperception that if we have started to go non-blocking, we cannot have any blocking calls whatsoever. While this assumption MIGHT stand for systems handling life-or-death situations, but for your usual business system nothing can be further from the truth (I’ve seen HUGE real-world systems which mixed blocking processing with non-blocking one - and with great success too).

Very briefly - for ALL the I/O processing, there are two DISTINCT scenarios. In the first scenario, we DO NOT want to process ANY potentially happening events while we’re waiting for the result of the outstanding call. In this case, IT IS PERFECTLY FINE TO USE BLOCKING CALL. Of course, there is a caveat that IF operation just MIGHT take too long even once in a really while, we DO want to process intervening events for it.

This might be the reason for the misunderstanding, but in reality, things tend to be quite simple: THERE ARE things out there which, if they happen to take too long, mean that THE SYSTEM IS ALREADY BROKEN BEYOND ANY REPAIR. For example, if we’re trying to read 1kilobyte from a local disk and it takes any observable-for-user time - the PC is most likely already dead; and on the Server-Side, IF our database (which normally responds within 10ms), starts to exhibit response times an order of magnitude longer - the system won’t work as-we-expect REGARDLESS of us using blocking or non-blocking call here. Real-world examples of such operations-which-we-MAY-handle-as-blocking, often include accessing the local disk and/or database,  and sometimes even operations in the Server-Side intra-datacenter LAN.

If, on the other hand, we MIGHT want to process something-which-has-happened-while-we’re-waiting - we SHOULD process it in a non-blocking manner; however, in this case, any complications are not because of the CODE being non-blocking, but because of the EXTERNAL REQUIREMENT to process those intervening events (and believe me, handling intervening events via mutex on a state is MUCH worse). One example of such operations which ALWAYS need to be handled as non-blocking - is ANY operation which goes over the Internet.

Fortunately, with “await” operator in mind, the non-blocking code looks REALLY simple. Not just that, but as we can see, it looks REALLY simple EVEN IN C++.

On the other hand, I would be cheating if not mentioning one very significant potential issue arising here. At the point of co_await, there can be a context switch to process a different event (hey, this is EXACTLY what we wanted when we added co_await in the first place). And this processing can CHANGE the state of our (Re)Actor, sometimes in an unexpected manner.

Let’s consider the following blocking code…

With this code, the assertion at the end always stands (well, with some not-so-unreasonable assumptions about readFile() function not having too drastic side effects)

Now, when moving to the non-blocking paradigm, the same assertion MAY FAIL. This can happen because while we’re waiting for the result of readFile(), another event could come in, and its processing can change m_x while we’re waiting on co_await operator.

OTOH, we have to note that:

• first, as discussed before, we DO NOT need to use non-blocking handling AS LONG AS we don’t need to process-events-while-waiting.
• second, this problem is INHERENT to ANY kind of handling-of-intervening-events-while-waiting-for-operation-to-complete. In other words, WHATEVER WE DO, there is no way to avoid this kind of interaction (and very often, it is exactly this interaction which we need when dealing with intervening events).
• third, there is still NO need for thread sync in the code on the slide (while context switches are possible, they can happen ONLY at those co_await points, so by the time when we’re running again, we’re GUARANTEED to have exclusive access to members of our (Re)Actor <phew />)
• last but certainly not least, this code is inherently MUCH SIMPLER and MUCH less error-prone than Shared-Memory approaches. Not only all the thread sync is gone (and it is a MAJOR source of non-debuggable problems), but also while we DO have a potential context switch here, it’s position is well-defined, while in a mutex-based program such a context switch can happen AT EACH AND EVERY POINT IN OUR PROGRAM (which makes reasoning about potential effects of such switches orders of magnitude more difficult).

As a nice side effect, the same approach can be used to ensure parallelization even while staying within the same (Re)Actor.

The idea here is very simple: we’re merely saying “hey, execute for me this-function-with-these-params (or ‘this lambda function using these captures’) and resume execution when you’re done.“ While lengthy calculations are performed - we can still process incoming events, and context switches and their implications here are exactly the same as for the non-blocking code discussed above, which in turn simplifies coding further.

The second not-so-obvious-feature of (Re)Actors (and the one which I happen to love a LOT <smile />) is determinism.

Strictly speaking, making (Re)Actor deterministic is not required to run a (Re)Actor-fest architecture, so we MAY have our (Re)Actors as non-deterministic; however - if we’ll spend additional effort on making them deterministic - we’ll get several very-useful-in-real-life properties, so I consider it as a VERY nice to have.

For our purposes, the following definition (which corresponds to “same-executable determinism” from [Nobugs17]) will do:

    The program is deterministic if

        we can write down all its inputs into inputs-log,

        and then replay this inputs-log using the same executable,

        obtaining EXACTLY THE SAME outputs.

We don’t have time to discuss the way HOW deterministic (Re)Actors can be implemented (last year, I made a 90-minute talk just about determinism at ACCU 2017), but I’ll still outline a few very basic points about it.

In general, there are three major sources of non-deterministic behavior:

• multithreading (which doesn’t apply to (Re)Actors, <phew />)
• Compiler/library/platform (which doesn’t apply to same-executable determinism, another <phew />)
• All kinds of system calls (starting from very-innocently-looking GetTickCount() and gettimeofday()).

To become deterministic (and in addition to recording all the input Events), we have to record return values of these System Calls to the same inputs-log; this is known as “Call Wrapping”. Then, when we’re replaying our inputs-log, as a result of deterministic behaviour up to the point of System Call, the call-during-replay will happen EXACTLY at that point where we have a record of the return value within inputs-log, so it is trivial to return recorded value - and to ensure determinism at this point. Then, determinism of the whole replay can be proven by induction.

“Call Wrapping” allows to handle ANY system call; however, alternatively, quite a few popular System Calls (such as time-related stuff) can be handled in a faster and/or more flexible manner using other techniques (which were discussed in detail in my talk a year ago).

An example implementation of Call Wrapping MAY look as follows.

In app-level code, we’ll have to replace all the system calls with their deterministic wrappers (or provide seamless replacements). After we do it - implementation of the wrapper becomes trivial: in “Recording” mode we’re calling system call, AND writing its return to the inputs-log, and in “Replay” mode we’re not calling anything from system-level, merely reading the corresponding frame from the inputs-log and returning the result back to the calling app.

Some further practical considerations when we’re implementing determinism:

• in production, storing inputs-log forever-and-ever is not practical (at least most of the time)
• to deal with it, we may use circular logging - but this in turn requires state serialization.
• Serialization can be done in different ways, but one of the most interesting ones (and certainly the fastest one) is the one which assigns allocator to the (Re)Actor, and serialises the whole allocator at CPU page level; while there are some caveats on this way, it seems to work pretty well in practice (unfortunately, this implementation is very new and wasn’t tested in massive deployments yet).

As soon as we make our (Re)Actors deterministic, we obtain the following all-important benefits:

• more meaningful testing. As non-reproducible testing is not exactly meaningful, higher-level test cases often have to ignore those-parts-of-the-output-which-depend-on-the-specific-run (such as time-based outputs). With deterministic (Re)Actors, it is not a problem.
• production post-mortem analysis. If we’re writing all the events (and returns of system calls) to an in-memory log, then, if our app crashes or asserts, we can have not only the-state-which-we-had-at-the-moment-of-the-crash but also last-N-minutes-of-the-program’s-life. As a result, most of the time we can see not only WHAT went wrong, but also WHY it went wrong. This has been seen to allow to fix 90+% of the bugs after the very first time they’re reported IN PRODUCTION - and this is a feature which importance is VERY difficult to overestimate.
• Low-latency Fault Tolerance and/or (Re)Actor Migration. We don’t have time to discuss it in detail, but we can say that at least the former is a very close cousin of Fault Tolerance via so-called “Virtual Lockstep” used in the past by VMWare.
• Replay-Based Regression Testing. This can be a real life-saver for major refactorings - but I have to note that it requires a bit stronger guarantees than same-executable determinism - which guarantees are usually still achievable in practice.

Now let’s see how (Re)Actor-based architectures (whether deterministic or not) satisfy our two Big Business Requirements.

When speaking about multi-coring and scalability - everything looks very very good for (Re)Actors (and more generally - for message-passing architectures). As we don’t share ANY state between different cores (threads, processes, whatever-else) - it means that we’re working under so-called Shared-Nothing model, which is EXTREMELY good for Scalability. Not only Shared-Nothing architectures do allow to use multiple cores - moreover (and unlike Shared-Memory-based approaches) they scale to the pretty much INFINITE number of cores in a not-so-nonlinear-manner. Sure, there are some corner cases where pure (Re)Actors don’t work too well (in particular, in case of one huge monolithic state), but these cases are rather few and far between (and as we’ll see in Part III, there are not-too-ugly workarounds for it too).

When speaking of guarantees against blocking, under (Re)Actors it is achieved via (Mostly)-Non-Blocking I/O we just described. Indeed, if our current thread can process other inputs while that-I/O-call-which-can-take-5-minutes is in progress - we won’t have any problems with our app being responsive-enough for usual end-user.

In practice, I’ve seen (Re)Actors exhibiting latencies as low as 50us, though significantly reducing this number would be complicated, if possible at all.

Now we can try comparing two competing programming approaches: on the one hand it is Shared-Memory one, on another hand - it is (Re)Actors (or more generally, message passing).

First, let’s compare code complexity under these two approaches. As I already mentioned, IF intervening events do NOT need to be handled - both approaches are exactly the same. As for the handling of intervening events - with mutexes and stateful systems it is a surefire recipe for the disaster, but for (Re)Actors it is perfectly manageable.

Now, let’s compare our adversaries from the point of view of “how easy it is to write error-free programs under these paradigms” (which is the opposite of being error-prone). On this front, (Re)Actors win hands down. If there are no mutexes in our programs - there is no chance to forget to lock them when it is necessary, and there is no chance to lock them in the wrong order (opening the door for deadlocks). Sure, there are still ways to mess our program up - but they won’t be specific to (Re)Actors. In the real-world, (Re)Actor-based systems were seen to exhibit 5x less unplanned downtime than industry average (I’ll discuss it in a bit more detail in part III of the talk).

Our next criterium is testability. In general, Shared-Memory programs are inherently untestable, which in turn lets LOTS of bugs into production. OTOH, for (Re)Actors testability is generally very good even without taking any special measures such as determinism; and with determinism implemented they become perfectly testable.

The next line is closely related to the previous one, but it is soo important in practice that I have to mention it separately: it is “how we deal with bugs in production”. For Shared-Memory systems, there are LOTS of cases when we have no idea why it crashed - and worse, don’t even know how to find it out, causing LOTS of delays in fixing those all-important production bugs. OTOH, for (Re)Actors, simple text-based logging tends to help A LOT, allowing to fix 80+% of the bugs after their first manifestation; and if we have our (Re)Actors deterministic - the number can go to 90-95%).

The fifth line in our comparison table is Scalability. By definition, Shared-Memory systems DO NOT scale beyond one single box. OTOH, with (Re)Actors we’re speaking about POTENTIALLY-UNLIMITED scalability.

Then, there is performance (which is often a VERY different beast from Scalability). Shared-Memory approaches MAY provide ALL the spectrum of performance, ranging from Poor to Excellent (more on it in a jiff); OTOH, (Re)Actors, tend to exhibit Good to Excellent performance. In the almighty real-world, I have seen examples when (Re)Actor-based systems were performing about 30 TIMES more useful work per server box (again, we’ll briefly discuss it in Part III).

And on the last line, I WILL mention that-only-thing which does NOT allow to eliminate Shared-Memory Approaches entirely; it is Latencies. NON-blocking Shared-Memory allows reaching latencies which are measured in single-digit microseconds. As for (Re)Actors, they’re limited to some tens-of-microseconds. There are applications out there (notably HFT) which DO need this kind of latencies, but for 99.99% of all the business apps out there, double-digit microseconds are good enough.

Speaking in a bit more detail about performance and latencies, I have to note that generic Shared-Memory approach has two flavors which tend to be DRASTICALLY different from the performance point of view.

The first one is BLOCKING Shared-Memory, the one which uses mutexes (or any other blocking mechanism) at app-level to synchronize between threads. This is by far the most popular option for Shared-Memory systems, and it tends to fail BADLY in terms of performance (as it was already noted several times during this conference, one of the authors of POSIX threads has said that they should have named “mutex” a “bottleneck” to reduce chances for it to be misused).

This happens due to context switch costs, contention, and starvation (and BTW, by its very definition mutex is not really a way to make programs parallel, but a way to make them serialized).

Latency-wise, blocking Shared-Memory tends to be very uneven (and usually very unfair too) - and requires to specify acceptable latencies in terms of PERCENTILES.

Performance problems with blocking Shared-Memory are long recognized, and over the time several ways were invented to make Shared-Memory perform better (using stuff such as memory fences, atomics, and RCU). This indeed is known to perform really well (at least within one NUMA node).

Back to (Re)Actors and more generally - to Message Passing. Strictly speaking, Message-Passing also has two flavors: the first one is “classical” message passing (which includes (Re)Actors, Message Passing by Erlang, MPI, and so on). It tends to have very good performance (well, MPI is successfully used for HPC for decades). Latency-wise, it can provide latencies which are as low as double-digit microseconds. While this performance (and especially latency) is usually not AS GOOD as that of the NON-BLOCKING Shared-Memory - it is perfectly-sufficient for 99.9% of real-world apps.

The second flavor of Message-Passing is so-called Message-Driven (a.k.a. Data-Driven) programming, which is used in particular in an HPX library. It tends to improve calculation performance even further - making it a direct rival to much more complicated non-blocking Shared-Memory stuff performance-wise. In terms of latencies, while I don’t have any hard data on message-driven stuff, I don’t expect it to be AS GOOD as that of the non-blocking shared-memory systems.

To make this table complete, we also have to mention scenarios where we can PRETEND that we don’t care about the synchronization at all (it happens when contention is so low that we can assume it is negligible, so ANY synchronization will work with about the same performance). In this case, as the synchronization is virtually non-existent - we can get near-perfect performance.

To briefly summarise this table - I am sure the ONLY technology which we MIGHT have to rule out due to performance issues, is mutex-based Shared-Memory; performance of ALL the other architectures is more-or-less comparable, and each of them happens to have their own virtues (and applicability fields).

To summarise Part II of my talk today, (Re)Actors:

• (a) enable reasonable writing of app-level code, which is either SAME or SIMPLER than equivalent shared-memory code (and especially so with await in mind)
• (b) enable a very wide range of deployment options
• (c) are less error-prone, are testable, and allow to handle production bugs very efficiently
• (d) scale and perform very well
• and (e) can be made deterministic, which enables lots of its own goodies

Now, as we’re done with describing an individual (Re)Actor, we can proceed to discuss “how to build THE WHOLE SYSTEM consisting of nothing but (Re)Actors” (well, “almost nothing”, as usually - though not necessarily - Infrastructure Code happens to be NOT (Re)Actor-based).

First, let’s discuss (Re)Actor-fest architectures on the Client-Side. Here, I have to note that MOST of the Client-Side Apps are event-driven to start with; in fact, I don’t know of one single GUI framework which is NOT event-driven.

For a usual business app one single core (and therefore, single-threaded (Re)Actor) is usually more than enough; however, let’s consider something more complicated - a Game Client (or a Stock Exchange Client, which, believe it or not, is not THAT different).

As we can see, everything is quite simple here: we have a bunch or (Re)Actors, which communicate with each other via queues; in addition, some of these queues receive information from the outside world (user input, and network via select()). Inside, most of the (Re)Actors MAY call system-level functions to perform certain operations. If the call is guaranteed to complete “fast enough” (such as most of graphics calls) - then we can rely on that “(Mostly-)” prefix in Mostly-non-blocking, and call it in a blocking manner.

However, if the call is of indeterminate length (such as ALL Internet-related calls in Communications (Re)Actor) - we will issue a NON-blocking call and will wait for the result asynchronously. For Communications (Re)Actor, within our Infrastructure Code we’ll usually wait for select()-like function (in practice - it can be poll(), epoll(), WaitForMultipleObjects(), etc). When select()-like function returns - our Infrastructure Code will EITHER convert network packet into an event, OR will ensure that it is received by app-level code after respective await operator returns (in C++, it is done by implementing await_resume() function of an awaiter object).

As always with (Re)Actors, all kinds of different deployment options are possible. If our game is a social game or a stock exchange - the following single-threaded configuration is possible. It is about the same thing - but with EXACTLY THE SAME (Re)Actors moved to the main thread of the program.

On the other side of the spectrum is much more heavier system with one (Re)Actor per operating system process; these might be useful if we’re fighting with memory corruptions (either to debug them, or - in case of a buggy 3rd-party library which we have to use for whatever reason - to isolate them).

And, of course, between these two extremes (“everything in one single thread”, and “each (Re)Actor running in its own dedicated process”) there is pretty much everything else.

One interesting variation of (Re)Actors is a hybrid between Message-Passing and Shared-Memory architectures; I prefer to name it “(Re)Actor-with-Extractors”.

To the best of my knowledge, (Re)Actor-with-Extractors was first used by Bungie for their AAA game “Halo:Reach” [Tatarchuk]. The idea behind (Re)Actor-with-Extractors is simple:

• essentially, we have two phases in the processing (in games, both these phases usually fit into one single frame).
• during phase one (between dashed lines), state of our (Re)Actor stays CONSTANT (just because all the parties agreed not to modify it); as a result - it is perfectly safe to read it from multiple threads. In particular, “Extractors” may “extract” information-which-they-will-need-for-further-processing.
• as soon as each of extractors is done with extraction - it notifies Main Thread that it is done, and can process extracted data in its own thread, with no interaction with the state of our (Re)Actor.
• and as soon as ALL the extractors are done extracting data, Main Thread can proceed with modifying part of the (Re)Actor::react() (or the whole (Re)Actor::react() if we didn’t separate its read-only part).

This way, we can mitigate that Huge-State-Problem mentioned before (instead of separating it to multiple interacting (Re)Actors) - all while keeping ALL the benefits of (Re)Actors (there is NO thread sync within react(), it can be made deterministic, etc. etc.).

(Re)Actors-with-Extractors are known to be used in real-world (at least in gamedev industry) - however, I’ve seen them only on the Client-Side (my guess is that it can be attributed to an observation that ANY shared-memory approach cannot possibly scale beyond one single box, which is rarely sufficient for the Server-Side).

On the Server-Side, things tend to be even more interesting. Let’s start with a classical (NOT (Re)Actor-based) Server-Side for your usual web app.

The whole point of this architecture is that ALL the state we have is stored in the database. It means that EACH AND EVERY USER CLICK MODIFYING SOMETHING (beyond purely Client-Side stuff) - has to go to the DB. Since time immemorial, it was argued that this architecture, being stateless on the app side, BOTH allows for fault tolerance (which it does), AND scales well (which it doesn’t). The problem with the scalability of this architecture is that while scaling stateless apps is indeed trivial, it goes at the cost of pushing ALL the load to the database, and pushing all the load to one single point is rarely a good idea scalability-wise.

Practically, while this architecture does work for not-so-loaded projects, there are three big problems with this architecture as the load grows.

• first, as the load grows, in the DB we will have to use “transaction isolation levels” different from “serializable” (for performance reasons); and going into non-“serializable” isolation, in turn, happens to be about as error-prone and fundamentally untestable as multithreaded programming. This includes ALL THE SAME issues such as explicit locking to avoid data races (which is known as “SELECT FOR UPDATE” in database world), and ensuring order of locks system-wide to avoid deadlocks.
• Second, when we’re throwing ALL the data we have, at one single DB, no RDBMS out there scales in a linear fashion. In other words, DB becomes a CHOKE POINT of the whole architecture (and we have NOBODY to blame for it but ourselves). I have seen plenty of real-world examples of this problem, but the best publicly available one is a well-known problem of Uber, which first migrated from MySQL to Postgres (in hope to fix scalability issues), only to migrate back two years later or so. However, none of those migrations was really necessary: as we’ll see a bit below, Uber just doesn’t have enough write-transactions-which-HAVE-to-be-durable to cause any trouble even on a single(!) box; it is merely a question of architecting app with scalability in mind instead of blindly following stateless-app model pushed down our throats for last 20 years.
• Third, there are cache coherency issues. As SQL writes go directly to RDBMS, bypassing back-end cache, it makes back-end cache not really coherent (which tends to cause quite a few visible-in-end-user-space issues); and attempts to make it coherent at app-level are EXTREMELY error-prone (as Phil Karlton has reportedly said: “There are only two hard things in Computer Science: cache invalidation and naming things.”).
• And last but not least, latencies of the order of dozens of milliseconds or less are very difficult to achieve. As EACH AND EVERY writing request MUST go to DB - it has to go all the way to DB and back (and DBs are not known for being latency-friendly, especially under high load).

To make things even worse, these problems tend to develop EXPONENTIALLY as the load grows in the linear fashion. In other words - your DB will fail to cope with load (or even worse - exhibit that-multithreading-like problem in your code which never manifested itself before -  EXACTLY on your BIG DAY🙁.

Very roughly, speaking of a single DB where trivial partitioning doesn't exist, as long as you have fewer than 100 millions transactions per year - there is not much to worry about even in this architecture.

Between 100 million and 1 billion write transactions per year (with each transaction handling on average 10 rows) - things become MUCH worse. First, LOTS of previously-unseen transaction-isolation bugs start to surface - as noted above, in a non-linear fashion; and second - it starts to require LOTS of boxes as DBs do NOT scale linearly (that is, unless trivial sharding exists, but as soon as we have at least SOME interactions-just-because-user-A-wants-to-interact-with-userB - it doesn’t).

Between 1 billion and 10 billion transactions/year - things become even worse (to the point of this architecture becoming barely manageable).

And beyond 10 billion write transactions per year - with these architectures, things tend to become REALLY bad even IF (like with Uber) data is inherently trivially shardable.

Of course, all the numbers have to be taken with a HUGE grain of salt, and even in the very best case are accurate only within an order of magnitude.

As no single non-trivially-shardable DB can realistically handle over roughly 10 billion transactions per year, there is pretty much the only thing we can realistically do to cope with this kind of load - introduce an in-memory state (at least as a write-back 100%-coherent cache).

Such an architecture is shown on the slide; what we REALLY need to do at app-level, is to decide which parts of our Server-Side state MUST be durable (i.e. MUST NOT be lost even if the whole thing goes down), and which parts of the state can be NON-durable - which means that the sky won’t fall if we lose them during a once-per-several-years crash.

For example, for Uber, it is only RIDES (and maybe ORDERS) which MUST be durable (and with the number of Uber rides being less than half a billion per year - load-wise there is absolutely nothing to speak about, it is certainly possible to handle this kind of load on one single core).

What causes most of Uber troubles, is writing of all those location data from all the drivers and users - but it is NOT kind of data which has to be durable! That is, if, once in a year, this data is lost - well, it will amount to a 30-seconds delay, which is not TOO bad; note that this data MAY still have to be made persistent for analytical purposes, but analytics is a completely different story which we do NOT care about at this point (at least - we can be sure that losing 30 seconds of analytics once a year won’t cause too much trouble).

And as soon as we split our data into “durable” and “nondurable” - our architecture becomes very straightforward: we have web servers which deal with State Cache (which is NOT durable, so DB behind State Caches is actually OPTIONAL), and DB Server App, which is responsible for dealing with DURABLE and highly-critical data.

Under this architecture, the introduction of the State Cache allows to reduce the load on DB Server App - usually by a factor of 10x to 100x, which already makes HUUUUGE difference. In addition, as DB Server App is a SINGLE point of access to DB - it has an option to have a 100% coherent read-only Back-End Cache (which was seen to speed things up further by a factor of 10x, further reducing the effective load on DBMS by another order of magnitude).

As a result, by using an in-memory state, we can reduce DB load by anywhere from 10 times (which, due to highly non-linear issues, already qualifies as A DAMN LOT) to 1000 times (which can easily solve all your problems forever-and-ever - even if you’re Uber, Facebook, or Twitter).

Oh, and from the end-user point of view, there are absolutely no coherency issues (we do NOT have any non-coherent caches in the picture, except for static stuff).

From our current perspective, the most important point of the previous slide is that as soon as we introduce in-memory state, we can already express business-logic as <da-dum! /> (Re)Actor (don’t tell that you didn’t see it coming).

Actually, Infrastructure Code running on our Web Server from the previous diagram, can perform the same thing which we already briefly mentioned in the context of different deployment options for our (Re)Actors. Indeed, nothing prevents us from storing the state of our (Re)Actor externally in Redis or memcached, deserializing the state before, and serializing it back after (Re)Actor::react() call. To ensure coherency, we’ll have to implement some locking (usually I tend to prefer optimistic one based on Compare-and-Swap by Redis/memcached), but this locking can and SHOULD be implemented completely by Infrastructure Code, and without ANY app-level logic involved (which IS important as changing Infrastructure Code is usually possible, while rewriting million-LoC app-level logic is usually a non-starter).

And as soon as we introduce in-memory state and (Re)Actors, moving further becomes a cinch. Here is how a typical MOG game server looks like (with Game Servers running classical (Re)Actors). BTW, quite a few stock exchanges look about the same (while for stock exchanges all transactions have to be durable, they still benefit from reduced latencies, effectively using (Re)Actors as read-only caches).

Note that migration from/to our previous diagram can be implemented without ANY changes to our (Re)Actor code (in other words, whether we want to use an external central cache such as Redis, or intra-(Re)Actor state, is a DEPLOYMENT-TIME DECISION).

As for real-world performance and scalability - such a system has been observed to handle a billion messages per day on mere four server boxes (though this number does not account for two dozens of stateless Front-End Servers to handle purely communications traffic); as for latencies - it has been seen to achieve latencies at single- to double-digit millisecond levels.

By this time, we got a perfectly-debuggable and very-reliable (Re)Actor-based system; however, a question “how to scale our database” remains; while it IS ten-to-a-thousand-times easier to scale than usual stateless web-like architectures, for really-loaded systems it still CAN become a problem.

Strictly speaking, DB Server App is an implementation detail, so to stay within the (Re)Actor-fest model, we are NOT required to implement it in any particular manner.

However, USUALLY, I am suggesting to go the route of gradual development of the DB Server App, where DB Server App is implemented as (no surprise here) a yet another (Re)Actor. While it does sound as an Ultimate Fallacy for anybody with traditional DB experience, it DOES work in the real world for several systems which have already processed hundreds of billions (that’s 11 zeros) of durable write transactions and made some billions of dollars to the respective companies in the process.

The very first step on the road to hundreds of billions of durable transactions looks as follows. Here, we have a DB Server (Re)Actor, which processes all the incoming requests one-by-one, over a single writing database connection; note that all the requests coming to DB Server (Re)Actor have to be expressed in terms of business logic (such as “move money from account X to account Y”), rather than in terms of SQL. Also, ALL the requests MUST be inherently atomic (so there should be no half-baked transactions or, Codd forbid,  cursors after the request is completed).

This allows us to process incoming requests in a (Re)Actor fashion, one by one, and without worrying about “SELECT FOR UPDATE” or deadlocks. It also tends to work very fast and very predictably (there is zero contention by definition, and unless some read-only connection managed to poison RDBMS caches, there is absolutely nothing which can delay our simple requests). Note that we CAN have multiple read-only connections (preferably - with Read Uncommitted isolation, to avoid locks-which-may-affect-our-DB Server (Re)Actor)

When deploying such systems (which are very sensitive to latencies), there is some not-so-usual trickery; in particular, at least for DB logs, we MUST use a local RAID array with a RAID card having so-called BBWC (battery-backed write cache). As soon as this is satisfied - the whole thing works like a charm, and is extremely reliable too: first, the whole thing is debuggable (and can be made deterministic), so ANY issue is identifiable and fixable VERY quickly - and second, even internal races within RDBMS are rather unlikely to manifest themselves (as we’re not using MOST of race-dangerous RDBMS features, bugs in them become irrelevant to our deployment); overall, one specific RDBMS was seen to crash as rarely as once-per-30-to-50-billion-transactions - that is, under this specific configuration (and FWIW, multiple writers, used in replicas, made the same RDBMS crash MUCH more frequently).

After usual DB optimizations such as indexes, proper DB-level caching, etc., such systems have been seen to process about 1 billion transactions/year. As the load grows beyond this number, this system has to be refined.

To support loads up to about 10 billion transactions/year (that’s 10 zeros), the following Take 2 (as a gradual evolution of the previous one) has been seen to do the trick. We still have our single-writing connection DB (Re)Actor, but - we’re doing two major improvements here.

First, as we have the single point of access to the database - we can have 100%-coherent app-level cache. This alone has been seen to improve overall performance by about an order of magnitude (especially caching USERS table tends to be beneficial, as there are traditionally LOTS of checks against this table while processing requests).

The second improvement is that instead of running reporting over the main DB, we’re creating a replica, and it is replica which is used to run reporting. This solves two problems: (a) it prevents cache-poisoning-caused-by-rogue-report, from ruining the performance of the main DB, and (b) it allows to limit the size of main DB (which in turn allows to keep it 100% cached, which is a Good Thing(tm)). Note that to limit the size of main DB, our replicas have to become “super-replicas” (i.e. when removing the historical data from the main DB, this data is still kept in super-replicas), but this has been seen working like a charm too.

Such systems (after LOTS of optimizations), have been seen to handle up to 10B transactions/year, writing up to 100 billion rows (that’s over ONE SINGLE DB CONNECTION, AND staying within the deterministic guaranteed-to-be-race-free processing).

If even 10 billion durable writing transactions/year are not enough (which BTW I have seen in my career only once) - then we can go even further (while still staying within (Re)Actor paradigm). The idea would be to split our single monolithic database into multiple independent ones (it has been seen to work MUCH better than attempting to parallelize within the same DB).

The basic idea here is that when somebody wants to play, say, in a Tournament, first we’re making an ASYNCHRONOUS EVENTUALLY-CONSISTENT TRANSFER of the required attributes (such as “gold-required-to-play”) from USERS DB into respective TOURNAMENT DB, and then TOURNAMENT DB can do whatever-it-needs-to-do within its boundaries. When the tournament finishes - we’re doing another ASYNCHRONOUS EVENTUALLY-CONSISTENT TRANSFER from TOURNAMENT DB back to USERS DB. That’s pretty much it. As the transfers are inherently ASYNCHRONOUS - there is no blocking involved, so each of DBs still works at the maximum possible efficiency.

The way HOW EXACTLY to split DB depends on specifics (on the diagram, a split for a “typical” MOG is shown). OTOH, I don’t know of a system where such a split doesn’t exist. In particular, USERS table is present in a pretty much ANY business-like environment, and “places where users can interact” (on the diagram - “Game Worlds” and “Tournaments”) are also very typical. And as soon as we handled all the USERS and all the interactions between them - there isn’t much we have left; at this point the only thing we have to do, is to shard USERS horizontally (which usually becomes easy at this point, as all the inter-USER interactions are out of the picture) - and we have a PERFECTLY scalable database.

As for the performance - this kind of stuff can handle HUNDREDS OF BILLIONS OF TRANSACTIONS PER YEAR ON ONE SINGLE SERVER BOX. Of course, it WILL scale beyond one single box - but TBH, I know of ONLY ONE real-world system which really needs it; to put it into perspective - Twitter handles 200 billion tweets per year, and all Facebook comments+status updates+photos amount for 500 billion updates per year. The only system which goes above this level is post-2005 NASDAQ where lots of bot trading is happening, and which goes into single-digit thousands of billions quotes+trades per year.

Sure, one single box will NOT be sufficient to store THE HISTORY of all those transactions for more than a few months, so we WILL need historical super-replicas to store the history; however, for 100% COHERENT PROCESSING OF DURABLE TRANSACTIONS it will work (and this is THE ONLY place where we need 100% coherency, so everything else scales trivially).

On a question “hey, everybody is using those huge cloud clusters to handle the same kind of load, why?” I will tell that I don’t really know WHY. However, I can tell a real-world story in this regard. Once upon a time, in a certain online industry, there was an industry leader processing about 50% of the whole market; they made it on a bunch of 50 boxes or so (including both database and app servers). A direct competitor (with 99% of the logic being the same, and reporting available being MUCH weaker), had traffic which was about 4x lower AND used 400 boxes. That means that competitor used 32x MORE SERVER BOXES PER UNIT OF WORK. Moreover, downtimes of the industry leader were about 5x lower than the industry average. Needless to say, that at that point the industry leader was the only one in the industry using (Re)Actor-fest architecture (both at app-level, and at the database level).

Oh, and let’s observe that I DO NOT have ANYTHING against Big Data NoSQL databases: they are indeed REALLY GOOD for their purpose; it is just that they are NOT the best choice to handle durable transactions in a highly coherent manner; their best place is to handle that huge historical arrays of data stored in replicas.

Let’s summarise our findings of (Re)Actor-fest architectures:

• First, it IS possible to build a system using NOTHING but (Re)Actors at least at business logic level
• Second, such systems were seen to:
• • scale to at least to 10 billion network messages a day
  • (Re)Actor-based database handling can scale up to tens of billions writing DB transactions/year - that’s over a SINGLE DB connection
  • • there are systems which reached an equivalent of 100 billion writing DB transactions/year in testing, but AFAIK, the number has never been reached in practice
  • in addition, (Re)Actor-fest architectures were observed to (a) have 5x less unplanned downtime than a competition, and (b) use 30x less hardware than the competition (for the same unit of work, that is).

Of course, as with ANY technology, (Re)Actors (and (Re)Actor-fest architectures) have their own limitations.

Let’s expand our performance table (the one which was shown on a slide from Part II) with two new columns:

• A chance to handle a respective approach correctly in a typical business-level app, and
• types of Apps where the respective approach is usable to write app-level code

Now, let’s take a closer look at it. As discussed above, BLOCKING Shared-Memory (the one which uses mutexes at app-level to synchronize between threads), tends to fail badly BOTH in terms of performance AND in terms of being correct. As a result, I DO NOT see ANY room for mutexes in modern app-level programming - except when they’re VERY accidental to the task at hand.

Non-blocking shared memory (including rather exotic stuff such as non-blocking algorithms, memory fences, and RCU) are known to perform really well (at least within one NUMA node) - but complexity of these things for any non-trivial task (especially IF we’re not going to accept non-fixable bugs in production) is usually THAT high, that it is not really feasible for a vast majority of business programs out there (nor it is really necessary). Still, it remains a very reasonable choice for absolutely-latency-critical apps such as High-Frequency Trading (HFT).

“Classical” flavor of Message Passing (including (Re)Actors), tends to have very good performance- and excellent chances to handle it well in a typical business app. No wonder it is my personal preference to be used (except noted otherwise in this table).

The second flavor of Message-Passing, is Message-Driven (a.k.a. Data-Driven) programming, tends to improve calculation performance even further, but this performance gain is mostly HPC-oriented, and won’t make much difference for a typical business app (though IF you REALLY need heavy calculations, HPX would be my first recommendation).

As for no-contention stuff, it is widely used for web apps (in fact, assuming no-contention, though it is actually pushed down to the database level, with transaction isolation coming into play). Moreover, for LOW-WRITE-LOAD web apps I don’t object to this kind of architectures; it DOES work - as long as we have not-more-than a few writing transactions per second (or, more formally, as long as we can keep “serializable” isolation level for our DB connections). However, as we discussed above, in spite of all the theory saying that going for stateless no-sync apps is the only way to scale, such systems-taken-as-a-whole, are NOT really scalable. As a result, I am AGAINST trying to scale this kind of architecture beyond its limits (which happen to be around 100M writing transactions/year, but still covers LOTS of websites, including such monsters as BBC, CNN, etc., which have LOTS of reads but only VERY FEW writes).

One important thing to note is that this table is based only on anecdotal evidence (a.k.a. Real-World Experience), so Your Mileage May Vary.

To provide a bit different perspective to the table from the previous slide, it can be rewritten as follows:

• the ONLY field which I know where (Re)Actors won’t fly, is High-Frequency-Trading a.k.a. HFT. HFT guys-and-gals are dealing with latencies of the order of hundreds-of-nanoseconds, and traditional (Re)Actors are not AS good latency-wise. On the other hand, I have to mention “CAS-size (Re)Actors” [NoBugs17a] in this context, which MIGHT simplify development of non-blocking algorithms (CAS (Re)Actors is a very interesting subject per se, but unfortunately, we don’t have time to discuss them).
• For HPC, Message-Passing MPI is successfully used for ages, but I do agree that HPX-like approach of being Message-Driven is more promising (still, mutexes are outlawed there).
• For low-write-load web apps, a classical approach which essentially ignores all the synchronisation problems - DOES work, but ONLY as long as write load is relatively low. As soon as the write load goes higher - this approach doesn’t really scale (causing ALL kinds of trouble, from impossible-to-find data races to poor scalability).
• And for everything-else-out-there, IMNSHO (Re)Actors are THE way to go. In particular, real-world experience has shown that (Re)Actor-based systems have been seen to perform 30x better, and 5x more reliably than the competition, while processing billions of messages per day, and tens of billions DB transactions per year.

WHAT ARE YOU WAITING FOR? ARCHITECT YOUR NEXT SYSTEM AS A (RE)ACTOR-FEST!

To wrap it up, I also have to note that the subject of the (Re)Actors and related architectures (especially when speaking about fine details of implementing it) is a huuuge subject. As a result - it isn’t possible to fit all of this discussion into one 90-minute presentation, so pretty much inevitably there will be remaining questions. For further information, you can either to…

contact ‘No Bugs’ over e-mail (or via his website), or to wait for Vol. II/Vol.III of his upcoming book on “Development & Deployment of Multiplayer Online Games”. In Vol. II, he discusses (Re)Actors as such and Client-Side (Re)Actor-fest Architectures, and in Vol. III - he speaks about the Server-Side.

Questions?