I have always loved simple designs, being able to look at something and see immediately how it works without needing to ask the person who built it. I believe that is one of the cornerstones of good architecture whether it is buildings, machinery or software.
When working in a company, like bet365, where normal throughput exceeds 1000s of transactions per second, and there must be consideration for every millisecond (or microsecond) used, multi-threaded code is a necessity.
The challenge, from an architecture point of view, is to establish a standard approach in C# to define and organise highly concurrent code in a way that has a shallow learning curve.
One thing I do not like about multi-threaded C# code is that there are infinite ways of defining and structuring it, such that one application or service can be completely different from a similar one, just because two different people wrote them.
So, when I was designing a new highly concurrent message transmission service I took the opportunity to examine how we can simplify this part of our architecture with Akka.NET (see http://getakka.net/).
The Actor Model
The concepts of Akka.NET go back to the actor model, which Carl Hewitt first published in 1973 (see https://en.wikipedia.org/wiki/Actor_model). This influenced a huge number of software concepts and languages, eventually resulting in Scala for Java, leading to Akka and its .NET port.
The actor model adopts the philosophy that “everything is an actor”. This is similar to the “everything is an object” philosophy used by some object-oriented programming languages. An actor can respond to messages and send messages to other actors. Each actor’s actions are small and isolated so the overall system is easier to follow, less prone to bugs, and (this is what attracted me to this model) ideally suited for asynchronous concurrency – a messaging service.
Messages
Akka.NET requires that you create specific classes to pass actions from one actor to another. To send a Message we will use the following class.
public class Message
{
public long MessageId { get; private set; }
public string Destination { get; private set; }
public string Message { get; private set; }
public DateTime SendTime { get; private set; }
public DateTime ExpiryTime { get; private set; }
public Message(long messageId, string destination,
string message, DateTime sendTime, DateTime expiryTime)
{
MessageId = messageId;
Destination = destination;
Message = message;
SendTime = sendTime;
ExpiryTime = expiryTime;
}
}
As you can see, you can only set the public properties with the internal code, and only on instantiation. Once created, you cannot alter these properties from the outside; it means that this object will be thread-safe no matter where its instance occurs in the application.
Actors and Actions
Actors are just classes in C# and, just like the Message class above, they have a similar pattern. This is the Actor, which will send an individual message.
class MessageSender : ActorBase {
public MessageSender() {
Receive<Message> (message => HandleMessage(message));
}
private void HandleMessage(Message message) {
//...do the sending here
}
}
From a developer’s point of view you just need to inherit from ActorBase and Akka.NET provides all that you need in your class. You just need to define which “Actions” this class will respond to, in this case the arrival of a Message instance. Again, there does not appear to be any public communication allowed with this MessageSender once created but this is where Akka.NET really kicks in.
Building the service
With the basics in place, starting an action is just a matter of instantiating the backbone of the Akka framework.
SystemRoot = ActorSystem.Create("MessageService", config);
MessageSenderActorPool = SystemRoot.ActorOf(Props.Create(() => new Actors.MessageSender())
.WithRouter(FromConfig.Instance), "MessageSender");
Here, the SystemRoot is the core Actor System service and the governor for all the Actors created within it. The SystemRoot object will manage all Actors that need instantiating, handling or destroying.
The MessageSenderActorPool manages the queue of messages for the MessageSender.
Controlling the flow
Akka works in a tree structure with the SystemRoot at the top and the individual Actor instances at the bottom. In order to keep track of the active Actors, it is best practice to create a Supervisor, a parent actor that will be the main entry point and messages then distributed from there.
Supervisor = SystemRoot.ActorOf(Props.Create(() => new Actors.SenderSupervisor()),
"SenderSupervisor");
So, if I want to send a message, all I need to do is.
Message sendMessage = new Message(<parameters>);
Supervisor.Tell(sendMessage);
Then the Supervisor passes on the message to its pool of Actors.
MessageSenderActorPool.Tell(message);
If the software needs to shutdown we just send a terminate message to the SystemRoot which in turns tells the tree to complete and finish.
SystemRoot.Terminate();
Likewise, if an actor needs to communicate with its parent, e.g. from the Message Actor to the Supervisor, it just needs to use a class for the content of the conversation and pass it to Sender, a pointer to the parent actor.
Feedback feedback = new Feedback(<parameters>);
Sender.Tell(feedback);
Hold on, what about multi-threading?
It is a good question; I have spent all this time talking about Akka and not one mention of threading code. And that’s the answer. With Akka managing the actors and messages, the developer does not need to write a line of code for threading – the Akka base classes supply all this, through its queues and configuration.
The configuration is flexible but can be as simple as how many threads you want to process messages through the Pool. You add further complexity by making the Pool represent a remote server or a cluster of servers but, as far as the calling code is concerned, it is still a single line of code to tell the Pool to process the message.
The flexibility of having implied two-way communication with threads is an extremely powerful concept and frees up the developer to think about what needs to converse within the application, and not how to make it happen.
The end result
This is a summary of the design for the Messaging service we eventually built.
Within the design are a number of queue pools (denoted with 1..n) to let us tune the performance and smooth out bottlenecks to external suppliers and our databases without needing to change code. The yellow boxes represent JSON RESTful Services.
As the design evolved, we tested to determine how the Akka framework performs under load. The results were extremely encouraging showing that the code running on a VM constrained dual-core CPU could handle 4,000 messages per second, and would be only limited by the number of CPUs. I.e. if we wanted more throughput we could just add more cores. In the Akka world, we could configure the Supervisor to be part of a Cluster (see https://getakka.net/articles/clustering/cluster-overview.html) and the architecture would flex across multiple machines.
However, the best result in all of this was an implementation that:
- Needed no threading code to be written
- Was simple to follow and understand
- Could be performance tuned through configuration
- Only required a total of 600 lines of C#
Yes, the production system only has 600 lines of C# code.
I recommend you try Akka.NET out.