Albion Online: The Technology of Albion
To work on a successful game is the dream of many Indies and beginners in the games industry. On the next four pages David Salz from Sandbox Interactive shares his dream with us.
The opportunity to make Albion Online was a dream coming true. At the same time, it was frightening. Even though I had many years of game development experience, some of it as a partner in my own company, I had never done an MMO before. In fact, I did not personally know anyone who had. What followed was a great learning experience (and I learned many things the hard way).
We started Albion Online with just two coders. Knowing that we knew nothing, we went through every book and article about online and multiplayer game development we could find. There is shockingly little, especially if you compare it to other fields like graphics programming. We also reached out to the few people in Germany who had MMO experience. Their very helpful input took the worst of our fears away, and we simply got started.
We use the Unity3D game engine to make Albion Online. Unity was up-and-coming when we started; it is powerful, flexible and inexpensive. Unity’s biggest strength, however, is the ability to publish on a huge number of platforms with very little additional effort. Even though we originally planned a PC-only game, we quickly realized that tablets and other mobile devices would be a crucial market in the future. Hence, Albion Online became a true cross-platform project. We started with Unity 3.5 almost 7 years ago, but we have made it a habit to always upgrade to the latest version of Unity as quickly as possible. These days upgrading the engine is a huge testing effort, and often new Unity bugs prevent us from using a particular new release. Finding a version that is stable enough for us has been getting more and more tricky the larger our game has become. On the other hand, there is no alternative since new OS versions (mostly OSX and mobile) often require new Unity versions.
As a network middleware, we use Photon. Photon comes with a server framework; the core is written in C++ for performance reasons, but the application code is C# – which is good because it means we can write the entire project in a single language and share code between server and client. C# is, in fact, an excellent choice for server development – very easy to use, fast, efficient and very stable thanks to garbage collection and exception handling.
Most of the game uses Photon’s reliable UDP protocol, and movement messages are sent as unreliable UDP messages. For the chat server and for inter-server-communication we use TCP. We use Photon on the lowest possible level, i.e.,, only the server framework and the network protocols. There is also a cloud-based hosting solution, a chat system, and even an MMO framework, but we decided to use none of that. Instead, we wrote as much as possible ourselves, and we decided to host the game on dedicated bare-metal servers.
Finally, we needed databases. For the game, the biggest concern was the ability to scale the game database for the potentially “infinite” game world size. We decided to give Apache Cassandra a try. Cassandra is a NoSQL database, i.e.,, it sacrifices the ability to do complex queries in favor of speed, high throughput, and the possibility to scale horizontally, i.e.,, by simply adding more servers.
Cassandra is essentially a distributed hash table. You can retrieve data only based on its hash key. There is no ‘SELECT … WHERE’ mechanism like in SQL. You can, however, build compound keys out of multiple columns and retrieve data based on partial keys. That way Cassandra forces you to design your tables around the queries you are going to make. This takes some time to learn, but the upside is that inefficient queries are almost impossible in Cassandra.
You can build a Cassandra cluster out of as many server instances (‘nodes’) as necessary. They will automatically split the hash space with some redundancy, so a couple of nodes may fail and the cluster will still stay operational. A caveat is that Cassandra is only ‘eventually consistent.’ Due to the distributed nature of the database cluster, it is possible to read ‘outdated’ data from the cluster, even though the ‘latest’ data will eventually be synchronized to all nodes. This can be fixed by either writing to or reading from a quorum of the database nodes, i.e.,, if you need consistent information, you can choose between slower reads or slower writes. Albion Online keeps most data in the game server’s memory while a player is logged in, so we choose slow reads but very fast writes. Player data is only read when the player logs in or changes the game server, but it is written every time something changes – just in case the game server crashes.
The biggest downside of NoSQL databases is the inability to do complex queries, i.e., JOIN and WHERE commands. For some purposes, e.g., the in-game marketplaces, queries are important, so we use a couple of Postgres SQL databases as well.
Doing a Unity game… without Unity
It was clear to us from the beginning that we did not want to run Unity on the game servers. Unity is a closed-source environment we did not believe it was flexible, scalable and stable enough to support a game server for thousands of concurrent players. Also, we were simply not comfortable with having an uncontrollable ‘black box’ in such a key spot.
As a consequence of that, the core game systems would have to work without Unity, and we ended up coding our own technology even in areas where Unity offers good solutions. We have, for example, our own level file format, our own collision system, our own pathfinding, and our own game object management.
While that sounds like re-inventing the wheel, I would rather see it as cherry-picking. While it is somewhat crazy to write your own engine just because you can, it is equally crazy to use an engine feature just because it exists. You should rather ask yourself ‘Is this exactly the solution I need?’ and ‘How long would it take to write something myself that more precisely matches my needs?’ Owning and controlling key parts of the code has many advantages. Not only does our game server work without Unity, but we, in fact, also have a fully functional command-line version of the game client that runs without Unity! It has no graphics and no UI but is otherwise fully capable of logging in and playing the game. You can easily launch 50 to 100 of those on a single machine, and we often use it as a scriptable bot for stress tests.
Model – View – Controller
A key design principle of Albion Online is a strict Model-View-Controller pattern. In an online game, the server must have absolute authority, meaning that all game world manipulations happen on the server and only the server can decide whether a player action is valid or not. We keep game data in memory and use databases as permanent storage; hence the server acts as the Model and the Controller here.
The Albion Online client is just a library (i.e., a DLL) that is able to communicate with the server. It will mirror the part of the data model that the player can “see” at a given point in time and offer a controller interface similar to that of the server. The client does validity checks and sometimes even prediction, but will ultimately send all valid requests to the server. The client library knows nothing about Unity. This allows us to have the command-line client mentioned earlier. The client code is maintained as a separate Visual Studio project that has no Unity dependencies, so it is easy to keep the separation in everyday work.
Unity comes into play as a pure View component. It will subscribe to various events the client library offers, create a view for every game object and update the view whenever the game object changes. All of the UI and input code is in Unity as well, but the real business logic is in the client library and not in Unity.
Maps and Collision
Unity is very good for map and level editing and extensible, too, so we had every intention of using it that way. On the other hand, the game server would need to be able to load the map files as well. Certainly, the server would not need the graphical assets, but it would need to know to location and properties of all interactive objects, the shape of the terrain, object collisions, etc. Collision was the next issue. Unity comes with a sophisticated physics system, so you could leave player-vs-environment collision completely to the physics module, or at least use ray casts against the physics geometry to determine where a player can go and where not. Today Unity even has built-in navigation meshes for pathfinding, that did not exist in 3.5 when we started. While that approach works great for an action-adventure, it seemed too complicated for the kind of MMORPG that we had in mind. We would need walkable and non-walkable terrain, line-of-sight checks for projectiles and pathfinding for monsters.
All of that should be super fast because it would have to executed or at least double-checked on the server for thousands of players and monsters. Physics-based collision does not give you that kind of performance. Another problem with physics engines is that they sometimes ‘tick out.’ We have all occasionally fallen through the ground or have been catapulted in the air in our favorite games just because we walked into an odd corner at an odd angle. For a level designer, such problems are very annoying to fix because it is often not obvious why the physics engine has an issue with a particular piece of geometry.
Considering all that it made sense to write our own collision and pathfinding system. In a drastic first step, we decided to regard the whole game as essentially 2D. The top-down perspective already forces the level design into a 2D mindset, and while the maps would not be totally flat, they would be flat enough to get away with this simplification.
The world of Albion consists of ‘tiles.’ A tile can be a piece of ground or an object. Ground tiles are typically 10 by 10 meters (in respect to the size of the characters). Objects are everything that sits on top of the ground, from decoration to interactive objects like trees or buildings. All tiles have a hand-painted 2D collision in a 1×1 meter grid. The collision system allows a small number of geometrical primitives like triangles, circles and infinitely thin lines.
Maps are built in Unity but exported into a proprietary file format. Every tile is a Unity prefab plus some meta information like size, collision, etc. A map is essentially just a huge list of tiles, often up to 30,000. Server and client load the same file. While the server only cares about the meta information, the client actually instantiates all the Unity prefabs. To our surprise, this form of loading is actually faster and gives us more control than using Unity scenes! We still use Unity scenes to store the maps for the level editor, but not in the game.
The Server Farm
The server farm consists of several different server types and databases. We host everything on bare metal machines since we need strong CPU time and memory guarantees. Albion is a single, unique game world for all players. We use instancing only for player islands and certain types of dungeons, so we would not benefit much from the ability to scale to dynamically allocated virtual machines. Instead, we simply distribute the game world across a fixed number of physical machines.
Most of the heavy lifting is done by the Game Servers. Depending on your character’s location in the game world your client will connect to a different Game Server. As you move through the world, you may have to connect to a different game server.
Your client will also maintain a connection to the Chat and Login Server. The Login Server handles all account operations like login, logout, character creation, etc. and will tell the client which Game Server to connect to.
The World server is the single authority for all game features that work across Game Servers, for instance, parties, guilds, guild vs. guild battles, etc. The Backoffice server controls farm startup, shutdown and provides an API to “talk” to the entire farm.
The Game Server
Our game world consists of zones (“clusters”) that are roughly 1 x 1 km in size. There are currently around 600 of these. When you change zones, your client may have to connect to a different game server, and you will see a short loading screen on the client. The game logic of a cluster is handled by a single thread. Making this multi-threaded would be very error-prone and not necessarily more efficient since there are numerous complicated interactions between game objects which would require locking. Instead, all incoming player commands are put into a synchronized event queue and processed in order of arrival. Any work that can be offloaded – like database operations, pathfinding or logging – are handed to separate thread pools that complete their respective tasks asynchronously and put the result back into the event queue.
A second mechanism we use extensively is scheduled events (or simply put: timers). There is no real game loop in Albion Online, meaning that game objects have no Update() functions that are called every so often. Instead, objects will create timers whenever they need to. Monsters will create a recurring AI update timer as soon as a player gets near but delete the timer again when the player is gone.
Interest Management is the ability to efficiently determine which game object can “see” which other objects at a given point in time. This is important for multiple reasons.
First of all, we want to minimize the network traffic for each client by sending the client only updates of objects it can actually see right now. Second, players do cheat. Any information that is sent to the client can and will be extracted by cheaters. That means we absolutely need to restrict client knowledge to objects that are on the screen or at least very close to it. The whole mechanism is also useful for many other purposes; for instance, monsters are activated when a player comes near and deactivated when the player leaves.
All this needs to be as efficient as possible. Let’s illustrate this with some numbers: A single (1 x 1 km) cluster may contain more than 300 players, 500 monsters and more than 10.000 other interactive objects (most of which are trees, because you can cut down every single tree in Albion Online).
We achieve the efficiency by putting all objects into a grid-based hash. Each grid cell is roughly 10 x 10 meters and contains a list of objects inside. Very large objects may be in multiple cells at once, but that is rare. The object itself is responsible for updating the hash if it moves. The grid cells fire events when objects are added or removed, so subscribing to these events already gives you a very efficient way of monitoring an area.
Each player is surrounded by something we call an Interest Area. It consists of two grid-aligned rectangles. When an object enters the inner, smaller rectangle, the player will start “seeing” the object, i.e., the server will send the client an initial “new object” message (“look, a monster!”) and subscribe to the various event the object itself offers. Whenever the object does something interesting (e.g. the monster attacks someone) it will fire an event, that event will be forwarded to the Interest Area of the nearby player, and the Interest Area will forward the event to the client.
An object will be watched until it leaves the slightly larger blue area. When that happens, the client will receive an “object gone” message and stop receiving updates. The blue area is larger in order to keep the visible object set more stable over time.
I hope I was able to shed light on some of the more interesting technical aspects of Albion Online. It has been quite a journey, and we keep learning new things all the time. If you have questions, feel free to contact me!
Example of an object life cycle
Here is a short example of an object life cycle. The server knows all objects (players,mobs,trees etc.) and their properties. The Interest Management system checks which client can see which objects. Whenever an object becomes visible (or invisible) to a client, an enter (or leave) message is sent to the client which will create a local copy of the object (or destroy it again). The server will also send update messages for all visible objects should their properties change.
Unity sits on top of the client as a pure view component. For every object created by the client, it will create an object view in the form of a Unity GameObject. The view will subscribe to any changes to the original object and will update graphics, sound and animation accordingly. The view takes input, i.e., interprets Mouse Clicks etc. into game commands. Commands are forwarded to the client game object, which checks the command for plausibility, and ultimately forwarded to the server which executes the command. Should game objects change as a result of the action, the server will send update messages as described above.
CTO, Sandbox Interactive
David Salz has been working as a software engineer in the games industry for 18 years. He is CTO and one of the founders of Sandbox Interactive. Before he entered the world of MMORPG gaming, David co-founded another company; his previous studio Bitfield produced several games for PC and Nintendo DS.
Albion Online: The Technology of Albion