Path to a Cluster
What keeps zone-server on one process today, which seams already help, the two ways to grow (realm shards, spatial cells) and the order of work. Design only; none of it is built.
One process owns all players, all rows and all global counters. There is little cluster infrastructure yet: the zone runs a Valkey for the API Server and the login server’s kicks, with a directory of which node holds an account, but there is no etcd and no link between two zones. Some seams already help, and a few pieces of state stand in the way.
A target shape
graph TB
C["Game clients"] -->|TLS| GW["<b>Gateway</b><br/>auth · rate limits · idle checks<br/>per-client batches · session epoch"]
subgraph SIM["Simulation nodes"]
direction LR
A["Sim node A<br/>owns chunk cells 0…k"] <-->|"ghosts,<br/>handoff"| B["Sim node B<br/>owns chunk cells k…n"]
end
GW <-->|"commands,<br/>encoded batches"| SIM
GW -->|route lookup| DIR["Directory<br/>account → node · region → node<br/>node ids · leases"]
SIM -.->|lease| DIR
SIM <-->|events| BUS["Message bus<br/>party · chat · trade escrow · economy"]
SIM -.->|write-behind| DB[("Database<br/>owner column · @Version")]
W["Companion app"] -->|"REST · WebSocket"| API["<b>API server</b><br/>REST · live push · map tiles"]
API -->|route lookup| DIR
API <-->|"requests,<br/>pushes"| BUS
This is a target shape, not a spec. The boxes are roles, not processes: the first version runs the gateway role inside every zone node. Clients only talk to the gateway, so an entity can move between nodes without a reconnect. Each node is the single writer for its chunk cells. Neighbours see border entities as read-only ghosts, and they replicate and hand over directly between nodes, not through a broker. Party, chat, trade and economy leave the World and talk over a bus. Spatial Cells explains the nodes, the gateway role and the handoff. The companion app does not use the gateway. A separate API Server serves its REST calls and live pushes, as a second client of the directory and the bus.
What blocks it today
Each row is state that assumes one process. The last column says which way to grow needs it fixed. A realm is one node with its own database, so state that stays inside one node is fine there.
| State | Problem | Change | Paths |
|---|---|---|---|
WorldWorld | The tick thread is the only writer, but a World belongs to one process. Nothing copies or moves a whole entity to another process. There is no ghost, no region and no silent removal. | One single-writer World per region, with a handoff | Cells |
SessionsConnectionInfoService | A per-process map decides which entity an account controls. It has no epoch. A stale teardown is stopped only by channel identity in ChannelRegistry. | Directory with lease and session epoch | Cells |
Channels, ticketsChannelRegistry, HttpTicketService | “One connection per account” holds only inside one node, and a ticket is known only to the node that minted it. The login server already reaches every node: its kicks go to a stream each node reads in a group of its own, because it cannot see where the player is. | Gateway role; the directory answers “which node” | Cells |
Party, trade, whisperPartyService, TradeService, ChatHandler | State sits in per-process maps. Each one reads and writes both players in the local World. | Bus service; trade as two-phase escrow | Cells |
World coin reserveWorldReserve | The balance lives in memory and is saved as one absolute value. With two nodes, the last writer wins. | Atomic SQL deltas, or one owner | Cells |
Rumour idsRumourRegistry | The next id is max(id) + 1 at boot. Two nodes collide. | DB sequence or snowflake | Cells |
Terrain editsChunkService | Held in a MemoryBlobStore: node-local and lost on restart. See Storage. | Shared durable delta store with revisions | Both |
Boot loading and cleanupEntityLoaderBootRunner, TradeReservationCleanupBootRunner | Every node loads all rows: the mob, loot and script persisters read their whole kind at boot, with no owner or region filter. The cleanup frees the trade reservations of the whole table. | Owner and region columns; one coordinator | Cells |
Entity idsEcsConfiguration | The node part is zone.shard-id, a static 1 in the config. It is clamped to 0–255 without an error, and nothing checks that nodes differ. Two nodes from one config mint the same ids. | Lease node ids; fail fast at boot | Cells |
Login tokenLoginTokenValidator | The audience is just zone, so any zone accepts it. Single-use ids are kept per process, so each node accepts the same token once. The key is a shared HMAC secret, so a zone could mint tokens. | Zone claim; one place for the single-use check; asymmetric keys | Both |
Global sweepsSettlementEconomySystem and six more | Seven systems work on the whole world or a whole table, wherever the players are: SettlementEconomySystem (catchUpAll every 30 s), RumourExpirySystem, IndoorEmergenceSystem, MacroGraphMaintenanceSystem, GroundLevelDecaySystem, ScorchRegrowthSystem and GroundFireSystem. Each node would run every sweep, over state that is not its own. | An owner for each sweep, best sharded by settlement region | Cells |
World-wide factsSharedMemoryService, MacroGraphService | The AI keeps one world blackboard and one per faction. The macro graph has one version counter. The spawner and townsfolk residency maps are per process too. With several nodes a world-wide fact stops being world-wide, and a pack that a border splits gets two boards. | An owner for each fact; boards scoped to a region | Cells |
Seams that already help
- Entity ids are 64-bit snowflakes (
timestamp | node | sequence). The node part is 8 bits, so 256 nodes can mint ids with no coordination. A master gets its id when it is created and keeps it across sessions. - Interest is already “who holds this chunk”.
ChunkSubscriptionServicekeeps chunk → accounts, andChunkEntityVisibilitykeeps chunk → resident entities and answersobserversOf. A chunk cell is the natural unit of ownership. - Audiences are declared, not computed. Each syncable component returns
SyncTargets:PublicInRange,OwnerOnlyorAccounts(ids). Two of the three name their audience by account id, so they stay correct across nodes as soon as message routing exists. - Output has one seam. All game output goes through
OutMessageProcessortoOutMessageHandler.ChannelRegistryis its one production implementation, and onlysocket/knows Netty channels. - Terrain needs no sharing. A chunk is a base plus player edits. The base is a pure function of the world seed, so every node can generate any chunk. Only the edits have to travel.
- Persister snapshots are plain data, copied out with the world to itself. They are a start for moving an entity between nodes, but they only carry mutable state such as position and HP. A handoff needs more.
- One writer per World. The tick thread owns the World, and handlers reach it through the inbox. See Threads and the Tick.
- Login and zone share no database, only a token. The login server already reaches every zone node, through
the
kicksstream in Valkey.
Two ways to grow
Realm shards. Run many separate copies of the world, one node each, with one database per realm. Players connect straight to their realm, as they do today, so this path needs no gateway process. What is missing:
- The login server must know the realm list and hand out the right socket endpoint. It knows no zone at all today: its kicks go to every node through the stream.
- The login token must name its realm.
- Terrain edits must survive a restart.
The last two points are the rows marked “Both” above. This is the cheaper path.
Spatial cells. Run one large world across many nodes. You need every row above, plus ghost entities, a handoff and cross-node global services. The design is on Spatial Cells.
The single writer per World is done. Measure before you choose: a load test with simulated clients shows how many players one node holds. If one node carries the players you expect, realm shards are enough.
Order of work
Each step can ship on its own, and each one also helps a single node.
Already in the code
- One writer per World: the tick thread, the inbox and leases.
- Inbound messages run off the Netty event loops (
AccountInbox). - One batch per account per tick (
TickOutbox). - Audiences follow chunk subscriptions (
EntityAudience,sendToObserversOf). - Player bestias are saved when their owner leaves (
PlayerBestiaEntityPersister). - Login tokens are single use, per process.
Both paths
- Put a zone claim in the login token, check single use in one place, and sign with an asymmetric key.
- Persist terrain edits in a durable store.
- Run a load test with simulated clients, and choose.
Realm shards
- Give the login server a realm list with the socket endpoint of each realm.
- Run one database per realm.
Spatial cells
- Make the gameplay layer toroidal (
Vec3L.distance,AreaOfInterestService,SpawnerCellIndex). Do this first, on its own, with tests. It needs no cluster technology. - Move the last radius sender,
BestiaTrapSystem, tosendToObserversOf, and delete the radius path. Then encode entity updates once per audience, as chunk data already is. - Make healing an intent, like damage.
Heal, restorative items and regeneration still writeHealthdirectly, and a remote caster must be able to heal through a message. - Add
World.evictbesidedestroy, and a negative term onQuery. - Give
ChunkStreamSysteman explicit anchor source. It finds the anchor with a local query today. - Write the full-entity codec for the handoff, with a boot check that covers every component type.
- Check
zone.shard-idat boot. Then lease node ids. - Give each global sweep and each world-wide fact an owner. Move party, chat, trade and economy onto a bus.
- Build the gateway role, the direct node-to-node link and the directory. Then ghosts, then the handoff.
Last updated 09 Nov 2026, 23:57 +0100 .