Skip to content

DeepDame: Server-Authoritative Checkers Engine

shipped

Live games stay in Redis and are written to MongoDB once, when they end. One k3s node held 6,000 concurrent sessions before saturating.

Real-time multiplayer checkers built by a team of five: I owned the game side of the backend and led the Flutter client.

Spring BootRedisMongoDBPostgreSQLFlutterRiverpodSTOMPKubernetesk6
View on GitHub

A Flutter mobile client talks to a Spring Boot backend over STOMP/WebSockets. The backend ran on a VM on my Proxmox homelab (the desktop in my room), exposed through a Cloudflare Tunnel.

The development story is a separate post. This document details the system architecture, state boundaries, and failure modes.

Subsystem Boundaries

I engineered the game state machine, the Dart engine port, the game service layer, the WebSocket security handshake (JwtWebSocketInterceptor), the AI orchestration pipeline with simulated thinking delays, matchmaking, in-game chat, and the k6 load testing suite. On the client, I built the Flutter core architecture: Riverpod state providers, the STOMP transport with automatic subscription rehydration, dynamic server URL switching, the audio engine, and the game screen with turn-timer heartbeat animations.

Teammates handled REST authentication endpoints, the social/friends graph, Redis pub/sub listener wiring, core Prometheus/Loki/Grafana k8s manifests, and the Java engine’s legal move generation rules (including forced jumps). I authored the Ollama Kubernetes deployment (Deployment, PVC, Service) and assisted in the k3s cluster migration.

Architecture

CLIENT Flutter app holds no authority: it predicts which moves are legal so a tap can respond without a round trip, and never decides what happened EDGE Cloudflare tunnel the only way in: the host publishes no inbound ports REST over HTTPS (cookie auth) STOMP over SockJS (JWT auth) APPLICATION Spring Boot, stateless (1 replica in prod) owns the game lifecycle and decides every move keeps no session state, so any replica can serve any request AI provider one bean, chosen at startup by ai.provider reached only on the two harder difficulties game-engine the rules as a pure function of state and move its own Maven module; couples only to spring-data-commons STORES Redis the live board while a game is played, plus the channels instances fan out over MongoDB games that have finished, and the player statistics derived from them PostgreSQL everything that outlives a game: accounts, friends, the general chat state and pub/sub one write per match (@Async stats) JPA over Hikari OBSERVABILITY Prometheus finds the backend through Kubernetes service discovery; one pod in production Loki receives logs the application pushes to it directly, with no agent in between Grafana reads both, so metrics and logs sit on one dashboard Metrics are pulled and logs are pushed, which is why nothing ships logs on the app's behalf. No Tempo and no Mimir: there is no distributed tracing, and Prometheus keeps its own storage. An AI failure is not uniformly survivable. A provider that fails is replaced by a random legal move, but anything thrown around that is swallowed and returns null, and no move is published at all. A game against the AI in that state waits forever.

Storage is partitioned across three engines based on access patterns:

  • PostgreSQL: Relational data (users, credentials, friendships, general chat).
  • Redis: Ephemeral state (live matches, active lobbies, chat lists, user session pointers).
  • MongoDB: Match history written strictly once at game completion.

The rules module (game-engine) manages board representations, piece mutations, promotions, and win conditions. However, the domain model directly couples to spring-data-commons: GameState and Board carry @PersistenceCreator annotations so Spring Data can hydrate documents straight from MongoDB. This trades domain isolation to eliminate mapping boilerplate.

Post-match statistics are isolated from the critical path: StatisticsServiceImpl.updateStatsAfterGame is annotated with @Async("taskExecutor"), preventing cold disk writes in MongoDB from delaying the real-time match termination broadcast.

Ephemeral State & Lazy Pruning

While a match is active, state transitions occur entirely in memory within Redis:

  • game:{id}: Serialized match document (board, history, player IDs) with a rolling 5-minute TTL refreshed on every legal move.
  • user:{id}:active_game: String pointer enforcing a single concurrent match per player, refreshed alongside game:{id}.
  • chat:game:{id}: Redis List of messages with a 24-hour TTL, deliberately outliving abandoned matches.
  • lobby:open_games: A Redis Set containing open match UUIDs. This key has no TTL.
lobby:open_games a SET of game ids no TTL: a set cannot expire members user:{id}:active_game one game id first written with no TTL at all game:{id} gone 5 minutes after the last write the TTL only moves when a move is saved, so a lobby nobody is playing in dies after five minutes WHAT THE PLAYER SAW Ghost lobbies Open games listed that no longer existed. A batch read came back with holes in it. Players locked out for good A pointer with no TTL outlived its game, and every entry point checks it before letting you in. REPAIRED ON THE READ PATH listing the lobby order the set into a list first, so the batch results line up with the ids behind them a hole in slot i means id i is dead drop every dead id, return what survived asking whether a user is playing give the pointer a TTL, refreshed on each move so its life tracks the game it points at follow it and confirm the game is really there if not, clear it and answer no And one read that was never repaired Matchmaking does not list the lobby. It draws a random member via SRANDMEMBER and fails immediately if stale, aborting the method on attempt 0 without removing the dead key. Because nothing removes it, the next attempt draws the same corpse. The repair covers the read that noticed the rot, not the read that trips over it.

Because lobby:open_games holds keys that expire independently under game:{id}, inactive lobbies produce dangling references:

  • Client Timer vs. Server Expiry: To mitigate this, the Flutter client runs a local lobby timer calibrated to slightly under 5 minutes, canceling the lobby and kicking the host back to the main menu before the backend key expires. However, mobile clients are ungraceful: locking the phone, backgrounding the OS, or dropping cellular signal halts the timer before the cancellation frame can send. The server’s 5-minute TTL lapses regardless, leaving an orphaned UUID in lobby:open_games.
  • Opportunistic Garbage Collection: Rather than running scheduled scans, fetchGamesBatch() implements self-healing reads. When fetching lobbies, it runs a multiGet() across all IDs; keys that return null (expired via their 5-minute TTL) are gathered and batch-evicted via opsForSet().remove().
  • Lockout Protection: When a player initiates a match, isUserPlaying() validates gameTemplate.hasKey("game:" + gameId). If the game expired, it inline-evicts the orphaned user pointer to prevent permanent account lockouts.

Matchmaking & Joining

Joiner Instance B Redis Instance A Host 1 subscribe to the reply queues, then send STOMP has no request-response, so a reply that beats its subscription is lost 2 ask to be matched 3 is this player already in a game? a pointer with no game behind it is repaired here rather than believed 4 draw a random open game id returned without checking that the game still exists If that id is stale, this is where a player gets stuck. Up to three draws, but the loop retries nothing: any missing game throws an immediate NotFoundException on attempt 0 and aborts. The stale id stays in the set because nothing removes it on this path, trapping subsequent matchmaking attempts on the same dead game. 5 seat the joiner, take the game out of the lobby the game, the joiner's pointer and the lobby set all change together 6 announce that the lobby lost a row 7 every instance relays it to the lobby topic so every client browsing the lobby drops that row 8 you are white, here is your opponent 9 your game filled up, you are black drops across replicas: in-memory broker lacks a Redis relay to reach Instance A 10 open the board, subscribe to the game topic that subscription re-registers itself on every reconnect Both players now sit on the same topic. Every board change from here reaches both of them the same way, including each player's own move.

Matchmaking prioritizes zero-coordination throughput over strict FIFO queueing:

  1. Random Draw & Unhandled Corrupted Keys: An incoming player runs SRANDMEMBER against lobby:open_games. If empty, they create a game and wait; if populated, they attempt to join. If the drawn ID has expired in Redis, the backend throws an unhandled exception immediately on attempt 0 and aborts the request. Because this crash path never calls srem, the dead ID remains in the set, causing subsequent quick-match attempts to draw the exact same dead key until someone visits the lobby browser and indirectly triggers the batch repair.
  2. Subscription Race: The client registers STOMP subscriptions before dispatching join frames. Because STOMP lacks request-response semantics, frames arriving before subscription confirmation are silently dropped.
  3. In-Memory Broker Boundary: The backend was designed to run as several replicas: broadcasts fan out across instances using discrete Redis channels (game-updates, game-over, game-chat, lobby-updates). Private user notifications (convertAndSendToUser), however, rely on Spring’s in-memory SimpleBroker, so with more than one replica a directed message would fail whenever the recipient is connected to another pod. In production the backend ran as one replica on one node, so this limit was never hit.

Move Validation & State Authority

Player A Instance A Redis Instance B Player B 1 tap a piece: the local engine highlights where it can go no network. Prediction only, and never allowed to move anything. 2 send the move the board does not change yet 3 the mover is taken from the session, not the message so a client cannot submit a move on behalf of its opponent 4 read the game nothing there means the key expired, and the move is refused as not found 5 refuse the move unless it is this player's turn an illegal move goes back to this player alone and nothing is published 6 write the game back, push every TTL out again reading then writing is safe only because one player can legally write 7 publish the move to channel "game-updates" 8 delivered to every subscriber, the publisher included One Redis channel (game-updates) carries moves for every game. Each instance unpacks the stream and relays moves only to the matching /topic/game/{id} destinations for its local sessions, ensuring A's move returns via the same fan-out. 9 relayed to /topic/game/{id} by each instance for its sessions 10 the local engine applies it and the board finally moves the only place either client ever changes the board If that move ended the game Move-triggered game over saves to Mongo, recomputes stats asynchronously, and publishes to "game-over" so both clients receive the frame. In contrast, disconnect auto-surrender bypasses the controller and clears Redis silently, leaving the opponent stranded on the board. Against AI, moves publish to "game-updates" asynchronously after an artificial think delay.

The rules exist twice, in Java on the server and in Dart on the client, with strict server authority:

  • Zero Optimistic UI: The client engine runs locally solely to calculate reachable squares and render tap targets. When a player taps, submitMove() emits the STOMP frame and clears selection state. The local board updates only when the server broadcasts the applied move back on /topic/game/{id}.
  • WebSocket Security Handshake: Because standard HTTP session security does not bridge to STOMP over SockJS, a custom JwtWebSocketInterceptor intercepts StompCommand.CONNECT frames, extracts and validates the Authorization: Bearer <jwt> header, and binds the principal to the STOMP session.
  • Concurrency Model: Move execution runs a read-modify-write cycle (getGame -> applyMove -> saveGame) without Redis WATCH or distributed locks. Concurrency safety relies entirely on sequential turn validation: only one player can legally write at any moment.

Disconnects & The Silent Surrender Bug

Disconnect events and client reconnection logic produce a critical edge-case conflict:

  1. Client Auto-Rehydration: If a network blip drops the socket, Flutter’s StompService reconnects and replays an internal registry of active subscriptions (_activeSubscriptions). However, it does not fetch the latest board state; any move broadcast during the drop window is permanently lost.
  2. The Disconnect Trap: If the socket closes completely, WebSocketPresenceEventListener auto-surrenders the match to prevent orphaned state.
  3. The Silent Game-Over: The disconnect listener calls gameService.surrenderGame() directly, bypassing GameSocketController.notifyGameOver(). As a result, the match is persisted to Mongo and cleared from Redis, but no game-over frame is published to the connected opponent, leaving them stranded on an active board waiting for a move that will never arrive.

AI Opponent & Thinking Engine

The AI integration runs asynchronously to avoid blocking server worker threads:

  • Dynamic Provider Strategy: The backend supports pluggable providers via an ai.provider configuration.
  • Engine Fallbacks: When calling external models (Medium/Hard), AiOrchestrator traps network timeouts and schema failures, falling back to a random legal engine move (getRandomMove). However, GameServiceImpl.triggerAiMove() wraps the entire dispatch in an empty catch (Exception e) {} block that returns null; an uncaught engine or serialization bug causes the game to hang silently.
  • Simulated Think Delay: Fast local models and deterministic moves executed in under 50ms, jarring user pacing. ensureMinimumDelay() enforces a configurable delay (ai.min-think-time-ms) to simulate real human consideration before dispatching moves.

Flutter Client Architecture

The mobile app is organized by feature modules under lib/features/:

  • Runtime Endpoint Configuration: Replaced static compile-time constants with BackendSetupScreen and Riverpod controllers, allowing the APK to target local LAN, Docker, or Cloudflare Tunnel environments at launch.
  • Turn Timer & Heartbeat Glow: Enforces turn deadlines client-side with a custom pulsating heartbeat animation when time drops below threshold, triggering automatic surrender on expiration.
  • Audio Layer: Centralized SoundService maps discrete game events (moves, captures, promotions, game over) to dedicated sound assets and haptic feedback profiles.

Load Testing & Benchmarks

Performance testing was executed against a single-node k3s cluster using k6.

Because k6 lacks native STOMP protocol support, I developed a custom test harness (k6/loadtest.js) that manually constructs raw byte-level frames: formatting headers, injecting JWTs, tracking subscription IDs, and appending explicit null byte terminators (\x00).

The scenario simulated complete player lifecycles (authentication, lobby polling, board subscriptions, move exchanges, and in-game chat). The single node reached saturation at 6,000 concurrent active STOMP sessions before encountering OS socket and thread pool constraints.

← Back