Multi-server Swarm

A Redelay project scales out by running its app tier on several Swarm nodes — often in different regions — next to replicated data services. That only pays off if every hop on a request's path stays local. This guide is the checklist.

The one number to keep in mind: a request makes several sequential data calls, so cross-region round trips multiply. A replica 147 ms away from the database primary turned a 59 ms p50 list call into 740 ms — five reads, each crossing the ocean. Measure the RTT between your nodes before placing anything.

1. What is local, what is not

{{.Node.Hostname}} is a Swarm template: the task's env var is rendered with the node it lands on, so one stack file gives every replica its local member. Name the per-node services to match (clickhouse-<hostname>, redis-<hostname>).

2. MongoDB: primary by default, Nearest per read

The client default stays MONGO_READ_PREFERENCE=primary — never flip it globally. Replication is asynchronous: a blanket secondary preference breaks read-your-writes for every endpoint at once (the user saves something, the next request lands on a lagging member, and it seems to vanish).

Opt in per read, where staleness is acceptable:

import rtmongo "github.com/redelay/go-framework/runtime/mongo"

ctx = rtmongo.Nearest(ctx) // catalogue, browse, public lists — not what the user just wrote
items, err := svc.store.List(ctx, params)

• crud.MongoCRUD honours the context preference on every read (FindByID, FindByField, FindOne, Exists, List, ListAll, Count) from go-framework v0.16.6; before that only List/ListAll/Count did.
• A raw *mongo.Collection ignores the context. Wrap it with rtmongo.Coll(ctx, coll) (go-framework v0.17.0), which applies the context's preference through a cached clone: rtmongo.Coll(rtmongo.Nearest(ctx), coll).Find(ctx, filter). Most hot paths use raw collections, and before v0.17.0 Nearest(ctx) silently did nothing for them.
• rtmongo.Secondary(ctx) (SecondaryPreferred) is for reports and exports.
• The framework's auth middleware is JWT-only — no database read per request.

Classify every read on a hot path:

Read-your-writes on a nearby replica

1. Recent-write window (simple, built in since go-framework v0.17.0). A write by user U marks U in shared Redis for RecentWindow (10 s); U's own reads go to the primary while the mark exists, and to the nearest member otherwise.
   goCopy

   import (
       rtmongo "github.com/redelay/go-framework/runtime/mongo"
       "github.com/redelay/go-framework/runtime/mongo/recentwrite"
   )
   
   // at startup: the marker on the shared Redis Cluster; this node's master host
   rtmongo.SetMarker(recentwrite.New(clusterClient, "myapp:shared:", "redis-fp-"+nodeHostname))
   
   // after authentication (or rtmongo.RecentWriteMiddleware(userIDFromRequest)):
   ctx = rtmongo.Request(ctx, userID, r.Method) // a write marks U before it runs
   
   // a read of U's own data:
   cur, err := rtmongo.Coll(rtmongo.ForUser(ctx, userID), favourites).Find(ctx, filter)
   

   • Mark before the write runs, not after: a read racing the write, and the write's own read-back, must go to the primary.
   • Ask once per request, and only if a user-owned read happens. Request memoises it; a hot path that checks per read pays one Redis round trip per read.
   • Keep the check local. A Redis Cluster without replicas keeps a key on one master, maybe in another region, and a 147 ms check cancels the Nearest read it decides. recentwrite writes the mark to every master (hash tags, in parallel: one round trip, writes only). The check reads the node's own master (REDIS_SHARED_LOCAL_NODE=redis-fp-{{.Node.Hostname}}).
   • Fail to the primary: Redis down, or no marker set, means user-owned reads stay on the primary. That is slower, never stale.
   • Login / refresh / signup carry no user before they run: they read the primary explicitly, not through ForUser.
2. Causal consistency (exact). Writes use w: majority; the response carries the operation time, which the client (or a per-user Redis key) keeps. The next request starts a causally consistent session, AdvanceOperationTime / AdvanceClusterTime to that point, and reads Nearest with readConcern: majority — the nearby member waits until it has replicated the user's write, so it can never serve something older.

3. Request routing: edge → Traefik → task

With proxied DNS (e.g. Cloudflare) holding one A record per node, visitors are spread across nodes without regard to location. Traefik runs on every node (global, host ports) and load-balances each service across all of its tasks Swarm-wide. So a request can enter in one region and be served in another, and there is no "prefer the local task" setting in Traefik's Swarm provider.

Pick one model per project:

Model B needs both halves: regional routing and Nearest reads. Regional tasks that still read the remote primary are just as slow; Nearest reads behind round-robin DNS still cross regions at the edge.

Client-chosen region (alternative to an edge load balancer)

Free, and it picks by each user's measured latency rather than geography:

1. One hostname per region, each reaching only that region's tasks: a proxied A record per region (eu.example.com → EU nodes, na.example.com → NA node) and one Swarm service per region (web-eu, web-na, api-eu, api-na, constrained by node.labels.location), each behind a router on its regional host. With a single service, Traefik would still balance across every region's tasks.
2. Regional hosts serve only the API path (Host(na.example.com) && PathPrefix(/api/)), with X-Robots-Tag: noindex. HTML and SEO stay on the canonical host.
3. The client chooses: after hydration (or at native app start) it calls a tiny uncached GET /api/.../region on every regional host in parallel, keeps the fastest (stored with a TTL, re-measured in the background), sends API calls there, and on a network error or 5xx switches to the next region (retrying idempotent requests only).
4. Call the regional proxy, not the Go API, when the web app talks through a Nitro proxy or BFF that adds headers, locale or credentials. The proxy in each region calls its own region's API service over the overlay.
5. Cross-origin plumbing: CORS on the regional hosts for the canonical origin (with credentials if cookies are used); session cookies scoped to the parent domain (Domain=.example.com) or bearer tokens; per-IP limits and sessions in shared Redis, so they don't split per region.
6. The HTML still comes from the canonical host, so keep its origins in the primary's region (§3 model A). Server-side rendering there stays local; only the browser's API calls go regional.

It only pays off when the regional API reads Nearest with a read-your-writes pattern (§2). The client sticking to one region also gives session-affinity consistency.

Lessons from a BFF rollout (Nameniac, 2026-10-15)

• Moving session cookies to Domain=.example.com leaves old host-only copies behind.
  • A browser holding both sends both. With equal paths the older one comes first, and most cookie parsers keep the first value, so a stale refresh token wins.
  • Every write from the canonical host must delete the host-only copy too. Only that host can.
  • Migrate once with a marker cookie.
  • Repair browsers that already have duplicates: when a name appears twice, keep the last value (the newest) and delete the host-only copy.
• Delete before you set. A response that sets the domain cookie and then deletes the host-only one is fine for browsers, which key by name + domain. A client keyed by name alone (a synthetic probe, a script) applies the delete last and loses the session.
• Server-render calls must forward the page's host. An in-process SSR call that refreshes the session computes the cookie Domain from its own Host (localhost) and sets host-only cookies again. Forward X-Forwarded-Host.
• CSRF: Sec-Fetch-Site: same-site is not enough. Every *.example.com is same-site (old., admin., api.). Accept same-site only with an Origin that is exactly the canonical origin(s).
• CORS preflights must be answered before your guards. A CSRF or members-only check refuses a bodyless OPTIONS. Answer exact origins only, with credentials and Vary: Origin.
• A new route on a catch-all redirect host (e.g. "unknown paths → the old site") gets redirected away until it is on the own-path list. Test the regional routes through the real chain.
• Pre-production gates: a header-routed candidate can't see browser preflights (an OPTIONS carries no custom header). Cover the regional hosts at the HTTP level (explicit Origin, Sec-Fetch-Site and cookies), plus a router for the candidate on each regional host.
• Deploy checks must not assume placement. "Exec into a task on this node" fails the day the scheduler moves it. Attachable overlays are rare; ask from a service pinned to the runner's node.

4. Sticky sessions — what they do and don't fix

Traefik sticky cookies pin a browser to one task:

- traefik.http.services.<svc>.loadbalancer.sticky.cookie=true
- traefik.http.services.<svc>.loadbalancer.sticky.cookie.name=<svc>_srv
- traefik.http.services.<svc>.loadbalancer.sticky.cookie.secure=true
- traefik.http.services.<svc>.loadbalancer.sticky.cookie.httponly=true

Rule: make app tasks stateless (shared state in Redis/Mongo), choose consistency per read (§2), choose region at the edge (§3). Add a sticky cookie only for genuinely per-task state.

5. Placement

• Constraints are hard; preferences only spread. placement.preferences: [spread: node.labels.X] balances tasks evenly across label values — there is no soft "prefer this node, fall back to that one". Two replicas over two eligible nodes always land 1 + 1.
• Label nodes by role and location (location=<region>, api=true, clickhouse=true …) and constrain on labels, not hostnames.
• A service pinned to one region has no automatic fallback. Either run it in two regions (and accept their latency), or keep the failover step written down (relabel the node, redeploy).
• max_replicas_per_node keeps replicas apart within a region.
• Rolling updates: update_config.order: start-first plus SHUTDOWN_DRAIN_SECONDS (go-framework ≥ v0.15.4) so the old task keeps serving until the proxy has noticed the new one.

6. Events across nodes

• Work queues: one consumer group per handler; Kafka spreads partitions across replicas. Publishing from a request is a remote call — do it asynchronously or after the response where possible.
• Broadcasts (cache invalidation, template reload): every replica must get every message. go-events v0.4.5 (v0.3.1 on the 0.3 line; avoid v0.4.1–v0.4.4, whose topic checks stall startup for seconds per topic) consumes the known broadcast group prefixes (KAFKA_BROADCAST_GROUP_PREFIXES) without a consumer group, so restarts leave nothing behind on the broker.

7. Measure, per replica

Average latency hides a slow replica. Break request histograms down by instance and map instances to nodes:

histogram_quantile(0.5,  sum by (le) (rate(http_request_duration_seconds_bucket{job="<app>",service="<svc>",instance="<task-ip>:<port>"}[1h])))
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="<app>",service="<svc>",instance="<task-ip>:<port>"}[1h])))

docker service ps <stack>_<svc> --filter desired-state=running --format '{{.Node}} {{.ID}}'
docker inspect <task-id> --format '{{range .NetworksAttachments}}{{.Network.Spec.Name}}={{.Addresses}} {{end}}'
ping -c 5 <other-node>    # node-to-node RTT; every remote data call costs at least this

Count the sequential data calls per request (Mongo, shared Redis, synchronous publishes). calls × RTT is the floor a remote replica adds.

Testing read preference

• Test on a replica set. A standalone server or directConnection=true makes the driver drop $readPreference, so a test there passes for the wrong reason. A single-member set is enough:
  shellCopy

  docker run -d -p 27199:27199 mongo:7 --replSet rs0 --port 27199
  mongosh --port 27199 --eval "rs.initiate({_id:'rs0',members:[{_id:0,host:'localhost:27199'}]})"
  # mongodb://localhost:27199/?replicaSet=rs0
  

  
  The port must match inside and outside the container: a member must find itself in its own config. In CI, pick a free port first.
• Assert on the wire: use a CommandMonitor, and check $readPreference on the command each classified read sends.
• Watch for TTL time bombs. Real-data fixtures with fixed dates fall out of a table's TTL (e.g. 7 days) and ClickHouse drops them on insert; the tests start failing on a date, on every commit. Create test tables without the TTL.

Checklist

• Node labels for role and location; services constrained by labels.
• Per-node services named <svc>-<hostname>; app env uses {{.Node.Hostname}}.
• MONGO_URI lists every member with replicaSet=; default read preference stays primary.
• go-framework ≥ v0.16.6 if any read uses Nearest; raw collections cloned with the preference.
• Every hot-path read classified (Nearest / primary / read-your-writes pattern), recorded with the code.
• Routing model chosen: A (app next to primary), B (regional services + geo-steering edge), or client-chosen region (regional hosts + client RTT choice).
• Sticky cookies only where a task holds per-request state.
• go-events ≥ v0.4.5 (or v0.3.1): broadcast consumers, and no per-topic startup stall.
• Per-replica p50/p95 dashboards; RTT between nodes known.