Regions

A Redelay app can serve, say, Europe and North America from nodes in both, with each member's reads and writes staying in their own region. This page is the design: the one rule every datastore follows, what the framework gives you, and what is still to come. Node placement and Swarm details are in Multi-server Swarm.

The overview with diagrams and measured round trips is the Multi-region page.

The rule: a write waits for its majority partner

Every replicated store (MongoDB, Kafka, ClickHouse Keeper, Redis Cluster failover) commits a write once a majority has it. The write is as fast as the nearest member of that majority, so:

• put a write's majority inside the region that writes it;
• never let a write wait for a member across an ocean;
• add one tie-breaker vote at a third site (non-electable) so a region keeps a majority when one of its nodes is down — it never slows normal writes, because a majority of three is the primary plus the fastest acknowledgement.

Measured on a two-region deployment (Europe ↔ North America, ~120 ms apart):

MongoDB — regional shards (runtime/mongo, runtime/region)

Member-owned collections are zone-sharded on {home_region: 1, user_id: 1}; each zone's shard has its primary and write majority in its region. Global data (catalogue, CMS, settings) stays unsharded on the primary shard and is read from local secondaries with readPreference=nearest&maxStalenessSeconds=90. Public rows in a mixed collection use home_region: "global".

Rollout order per collection: (1) stamp new rows (StampRegionStrict, write filters through OwnFilter); (2) backfill (BackfillRegion in a migration) and create the shard-key index {home_region: 1, user_id: 1}; (3) run the backfill again, then MONGO_HOME_REGION_FILTER=true — from here every filter carries the shard key, which upserts on a sharded collection need; (4) rebuild unique indexes with the shard key first, shardCollection, zone ranges. Turning the switch on before the backfill would make upserts miss older rows and insert copies. Apps connect through mongos before step 4: once chunks move, a client connected directly to one shard sees only part of the data (MongoDB 8.0 refuses direct operations on a shard unless the user has directShardOperations).

Settings that matter: settings.chainingAllowed: false on every replica set (a secondary syncing through a tie-breaker doubled majority-write time in our benchmark), identity (email, password, sessions) unsharded (a unique index on a sharded collection must start with the shard key, so a global unique email cannot live there), and backups per shard plus config servers.

Last write majority {#last-write-majority}

Writes to one replica set are applied in order. A request that writes several times only needs to wait for the last one: once it is majority-committed, every earlier write on that primary is too.

fast := rtmongo.W1(orders)                    // w:1 — no wait for the majority
_, _ = fast.InsertOne(ctx, line1)
_, _ = fast.UpdateOne(ctx, f, u)
_, err := orders.InsertOne(ctx, summary)       // the collection's own concern (majority): waits once

Three majority waits of ~30 ms become one. Only within ONE replica set (one shard): writes that land on different shards each need their own majority write. If the primary fails before the last write is acknowledged, the earlier w:1 writes may be rolled back with it — the request fails either way, as it would have with three majority writes. Prefer BulkWrite when the writes are to one collection.

Adopting home regions in an app {#adopting}

Any number of regions: HOME_REGIONS="na=US,CA;apac=JP,AU,SG" plus the default gives three, each its own zone (one shard can hold several zones until a region gets its own); nothing in the framework assumes two (tested with eu/na/apac).

Home regions are optional. A single-region project sets nothing: with HOME_REGIONS unset, region.Enabled() is false and every helper below is a no-op (no users lookup, no home_region field, no index), so framework modules can call them unconditionally. A multi-region project turns them on with HOME_REGIONS (+ HOME_REGION_DEFAULT); HOME_REGIONS_ENABLED=false turns them off again.

Order, per app (each step its own release; verify by behaviour):

1. Expand. Members get home_region at sign-up (region.ForCountry) and a backfill (none → default). Every write of a member collection stamps the owner's region (Owner + StampRegionStrict, or StampOwner) and its upsert filters go through OwnFilter. Migrations: PrepareRegionCollection per collection.
2. Recheck + filter. The same migration under a new id (rows the previous image wrote during the roll), then MONGO_HOME_REGION_FILTER=true. Run the test suite with the flag on first: a row without the region makes an upsert miss it and fail on a unique index.
3. Shard-ready. Member collections use the simple collation (store names lowercased): a non-simple collation on a collection sharded on a string key goes to every shard. Unique indexes become {home_region, owner, …} with simple collation (ReplaceUniqueIndex) — MongoDB refuses to shard otherwise. CheckShardable must return nil for each collection.
4. Shard. Through mongos only (every client of the database must connect through mongos first; MongoDB 8 refuses direct operations on a shard without directShardOperations). ShardByRegion with the zones whose shards should hold data now — add a region's range only when it has members: an empty range on a remote shard turns every query without the shard key into a cross-region round trip.
5. Move members of a region (add its range; the balancer moves chunks).

Per collection, check: owner field and type (ObjectID or hex string — both match); every insert path, including background and late writers (stamp from the parent row's owner: a journal op from its session's owner); every upsert filter; unique indexes; queries without the shard key (fine, but they reach every shard holding chunks); TTL indexes (fine). Identity (users, sessions, tokens) stays unsharded: a global unique email cannot live on a sharded collection. Catalogue rows in a mixed collection get home_region: "global" in their own migration first.

Testing on a sharded cluster {#testing-sharded}

docker network create shtest
docker run -d --name sht-cfg --network shtest mongo:8 --configsvr --replSet cfgrs --port 27019 --bind_ip_all
docker run -d --name sht-a   --network shtest mongo:8 --shardsvr  --replSet rsa   --port 27018 --bind_ip_all
docker run -d --name sht-b   --network shtest mongo:8 --shardsvr  --replSet rsb   --port 27018 --bind_ip_all
docker exec sht-cfg mongosh --port 27019 --eval "rs.initiate({_id:'cfgrs',configsvr:true,members:[{_id:0,host:'sht-cfg:27019'}]})"
docker exec sht-a   mongosh --port 27018 --eval "rs.initiate({_id:'rsa',members:[{_id:0,host:'sht-a:27018'}]})"
docker exec sht-b   mongosh --port 27018 --eval "rs.initiate({_id:'rsb',members:[{_id:0,host:'sht-b:27018'}]})"
docker run -d --name sht-mongos --network shtest -p 27100:27017 mongo:8 mongos --configdb cfgrs/sht-cfg:27019 --bind_ip_all --port 27017
docker exec sht-mongos mongosh --eval "sh.addShard('rsa/sht-a:27018'); sh.addShard('rsb/sht-b:27018'); sh.addShardToZone('rsa','eu'); sh.addShardToZone('rsa','global'); sh.addShardToZone('rsb','na')"
REDELAY_TEST_MONGO_SHARDED_URI='mongodb://localhost:27100' go test ./...

Kafka — replicas stay in one region (go-events)

acks=all waits for every in-sync replica, not min.insync.replicas. One replica across an ocean makes every publish ~1 s. Keep a topic's replicas in one region (RF 3 with a third-site broker, or RF 2 in-region); producers in the other region pay one round trip. KAFKA_TOPIC_RACK (go-events ≥ v0.7.3) creates new topics only on brokers of that broker.rack, so a topic created by an app can't land in the other region. Planned: EVENTS_REGION with a region prefix for topics produced and consumed in one region, a consumer scope (local | all), and the outbox for publishes made inside a request so a request never waits for Kafka.

ClickHouse — nearest Keeper, batched inserts

Replicating the data is asynchronous; an insert waits for ~8 Keeper writes, each a Keeper majority round. Batch inserts (or async_insert=1) and give each server its nearest Keeper first: zookeeper_load_balancing=in_order with a per-server node order — the default is random, and one session on a Keeper across the ocean made 9 ms inserts take 800 ms in our benchmark.

Redis — shared cluster with replicas and an ACL user

The shared cluster (locks, rate limits, coordination, event dedupe) needs a replica per master on another node so one node down is not CLUSTERDOWN. Turn auth on with a named ACL user (REDIS_SHARED_USERNAME / REDIS_SHARED_PASSWORD, go-modules ≥ v0.68.0; go-events ≥ v0.7.2 for the dedupe store): a password sent to a server without one is an error, so clients log in as the user first and the default user is switched off last. Code that opens its own client on the cluster uses SharedClusterOptions(). The recent-write marker (runtime/mongo/recentwrite) reads this node's own master; on a node that runs only a replica (yul, a replica of amer's master) it reads the master with the lowest measured round trip (go-framework ≥ v0.27.0) instead of whichever master holds the user's key.

Still to come