Sharding can scale throughput but risks breaking cross-shard atomicity, the guarantee that a multi-shard transaction either fully commits everywhere or aborts everywhere. Robust designs combine strong local agreement, coordinated commit protocols, and deterministic global ordering to preserve atomicity without negating parallelism.
Design primitives
A foundational choice is to run shard-local consensus using proven algorithms so each shard behaves like a reliable single node. Leslie Lamport at Microsoft Research formalized consensus principles that underpin Paxos, and Diego Ongaro and John Ousterhout at Stanford popularized Raft as an engineer-friendly alternative; both patterns give shards a consistent view of local state. On top of that, a reliable atomic commit protocol coordinates changes across shards. Classical two-phase commit described by Jim Gray at Microsoft Research remains a clear model: prepare and then commit, ensuring either global success or coordinated rollback. To reduce blocking, designs often combine two-phase commit with shard-local consensus so that a coordinator can rely on durable local decisions even if it fails.
Ordering and cross-shard messages
Another effective choice is deterministic ordering of transactions, where a global sequencer or deterministic routing assigns a total order that shards execute in a conflict-free way. Systems inspired by deterministic databases avoid run-time locking by preordering operations, reducing expensive cross-shard coordination. Alternatively, optimistic concurrency control lets transactions proceed and resolves conflicts at commit time with an atomic commit step; this favors latency when cross-shard conflicts are rare but requires robust rollback paths.
Using cryptographic or time-limited locks for cross-shard transfers introduces application-level guarantees rather than full transactional semantics. Blockchain research by Vitalik Buterin at Ethereum Foundation explores asynchronous message-passing approaches for sharded ledgers that preserve eventual consistency while trading strict atomicity for liveness under network asynchrony.
Trade-offs and consequences
These choices impose trade-offs. Strong coordination raises latency and operational complexity, and increases the attack surface for cross-shard denial-of-service. Deterministic or optimistic designs can improve throughput but require careful handling of contention and client retries. Culturally and territorially, systems spanning jurisdictions must also reconcile differing data residency and regulatory requirements, which can influence how much cross-shard state is allowed to move. In practice, combining shard-local consensus, a resilient atomic commit layer, and deterministic or optimistic ordering—implemented with clear failure and rollback semantics—preserves cross-shard atomicity while enabling scalable sharding. The precise mix depends on workload patterns, trust assumptions, and deployment constraints.