DISTRIBUTED COUNTER FOR DYNAMODB

Overview

An engineering project addressing DynamoDB's challenges with high-throughput atomic counters. DynamoDB uses optimistic concurrency control which can cause transaction failures under contention. Our hybrid solution combines a best-effort counter (immediate, may undercount but never overcounts) with an eventually consistent counter (processed asynchronously via DynamoDB Streams and Lambda). The best-effort counter serves as an immediate heuristic for enforcing creation limits, while the eventual consistency counter catches up in the background. This design achieves exactly-once processing guarantees through idempotency flags and transactional atomic operations.

Key Results

Methodology

Built a TypeScript API using Elysia on the Bun runtime with a DynamoDB single-table design. Resource creation immediately increments a best-effort counter and writes to DynamoDB with a counter_processed=false flag. DynamoDB Streams trigger Lambda functions that atomically increment the eventually consistent counter and set the flag. Deletion uses soft-delete (flag) followed by Lambda-driven decrement and hard-delete. Error handling uses a two-tier dead letter queue system. The entire stack is containerized with Docker using a local DynamoDB emulator for development.

What We Built

• ●Hybrid best-effort + eventually consistent counter architecture
• ●Exactly-once processing via idempotency flags and DynamoDB transactions
• ●Self-healing: both counters reset to zero when all resources are deleted
• ●Two-tier dead letter queue for fault tolerance and manual recovery
• ●DynamoDB Streams + Lambda for asynchronous counter reconciliation
• ●Zero new API error types — errors handled entirely asynchronously

Challenges

• ●Preventing double-counting in concurrent Lambda stream processing with DynamoDB transactions
• ●Local DynamoDB emulator lacks true parity with cloud (locking vs optimistic concurrency)
• ●Getting transaction failures to occur locally for testing — solved by emulating DynamoDB through local stack
• ●Sparse AWS documentation for DynamoDB Streams behavior required reading original white papers

Outcomes

• ●Achieved over 2,000 TPS — exceeding DynamoDB's standard one-millisecond counter guarantee
• ●Designed a counter that can never overcount, only temporarily undercount during reconciliation
• ●System is infinitely scalable via Lambda functions that spawn to meet demand
• ●Created a production-ready pattern applicable to any resource-counting API on DynamoDB

Papers & Reports