Introduction
I have had two experiences of improving systems by introducing queues in practice. This article covers the first case, focusing on "Why was a queue needed, how was it applied, and what changed?"
(The cryptocurrency names and code examples are for understanding purposes and not from actual services. Also, blocks frequently mentioned in the article refer to bundles of cryptocurrency transactions, and transactions refer to cryptocurrency transaction records.)
Service Introduction
The service covered in this article is a cryptocurrency transaction explorer. Simply put, it's similar to Etherscan (https://etherscan.io/), which allows you to view all Ethereum transaction records.
To implement the explorer, we need to periodically query transaction records from external blockchain networks and store them in the service's internal database. For this purpose, we operated separate query servers and processing servers, and transaction records go through multiple parsing processes before being finally stored, not just simple storage.
Reasons for Introducing Queues
There are various reasons for using queues. Usually, there are multiple reasons such as asynchronous processing, load distribution, reduced coupling between systems, and fault isolation.
The core reasons for introducing queues in this service were three:
Load Buffering – The block collection server fetches new data every second, but the processing server often couldn't keep up with that speed. The queue acted as an intermediate buffer, separating collection and processing to ensure stability.
Scalability – Currently, we only synchronize one type of cryptocurrency, but we needed to process multiple assets simultaneously in the future. With a queue-based structure, we could easily scale by separating topics or increasing partitions.
Parallel Processing Utilization – Using Kafka Consumer Groups allows us to utilize processing server instances in parallel, resolving bottlenecks that occurred in single servers.
In other words, the purpose of introducing queues was to absorb the speed difference between producers (query servers) and consumers (processing servers) and structurally open up scalability possibilities.
However, there was a separate problem that block data needed to be stored in order by number, which couldn't be solved by queues alone. So we later introduced Redis Sorted Set to guarantee order.
Problem Occurrence
At that time, the system had separate transaction record query servers and transaction record processing servers, and the two servers were connected through synchronous HTTP communication.
The operation flow was as follows:
- The block collection server periodically queries new blocks.
- It sends an HTTP POST request with the queried block data to the processing server.
- The processing server parses transactions and stores them in the database.
- Only after receiving a processing completion response can it process the next block.
The problem was that the processing server's processing speed couldn't keep up with the collection server's collection speed. Due to the characteristics of the synchronous structure, when the processing server slowed down, the query server would cause timeouts, eventually skipping some blocks and causing data loss.
Why Was the Synchronous Processing Method Problematic?
The core causes were the constraints of synchronous communication and low resource utilization.
- Block collection server: Single instance (single thread)
- Processing server: 4 instances (cluster mode)
- Actual utilization: Only 1 instance was always used, the other 3 were idle
In other words, the query server couldn't send other requests because it was waiting for the processing server's response, causing the overall throughput to remain at a single server level. Moreover, when external API calls or operations were delayed in the processing server, the query server would cause timeouts and skip blocks, creating data consistency problems.
Introducing Infrastructure (Kafka)
To solve this problem, we decided to place a queue between the query server and processing server. There were queue-based tools like Kafka, BullMQ, etc., but we ultimately chose Kafka for a simple reason:
It was the tool the team was most familiar with, and it was also the tool I understood relatively well.
(Looking back, I realized again that in practice, choosing a tool the team can handle best is more important than technical superiority.)
Applying Kafka
By placing Kafka in between, we simplified the query server to only act as a Producer and the processing server to only act as a Consumer.
1) Consumer Group Configuration
this.consumer = this.kafka.consumer({
groupId: 'transaction-processors', // 1 Consumer Group
partitionAssignmentStrategy: 'RoundRobin', // Equal distribution
enableAutoCommit: false, // Manual commit for precise control
});- 4 Consumer instances → grouped into the same Consumer Group
- Each instance handles 1 partition → parallel processing possible
We designed it so that when cryptocurrency types increase in the future, we can scale by creating new topics for each asset and adjusting the number of partitions.
2) Kafka Partition Design
Since the processing server consisted of 4 instances, we matched the topic's partition count to 4. Kafka guarantees message order within a partition, but only one Consumer can read from a partition. Therefore, if there's only one partition, even if there are multiple Consumers in the Consumer Group, only one Consumer actually works.
👉 To utilize parallel processing, the number of partitions ≥ number of Consumers must be satisfied.
// Block height-based partition routing
const partition = blockHeight % 4; // 0, 1, 2, 3
await this.producer.send({
topic: 'block-processing',
partition: partition,
messages: [
{
key: blockHeight.toString(),
value: JSON.stringify(blockData),
},
],
});By increasing partitions to 4, the 4 instances in the Consumer Group could each take one and process messages in parallel. In other words, we could escape the situation where only one process worked while the others were idle.
Problem 1: Block Order Guarantee
Through Kafka, we solved the processing server's load problem, but a new problem arose where block data wasn't stored in order in the database.
Since Kafka only guarantees order within partitions, data processed in parallel across 4 partitions would inevitably be mixed up overall. → We needed a separate mechanism to guarantee block number order.
Introducing Infrastructure (Redis)
To solve this, we introduced Redis Sorted Set. Sorted Set can store data sorted by Score, and we used block numbers as Score here.
Of course, we could simply use ORDER BY block_height in the DB to maintain order. But this approach incurs sorting costs every time during large-scale synchronization processes, increasing DB load and eventually becoming a bottleneck in the entire pipeline. The parallel processing benefits of Kafka would also be diluted. So we chose to use lightweight memory-based Redis Sorted Set to organize order before putting it into the DB.
Applying Redis
The final structure is as follows:
The application method was as follows:
-
When a Consumer receives a message, it stores only transaction data in the DB and puts block information in Redis Sorted Set.
-
A separate process takes data from Redis and stores it in the DB in block number order.
-
Considering that cryptocurrencies will increase, we separate queue names by asset.
1) Queue Name Design
// Generate queue names by cryptocurrency
private getQueueName(assetType: string): string {
const baseQueueName = 'block_queue';
return `${baseQueueName}_${assetType.toLowerCase()}`;
}
// Example queue names
// block_queue_bitcoin
// block_queue_ethereum
// block_queue_ripple2) Order Guarantee Logic
// Use block height as Score for automatic sorting
await this.redis.zadd(queueName, blockHeight, JSON.stringify(blockData));
// Atomically get the block with the lowest height
const result = await this.redis.zpopmin(queueName);Thanks to this method, Redis could act as a sorting buffer and rearrange data that was mixed up due to Kafka's parallel processing back into order.
Final Architecture
Results
- Improved transaction processing speed: Block-level processing delays were greatly reduced, improving overall synchronization speed by several times
- Eliminated timeouts/data loss: The problem of query servers timing out while waiting for processing server responses disappeared, enabling stable storage without block omissions
- Secured operational stability: Even when load temporarily concentrated, Kafka acted as a buffer, blocking fault propagation
Disaster Recovery
In the current structure, if failures occur in Kafka or Redis, we need to manually check the last stored data and recover manually.
Of course, building clustering for both Kafka and Redis can greatly increase stability. But considering the service scale and data volume at that time, introducing infrastructure at that level was an excessive choice.
In other words, while knowing there were technically "better" options, we chose a realistic compromise considering business situations and operational resources. This decision wasn't simple technical greed but a balanced choice appropriate for the service scale.
Future Clustering Plans
When the service grows, we plan to introduce clustering as follows.
Kafka Cluster
# Configure cluster with 3 brokers
Broker1: Partition 0, 3 leader
Broker2: Partition 1 leader
Broker3: Partition 2 leader
# Service continues even if 1 broker failsRedis Cluster
# Redis Sentinel configuration
Master: Redis1
Slaves: Redis2, Redis3
Sentinel: 3 units (failure detection and automatic failover)Reflection on Over-Engineering
Looking back, there were excessive parts in the design at that time. Of course, considering future expansion is important, but thinking about it now, solving immediate problems with simpler methods might have been more appropriate.
If I faced the same situation again, I would reduce unnecessary infrastructure expansion and make light and efficient choices appropriate for the service scale and actual requirements. I think my personal desire to try new technologies was mixed into the decisions at that time, so I personally feel some regret.
Through this experience, I learned that appropriate compromises suitable for the situation might be wiser choices than technically perfect designs. Specifically, I gained these lessons:
- Introduce expansion step by step when needed
- Choosing tools the team can handle best ultimately increases productivity
- Balance appropriate for service requirements takes priority over technical greed
Actual Operation Status and Personal Meaning
This architecture was tested under conditions identical to the production environment and received team review and approval. However, it couldn't be applied to actual operation because the company went out of business due to financial difficulties. It was a project that was discontinued due to external factors, not technical limitations.
Although it didn't make it to production, it had great personal meaning as my first practical architecture experience of defining real-world problems myself and leading the design, implementation, and verification of solutions.
