샤딩(Sharding)은 대용량 데이터를 여러 개의 데이터베이스에 수평 분할(Horizontal Partitioning)하여 저장하는 기법입니다. 각각의 분할된 데이터베이스를 샤드(Shard)라고 부릅니다.
샤딩 개념 예시:
모놀리식 애플리케이션에서도 샤딩은 매우 유용합니다:
전체 아키텍처 구성:
Application Server (애플리케이션 서버)
Meta DB (메타 데이터베이스)
shard_info 테이블: 각 샤드의 DB 접속 정보user_shard_mapping 테이블: 사용자와 샤드 매핑 정보Shard DBs (샤드 데이터베이스들)
동작 흐름:
1. 애플리케이션이 사용자 요청을 받음
2. Routing Logic이 샤드 키(UUID)로 샤드 ID 계산
3. Meta DB에서 해당 샤드의 접속 정보 조회
4. 계산된 샤드 DB에 접속하여 데이터 처리
1. shard_info 테이블
CREATE TABLE shard_info (
shard_id INT PRIMARY KEY,
db_host VARCHAR(255) NOT NULL,
db_port INT NOT NULL,
db_name VARCHAR(100) NOT NULL,
db_username VARCHAR(100) NOT NULL,
db_password VARCHAR(255) NOT NULL,
status VARCHAR(20) DEFAULT 'ACTIVE',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
2. user_shard_mapping 테이블 (선택사항)
CREATE TABLE user_shard_mapping (
user_id VARCHAR(36) PRIMARY KEY,
shard_id INT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (shard_id) REFERENCES shard_info(shard_id)
);
가장 일반적인 샤딩 키 결정 방식입니다:
// 샤드 결정 로직
String userId = "550e8400-e29b-41d4-a716-446655440000";
int totalShards = 4;
// UUID를 해시하여 샤드 ID 계산
int shardId = Math.abs(userId.hashCode()) % totalShards;
// shardId = 0, 1, 2, 또는 3
해시 기반 샤드 ID 계산 과정:
550e8400-e29b-41d4-a716-446655440000-1234567890 (예시)12345678902예시:
a1b2c3d4-... → hash % 4 → Shard 0e5f6g7h8-... → hash % 4 → Shard 1i9j0k1l2-... → hash % 4 → Shard 2m3n4o5p6-... → hash % 4 → Shard 3장점:
단점:
샤드 추가/제거 시 데이터 이동을 최소화하려면 Consistent Hashing 사용 권장:
// Consistent Hashing 예제
public class ConsistentHash {
private final TreeMap<Long, Integer> ring = new TreeMap<>();
private final int virtualNodes = 150;
public void addShard(int shardId) {
for (int i = 0; i < virtualNodes; i++) {
long hash = hash(shardId + "-vnode-" + i);
ring.put(hash, shardId);
}
}
public int getShardId(String key) {
long hash = hash(key);
Map.Entry<Long, Integer> entry = ring.ceilingEntry(hash);
return entry != null ? entry.getValue() : ring.firstEntry().getValue();
}
}
src/main/java/com/example/sharding/
├── config/
│ ├── DataSourceConfig.java
│ └── ShardDataSourceConfig.java
├── aspect/
│ └── ShardingAspect.java
├── annotation/
│ └── Sharding.java
├── routing/
│ ├── ShardContextHolder.java
│ └── ShardRoutingDataSource.java
├── service/
│ ├── ShardService.java
│ └── UserService.java
└── repository/
└── UserRepository.java
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
implementation 'org.springframework.boot:spring-boot-starter-aop'
implementation 'com.zaxxer:HikariCP'
runtimeOnly 'com.mysql:mysql-connector-j'
compileOnly 'org.projectlombok:lombok'
annotationProcessor 'org.projectlombok:lombok'
}
package com.example.sharding.annotation;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface Sharding {
/**
* 샤드 키로 사용할 파라미터 이름
*/
String key();
/**
* 전체 샤드 개수
*/
int shardCount() default 4;
}
package com.example.sharding.routing;
public class ShardContextHolder {
private static final ThreadLocal<Integer> CONTEXT = new ThreadLocal<>();
public static void setShardId(Integer shardId) {
CONTEXT.set(shardId);
}
public static Integer getShardId() {
return CONTEXT.get();
}
public static void clear() {
CONTEXT.remove();
}
}
package com.example.sharding.routing;
import org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource;
public class ShardRoutingDataSource extends AbstractRoutingDataSource {
@Override
protected Object determineCurrentLookupKey() {
Integer shardId = ShardContextHolder.getShardId();
logger.debug("Current Shard ID: " + shardId);
return shardId;
}
}
package com.example.sharding.config;
import com.example.sharding.routing.ShardRoutingDataSource;
import com.zaxxer.hikari.HikariDataSource;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.jdbc.DataSourceBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import javax.sql.DataSource;
import java.util.HashMap;
import java.util.Map;
@Configuration
public class ShardDataSourceConfig {
/**
* Meta DB DataSource (샤드 정보 관리용)
*/
@Bean
@ConfigurationProperties(prefix = "spring.datasource.meta")
public DataSource metaDataSource() {
return DataSourceBuilder.create()
.type(HikariDataSource.class)
.build();
}
/**
* Shard 0 DataSource
*/
@Bean
@ConfigurationProperties(prefix = "spring.datasource.shard0")
public DataSource shard0DataSource() {
return DataSourceBuilder.create()
.type(HikariDataSource.class)
.build();
}
/**
* Shard 1 DataSource
*/
@Bean
@ConfigurationProperties(prefix = "spring.datasource.shard1")
public DataSource shard1DataSource() {
return DataSourceBuilder.create()
.type(HikariDataSource.class)
.build();
}
/**
* Shard 2 DataSource
*/
@Bean
@ConfigurationProperties(prefix = "spring.datasource.shard2")
public DataSource shard2DataSource() {
return DataSourceBuilder.create()
.type(HikariDataSource.class)
.build();
}
/**
* Shard 3 DataSource
*/
@Bean
@ConfigurationProperties(prefix = "spring.datasource.shard3")
public DataSource shard3DataSource() {
return DataSourceBuilder.create()
.type(HikariDataSource.class)
.build();
}
/**
* 동적 라우팅 DataSource
*/
@Primary
@Bean
public DataSource routingDataSource() {
ShardRoutingDataSource routingDataSource = new ShardRoutingDataSource();
Map<Object, Object> targetDataSources = new HashMap<>();
targetDataSources.put(0, shard0DataSource());
targetDataSources.put(1, shard1DataSource());
targetDataSources.put(2, shard2DataSource());
targetDataSources.put(3, shard3DataSource());
routingDataSource.setTargetDataSources(targetDataSources);
routingDataSource.setDefaultTargetDataSource(shard0DataSource());
return routingDataSource;
}
}
spring:
datasource:
meta:
jdbc-url: jdbc:mysql://localhost:3306/meta_db
username: root
password: password
driver-class-name: com.mysql.cj.jdbc.Driver
hikari:
maximum-pool-size: 10
shard0:
jdbc-url: jdbc:mysql://localhost:3307/shard_0
username: root
password: password
driver-class-name: com.mysql.cj.jdbc.Driver
hikari:
maximum-pool-size: 20
shard1:
jdbc-url: jdbc:mysql://localhost:3308/shard_1
username: root
password: password
driver-class-name: com.mysql.cj.jdbc.Driver
hikari:
maximum-pool-size: 20
shard2:
jdbc-url: jdbc:mysql://localhost:3309/shard_2
username: root
password: password
driver-class-name: com.mysql.cj.jdbc.Driver
hikari:
maximum-pool-size: 20
shard3:
jdbc-url: jdbc:mysql://localhost:3310/shard_3
username: root
password: password
driver-class-name: com.mysql.cj.jdbc.Driver
hikari:
maximum-pool-size: 20
jpa:
hibernate:
ddl-auto: update
show-sql: true
properties:
hibernate:
format_sql: true
package com.example.sharding.aspect;
import com.example.sharding.annotation.Sharding;
import com.example.sharding.routing.ShardContextHolder;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.reflect.MethodSignature;
import org.springframework.stereotype.Component;
import java.lang.reflect.Method;
import java.lang.reflect.Parameter;
@Slf4j
@Aspect
@Component
@RequiredArgsConstructor
public class ShardingAspect {
@Around("@annotation(com.example.sharding.annotation.Sharding)")
public Object routeToShard(ProceedingJoinPoint joinPoint) throws Throwable {
try {
MethodSignature signature = (MethodSignature) joinPoint.getSignature();
Method method = signature.getMethod();
Sharding sharding = method.getAnnotation(Sharding.class);
// 어노테이션에서 지정한 파라미터 이름 가져오기
String keyParamName = sharding.key();
int shardCount = sharding.shardCount();
// 메서드 파라미터에서 샤드 키 값 추출
Object shardKeyValue = extractShardKeyValue(
signature.getParameterNames(),
joinPoint.getArgs(),
keyParamName
);
if (shardKeyValue == null) {
throw new IllegalArgumentException(
"Shard key parameter '" + keyParamName + "' not found or is null"
);
}
// 샤드 ID 계산
int shardId = calculateShardId(shardKeyValue.toString(), shardCount);
log.info("Routing to shard: {} (key: {}, value: {})",
shardId, keyParamName, shardKeyValue);
// ThreadLocal에 샤드 ID 설정
ShardContextHolder.setShardId(shardId);
// 실제 메서드 실행
return joinPoint.proceed();
} finally {
// ThreadLocal 정리
ShardContextHolder.clear();
}
}
/**
* 파라미터 이름으로 실제 값 추출
*/
private Object extractShardKeyValue(String[] paramNames, Object[] args, String targetParamName) {
for (int i = 0; i < paramNames.length; i++) {
if (paramNames[i].equals(targetParamName)) {
return args[i];
}
}
return null;
}
/**
* UUID를 해시하여 샤드 ID 계산
*/
private int calculateShardId(String key, int shardCount) {
int hash = Math.abs(key.hashCode());
return hash % shardCount;
}
}
package com.example.sharding.service;
import com.example.sharding.routing.ShardContextHolder;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
@Slf4j
@Service
public class ShardService {
private static final int TOTAL_SHARDS = 4;
/**
* UUID 기반 샤드 ID 계산
*/
public int calculateShardId(String uuid) {
int hash = Math.abs(uuid.hashCode());
int shardId = hash % TOTAL_SHARDS;
log.debug("Calculated shard ID {} for UUID: {}", shardId, uuid);
return shardId;
}
/**
* 수동으로 샤드 설정
*/
public void setShardId(int shardId) {
if (shardId < 0 || shardId >= TOTAL_SHARDS) {
throw new IllegalArgumentException(
"Invalid shard ID: " + shardId + ". Must be between 0 and " + (TOTAL_SHARDS - 1)
);
}
ShardContextHolder.setShardId(shardId);
}
/**
* 현재 샤드 ID 조회
*/
public Integer getCurrentShardId() {
return ShardContextHolder.getShardId();
}
/**
* 샤드 컨텍스트 정리
*/
public void clearShardContext() {
ShardContextHolder.clear();
}
}
package com.example.sharding.entity;
import jakarta.persistence.*;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.time.LocalDateTime;
@Entity
@Table(name = "users")
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class User {
@Id
@Column(length = 36)
private String userId; // UUID
@Column(nullable = false, length = 100)
private String username;
@Column(nullable = false, length = 255)
private String email;
@Column(length = 20)
private String phone;
@Column(nullable = false)
private LocalDateTime createdAt;
@Column
private LocalDateTime updatedAt;
@PrePersist
protected void onCreate() {
createdAt = LocalDateTime.now();
}
@PreUpdate
protected void onUpdate() {
updatedAt = LocalDateTime.now();
}
}
package com.example.sharding.repository;
import com.example.sharding.entity.User;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;
import java.util.Optional;
@Repository
public interface UserRepository extends JpaRepository<User, String> {
Optional<User> findByEmail(String email);
boolean existsByEmail(String email);
}
package com.example.sharding.service;
import com.example.sharding.annotation.Sharding;
import com.example.sharding.entity.User;
import com.example.sharding.repository.UserRepository;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.util.Optional;
import java.util.UUID;
@Slf4j
@Service
@RequiredArgsConstructor
public class UserService {
private final UserRepository userRepository;
private final ShardService shardService;
/**
* 사용자 생성 (자동 샤딩)
*/
@Transactional
@Sharding(key = "userId", shardCount = 4)
public User createUser(String userId, String username, String email, String phone) {
log.info("Creating user with ID: {}", userId);
User user = User.builder()
.userId(userId)
.username(username)
.email(email)
.phone(phone)
.build();
return userRepository.save(user);
}
/**
* UUID 생성 후 사용자 생성
*/
@Transactional
public User createUserWithGeneratedId(String username, String email, String phone) {
String userId = UUID.randomUUID().toString();
return createUser(userId, username, email, phone);
}
/**
* 사용자 조회 (자동 샤딩)
*/
@Transactional(readOnly = true)
@Sharding(key = "userId", shardCount = 4)
public Optional<User> getUserById(String userId) {
log.info("Fetching user with ID: {}", userId);
return userRepository.findById(userId);
}
/**
* 사용자 업데이트 (자동 샤딩)
*/
@Transactional
@Sharding(key = "userId", shardCount = 4)
public User updateUser(String userId, String username, String email, String phone) {
log.info("Updating user with ID: {}", userId);
User user = userRepository.findById(userId)
.orElseThrow(() -> new RuntimeException("User not found: " + userId));
user.setUsername(username);
user.setEmail(email);
user.setPhone(phone);
return userRepository.save(user);
}
/**
* 사용자 삭제 (자동 샤딩)
*/
@Transactional
@Sharding(key = "userId", shardCount = 4)
public void deleteUser(String userId) {
log.info("Deleting user with ID: {}", userId);
userRepository.deleteById(userId);
}
/**
* 이메일로 사용자 검색 (모든 샤드 검색 필요)
* 주의: 이 방식은 성능이 좋지 않으므로 실제로는 Meta DB에 이메일 인덱스를 유지하는 것이 좋음
*/
@Transactional(readOnly = true)
public Optional<User> findByEmail(String email) {
log.info("Searching for user by email: {}", email);
// 모든 샤드를 순회하며 검색
for (int shardId = 0; shardId < 4; shardId++) {
try {
shardService.setShardId(shardId);
Optional<User> user = userRepository.findByEmail(email);
if (user.isPresent()) {
return user;
}
} finally {
shardService.clearShardContext();
}
}
return Optional.empty();
}
}
package com.example.sharding.controller;
import com.example.sharding.entity.User;
import com.example.sharding.service.UserService;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;
import java.util.UUID;
@Slf4j
@RestController
@RequestMapping("/api/users")
@RequiredArgsConstructor
public class UserController {
private final UserService userService;
/**
* 사용자 생성
*/
@PostMapping
public ResponseEntity<User> createUser(@RequestBody CreateUserRequest request) {
User user = userService.createUserWithGeneratedId(
request.getUsername(),
request.getEmail(),
request.getPhone()
);
return ResponseEntity.status(HttpStatus.CREATED).body(user);
}
/**
* 사용자 조회
*/
@GetMapping("/{userId}")
public ResponseEntity<User> getUser(@PathVariable String userId) {
return userService.getUserById(userId)
.map(ResponseEntity::ok)
.orElse(ResponseEntity.notFound().build());
}
/**
* 사용자 업데이트
*/
@PutMapping("/{userId}")
public ResponseEntity<User> updateUser(
@PathVariable String userId,
@RequestBody UpdateUserRequest request) {
User user = userService.updateUser(
userId,
request.getUsername(),
request.getEmail(),
request.getPhone()
);
return ResponseEntity.ok(user);
}
/**
* 사용자 삭제
*/
@DeleteMapping("/{userId}")
public ResponseEntity<Void> deleteUser(@PathVariable String userId) {
userService.deleteUser(userId);
return ResponseEntity.noContent().build();
}
/**
* 이메일로 사용자 검색
*/
@GetMapping("/search")
public ResponseEntity<User> findByEmail(@RequestParam String email) {
return userService.findByEmail(email)
.map(ResponseEntity::ok)
.orElse(ResponseEntity.notFound().build());
}
}
// DTO 클래스들
@Data
class CreateUserRequest {
private String username;
private String email;
private String phone;
}
@Data
class UpdateUserRequest {
private String username;
private String email;
private String phone;
}
package com.example.sharding.service;
import com.example.sharding.entity.User;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import java.util.UUID;
import static org.assertj.core.api.Assertions.*;
@SpringBootTest
class UserServiceTest {
@Autowired
private UserService userService;
@Autowired
private ShardService shardService;
@Test
void testCreateAndRetrieveUser() {
// Given
String userId = UUID.randomUUID().toString();
String username = "testuser";
String email = "test@example.com";
String phone = "010-1234-5678";
// When
User createdUser = userService.createUser(userId, username, email, phone);
// Then
assertThat(createdUser).isNotNull();
assertThat(createdUser.getUserId()).isEqualTo(userId);
assertThat(createdUser.getUsername()).isEqualTo(username);
// Verify retrieval
User retrievedUser = userService.getUserById(userId).orElseThrow();
assertThat(retrievedUser.getEmail()).isEqualTo(email);
}
@Test
void testShardCalculation() {
// Given
String userId1 = "550e8400-e29b-41d4-a716-446655440000";
String userId2 = "6ba7b810-9dad-11d1-80b4-00c04fd430c8";
// When
int shard1 = shardService.calculateShardId(userId1);
int shard2 = shardService.calculateShardId(userId2);
// Then
assertThat(shard1).isBetween(0, 3);
assertThat(shard2).isBetween(0, 3);
// 동일한 UUID는 항상 동일한 샤드로 라우팅되어야 함
assertThat(shardService.calculateShardId(userId1)).isEqualTo(shard1);
}
}
샤드 개수를 변경해야 할 때의 전략:
여러 샤드에 걸친 JOIN은 애플리케이션 레벨에서 처리 필요
분산 트랜잭션 대신 Saga 패턴 또는 보상 트랜잭션 고려
각 샤드별 독립적인 백업 전략 수립
샤딩은 대용량 데이터를 효과적으로 처리하기 위한 강력한 기법입니다. Spring Boot와 AOP를 활용하면 어노테이션 기반으로 깔끔하게 샤딩 로직을 구현할 수 있습니다.
핵심 포인트:
이 구조를 기반으로 프로젝트 요구사항에 맞게 확장하여 사용하시면 됩니다! 🚀