[fix] 대규모 팔로워 데이터 수집 시 DB 병목 및 타임아웃 장애 개선 - #97
Merged
Merged
Conversation
- PR 머지 커밋(Merge Commit) 푸시 시 Git 기본 동작으로 인해 변경 파일이 빈 값으로 판별되던 결함 해결 - 머지 커밋의 부모(HEAD^1) 또는 이전 기준 커밋(github.event.before)을 명시적으로 비교하도록 git diff 기반 감지 로직 적용 - Actions 실행 로그에 변경 파일 목록을 명확히 출력하여 배포 모듈 추적성 향상
- 스트리머 통계 및 대시보드 내 텍스트 색상 명도 상향 (text-gray-400 -> text-gray-300/200) - 캘린더 타임라인 및 툴팁 가독성 개선을 위한 폰트 크기 및 두께 보정
- [engine] 2.2만 건 수집 데이터를 1,000건 단위 순차 청크로 분할 전송하도록 스케줄러 개선하여 수신측 유입 부하 완화 - [api-server] 기존 건별 단건 쿼리(N+1, 9만 회)를 PostgreSQL Native ON CONFLICT Bulk Upsert 파이프라인으로 전면 개편 (DB 왕복 99.9% 감소) - [api-server] 직전일(D-1) 스냅샷을 IN 절로 일괄 메모리 로드 후 증감량을 계산하여 트랜잭션 소요 시간 및 Row Lock 점유 시간 단축 (420초 -> 15ms) - [api-server] 단위 및 통합 테스트 케이스 추가/수정 (StreamerFollowerCommandServiceTest, StreamerFollowerJdbcRepositoryTest 등 100% 통과 검증)
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: Repository: Firedrago95/stream-engine/.coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: ⛔ Files ignored due to path filters (9)
📒 Files selected for processing (10)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Owner
Author
|
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== PR metadata and files ==='
gh pr view 97 --repo Firedrago95/stream-engine --json title,baseRefName,headRefName,files --jq '.title, .baseRefName, .headRefName, (.files[] | "\(.path)\t+\(.additions) -\(.deletions)")'
echo '=== changed file names ==='
git diff --name-only HEAD^ HEAD
echo '=== stat ==='
git diff --stat HEAD^ HEAD
echo '=== diff ==='
git diff --find-renames --find-copies --unified=80 HEAD^ HEADLength of output: 57975 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== local graph and merge base ==='
git log --oneline --decorate -8
git branch -a --contains HEAD | sed -n '1,40p'
git merge-base HEAD origin/main || true
BASE=$(git merge-base HEAD origin/main)
echo "BASE=$BASE"
git diff --name-status "$BASE"...HEAD
echo '=== complete GitHub PR diff file headers ==='
gh pr diff 97 --repo Firedrago95/stream-engine --name-only
echo '=== candidate source files ==='
fd -a 'StreamerFollowerSnapshotEntity.java|StreamEntity.java|ApiServerClient.java|FollowerSnapshotRecord.java|FollowerSnapshotRecordDto.java' .
echo '=== SQL migrations and schema references ==='
rg -n -i -C 4 'streamer_follower_snapshots|CREATE TABLE streams|follower_count|last_follower_updated_at' api-server -g '*.sql' -g '*.java'
echo '=== follower API routes and caller implementation ==='
rg -n -C 5 'recordFollowers\s*\(|sendFollowerSnapshots\s*\(|FollowerSnapshotRecordDto|FollowerSnapshotRecord' api-server engine -g '*.java'
echo '=== full base diff for operational files ==='
git diff --find-renames --unified=40 "$BASE"...HEAD -- \
api-server/src/main/java/io/slice/stream/apiserver/streamer \
engine/src/main/java/io/slice/stream/engine/ingestion \
.github/workflows/deploy.ymlLength of output: 2232 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== PR file list ==='
gh pr diff 97 --repo Firedrago95/stream-engine --name-only
echo '=== PR full diff: follower and deployment changes ==='
gh pr diff 97 --repo Firedrago95/stream-engine --patch | \
awk '
/^diff --git / { show = ($0 ~ /(StreamerFollower|DailyFollowerCollector|ApiServerClient|FollowerSnapshot|deploy\.yml)/) }
show { print }
'
echo '=== locate relevant source files ==='
fd -a . api-server engine | rg '(StreamerFollowerSnapshotEntity|StreamEntity|ApiServerClient|FollowerSnapshotRecord|FollowerSnapshotRecordDto|DailyFollowerCollectorScheduler|application\.ya?ml)$'
echo '=== relevant type and API declarations ==='
rg -n -C 8 'class StreamerFollowerSnapshotEntity|class StreamEntity|interface ApiServerClient|class ApiServerClient|record FollowerSnapshotRecord|record FollowerSnapshotRecordDto|recordFollowers\s*\(' api-server engine -g '*.java'
echo '=== schedule, executor, and request size configuration ==='
rg -n -i -C 4 'follower\.cron|follower.*collect|virtual.*thread|executor|multipart|max.*request|max.*body' api-server engine -g '*.yml' -g '*.yaml' -g '*.properties' -g '*.java'Length of output: 180588 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== entity mappings ==='
sed -n '1,220p' api-server/src/main/java/io/slice/stream/apiserver/streamer/infrastructure/entity/StreamerFollowerSnapshotEntity.java
sed -n '1,220p' api-server/src/main/java/io/slice/stream/apiserver/stream/infrastructure/entity/StreamEntity.java
echo '=== API client ==='
sed -n '1,240p' engine/src/main/java/io/slice/stream/engine/ingestion/infrastructure/apiServer/ApiServerClient.java
echo '=== controller and security path ==='
sed -n '1,160p' api-server/src/main/java/io/slice/stream/apiserver/streamer/presentation/StreamerFollowerInternalController.java
rg -n -C 5 'follower-snapshots|X-ENGINE-SECRET|internal/' api-server/src/main/java -g '*.java'
echo '=== database migrations ==='
fd -a -e sql . api-server | sort | xargs -r rg -n -i -C 5 'streamer_follower_snapshots|CREATE TABLE streams|last_follower_updated_at|follower_growth'
echo '=== RestClient construction and timeouts ==='
rg -n -C 8 'RestClient\.builder|RestClient\.Builder|setConnectTimeout|setReadTimeout|requestFactory|HttpComponents|JdkClientHttpRequestFactory' engine/src/main/java engine/src/main/resources -g '*.java' -g '*.yml'
echo '=== all source references to target query ==='
rg -n -C 4 'findDistinctStreamIdsByStartedAtAfter|findAllBySnapshotDateAndStreamIdIn' api-server/src/main/java -g '*.java'Length of output: 39761
Critical / Major 결과: 발견하지 못했습니다. 확인한 핵심 항목입니다.
후속 개선 후보도 확인했습니다. 엔진의
|
- [engine] RestClientConfig 내 apiServerRestClient 전용 JdkClientHttpRequestFactory 타임아웃 적용 (연결 5초, 읽기 15초) - [engine] 외부 치지직 API 클라이언트와의 격리를 위해 전역 Bean이 아닌 전용 팩토리 메서드로 분리하여 사이드 이펙트 차단 - [engine] RestClientConfigTest 단위 테스트 추가 및 타임아웃 격리 검증 완료
Owner
Author
|
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== PR metadata and current diff files ==='
gh pr view 97 --repo Firedrago95/stream-engine --json headRefOid,baseRefOid,updatedAt,mergeable,mergeStateStatus,statusCheckRollup --jq '{headRefOid,baseRefOid,updatedAt,mergeable,mergeStateStatus,statusCheckRollup: [.statusCheckRollup[]? | {name,conclusion,status}]}'
gh pr diff 97 --repo Firedrago95/stream-engine --name-only
echo '=== timeout-related patch ==='
gh pr diff 97 --repo Firedrago95/stream-engine --patch | awk '
/^diff --git / { show = ($0 ~ /(ApiServerClient|HttpRequestFactory|RestClient|application\.(yml|yaml|properties))/) }
show { print }
'
echo '=== candidate client source map ==='
fd -a 'ApiServerClient.java' .
ast-grep outline engine/src/main/java/io/slice/stream/engine/ingestion/infrastructure/apiServer/ApiServerClient.java --items all 2>/dev/null || true
echo '=== API client implementation and direct callers ==='
sed -n '1,280p' engine/src/main/java/io/slice/stream/engine/ingestion/infrastructure/apiServer/ApiServerClient.java
rg -n -C 5 '\b(sendFollowerSnapshots|recordFollowers)\s*\(' engine/src/main/java api-server/src/main/java -g '*.java'
echo '=== timeout construction and error handling references ==='
rg -n -C 8 'JdkClientHttpRequestFactory|setConnectTimeout|setReadTimeout|RestClient\.builder|RestClient\.create|RestClientResponseException|ResourceAccessException|HttpStatusCodeException' engine/src/main/java -g '*.java'Length of output: 36207
확인 결과,
이번 변경에서 Critical 또는 Major 문제는 발견하지 못했습니다. GitHub의 mergeability 상태가 계산 완료된 후 충돌이 없으면 머지할 수 있습니다. If you found this review helpful, would you consider giving us a shout-out on X? Thank you for using CodeRabbit! You are interacting with an AI system. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 개요 (Overview)
✨ 반영 내용 (Key Changes)
ON CONFLICT기반 단일 쿼리로 1,000건씩 일괄 저장JdkClientHttpRequestFactory)🧪 검증 결과 (Verification)
./gradlew test,npm run build성공)🚀 향후 과제 (Optional)
Summary by CodeRabbit