Conversation
P1: 多级 rowSlicePool 内存优化
- 将单级固定 128 容量的 rowSlicePool 改为 4 级 size-class 池
- Level 0: 1-8 列,Level 1: 9-32 列,Level 2: 33-128 列,Level 3: 129-256 列
- Fallback: >256 列直接分配(带告警)
- 零长度预分配设计:make([]interface{}, 0, capacity) 避免未使用元素浪费
- 小表场景内存利用率提升 70-90%
P2: 无锁进度聚合(锁竞争优化)
- 使用 buffered channel + 专用消费者 goroutine 替代 mutex
- Channel buffer 大小 = concurrency(默认 10)
- 满时丢弃策略:select { default: } 不阻塞工作 goroutine
- 基准测试:51 倍速度提升(9155ns → 178ns),96% 内存减少
基准测试结果:
- Mutex 方案:9155 ns/op, 3059 B/op, 69 allocs/op
- Channel 方案:178 ns/op, 80 B/op, 10 allocs/op
文件变更:
- internal/postgres/connection.go: 多级 rowSlicePool + getRowSlice/putRowSlice
- internal/postgres/connection_test.go: P1 基准测试
- internal/converter/postgres/sync_data.go: 无锁进度聚合 + displayProgressNoLock
- internal/converter/postgres/sync_data_test.go: P2 基准测试(新建)
- internal/converter/postgres/manager.go: 传递 progressChan
- 添加多级行切片池(4 级 size-class 内存池)说明 - 添加无锁进度聚合(51 倍速度提升)说明 - 更新批量处理大小从 10,000 行到 50,000 行 - 同步更新英文版和中文版 README 性能优化详情: - 小表场景内存分配减少 70-90% - 进度更新速度提升 51 倍(9155ns → 178ns) - 内存减少 96%
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
P1: 多级 rowSlicePool 内存优化
P2: 无锁进度聚合(锁竞争优化)
基准测试结果:
文件变更: