Skip to content

fix: add SQLite busy_timeout and WAL autocheckpoint to prevent CPU 100% epoll busy wait (Closes #8056) #8111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Pleiades1726 wants to merge 1 commit into from
+17 −5
Closed

fix: add SQLite busy_timeout and WAL autocheckpoint to prevent CPU 100% epoll busy wait (Closes #8056) #8111

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions astrbot/core/db/sqlite.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@ async def initialize(self) -> None:
await conn.execute(text("PRAGMA cache_size=20000"))
await conn.execute(text("PRAGMA temp_store=MEMORY"))
await conn.execute(text("PRAGMA mmap_size=134217728"))
await conn.execute(text("PRAGMA busy_timeout=5000"))
await conn.execute(text("PRAGMA wal_autocheckpoint=500"))
Comment on lines +59 to +60
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot May 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

PRAGMA busy_timeout=5000 (5秒) 与 BaseDatabase 构造函数中设置的 timeout=30 (30秒) 不一致。执行此 PRAGMA 会覆盖连接级别的设置。建议统一使用 30000ms 以保持一致性，确保在高负载下有足够的等待时间避免 SQLITE_BUSY 错误。

此外，这套 SQLite 优化参数在多个文件（sqlite.py, document_storage.py, kb_db_sqlite.py）中重复出现。根据通用规则，建议将其重构为共享的助手函数以提高代码复用性和可维护性。

Suggested change
await conn.execute(text("PRAGMA busy_timeout=5000"))
await conn.execute(text("PRAGMA wal_autocheckpoint=500"))
await conn.execute(text("PRAGMA busy_timeout=30000"))
await conn.execute(text("PRAGMA wal_autocheckpoint=500"))
References
  1. When implementing similar functionality for different cases, refactor the logic into a shared helper function to avoid code duplication.

await conn.execute(text("PRAGMA optimize"))
# 确保 personas 表有 folder_id、sort_order、skills 列（前向兼容）
await self._ensure_persona_folder_columns(conn)
Expand Down
11 changes: 11 additions & 0 deletions astrbot/core/db/vec_db/faiss_impl/document_storage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ def __init__(self, db_path: str) -> None:
async def initialize(self) -> None:
"""Initialize the SQLite database and create the documents table if it doesn't exist."""
await self.connect()
await self._apply_pragma()
async with self.engine.begin() as conn: # type: ignore
Comment on lines +62 to 63
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot May 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

在此处单独调用 _apply_pragma() 会导致在主初始化块之前额外开启和关闭一次数据库连接。建议将 PRAGMA 设置逻辑合并到下方的 engine.begin() 块中，以减少不必要的连接开销。

# Create tables using SQLModel
await conn.run_sync(BaseDocModel.metadata.create_all)
Expand Down Expand Up @@ -197,13 +198,23 @@ async def connect(self) -> None:
self.DATABASE_URL,
echo=False,
future=True,
connect_args={"timeout": 30},
Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai Bot May 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Clarify whether both connect_args["timeout"] and PRAGMA busy_timeout are needed.

You’re now setting the lock wait in two places: connect_args={"timeout": 30} (seconds) and PRAGMA busy_timeout=5000 (milliseconds). Consider choosing one mechanism or clearly aligning/documenting the values so the effective timeout is unambiguous.

Suggested implementation:

                self.DATABASE_URL,
                echo=False,
                future=True,
                # timeout is in seconds; aligned with PRAGMA busy_timeout (30_000 ms)
                connect_args={"timeout": 30},
            )
            await conn.execute(text("PRAGMA journal_mode=WAL"))
            await conn.execute(text("PRAGMA synchronous=NORMAL"))
            # 30_000 ms = 30 seconds; aligned with connect_args["timeout"]
            await conn.execute(text("PRAGMA busy_timeout=30000"))

)
self.async_session_maker = sessionmaker(
self.engine, # type: ignore
class_=AsyncSession,
expire_on_commit=False,
) # type: ignore

async def _apply_pragma(self) -> None:
"""Apply SQLite PRAGMAs for performance and concurrency."""
async with self.engine.begin() as conn:
await conn.execute(text("PRAGMA journal_mode=WAL"))
await conn.execute(text("PRAGMA synchronous=NORMAL"))
await conn.execute(text("PRAGMA busy_timeout=5000"))
await conn.execute(text("PRAGMA wal_autocheckpoint=500"))
await conn.commit()
Comment on lines +209 to +216
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot May 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

此新方法存在以下问题：

  1. 事务冲突：PRAGMA journal_mode=WAL 无法在事务内修改。self.engine.begin() 会自动开启事务，这可能导致该命令失效或被忽略。建议改用 self.engine.connect() 或在非事务环境下执行。
  2. 不一致性：busy_timeout=5000 与第 201 行设置的 timeout=30 冲突。建议统一使用 30000。
  3. 冗余代码：在 engine.begin() 块内调用 await conn.commit() 是多余的，因为该上下文管理器在退出时会自动提交。
Suggested change
async def _apply_pragma(self) -> None:
"""Apply SQLite PRAGMAs for performance and concurrency."""
async with self.engine.begin() as conn:
await conn.execute(text("PRAGMA journal_mode=WAL"))
await conn.execute(text("PRAGMA synchronous=NORMAL"))
await conn.execute(text("PRAGMA busy_timeout=5000"))
await conn.execute(text("PRAGMA wal_autocheckpoint=500"))
await conn.commit()
async def _apply_pragma(self) -> None:
"""Apply SQLite PRAGMAs for performance and concurrency."""
async with self.engine.connect() as conn:
await conn.execute(text("PRAGMA journal_mode=WAL"))
await conn.execute(text("PRAGMA synchronous=NORMAL"))
await conn.execute(text("PRAGMA busy_timeout=30000"))
await conn.execute(text("PRAGMA wal_autocheckpoint=500"))


@asynccontextmanager
async def get_session(self):
"""Context manager for database sessions."""
Expand Down
9 changes: 4 additions & 5 deletions astrbot/core/knowledge_base/kb_db_sqlite.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ def __init__(self, db_path: str | None = None) -> None:
echo=False,
pool_pre_ping=True,
pool_recycle=3600,
connect_args={"timeout": 30},
)

# 创建会话工厂
Expand All @@ -68,15 +69,13 @@ async def initialize(self) -> None:
async with self.engine.begin() as conn:
# 创建所有知识库相关表
await conn.run_sync(BaseKBModel.metadata.create_all)

# 配置 SQLite 性能优化参数
# 配置 SQLite 参数以优化并发性能
await conn.execute(text("PRAGMA journal_mode=WAL"))
await conn.execute(text("PRAGMA synchronous=NORMAL"))
await conn.execute(text("PRAGMA cache_size=20000"))
await conn.execute(text("PRAGMA busy_timeout=5000"))
await conn.execute(text("PRAGMA wal_autocheckpoint=500"))
await conn.execute(text("PRAGMA temp_store=MEMORY"))
Comment on lines +76 to 78
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot May 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

busy_timeout 的值 (5000ms) 与第 45 行设置的 timeout=30 不一致。此外，本次修改删除了原有的 PRAGMA mmap_size 和 PRAGMA optimize；这些设置对 SQLite 性能（尤其是知识库这类大数据量场景）有显著帮助，建议保留。

Suggested change
await conn.execute(text("PRAGMA busy_timeout=5000"))
await conn.execute(text("PRAGMA wal_autocheckpoint=500"))
await conn.execute(text("PRAGMA temp_store=MEMORY"))
await conn.execute(text("PRAGMA busy_timeout=30000"))
await conn.execute(text("PRAGMA wal_autocheckpoint=500"))
await conn.execute(text("PRAGMA temp_store=MEMORY"))
await conn.execute(text("PRAGMA mmap_size=134217728"))
await conn.execute(text("PRAGMA optimize"))

await conn.execute(text("PRAGMA mmap_size=134217728"))
await conn.execute(text("PRAGMA optimize"))
await conn.commit()

self.inited = True

Expand Down
Loading