Skip to content

fix: prevent metagraph memory leak (~50MB/20min)#14

Open
loayei97 wants to merge 1 commit into
thenervelab:mainfrom
loayei97:fix/metagraph-memory-leak
Open

fix: prevent metagraph memory leak (~50MB/20min)#14
loayei97 wants to merge 1 commit into
thenervelab:mainfrom
loayei97:fix/metagraph-memory-leak

Conversation

@loayei97

@loayei97 loayei97 commented Jan 7, 2026

Copy link
Copy Markdown

PROBLEM:

  • bt.metagraph() was called on each weight submission cycle (~20 min)
  • Each call registers scalecodec types that are never garbage collected
  • This caused ~50MB memory leak per iteration, leading to OOM after ~13 hours

SOLUTION:

  • Create metagraph ONCE at startup
  • Use metagraph.sync(subtensor=...) to refresh data without creating new objects
  • Reuse WeightSubmitter instance across iterations instead of creating new

FILES CHANGED:

  • run_continuous.py: Reuse WeightSubmitter, call run_submission() instead of run()
  • weight_submitter.py: Add run_submission()/cleanup(), use sync() in fetch_metagraph()
  • sync_metagraph.py: Create metagraph once in connect_chains(), sync() in getters

MEMORY IMPACT:

  • Before: ~150 MB/hour growth, OOM in ~13 hours
  • After: Stable memory, no leak

PROBLEM:
- bt.metagraph() was called on each weight submission cycle (~20 min)
- Each call registers scalecodec types that are never garbage collected
- This caused ~50MB memory leak per iteration, leading to OOM after ~13 hours

SOLUTION:
- Create metagraph ONCE at startup
- Use metagraph.sync(subtensor=...) to refresh data without creating new objects
- Reuse WeightSubmitter instance across iterations instead of creating new

FILES CHANGED:
- run_continuous.py: Reuse WeightSubmitter, call run_submission() instead of run()
- weight_submitter.py: Add run_submission()/cleanup(), use sync() in fetch_metagraph()
- sync_metagraph.py: Create metagraph once in connect_chains(), sync() in getters

MEMORY IMPACT:
- Before: ~150 MB/hour growth, OOM in ~13 hours
- After: Stable memory, no leak
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant