Skip to content

Rework shared memory #54

@emilyalbini

Description

@emilyalbini

The current status of shared memory is a real mess, and I really need to solve it:

First of all, you can't easily use sub-dictionaries because they're not synchronized, and you need to do something like this, which is really ugly:

mydata = shared["mydata"]
mydata["test"] = "abc"
shared["mydata"] = mydata

Other than that, shared memory is directly available via the shared argument of any hook: this might seem convenient, but prevents adding more shared things, like a global memory (as requested in #22) and locks, which are currently implemented by adding the method at runtime.

In order to solve this problems, and allowing to create custom drivers more easily, a big rewrite of the whole thing is needed. The new API I want for it is:

# For a component's memory
shared.memory["mydata"] = shared.object("dict")
shared.memory["mydata"]["test"] = "yay"
with shared.lock("hello"):
    pass

# A custom bucket
bucket = bot.shared.bucket("test")
bucket.memory["mydata"] = "test

  • Move the functionality of shared into shared.memory
  • Use a custom proxy instead of the multiprocessing one, so it's easier to write custom drivers and the API is consistent between them
    • Add basic support for objects management
    • Add ability to export/import objects and switch drivers
    • Add ability to add memory preparers
    • Add support for the runner driver
  • Add some sort of garbage collection

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions