Spin off read/write into it's own package?

[adamgayoso]

As we are refactoring scvi, I'm wondering what the utility would be to spin off the i/o part of scanpy into it's own lightweight package that's more general for reading single cell pipeline outputs into anndata. For example, we'd like to use the scanpy read from 10x/visium functions, but don't necessarily want to have scanpy be a full requirement. It's also a bit confusing why something like read_umi_tools is in anndata but not read_10x_h5. The same goes for loom to some extent. I could imagine either moving such functionality to anndata or a standalone package that could be expanded to include support for other technologies like scATAC-seq.

This overall could be a big benefit to methods developers who would like to build on anndata.

[ivirshup]

This makes me think of a few things.

Moving 10x reading functions to anndata

Initially the idea was all single cell stuff should go into scanpy. Since we read loom into anndata, I think we'd be okay with putting the 10x readers into anndata, especially if there is broad consensus this would make development of other packages easier. However, they would need to be re-written to use h5py instead of pytables.

scanpy as a requirement

Is the requirement of scanpy so bad? If there are pain points here, should we be trying to make a basic scanpy installation lighter weight?

Splitting off new modules

Finally, the idea of having IO functions go into their own package. I think this is a much bigger change, and I'd like to see a more fleshed out case for it. This would add a fair bit of complexity to development, so I'd want to be sure it's worth it. Some general questions I have:

• What are the advantages/ disadvantages of having smaller sub-packages?
  • How does this impact users vs. developers?
• Is IO special, or should more parts go into sub-packages?
• What gets re-exported from "main" modules?
• Who manages the sub-packages?

A more specific question: how modular of a component is IO? In some cases, like reading a transcriptomic only datasets, I'd say very. For more recent developments, like visium, I'm not sure this is the case. What we read in, and how we represent it, is very tightly coupled to the methods we have. Until the data's been around for a bit longer, I think it would make sense to keep visium IO in the same package as methods for it.

[adamgayoso]

Moving 10x reading functions to anndata

I haven't worked much with h5py or tables, is it time-consuming to refactor these functions? It seems like moving to anndata is the most straightforward solution at least logically to me.

scanpy as a requirement

I like scanpy, but the only thing we really require in scvi is the data loading part. A user could take their scvi outputs and go use Seurat if that makes them happy. And then like the data loading functions are simple enough that we could just implement them ourselves. I'm sure a lot of people are currently doing this, which inspired the idea to have a standalone package.

Splitting off new modules

Your questions are very valid. I don't really have good answers for them. I could just see a standalone package being widely used and community driven, especially if there is some scanpy backing + maybe optional dependencies/functionality to get your objects ready for R analysis pipelines.

[ivirshup]

I haven't worked much with h5py or tables, is it time-consuming to refactor these functions? It seems like moving to anndata is the most straightforward solution at least logically to me.

In this case, I think it should be fine. It might not happen too soon if we're left to our own devices, so a PR is welcome.

I could just see a standalone package being widely used and community driven

What formats that aren't in anndata would you see in this package? I'm trying to get an idea of the kind of scope you're thinking of here. I think there are formats where there isn't one obvious "right way" to represent them as an AnnData object (e.g. visium), so having a canonical reading/ writing function is difficult.

[ivirshup]

pytables is starting to throw warnings, so it may be time for a rewrite and moving the function.

[LuckyMD]

I would love to see file I/O in Anndata. I imagine this would make things easier for episcanpy as well. That package can then focus more on setting up count tables where they are not nicely provided. Otherwise it becomes a bit difficult for the new user (me) to distinguish data loading and setting up new tables.

[giovp]

Splitting off new modules
Finally, the idea of having IO functions go into their own package. I think this is a much bigger change, and I'd like to see a more fleshed out case for it. This would add a fair bit of complexity to development, so I'd want to be sure it's worth it. Some general questions I have:

What are the advantages/ disadvantages of having smaller sub-packages?

method developer would just depend on those instead of (multiple) analysis package

How does this impact users vs. developers?

user none, as the analysis package would ofc have the IO as dep. developer would be impacted by a leaner dep tree

Is IO special, or should more parts go into sub-packages?

it kind of is imho, it's all about having whatever data there is in an anndata/mudata shape. I must say that I'd also think plotting could be it's own separate package but it would probably require a lot of refactoring across packages (thinking about duplication of scanpy/scvelo code)

What gets re-exported from "main" modules?

didn't get this sorry

Who manages the sub-packages?

the IO subpackage? everyone 😅

so beside being in favour, it might also be that other issues arise. For instance, for modality-specific formats we'd have to rely on specific external libraries which would then have to be lazily imported (as pointed out before). Would this create the premise of exponential growing of modality-specific lazy import libraries? probably yes. Is this best practice? I don't know.

[adamgayoso]

it kind of is imho, it's all about having whatever data there is in an anndata/mudata shape. I must say that I'd also think plotting could be it's own separate package but it would probably require a lot of refactoring across packages (thinking about duplication of scanpy/scvelo code)

@giovp and I are synced on these ideas :)

[adamgayoso]

Would this create the premise of exponential growing of modality-specific lazy import libraries? probably yes. Is this best practice? I don't know.

Could be, but even in this case, I think centralization is so so important and this package could receive a lot of community interaction....

[adamgayoso]

Here are instances that could have leveraged scio

1. PyLiger
2. Maestro
3. scvi-tools
4. scCloud
5. Simba
6. celltools
7. cellbender