Proposal: change how generated modules are named and nested by default

[dicej]

Currently, if you tell componentize-py to target a world that imports the interface foo:bar/baz@0.1.0, it will generate a Python module with one of the following names by default, depending on what other interfaces are included in that world:

• wit_world.imports.baz if there are no other interfaces named baz referenced by the world
• wit_world.imports.foo_bar_baz if there is at least one other interface named baz (e.g. x:y/baz@0.4.0) referenced by the world, but none in the same foo:bar package
• wit_world_imports.foo_bar_baz_0_1_0 if there is at least one other interface named baz in the foo:bar package with a different version (e.g. foo:bar/baz@0.2.0)

In other words, the generated module will be given a short name if unambiguous, but a longer, more specific name otherwise. The advantage of this approach is that it's concise for simple cases. The main disadvantage is that, if the world is changed later to add new (versions of) interfaces, previously-unambiguous names may become ambiguous, requiring changes even for code which isn't using the new interfaces. That's especially painful for library packages, turning semver-compatible, purely additive WIT changes into semver-breaking library changes. It also creates a name clash hazard if code is generated for multiple worlds using multiple componentize-py bindings commands.

Considering similar tools for other languages:

• ComponentizeJS always generates full names for modules, thereby avoiding such issues.
• componentize-go generates mostly full names, always including the namespace and package name, but omits the version number if unambiguous unless the --include-versions option is specified, so it still has the same hazard, although only with respect to versions, not package names.
• The Rust wit-bindgen nests the interface module in namespace and package modules, but omits version numbers if unambiguous, similar to componentize-go.

Personally, I think ComponentizeJS makes the right choice here, and we should update the tools for other languages to match. In the case of componentize-py, that would mean making the --full-names option the default and either adding a --short-names option to turn it off, or not support turning it off (but still allow overriding the names on a case-by-case basis using the --import-interface-name and --export-interface-name options).

In addition, I'd like to propose that instead of placing the modules generated for each interface under wit_world.imports.* by default, make them top-level modules. That would more closely match the ComponentizeJS experience, where you use e.g. import "foo:bar/baz@0.1.0". In Python, we'd do import foo_bar_baz_0_1_0 (or import foo_bar_baz_0_1_0 as baz if desired). We'd still use wit_world for world-level types and functions, but stop using it for interface-level ones. Also, we'd still put code generated for exports under an exports (bikeshed-able) module to avoid clashes when the same interface is both imported and exported.

For example, consider the following WIT world:

package foo:bar@0.1.0;

interface baz {
  wow: func();
}

world my-world {
  import baz;
  export baz;
  import yay: func();
}

Today componentize-py bindings would generate the following modules by default:

• wit_world.exports: contains a Baz abstract base class and baz submodule representing the exported interface
• wit_world.imports.baz: contains the wow function from the imported interface
• wit_world: contains the yay function imported at the world level

I'm proposing that changes to:

• exports: contains a FooBarBaz010 abstract base class and foo_bar_baz_0_1_0 submodule representing the exported interface
• foo_bar_baz_0_1_0: contains the wow function from the imported interface
• wit_world: contains the yay function imported at the world level

Thoughts? Opinions? Bikeshed colors?

[ricochet]

Strong agree on full names as the default and I also lean towards not supporting short-names at all.

On top-level modules: The wit_world.imports prefix acts as a synthetic namespace, sort of like what we had with cargo component bindings. I liked having a place to discover bindings (lived off of cargo-expand forever) and suspect the same will be true in this ecosystem. I do like the symmetry of bindings for imports/exports today. I understand why you wouldn't want a top-level imports. since then you'd have a stutter import imports.baz. So only soft push-back on this part of the proposal.

Couple other references I peaked at:

• Protobuf: package foo.bar => foo/bar/baz_pb2.py in Python. The buf styleguide recommends encoding the version in the package name, e.g. foo.bar.v1 will be foo_bar_v1/baz.py. An actual import looks like from connectrpc.eliza.v1.eliza_pb2 import SayRequest. So folks are very used to seeing the version in the import.
• openapi-generator: you specify a top-level package name and everything nests under it, e.g. openapi_client.models.pet, openapi_client.api.pets_api.

For both of those, I like that they have a synthetic root.

So if I were painting this shed, I'd probably opt for something very close to what you're proposing:

my_world.foo_bar_baz_0_1_0   # imported interface
my_world.exports.foo_bar_baz_0_1_0  # exported interface
my_world  # world-level functions

[dicej]

@ricochet Thanks for referencing the buf styleguide and openapi generator; definitely makes sense to follow an existing pattern rather than make up something new. Maybe we should follow the same scheme as buf, e.g. foo.bar.v0_1_0.baz given that it's already familiar to at least some programmers.

BTW, if you're wondering why we currently use wit_world as the root module name rather than the original WIT world name, see #149. Also, componentize-py supports providing any number of worlds in a single bindings or componentize subcommand, in which case there's not one single world name we can use. Finally, world names don't have any real meaning at the component model level (i.e. they don't appear anywhere in a generated component), so we've tended to leave them out of the generated source code.

[dicej]

Also, I tend to agree that having everything under a single, top-level module is tidy (which is why I did it that way to start with), but I've also received feedback that folks like the ComponentizeJS style of imports and would like to see something similar in Python. I could go either way, personally.

[ricochet]

Makes sense, thank you for the background. I think buf is the closest analog but they also don't support multiple simultaneous versions. They have their version in their definition as a style convention but it doesn't really have a practical impact.

Given that, foo.bar.v0_1_0.baz does seem like the most familiar. I don't feel super strongly. In addition to avoiding the issues you listed that ComponentizeJS style avoids, I think folks like ComponentizeJS style because it directly looks like what they see in WIT, e.g. from 'wasi:http/types@0.2.4'. I don't think that's something on the table here for python right? Also in poking about the ecosystem, I think that would also be pretty unfamiliar to end-users.

Couple other references:

• Just looked at the code gen for FastMCP where it will generate tool definitions from openapi. from mcp_petstore.tools.pets import list_pets, add_pet and from mcp_petstore.models.base import Pet. Only key learning there is that they fold the generated code under tools and models. Not the greatest reference for us.
• Smithy looks like com.amazonaws.s3#ListObjectsV2Request with generated binding as mypy_boto3_s3.type_defs.ListObjectsV2RequestTypeDef. Versioning is handled via @SInCE traits

[karthik2804]

As someone who like the way ComponentizeJS bindings look, I do like the foo.bar.v0_1_0.baz approach as it is close enough to WIT. I also like seperate modules for each namespace:package_name instead of a singular top level module.

For exports, we could do something similar like foo.bar.v0_1_0.exports.baz (?). Maybe this is not too discoverable?

For world level stuff I do not have strong opinions but the way ComponentizeJS does it is import greeter from 'greeter' which is something we could do in python but I am not sure if it is idiomatic or good practice in python.

[lukewagner]

My 2c:

+1 to never using world names in generated bindings b/c world names are semantically insignificant after bindgen and meant to be unioned/intersected at any stage; only the import/export (interface and plain) names are significant.

+1 to using a single synthetic top-level module for all WIT-generated bindings (maybe wit?). When we considered the same question in ComponentizeJS, the key difference was that there there are only a few namespace:s (node:, deno:, ...) added by runtimes, and they don't include, e.g., stdlib module and package names, so there seems like less of a collision hazard requiring a unique prefix.

+1 to putting versions on packages, not interfaces. CM/#573 suggests switching to wasi:http@0.3/types which does seem like a good idea.

For versions: I'd suggest considering using just the canonical version, which means chopping off the "minor" info (so e.g. _v1, _v0_1 or _v0_0_1), so that semver-compatible upgrades don't require bumping a bunch of versions unnecessarily. Not sure there are any extra constraints we have with bindgen, but at least for components, this seemed like the way to go to avoid having to build in a bunch of semver logic into a bunch of places (which hasn't been fully implemented yet, although @chenyan2002 is working on it).

For exports: instead of using inheritance as the signal, what about a @decorator attached to the class implementing the interface? This seems a little closer to an export keyword and to what ComponentizeJS is doing. So maybe:

@wit.export.foo_bar_baz_v1
class WhateverNameIWant:
  pass

and then for direct function exports (no interface), decorators can also be attached directly to functions:

@wit.export.foo_v1
def whatever_name_i_want():
  pass

and maybe, as an optional convenience, when the Python class/function name matches the export name:

@wit.export
class FooBarBaz_v1
  pass

@wit.export
def foo_v1():
  pass

? If we're syntactically scanning, decorators seem as-easy-if-not-easier to find. Or, if we're using dynamic Python semantics (I assume a syntactic scan will be approximate and unsound?), then the @wit.export decorators (which are functions defined by wit-bindgen) could dynamically register the given class/def (failing if there is already one registered) and then the final list could be written as output (perhaps during wizening) for use by the later stages of componentization.

[dicej]

For exports: instead of using inheritance as the signal, what about a @decorator attached to the class implementing the interface?

Small correction: we currently look for classes with a specific names -- not inheritance -- to resolve exports. Inheritance from the generated abstract base class is just an optional step to assist with linting.

But yeah, using decorators for this instead sounds good to me.

[brettcannon]

Any reason it isn't wit_world.imports.foo.bar.baz?

[lann]

Any reason it isn't wit_world.imports.foo.bar.baz?

Could you clarify what you're responding to here?

[brettcannon]

Any reason it isn't wit_world.imports.foo.bar.baz?

Could you clarify what you're responding to here?

The issue is about naming and how to handle various cases. I'm asking whether using Python's namespace support was considered as I think it would render this issue moot.