Concept: Modules and Scripts

[rgchris]

Two points of reference for this thread: Rebol 3 Modules, and How words are bound within a module.

Also one other point of note, there is a conceptual difference between the MODULE! datatype and the Module System designed to organize code. The MODULE! datatype is just a variation on OBJECT! with some subtle differences. Whereas the Module System determines how modules intended for reuse are created from source. A lot of the complexity around modules and code reuse stem from the the Module System and it is that I am addressing here, not the MODULE! type.

Note: I will likely revise this text as and when I have time to further elaborate on the details.

---

The Module System as-is largely serves two purposes, code isolation and organization. While it may seem intuitive that these go hand in hand, they do have a couple of divergent points. One is the use of unnamed modules to create scripts that run in an isolated namespace

Unnamed Modules as Temporaries:

A module that has no name field is by convention a temporary module. Without a name, the module will not be stored within the system module list, nor will its words be added to the system export list.

The advantage of such modules is that they can be garbage collected once there are no more active references to the words or contextual values (functions, objects) of the module.

Unnamed Modules

In some cases, you really want your code run for effect. In these cases having your code rerun every time is what you want to do. Maybe it's a script that you're running with do but structuring as a module to avoid leaking words. Maybe you're making a mixin, some utility functions that have some local state that needs initializing. It could be just about anything.

I believe this to break the intent of the Module System. Type: module should only relate to code that is intended for reuse, that would be an intuitive understanding of its purpose.

Recommendation: scripts that are intended to be run this way should use Type: private

(the MODULE! type is still used in implementation, but the resultant value is discarded upon evaluation)

Example:

Rebol [
    Type: private
]

a: 123

>> do %a-private-script.reb
== 123

>> a
** Script error: a has no value

This provides a clear pathway to create scripts that are to be run for effect but run in a private namespace.

(disambiguates wish #14, no longer need #1178—exports from such a script don't make sense)

This also removes complexity about how Type: module files are processed by DO, paving the way for all Type: module scripts are stored in system/modules. Which brings me to my second recommendation:

Recommendation: system/modules becomes a MAP! allowing for modules to be stored by name, file, and/or URL

This is exactly what URLs are designed to do: unambiguously differentiate between resources. Files can similarly be resolved (e.g. to-real-file) and unambiguously refer to a module within the filesystem so that unnamed modules are not rerun (see recent thread for an example of where current behavior breaks intuitive understanding).

select system/modules reb:xml
; by internal URL

select system/modules 'xml
; by Name

select system/modules clean-path %foo.reb
; by full filepath

(mostly solves #1175, #1176, #1187)

This also allows for:

Recommendation: DO of %a-module.reb creates a module if not already run, retrieves a module if it has

There is no longer any ambiguity about how this works: DO plays nice with the module system. Modules no longer require names in order to be 'saved.' While names can still be used as a reference, they are not suited by themselves for an extensively used Module System. Per the Module definition in the old R3 Dev Wiki, Modules invoked with DO are not imported, but, they don't need to be reloaded if a subsequent IMPORT call is made.

>> type? a-module: do %a-module.reb
== #(module!)

>> same? a-module do %a-module.reb
== #(true)

>> module? my-svg: do r3:rgchris:svg
== #(true)

>> same? my-svg do r3:rgchris:svg
== #(true)

(disambiguates wish #14)

Recommendation: MIXINs become the default word exchange mechanism for modules

Exported words are no longer added to LIB by default (if ever), they are only imported into the local context. This doesn't fully answer the question of how IMPORT works outside of the NEEDS header (it may need to be localized for each module) (I may have a better answer for this). This means there's no surprises with unexpected imports from some deep part of the dependency chain; there should be some other mechanism for exporting to LIB.

Consequences:

mod-a.reb

Rebol [
    Type: module
    Exports: [func-a]
]

mod-b.reb

Rebol [
    Type: module
    Needs: [%mod-a.reb]
    Exports: [func-b]
]

func-a  ; ok, imported
func-b: ...

script.reb

Rebol [
    Needs: [%mod-b.reb]
]
func-b  ; ok, imported
func-a  ; error, not imported

other-script.reb

Rebol [
    Needs: [%mod-a.reb %mod-b.reb]
]
func-b  ; ok, imported
func-a  ; ok, imported

(addresses wish #12)

Recommendation: A separate CORE module type that adds to LIB

I think there's value in having this non-mixin functionality, but having it as a distinct TYPE allows it to be handled differently. That may be purely semantic at this point, but it could potentially be used in substantive tracing/debugging.

a-core-module.reb

Rebol [
    Type: core
    Exports: [
        something-useful
        ; this is added to LIB
    ]
]

something-useful: func [thing] [thing + 1]

Recommendation: Some alternative to SYSTEM/SCRIPT for modules

Modules should be able to access their own headers and neighboring file system.

Concept: Modules are evaluated apart from all other code.

a-module.reb

Rebol [
    Type: module
]

system/script/header
; module header

system/script/path
; module file path

system/script/parent
; none, modules are run apart from scripts

Recommendation: Remove version checking from the Module System

Versions can be specified in Files and URLs, and if needed, by helper functions. In other systems, this is mostly handled by convention (though may require a reconsideration of file vs. email lexical space):

%some-module@1.1.reb
https://example.com/some-module@2.0.reb
https://example.com/some-module@2.0
reb:some-module@3.2

Recommendation: DO of a type other than NONE, MODULE, or PRIVATE does not evaluate the script directly

This doesn't directly pertain to the Module System, but makes good on Rebol's capacity to operate in different domains. As an example, DO of a script such as the following could conceptually be evaluated in a different context:

my-svg.reb

Rebol [
    Type: svg
    Needs: [
        reb:svg
    ]
]

size 400x400
circle red 200x200 150

>> do %my-svg.reb
** Script error: no handling for Type: svg
; temporary, it should be possible for reb:svg to add a Type: svg handler

While I don't have a recommendation on how this mechanism would work at this time, this is a feature that is a part of DO and should be part of design discussions at this stage. Part of the motivation for this is in consideration of the old Desktop Rebsite concept; do %index.reb could conceivably return a processed version of a Rebsite file.

(posted on this topic long ago on the Ren-C forum: Using the Type Header)

Recommendation: rename LIB to RUNTIME

Having an essential concept have such a throwaway name isn't useful. LIB is often referred to as the Runtime Library, and RUNTIME isn't a common word in scripts. I believe this would help clarify the purpose of the Runtime Library in general.

I'd propose a renaming for SYS, but I don't have a good candidate at this time. Possibly INTERNALS:

runtime/copy/part "thing" 3

internals/make-scheme [...]

---

These recommendations are designed to streamline the Module System and make modules more consistent for script and module authors. Important to note, this isn't the last word in Modules. It's possible to add further module types that may behave differently if the need for them emerges, they just won't be Type: module

---

As an aside, I dislike the term 'mixin'—Rebol has EXPORTS and should similarly have IMPORTS. 'Mixin' is a niche term.

[Oldes]

• using map to hold modules is good idea.
• %some-module@1.1.reb is an invalid email... it is possible to modify the lexer, but then all script where it would be used would not be backwards compatible, which may be confusing... for example in my github actions I often do use older Rebol versions.
• https://example.com/some-module@2.0.reb why the .reb at tail?

[rgchris]

• Re. lexer, obviously %"some-module@1.1.reb" works. The lexer issue is beside the point—would be better handled as a separate issue. The main concern is that the file is unique.
• 'why the .reb at tail?' It doesn't have to, it really depends on how a given server is configured. I don't propose any special handling, unless carve outs make sense for modules hosted on, say, GitHub (the less special handling, the better, imo). The main concern is, again, that a URL is Unique

It's possible that conventions could be codified (like @version or others), I don't see any need for that up front. A scheme like my experimental r3: scheme is a different matter, that can be very specific.

[rebolek]

URLs aren't unique at all. Without hash you don't know if code you're reading today is the same as yesterday. You can guarantee it on local system to some level but you can't trust anything that's from web.

[rgchris]

That's not really a concern for differentiating modules, most of all of this is predicated on only referring to resources that you trust. The only worry here would be that I used a different version of the URL in my module as you do in yours and thus two versions of the same module are imported. As an example, if I'm using one of my modules and one of yours in my script, in my module, I import reb:xml and in yours, you refer to reb:xml@1.1—it may refer to the same module but it'll be loaded twice. If mixins are the default, the only significant problem here is that the same module is loaded twice. Even then, that might be considered a feature and not a bug; if the URL is different, it's more than likely that a different module is intended.

[Oldes]

The lexer was modified to allow unescaped @ in files: 3ade0c3

[rgchris]

I think that's a positive change. Whether or not a potential module system uses this convention, it does exist in other domains. It has been used to denote image density, e.g. photo@2x.jpg indicating two times the 'regular' resolution. And of course those JavaScript libraries.

[rebolek]

I agree. I have the feeling that modules in Rebol are trying to solve two problems at once that I think should be needed to treat separately - namespace handling and packaging. So removing version from modules (for now) seems like a good idea. IMO modules should be all about namespace separation and inclusion and packages from modules is a problem that needs to be solved separately.

Packaging open whole new can full of different forms. Code signing, trust issues, how to prevent left-pad issue, how to prevent dependency hell, etc, etc. We won't solve it perfectly, because if it could be solved perfectly, it would be solved already, but at least we can learn from past mistakes. So that's why I believe it's a problem that should be solved separately.

[rgchris]

I still think packaging can entirely be deferred to a scheme. My r3: scheme is intended as an unsophisticated bridge to this approach, but another scheme could be developed that does full checking and management.

[rebolek]

Cool, I take a look. It's a very interesting area for me.

[Oldes]

If it helps, I have collected some old DocBase documents.. here is the one related to modules.

[rgchris]

Indeed, I have a mirror of DocBase as well. Somewhere I have the final backup of the MediaWiki database.

[Oldes]

Interesting... I had to parse it from the WebArchive manually.
Btw.. it is not possible to clone your wiki (at least on Windows) because use of file names with colons (:)... like:

error: invalid path 'Ports: Error Handling.mediawiki'
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.

But I may clone it from WSL

[rgchris]

I had no idea! I can update before you clone if you like.

[Oldes]

It would be nice to have some standard way for modules removal... for example now, when I want to test new console code update, I must first remove existing modules manually like here.

[rebolek]

That would be great. I usually comment type: module when testing something and enable it when done. But then new errors related to module behaviour would show up, so it's definitely not a good way.

[rgchris]

Some considerations:

• If mixins are the default, imported words will no longer be added to LIB, so no need to remove
• Removing modules has a whole layer of issues, not least the point of them being an isolated, stable, dependable code chain

I think a different concept is needed altogether, could call it 'development-module where you get the isolation and importability of modules, but it simply isn't added to system/modules thus is always reloaded if imported again. It could be that they are stored elsewhere in such a way that they share bindings with predecessors so that any exports are updated when the module is reloaded. This would be quite an unstable mechanism, but I think the risk would be implied by the concept name