`bin/crystal` vs `.build/crystal`

[straight-shoota]

I previously posted part of this in #15516 (comment), but I believe this topic deserves a dedicated discussion:

bin/crystal serves two primary purposes:

1. Run the local compiler (.build/crystal) configured with the local standard library (CRYSTAL_PATH=lib:src) and other
2. Run the system compiler (crystal; or any other compiler indicated by $CRYSTAL) configured with the local standard library (CRYSTAL_PATH=lib:src)

Both use cases are kind of merged into each other, and only on the existence of .build/crystal decides which one applies. If that file exists, it uses that and prints a notification message.

This behaviour is still not very transparent.
For example, integration specs like spec/primitives or spec/compiler-cli (proposed in #17033) should always use a local compiler build, not the system compiler. If a local compiler doesn't exist, it should be an error.
#15516 discusses this in more depth, regarding a specific CI setup.

If the local compiler would implicitly work without bin/crystal as a wrapper, it would all be more straightforward
and we could simplify the wrapper script itself as well as its integration.

Then there would be a clear separation:

• bin/crystal runs the system compiler with the local standard library.
• .build/crystal runs the local compiler (which uses the local standard library by default).

We'd have to figure out what that means for the places where we're currently use bin/crystal and its meaning depends on the state of the environment.
For example: If we run make std_spec in a clean environment, it runs bin/crystal using the system compiler. Then run make crystal to build .build/crystal. And now make std_spec runs bin/crystal using the local compiler.

It would be perhaps more clear if we would run the latter explicitly with make std_spec CRYSTAL=.build/crystal instead. Then it would implicitly fail if the local compiler is missing for whatever reason.

---

In order to achieve a stand alone .build/crystal, we need its CRYSTAL_PATH to point to src.

I don't think it would be feasible to require the caller to do that, as that would destroy all the convenience that bin/crystal provides. Unless perhaps we consider local environment variable configuration? For interactive use, this could be solved with direnv integration.

Currently, the compiler is always built with CRYSTAL_CONFIG_PATH config that's intended for a standard release package layout, where stdlib sits in $ORIGIN/../share/crystal/src (Unix) or $ORIGIN/src (Windows).
We could instead build development compilers explicitly with CRYSTAL_CONFIG_PATH=$ORIGIN/../src, and they'd automatically pick up the correct location.

However, that could cause some friction because currently we do not differentiate between builds that are meant for development vs. builds for release (the release=1 flag is just about codegen optimization and may be used for development builds as well). Changing CRYSTAL_CONFIG_PATH seems like it could lead to a foot gun, and you cannot use the same executable to test locally and then install to a different place.

As an alternative, the compiler could recognize the development environment at runtime. For example when the executable is placed in a directory named .build with a sibling called src, that sibling directory would be used as CRYSTAL_PATH.

---

Add a 👍 reaction to issues you find important.

[ysbaddaden]

The double behavior of bin/crystal is the point for working on the compiler, be it a developer or Makefile: use the locally built compiler when present and fallback to a global compiler. I see no reasons to break that.

If anything, compiler specs calling bin/crystal can be considered an abuse. Std specs calling bin/crystal is fine, they test std, not the compiler.

The spec helpers could just always target the locally built compiler (overridable with env, error when not found) and still configure the call — #compile_file handles the config and env but defaults to call bin/crystal (oops).

The locally built compiler could be configured with harcoded paths, but assuming the paths are hard coded is probably not a good idea either: with an env override, specs still need to tell where the src folder is. Better be explicit.

Preconfigured local paths in the locally built compiler would still be useful, as it would make using gdb or lldb to debug the compiler easier.

[straight-shoota]

Hm. Maybe we don't need to break the bin/crystal behaviour to call .build/crystal when available.

The main point is that I'd like to be able to call .build/crystal directly, without bin/crystal, in order to explicitly use a locally-built compiler. If that file doesn't exist, it should be a failure, not fall back to the system compiler.

[ysbaddaden]

Sounds good.