Skip to content

Add compiler detection of adjacent stdlib in dev folder #17071

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
straight-shoota wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Add compiler detection of adjacent stdlib in dev folder #17071

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
26254d3
Detect development use from executable path
straight-shoota Jun 4, 2026
cc884b1
Call `.build/crystal` directly instead of wrapper
straight-shoota Jun 4, 2026
57498a7
Do not set `CRYSTAL_EXEC_PATH` when using wrapper with local compiler
straight-shoota Jun 4, 2026
7d67736
Use `.build/crystal` in Windows CI
straight-shoota Jun 4, 2026
d428959
Makefile: Use `.build/crystal` directly for `primitive_spec`
straight-shoota Jun 4, 2026
68d92bd
Run `make -C samples` directly instead of `make samples`
straight-shoota Jun 14, 2026
0292d0d
Re-add `CRYSTAL_SPEC_COMPILER_BIN` in `win.yml`
straight-shoota Jun 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 3 additions & 6 deletions .github/workflows/mingw-w64-steps.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,20 +146,17 @@ jobs:
- name: Run compiler specs
shell: msys2 {0}
run: |
export CRYSTAL_SPEC_COMPILER_BIN="$(which crystal.exe)"
make compiler_spec
make compiler_spec CRYSTAL=.build/crystal

- name: Run interpreter specs
shell: msys2 {0}
run: |
export CRYSTAL_SPEC_COMPILER_BIN="$(which crystal.exe)"
make interpreter_spec
make interpreter_spec CRYSTAL=.build/crystal

- name: Run primitives specs
shell: msys2 {0}
run: |
export CRYSTAL_SPEC_COMPILER_BIN="$(which crystal.exe)"
make -o .build/crystal.exe primitives_spec # we know the compiler is fresh; do not rebuild it here
make -o .build/crystal.exe primitives_spec CRYSTAL=.build/crystal.exe # we know the compiler is fresh; do not rebuild it here
env:
SPEC_FLAGS: --tag=~external_commands # skip exec_external_command spec because it doesn't work with this setup

Expand Down
10 changes: 4 additions & 6 deletions .github/workflows/win.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ jobs:

- name: Set up environment
run: |
Add-Content $env:GITHUB_PATH "$(pwd)\build"
Add-Content $env:GITHUB_ENV "CRYSTAL=$(pwd)\build\crystal.exe"
Add-Content $env:GITHUB_ENV "CRYSTAL_SPEC_COMPILER_BIN=$(pwd)\build\crystal.exe"
Add-Content $env:GITHUB_ENV "LLVM_VERSION=${{ env.CI_LLVM_VERSION }}"
Add-Content $env:GITHUB_ENV "LLVM_TARGETS=${{ env.CI_LLVM_TARGETS }}"
Expand All @@ -262,7 +262,7 @@ jobs:
run: make -f Makefile.win cli_spec

- name: Build samples
run: make -f Makefile.win samples
run: make -f Makefile.win -C samples

test-std:
runs-on: windows-2025-vs2026
Expand Down Expand Up @@ -295,8 +295,7 @@ jobs:

- name: Set up environment
run: |
Add-Content $env:GITHUB_PATH "$(pwd)\build"
Add-Content $env:GITHUB_ENV "CRYSTAL_SPEC_COMPILER_BIN=$(pwd)\build\crystal.exe"
Add-Content $env:GITHUB_ENV "CRYSTAL=$(pwd)\build\crystal.exe"
Add-Content $env:GITHUB_ENV "LLVM_VERSION=${{ env.CI_LLVM_VERSION }}"
Add-Content $env:GITHUB_ENV "LLVM_TARGETS=${{ env.CI_LLVM_TARGETS }}"
Add-Content $env:GITHUB_ENV "LLVM_LDFLAGS=${{ env.CI_LLVM_LDFLAGS }}"
Expand Down Expand Up @@ -327,8 +326,7 @@ jobs:

- name: Set up environment
run: |
Add-Content $env:GITHUB_PATH "$(pwd)\build"
Add-Content $env:GITHUB_ENV "CRYSTAL_SPEC_COMPILER_BIN=$(pwd)\build\crystal.exe"
Add-Content $env:GITHUB_ENV "CRYSTAL=$(pwd)\build\crystal.exe"
Add-Content $env:GITHUB_ENV "LLVM_VERSION=${{ env.CI_LLVM_VERSION }}"
Add-Content $env:GITHUB_ENV "LLVM_TARGETS=${{ env.CI_LLVM_TARGETS }}"
Add-Content $env:GITHUB_ENV "LLVM_LDFLAGS=${{ env.CI_LLVM_LDFLAGS }}"
Expand Down
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,7 @@ $(O)/compiler_spec$(EXE): $(DEPS) $(SOURCES) $(SPEC_SOURCES)

$(O)/primitives_spec$(EXE): $(O)/$(CRYSTAL_BIN) $(DEPS) $(SOURCES) $(SPEC_SOURCES)
@mkdir -p $(O)
$(EXPORT_CC) ./bin/crystal build $(FLAGS) $(SPEC_WARNINGS_OFF) -o $@ spec/primitives_spec.cr
$(EXPORT_CC) ./$(O)/crystal build $(FLAGS) $(SPEC_WARNINGS_OFF) -o $@ spec/primitives_spec.cr

$(O)/cli_spec$(EXE): $(O)/$(CRYSTAL_BIN) $(DEPS) $(SOURCES) $(SPEC_SOURCES)
@mkdir -p $(O)
Expand Down
2 changes: 1 addition & 1 deletion Makefile.win
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ $(O)\compiler_spec.exe: $(DEPS) $(SOURCES) $(SPEC_SOURCES)

$(O)\primitives_spec.exe: $(O)\crystal.exe $(DEPS) $(SOURCES) $(SPEC_SOURCES)
@$(call MKDIR,"$(O)")
.\bin\crystal build $(FLAGS) $(SPEC_WARNINGS_OFF) -o "$@" spec\primitives_spec.cr
$(O)\crystal build $(FLAGS) $(SPEC_WARNINGS_OFF) -o "$@" spec\primitives_spec.cr

$(O)\cli_spec.exe: $(O)\crystal.exe $(DEPS) $(SOURCES) $(SPEC_SOURCES)
@$(call MKDIR,"$(O)")
Expand Down
21 changes: 11 additions & 10 deletions bin/ci
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,28 +106,29 @@ build() {
case $ARCH in
i386)
with_build_env 'make crystal threads=1'
with_build_env 'SPEC_SPLIT="0%4" make std_spec threads=1 junit_output=.junit/std_spec.0.xml'
with_build_env 'SPEC_SPLIT="1%4" make std_spec threads=1 junit_output=.junit/std_spec.1.xml'
with_build_env 'SPEC_SPLIT="2%4" make std_spec threads=1 junit_output=.junit/std_spec.2.xml'
with_build_env 'SPEC_SPLIT="3%4" make std_spec threads=1 junit_output=.junit/std_spec.3.xml'
with_build_env 'SPEC_SPLIT="0%4" make CRYSTAL=.build/crystal std_spec threads=1 junit_output=.junit/std_spec.0.xml'
with_build_env 'SPEC_SPLIT="1%4" make CRYSTAL=.build/crystal std_spec threads=1 junit_output=.junit/std_spec.1.xml'
with_build_env 'SPEC_SPLIT="2%4" make CRYSTAL=.build/crystal std_spec threads=1 junit_output=.junit/std_spec.2.xml'
with_build_env 'SPEC_SPLIT="3%4" make CRYSTAL=.build/crystal std_spec threads=1 junit_output=.junit/std_spec.3.xml'

parts=16
i=0
while [ $i -lt $parts ]; do
with_build_env "CRYSTAL_SPEC_COMPILER_THREADS=1 SPEC_SPLIT=\"$i%$parts\" make compiler_spec threads=1 junit_output=.junit/compiler_spec.$i.xml"
with_build_env "CRYSTAL_SPEC_COMPILER_THREADS=1 SPEC_SPLIT=\"$i%$parts\" make CRYSTAL=.build/crystal compiler_spec threads=1 junit_output=.junit/compiler_spec.$i.xml"
i=$((i + 1))
done

with_build_env 'make primitives_spec threads=1 junit_output=.junit/primitives_spec.xml'
with_build_env 'make docs threads=1'
with_build_env 'make CRYSTAL=.build/crystal primitives_spec threads=1 junit_output=.junit/primitives_spec.xml'
with_build_env 'make CRYSTAL=.build/crystal docs threads=1'
;;
*)
with_build_env 'make crystal primitives_spec std_spec compiler_spec docs threads=1 junit_output=.junit/spec.xml DOCS_OPTIONS="--json-config-url=/api/versions.json --canonical-base-url=https://crystal-lang.org/api/latest/"'
with_build_env 'make crystal'
with_build_env 'make CRYSTAL=.build/crystal primitives_spec std_spec compiler_spec docs threads=1 junit_output=.junit/spec.xml DOCS_OPTIONS="--json-config-url=/api/versions.json --canonical-base-url=https://crystal-lang.org/api/latest/"'
;;
esac

with_build_env 'make samples'
with_build_env 'CRYSTAL_OPTS=--debug make samples'
with_build_env 'make -C samples CRYSTAL=../.build/crystal'
with_build_env 'CRYSTAL_OPTS=--debug make -C samples CRYSTAL=../.build/crystal'
}

format() {
Expand Down
10 changes: 5 additions & 5 deletions bin/crystal
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,17 +185,17 @@ case "$(uname -s)" in
;;
esac

# CRYSTAL_EXEC_PATH determines the location of the `crystal` program for external
# compiler commands.
CRYSTAL_EXEC_PATH="$SCRIPT_ROOT"
export CRYSTAL_EXEC_PATH

if [ -x "$CRYSTAL_DIR/${CRYSTAL_BIN}" ]; then
__warning_msg "Using compiled compiler at ${CRYSTAL_DIR#"$PWD/"}/${CRYSTAL_BIN}"
exec "$CRYSTAL_DIR/${CRYSTAL_BIN}" "$@"
elif (! $PARENT_CRYSTAL_EXISTS); then
__error_msg 'You need to have a crystal executable in your path! or set CRYSTAL env variable'
exit 1
else
# CRYSTAL_EXEC_PATH determines the location of the `crystal` program for external
# compiler commands.
CRYSTAL_EXEC_PATH="$SCRIPT_ROOT"
export CRYSTAL_EXEC_PATH

exec "$PARENT_CRYSTAL" "$@"
fi
2 changes: 1 addition & 1 deletion spec/manual/find_executable_spec.cr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ describe "Process.run" do
src_fn = test_dir / "self_printer.cr"
exe_fn = test_dir / "self_printer.exe"
File.write(src_fn, "print #{name.inspect}")
Process.run(ENV["CRYSTAL_SPEC_COMPILER_BIN"]? || "bin/crystal", ["build", "-o", exe_fn.to_s, src_fn.to_s])
Process.run(ENV["CRYSTAL_SPEC_COMPILER_BIN"]? || File.join(".build", "crystal"), ["build", "-o", exe_fn.to_s, src_fn.to_s])
Dir.mkdir_p((base_dir / name).parent)
File.rename(exe_fn, base_dir / name)
end
Expand Down
12 changes: 12 additions & 0 deletions src/compiler/crystal/crystal_path.cr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ module Crystal
def self.default_paths : Array(String)
if path = ENV["CRYSTAL_PATH"]?
path_array = path.split(Process::PATH_DELIMITER, remove_empty: true)
elsif dev_path = dev_path?
path_array = [DEFAULT_LIB_PATH, dev_path]
Comment on lines +24 to +25

@ysbaddaden ysbaddaden Jun 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced. This would be hardcoded into every compiler while this is a configuration issue, and it's already handled by CRYSTAL_CONFIG_PATH and CRYSTAL_PATH.

I think the makefiles should export CRYSTAL_CONFIG_PATH=$PWD/src:lib (when unset), and call sites should just configure CRYSTAL_PATH. Some helpers calling to crystal already do it.

@straight-shoota straight-shoota Jun 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CRYSTAL_CONFIG_PATH is for configuring the stdlib source path for packaging. We could customize it for development builds to CRYSTAL_CONFIG_PATH=$$ORIGIN/../src, but then it wouldn't work when installed at /usr/share/crystal/src, for example. We'd need separate configuration for production builds.
You can either have a compiler build that picks up the local path, or one that is ready to be installed in a system path (or wherever).
Maybe, that's fine. But right now you can build a compiler, test it (with this patch, using .build/crystal), and then install it. The automatic discover just works.

Without automatic discovery, .build/crystal would not have any usable default configuration for CRYSTAL_PATH. The CRYSTAL_CONFIG_PATH value doesn't work because it's not in an install location yet.

Requiring explicit configuration is much more inconvenient compared to bin/crystal.

@ysbaddaden ysbaddaden Jun 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if anybody's using the install target to be honest. Distributions merely set CRYSTAL_CONFIG_PATH (e.g. alpine, debian, homebrew, ...).

@straight-shoota straight-shoota Jun 16, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does that matter? I don't think the usage makes much difference whether you run make install or install .build/crystal $out/bin/crystal 🤷

@ysbaddaden ysbaddaden Jun 18, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, if nobody is using the install target and instead configure CRYSTAL_CONFIG_PATH, then there's little point for the Makefile to keep using $$ORIGIN/../share/crystal/src by default.

@straight-shoota straight-shoota Jun 18, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think they are two independent dimensions: Using make install and relying on the default CRYSTAL_CONFIG_PATH are not directly related.
There are use cases involving either one or both of them.

elsif path = Crystal::Config.path.presence
path_array = path.split(Process::PATH_DELIMITER, remove_empty: true)
unless path_array.includes?(DEFAULT_LIB_PATH)
Expand All @@ -35,6 +37,16 @@ module Crystal
path_array
end

def self.dev_path? : String?
exe_path = Process.executable_path || return
dir_path = ::Path[exe_path].parent
return unless dir_path.basename == ".build"

src_path = dir_path.parent.join("src").to_s
return unless Dir.exists?(src_path)
src_path
end

def self.default_path : String
default_paths.join(Process::PATH_DELIMITER)
end
Expand Down
Loading