ERTS doctests#680
Open
garazdawi wants to merge 16 commits into
Open
Conversation
Reviewer's GuideAdds extensive doctest examples for file, atomics, counters, persistent_term and related modules; introduces compile:string/noenv_string and an in‑memory compilation path using ram files and epp’s new include_path_open option; extends ct_doctest to support compiler options and preprocessor-aware module blocks; adds a cooked ram file mode integrated with file_io_server; and wires all of this into new Common Test suites that execute the doctests. Sequence diagram for file:open/2 with ram and cooked modessequenceDiagram
actor Caller
participant file
participant file_io_server
participant OpenFun
participant ram_file
Caller->>file: open(Item, [ram, cooked, read, binary])
activate file
file->>file: mode_list(Item, Modes)
file-->>file: {HasRam=true, HasIoServer=true}
file->>file_io_server: start_handle(self(), OpenFun, [ram, cooked, read, binary])
activate file_io_server
note over file_io_server: do_start_handle(spawn, Owner, OpenFun, ModeList)
file_io_server->>file_io_server: parse_options(ModeList)
file_io_server-->>file_io_server: {ReadMode, UnicodeMode, Opts}
file_io_server->>OpenFun: OpenFun(ReadMode, Opts)
activate OpenFun
OpenFun->>ram_file: open(Item, [binary | Modes - {cooked}])
ram_file-->>OpenFun: {ok, RamFd}
deactivate OpenFun
file_io_server-->>file_io_server: Handle = RamFd
file_io_server-->>Caller: {ok, IoDevice}
deactivate file_io_server
deactivate file
Caller->>file_io_server: read_line(IoDevice)
file_io_server->>ram_file: read_line(RamFd)
ram_file-->>file_io_server: {ok, Line}
file_io_server-->>Caller: {ok, Line}
Sequence diagram for epp include handling with include_path_opensequenceDiagram
actor Compiler
participant compile
participant epp
participant IncludePathOpenFun
participant file
participant ram_file
Compiler->>compile: string(Source, [{include_path_open, Fun} | Opts])
activate compile
compile->>compile: internal({string,Source}, Opts)
compile->>compile: do_parse_string(Fd, Opts)
compile->>epp: open([{fd, Fd}, {include_path_open, Fun}, ...])
activate epp
epp-->>compile: {ok, EppHandle, Extra}
compile->>epp: parse_file(EppHandle)
epp->>epp: scan tokens, encounter -include("hdr.hrl")
epp->>IncludePathOpenFun: Fun(Path, "hdr.hrl", [read])
alt default Fun = file:path_open/3
IncludePathOpenFun->>file: path_open(Path, "hdr.hrl", [read])
file-->>IncludePathOpenFun: {ok, IoDev, FullName}
else custom in-memory Fun
IncludePathOpenFun->>ram_file: open(BinaryHeader, [ram, read, binary, cooked])
ram_file-->>IncludePathOpenFun: {ok, RamFd}
IncludePathOpenFun-->>IncludePathOpenFun: wrap as IoDev
IncludePathOpenFun-->>epp: {ok, IoDev, "hdr.hrl"}
end
epp-->>compile: Forms
deactivate epp
compile->>compile: internal_comp(Passes, Forms, ...)
compile-->>Compiler: {ok, Module, BeamBin}
deactivate compile
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- In compile.erl the -doc attribute for forms/2 was changed to start and end with four quotes (
""""), which will likely break doc parsing/rendering; this should be corrected back to a standard triple-quoted doc string. - In compile_SUITE:doctests/1, the fallback branch in PathOpen uses
file:open(~"", [ram, cooked | Modes]), but~""is not valid Erlang syntax and will fail to compile; replace it with a proper empty string/binary literal (e.g.<<>>or"").
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In compile.erl the -doc attribute for forms/2 was changed to start and end with four quotes (`""""`), which will likely break doc parsing/rendering; this should be corrected back to a standard triple-quoted doc string.
- In compile_SUITE:doctests/1, the fallback branch in PathOpen uses `file:open(~"", [ram, cooked | Modes])`, but `~""` is not valid Erlang syntax and will fail to compile; replace it with a proper empty string/binary literal (e.g. `<<>>` or `""`).
## Individual Comments
### Comment 1
<location path="lib/compiler/src/compile.erl" line_range="1330-1337" />
<code_context>
strip_columns(Forms)
end,
internal_comp(Ps, NewForms, Source, "", Compile);
+internal({string,String}, Opts0) ->
+ Bin = unicode:characters_to_binary(String),
+ {ok, Fd} = file:open(Bin, [ram, read, binary, cooked]),
+ try
+ do_parse_string(Fd, Opts0)
+ after
+ file:close(Fd)
+ end;
internal({file,File}, Opts) ->
{Ext,Ps} = passes(file, Opts),
</code_context>
<issue_to_address>
**issue:** compile:string/2 crashes on file:open/2 failure instead of returning a structured compiler error.
This pattern match on `{ok, Fd}` means any `file:open/2` failure (e.g. ram file allocation failure under memory pressure) will raise `badmatch` instead of returning the usual `{error, {errors, ..., ...}}` tuple. Please handle the `{error, Reason}` case explicitly and convert it into the same error format as `file/2` and `forms/2` so `compile:string/*` remains consistent and doesn’t unexpectedly crash on such failures.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
+1330
to
+1337
| internal({string,String}, Opts0) -> | ||
| Bin = unicode:characters_to_binary(String), | ||
| {ok, Fd} = file:open(Bin, [ram, read, binary, cooked]), | ||
| try | ||
| do_parse_string(Fd, Opts0) | ||
| after | ||
| file:close(Fd) | ||
| end; |
There was a problem hiding this comment.
issue: compile:string/2 crashes on file:open/2 failure instead of returning a structured compiler error.
This pattern match on {ok, Fd} means any file:open/2 failure (e.g. ram file allocation failure under memory pressure) will raise badmatch instead of returning the usual {error, {errors, ..., ...}} tuple. Please handle the {error, Reason} case explicitly and convert it into the same error format as file/2 and forms/2 so compile:string/* remains consistent and doesn’t unexpectedly crash on such failures.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
+935
to
+948
| -doc #{ equiv => forms(Forms, ?DEFAULT_OPTIONS) }. | ||
| -spec forms(forms()) -> CompRet :: comp_ret(). | ||
|
|
||
| forms(Forms) -> forms(Forms, ?DEFAULT_OPTIONS). | ||
|
|
||
| -doc """ | ||
| -doc """" | ||
| Analogous to [`file/1`](`file/1`), but takes a list of forms (in either Erlang | ||
| abstract or Core Erlang format representation) as first argument. | ||
|
|
||
| Option `binary` is implicit, that is, no object code file is | ||
| produced. For options that normally produce a listing file, such as | ||
| 'E', the internal format for that compiler pass (an Erlang term, | ||
| usually not a binary) is returned instead of a binary. | ||
| """. | ||
|
|
Comment on lines
+2441
to
+2447
| ram_include_path_open_try([Candidate | Rest], Files, Modes) -> | ||
| case maps:find(Candidate, Files) of | ||
| {ok, Content} -> | ||
| {ok, Fd} = file:open(Content, [ram, read, binary, cooked]), | ||
| {ok, Fd, Candidate}; | ||
| error -> | ||
| ram_include_path_open_try(Rest, Files, Modes) |
| true -> | ||
| file_io_server:start_handle( | ||
| self(), | ||
| fun(_, _) -> ram_file:open(Item, [binary | ModeList -- [cooked]]) end, |
Add an `include_path_open` option to `epp:open/1` that allows callers
to provide a custom function for opening include files. The function
has the same signature as `file:path_open/3` and is used for
`-include`, `-include_lib`, and `-doc {file, ...}` directives.
This enables fully in-memory preprocessing using ram files, where
both the main source and all included files can be served from
memory without touching disk.
This commit makes ram files work as I/O servers so that code that expects I/O server can use ram files, for example epp:open. Add read_line/1 to ram_file for reading lines from in-memory binary data, handling both LF and CRLF line endings. Add start_handle/3 and start_link_handle/3 to file_io_server, which accept an open function instead of a filename. Add a `cooked` option that can be combined with `ram` in file:open to return a pid-based I/O server instead of a raw file descriptor.
Add compile:string/1,2 and compile:noenv_string/2 that compile Erlang
source code from a string or binary. Unlike compile:forms, the source
goes through the Erlang preprocessor (epp), supporting -define, -include,
-ifdef, records, and all other preprocessor directives.
The `{include_path_open, Fun}` option is passed through to epp, enabling
fully in-memory compilation without file system access for include files.
garazdawi
force-pushed
the
lukas/erts/doctest
branch
from
a53f9ea to
13b55f8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by Sourcery
Add extensive doctest examples across file handling, atomics, counters, and persistent_term modules, introduce in-memory compilation and preprocessing support (including include_path_open) for strings and doctests, and extend test suites to exercise doctests and new in-memory behaviors.
New Features:
Enhancements:
Tests: