makefiles/info-global: speedup info-boards-* targets by using parallel execution#22355
makefiles/info-global: speedup info-boards-* targets by using parallel execution#22355crasbe wants to merge 8 commits into
Conversation
crasbe
requested review from
AnnsAnns,
aabadie,
jia200x and
kaspar030
as code owners
crasbe
added
Type: enhancement
github-actions
Bot
added
Area: doc
crasbe
changed the title
crasbe
requested review from
MichelRottleuthner and
leandrolanzieri
as code owners
|
Oh fun, this uncovered a bug in the The Now here comes the bug: The current implementation saves the That module requires |
That error is wild :D |
crasbe
added
CI: ready for build
|
I'm not sure why I'll let Otherwise we'll just set the number of threads to a lower value and live with only a 10-20x improvement over a 30-40x improvement 🤷 |
crasbe
force-pushed
the
pr/speedup_info-global
branch
from
04444ed to
0127f23
Compare
Couldn't trigger it after 4 hours 🤷 |
mguetschow
left a comment
mguetschow
left a comment
There was a problem hiding this comment.
I see that you really strive to become our buildsystem maintainer :P
| @$(MAKE) BOARD=$* INFO_OVERRIDE=1 info-boards-collect --no-print-directory | ||
|
|
||
| info-boards-supported: bin/.INFO_BOARDS_SUPPORTED $(BOARD_CANDIDATES) | ||
| @cat bin/.INFO_BOARDS_SUPPORTED | sort | xargs echo |
There was a problem hiding this comment.
| @cat bin/.INFO_BOARDS_SUPPORTED | sort | xargs echo | |
| @cat bin/.INFO_BOARDS_SUPPORTED | sort | xargs echo |
There was a problem hiding this comment.
Also, why don't you sort them in the first place before writing them to the file?
There was a problem hiding this comment.
That's the point why I use a file in the first place instead of printing it to stdout: make starts a submake for each board and depending on the complexity of the dependency resolution, it might be faster or slower. With -j1, everything will be done in order (but still not guaranteed), but with -j>1, the order can't be guaranteed anymore.
While it would be possible to specify the target as .NOTPARALLEL, that would defeat the purpose 😅
There was a problem hiding this comment.
oh, but doesn't that mean that the file is written to from several make instances? I thought you ruled that out in your response to @maribu above.
| ifneq (,$(filter info-boards-collect, $(MAKECMDGOALS))) | ||
| ifneq (1,$(INFO_OVERRIDE)) | ||
| $(error info-boards-collect should not be called directly!) | ||
| endif | ||
| endif |
There was a problem hiding this comment.
Could we use your newly introduced PRIVATE_GLOBAL_GOALS variable to check for any such "private global goal", probably outside of this file actually?
There was a problem hiding this comment.
I guess. Do you have any in mind?
There was a problem hiding this comment.
Any file, you mean? Maybe just in Makefile.include where you added the variable.
Co-authored-by: mguetschow <mikolai.guetschow@tu-dresden.de> Co-authored-by: crasbe <crasbe@gmail.com>
4 participants
Contribution description
Currently,
make info-boards-supportedis raaaather slow.Instead of calling a subroutine with
evalover and over for every board, which is not parallizable, I changed the targets in a way that a new RIOT environment is called for each board, which is perfectly parallizable. But even without multithreading, this is a significant performance boost.The downside is, that temporary files have to be created to save the results.
The gain is that this will speed up scripts like
compile_like_murdock.pysignificantlyTesting procedure
Current
master:This PR without multithreading:
This PR with multithreading:
Issues/PRs references
None.
Declaration of AI-Tools / LLMs usage:
AI-Tools / LLMs that were used are: