Conversation
sengir
added
Code Cleanliness
Note that this is still _very_ experimental. There *are* bugs. For more information on how to use this and what's next, please read `build.zig`.
There was a problem hiding this comment.
The name mangling issues could be fixed by passing the /Zc:wchar_t compiler flag. Would be much easier, that flag is basically enough to fix the exports for the normal Civ 4 so it shouldn't be much different here.
There was a problem hiding this comment.
We are actually intentionally disabling distinct wchar_t via Clang's cc1 flag -Xclang -fno-wchar (AKA MSVC's /Zc:wchar_t-). This is to make Boost.Python bindings not break on wchar_t being a distinct type from unsigned short int. Having to manually remangle wchar_t is certainly less than ideal, but I couldn't find a way to get Boost.Python to behave correctly otherwise. I'm not sure if it's because our masquerading as MSVC++ 2003 is confusing Boost or because the EXE's initialization of Boost.Python's converter registry is taking precedence over ours.
If you've got more insight into what exactly the EXE does with Boost.Python, I'd love to hear more. I do know the EXE binds a good number of "secret" types and functions (such as the Scaleform GFC stuff) regardless of what the DLL does.
There was a problem hiding this comment.
Having to manually remangle wchar_t is certainly less than ideal, but I couldn't find a way to get Boost.Python to behave correctly otherwise.
What error is that causing exactly on your side?
This is overstated because it's actually surprising how compatible clang has become. The msvc compiler especially from 2015 onwards is less compatible than clang because Microsoft made lots of runtime changes and the compiler relies heavily on the runtime. You could give https://github.com/backengineering/llvm-msvc a try should you encounter issues with the normal clang compiler. |
|
@alberts8:
Note: The entire PR description is from an old revision of this work. If you're interested in the current state of this branch, read the doc comment of Admittedly, yeah, this is completely hyperbolic. If you use the right incantation of compatibility flags, then it's not too bad. However, there are a good number of very important differences that even Clang's MSVC compatibility mode fail to achieve. The one that frustrates me the most is of course filesystem case sensitivity (and of course the open pull request for it is stale). There are still a number of small things that are really dumb but you have to discover via extended troubleshooting, such as missing
That's a really cool project, thanks for sharing! You're correct in that it's unsuitable due to how Zig (currently) integrates with LLVM. However, that may change once LLVM gets removed from the Zig binary (and turned into a build module?). In that case, it may actually make sense to maintain a fork of this hypothetical build module that's based on this llvm-msvc fork. In the meantime, it seems like I've got things mostly working on par with the current MSVC++ 2003 build. The major remaining issues are due to bugs in the original code that are only being turned into actual problems due to the new compiler not being as a lenient as MSVC. You know, things like obscene UB that should have never compiled in the first place. If you're curious about any of this, feel free to hop on by the WTP Discord server! I'd love to chat more about this if you are. |
| pContext: ?*anyopaque, | ||
| pDC: ?*anyopaque, | ||
| ) callconv(.C) c_int { | ||
| return __CxxFrameHandler(pExcept, pRN, pContext, pDC); |
There was a problem hiding this comment.
Calling the __CxxFrameHandler instead fixes compilation but the exception handling won't work. The program will terminate if a exception occurs. I'am having this issue as well and there are some options to fix that which I'am evaluating currently.
There was a problem hiding this comment.
This is caused by the magic number having a lower value in the msvc 7.1 runtime. The handler doesn't handle the newer number aka
https://github.com/llvm/llvm-project/blob/0fea00d48bcd2368f21ecbca80833093f3a96e64/llvm/lib/CodeGen/AsmPrinter/WinException.cpp#L735
You could try to reduce that number by 2 when calling the CxxFrameHandler maybe that will work. This is one possible solution which i haven't tried myself.
|
Little things like this line of code are something that can be frustrating when using clang. Mod/Project Files/DLLSources/CvGameTextMgr.cpp Line 9436 in ba1323a The format specifier and the second argument don't match. This works with msvc but when using clang the resulting formatted string is garbage. |
|
This one should be checked by sneed. |
2 participants
This
zigbranch is tracking my efforts to completely overhaul the build system using Zig. Zig is an upcoming new programming language, but more importantly, it's also a build system and toolchain for C, C++, and of course Zig.Zig Build System
What makes Zig the build system so compelling is that it lets you define a static build graph using an imperative language (in Zig, if you'd believe it), allowing you to easily model the build process without getting lost in a control flow back & forth between variable files, shell scripts, and globbing
nmakerule. The reason why everyone has stuck to these same old makefiles is because almost no one understands them. Similarly, we could even replace the Perl scripts with Zig scripts to cut our dependence on Strawberry Perl. Then if vendor a portable Zig installation, we could make the entire mod buildable without a single system dependency. Pretty cool!Additionally, the Zig build system integrates seamlessly with an efficient & robust caching system of any build artifact, not just intermediate object files. This not only allows us to shed our reliance on a precompiled header, but it also lets us model our codegen build scripts as regular build steps that depend on specific input files and produce cacheable output files. With this, we should be able to clean up the source code organization quite a bit.
Zig Toolchain
The real "killer feature" of Zig, though, is its capabilities as a C/C++ toolchain. While the Zig language is capable of natively interfacing with C,
zig cc/zig c++is an alternate mode of the Zig compiler that leverages an embedded Clang and LLD to provide a truly portable C/C++ compiler than can cross-compile from any system to any target system without the usual insanities: cross-compilers, sysroots, etc. (see here for more info). This functionality is so useful and novel that (Uber uses Zig to compile all of their C/C++ code, allowing them to migrate to arm64)[https://www.uber.com/blog/bootstrapping-ubers-infrastructure-on-arm64-with-zig/]. What this allows us to do is develop WTP from Linux (or macOS)! I personally run NixOS and typically run WTP through Steam under Proton, but the current WTP build tooling is incompatible with both Linux and WIne/Proton, requiring either dual booting or running a virtual machine to build the project. Not for long!The rub here is that this uses Clang to compile our C++ code. Believe it or not, an MSVC++ Toolkit 2003 project is absolutely not compatible with Clang whatsoever. However, I've been hard at work trying to make it compatible! So far, the project builds successfully and I can open up the game and screw around. However, there are many failed
FAsserts in this build. I believe this is mostly or perhaps even entirely due to latent Undefined Behavior that already existed in the codebase, but MSVC++ Toolkit 2003 was simply not smart enough to optimize that Undefined Behavior into a runtime bug while Clang on the other hand is. Yes, this does mean that this new build of WTP should be significantly more optimized thanks to two decades of compiler innovations. We also get two decades of compiler innovations in static analysis and warnings, which is perhaps even more beneficial.Another fun benefit of Clang is that it gives us C++23 & all of its goodies! Unfortunately, a lot of those goodies are missing since we're still including an STL from MSVC++ 2003. We can still take advantage of the many language-level features, but more interestingly we can use this as an opportunity to define our own sort of STL that caters to our more specific needs and opinionated views. For example, we could make immutability the default that must get opted out of instead of the typical opt-in, allowing us to abide by the principle of least privilege (see #768 #864 #696 etc). In my opinion, the most important gain here would be cleaning up all the monstrous enum-related metaprogramming that had to not only be MSVC++ 2003 compatible but also use codegen'd enum stuff and support optionally hardcoding enum lengths. Bravo, @Nightinggale for the hard work, I don't think I could've done it, but holy **** we've got to simplify that ASAP with C++23.
Next Steps
build.zigbuild.zigbuild script to accommodate every use casedevelopdevelopNote
If anyone is interested in playing around with this (or fighting the good fight against Undefined Behavior), then follow these steps to install Zig. The build system API got reworked recently but there hasn't been a tagged release since then, so make sure you select the master branch/nightly build of Zig. In the future, we'll use a vendored copy of Zig, but I'd like to at least wait until the next tagged release before thinking about vendoring anything. For any assistance whatsoever, feel free to message me on here or the Discord server. I'd be happy to help however I can!
NOTE: This is a draft branch; this effort is far from complete. This branch will undergo many changes, including liberal use of history rewriting and force pushes, so keep that in mind if you do checkout this branch.