What is your suggestion?

Intro

Hi!

In continuation of my previous issue I am creating this one specifically to discuss possibilities for vendoring mechanism extention.

I noticed a new vendoring feature quite a time ago and it appeared like something we can use in our case.
However, we quickly realized that unfortunately it lacks some flexibility, which is critical to us.

The main limitation from our point of view, is inability to enable it conditionally.

Background

Our use case includes about a hundred of different C/C++ components, most of them are applications.
Lot's of them require heavy compilation process often taking several hours to complete.
As a result, developers want to consume these components in a prebuilt binary form.
That is the case 90% of time. However, 10% of time there is a demand to recompile certain components with different transitive components version, specified manually.
Let's take a look on a couple of possible examples:

1. Having a SPEC benchmark to test new hardware or simulators. 90% of time users want to use a precompiled benchmark to test new FPGA bitstream or QEMU simulator. However, LLVM compiler team wants to regularly rebuild SPEC on-the-fly with just built LLVM in CI pipeline.
2. Having a linux kernel to test benchmarks, simulators etc. 90% of time should be comsumed precompiled, 10% of time recompile with freshly built LLVM or GCC compilers. All the other linux dependencies should be left intact -- for example originally used version of rootfs should be defined by linux itself, not its consumer by default.
3. Having llvm-snippy test generator. 90% of time should be consumed precompiled. 10% of time should be rebuilt with different sail library version. All of other dependencies like gmp, zlib etc. should be left intact, i.e. versions of these dependencies should be defined by llvm-snippy, not by consumer by default.

All mentioned components here including compiler are conan packages.

Since it is not possible to conditionally enable vendoring based on some option we are unable to use it in our case.

conan install --requires spec/1.2.3 # great, works with vendoring
conan install --requires spec/1.2.3 -o vendor=False --build=spec/* # Does not work

Vendoring continuity

And even if it would be possible to enable conditional vendoring, it would not be enough.
Switching vendoring on/off is actually still too coarse granularity!

What we would like to see is something like "vendoring continuity".
That is:

1. By default, consuming vendored packages is a peace of cake and does not require from user to compile anything:

   conan install --requires spec/1.2.3 # great, just works

2. If needed, user can specify which compiler version (or any other dependency version) to use to recomile the benchmark:

   conan install --requires spec/1.2.3 -o llvm=4.5.6  --build=spec/*

   An only compiler will have custom version. All of the other spec dependencies will be untouched and taken as they were on initial spec package creation, from spec's lockfile, not user's one.

   Also, it is important that user has a precise control over which compiler is used.
   By default conan will pick a first suitable compiler version from the lockfile, which is not user wants. There are might be a dozen different compiler versions coming from different dependencies, yet user wants to use a very specific one.

Our implementation

In order to achieve desired behaviour, we implemented some sort of emulated vendoring, which can be approximately described by this example:

from conan import ConanFile

class Tool(ConanFile):
    name = "linux"
    version = "1.0.0"

    options = {
        "llvm": [None, "ANY"],
        "rootfs": [None, "ANY"],
    }

    def requirements(self):
        # LLVM and rootfs itself has "emulated vendoring".
        llvm = self.options.llvm or "1.2.3#13c96f538b52e1600c40b88994de240f"
        rootfs = self.options.rootfs or "4.5.6#3d2b40ef4ac9a33400b929e1cdfd4aae"

        self.requires(f"rootfs/{rootfs}", visible=False)
        self.tool_requires(f"llvm/{llvm}")

    def package_id(self) -> None:
        self.info.requires.clear()
        self.info.build_requires.clear()

        # Also, remove all python_requires, to exclude potential problems
        self.info.python_requires.clear()

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide