RFC: Additional Guidance on CI and Deployment

[nickwesselman]

Helix already includes some guidance on DevOps and development lifecycle management. However we have received feedback that Helix needs more content/guidance on these topics. What are the gaps here that need to be addressed?

Modular / partial deployments come up a lot with Helix, even though it's explicitly stated to be not a goal of the architecture. How often do your implementations merit this, and does the typical partner/customer have the DevOps expertise and infrastructure to implement?

[asmagin]

We implement CI/CD for every project that we have, that covers both server- and client-side code. and were doing that before Helix appeared.

As it was mentioned in a separate thread, Helix is doesn't really align with Reuse Release Equivalence principle in the way it defines modules. As most of the time, the whole solution is installed at once, while it might have tons of Helix modules in it.

What we also found that it is extremely difficult to create a package that will be installable into Sitecore and referenceable in Visual Studio (I'm not even talking about client-side code). It all should be done via different types of packages and mechanisms.

That said, it is more a limitation of a product and it should be fixed there first.
E.g.: change packages format to NuGet, with serialized items in App_Data folder, instead of serialization outside of a project. Such package should be referenceable in VS and installable to Sitecore server.
Going even further than that, NuGet can even be used in Sitecore server itself to track dependencies or pull modules.

[richardszalay]

I'm not sure if "tooling" is a valid point of discussion, but I'm going to shout out Helix Publishing Pipeline, with the obvious disclaimer that it's my project.

Let me know if either of those two are out of line and I'll nuke this comment.

Basic elevator pitch: add a nuget project to your Project (or a top-level "Website" project if you have multiple) and all of your module content (configs/views) will get published in the regular "Visual Studio Publish" flow (file system, zip, azure, Docker, etc)

Extra bits include:

• Advanced path mapping (eg. ../serialization/**/* to App_Data\unicorn\(Module)\(RecursiveFilename))
• Support for configuring extremely low-friction/latency local deployments on every build (+2-3 seconds, applies config transforms but avoids recycling app pool)
• Per-module Web.config transforms (/Web.Helix.config in each module is merged together and applied to /Web.config in the root project)
• Per-module parameters.xml (for Web Deploy packages)
• Exclude publishing of Sitecore assemblies (using the official list files that come with the releases, not just "Sitecore.*.dll")
• Pattern-whitelisted "remove all files" for local deployments

It's primarily focused on "full solution" deployments. It does actually have a module-level package that's updated for each release, but the docs no longer talk about it.

I use it on a number of projects that have build-once CI/CD pipelines, and I've gotten really good feedback from the community so far on their experiences with it.

[richardszalay]

Client side recommendations would be tricky as many people have their own processes. For SXA, we ended up using Webpack and transforming the output to Rainbow so that we could use transparent sync and not have to upload individual files.

Non-SXA would depend entirely on what libraries/frameworks people are using.

[lakesol]

How often do your implementations merit this, and does the typical partner/customer have the DevOps expertise and infrastructure to implement?

In my experience partial deployment of systems leads to a whole world of pain. We follow principles of deploy all, deploy often and use the Octopus Deploy tool set (which is geared towards that style of deployments) to manage that process.

IMO Whilst partial deployments are great in theory, the real world issues cause it to be high risk, complex and costly in terms of testing. Our clients value predictability of deployments over a few seconds/minutes saved during a deployment cycle.

To manage granular feature deployment we have a strong source control management approach (mostly git flow). This allows features to be a added to any branch/version/hotfix etc when needed.

I am not sure how best to update the docs. I would even say that its not really the place of helix to define how build/deploy should happen (not really part of the scope + too many variances of environment/use etc)?

[asmagin]

@lakesol, partial deployment should use individual versioning of Helix modules, otherwise you cannot really control your dependencies.
In this case, you might leverage predictable dependencies control like in NPM or NuGet.

Gitflow is nice if you deploying the whole project (and do not have artifacts storage) or individual for module.

However, I would agree that partial deployment doesn't provide assurance in "what exactly is on server now" as many dependencies in solution are not versionable and done by convention (e.g. c# and js code, or Sitecore items and c# code)

[lakesol]

@asmagin Agree 100%. However what we have done in the past is where a helix module is shared, we include this through NuGet into a specific customer solution. Then the solution is built and deployed as a single entity. Essentially we have moved the reuse of a module up a level from the deployment level to solution level.

Whilst this is not partial deployment, it does offer us the benefits of code (module) reuse, high levels of deployment predictability and support for granular functionality roll out where needed.

[dthunziker]

This was discussed during a DevOps panel at the Las Vegas Symposium last year. The consensus was that deploying each module seperatly can only be done if you have super tight control over dependencies between modules and have a close watch over versioning. For most, it will make sense to version the entire application and test/deploy all modules together at once. I think this topic often comes up due to the project-per-module structure. When people first see it, their natural tendency is to think that each module can/should be deployed seperatly, which the Helix documentation discourages.

[dsolovay]

I think the guide goes a bit far in arguing against code reuse. Especially in agency development scenarios, it makes sense to treat at least some foundation modules as independently revisable and releaseable (e.g. through an internal NuGet feed, or through a GitModule), so that improvements and capabilities may be shared across implementations ("The new version of the Search foundation module does what you need so pull it down from the NuGet server."). This seems keeping in spirit with the spirit of the packaging principles, and not addressing these scenarios seems like a miss.