[META] Support for FIPS 140-3 enforced mode

[nibix]

Intro

The purpose of this meta issue is supporting the ongoing efforts for supporting FIPS 140-3 enforced mode in OpenSearch. It shall serve two core functions:

• to document fundamental architectural decisions
• to document open to-do items and to keep track of them

This issue shall be a living document, we should try to keep the information here up to date. Discussions in the comments section are welcome - still, any fundamental decision in the comments section should be reflected here.

Note: Unfortunately, the GitHub issue permission model only allows me and maintainers to edit this issue. I would have hoped that there would have been some way to allow more users editing, but that does not seem to be the case. I will try to do my best to keep the issue up to date. Still, if anyone has ideas on how to improve this, please speak up!

Note: This is just my first shot. There is a 200% chance that there are omissions and inaccuracies. Please comment or edit in order to complete this list.

Fundamental decisions

• There will be only one OpenSearch distribution, which always bundles bc-fips but does not enable FIPS approved only mode by default.
• To enable FIPS approved only mode, these actions must be performed:
  • the script opensearch-fips-demo-installer needs to be executed
  • the environment variable OPENSEARCH_FIPS_MODE needs to be set (this will be converted by the opensearch-env script into the JVM property org.bouncycastle.fips.approved_only=true).
• The first version of this will be shipped with OpenSearch 3.6. However, enabling FIPS approved only mode will be only fully supported for the OpenSearch-min distribution (without any plugins, especially also without security plugin). Some further plugin functionality might work, but no guarantee is given.
• Full support, including security plugin and other plugins will be delivered in a later version of OpenSearch.

Assumptions

Please speak up if these assumptions are wrong.

• All plugins in the OpenSearch distribution have been already adapted to be aware of the bundled bc-fips library (This is only about the presence of the component, not about enabling FIPS approved only mode)
• The OpenSearch min distribution already runs without issues in FIPS approved only mode. All necessary PRs for this have been already merged to main.

Supporting References

opensearch-project/technical-steering#77

Issues

For OpenSearch 3.6.0:

• Bundle bc-fips with distribution: Toggle -Pcrypto.standard=FIPS-140-3 by default on all build scripts (Build) opensearch-build#5995
• Support flag to activate FIPS approved only mode: Use env variable (OPENSEARCH_FIPS_MODE) to enable opensearch to run in FIPS enforced mode instead of checking for existence of bcFIPS jars #20625
• Automated (smoke) tests for OpenSearch min in FIPS approved only mode. (Do we already have any tests?)
• User documentation
• Any other implementation work necessary to achieve the minimal approved only mode support?

For later OpenSearch release

• Finish adaptions of security plugin necessary for . (Note: if important decisions need to be made for these adaptions (such as not supporting SAML), let's document these here as well.)
• Finish adaptions of other plugins (Note: If there are other plugins which require non-trivial adaptions, please let me know. We should list these individually.)
• Automated tests for OpenSearch in FIPS approved only mode.
• User documentation
• Any other implementation work necessary to achieve the FIPS approved only mode support for the full distro?

[nibix]

@cwperks @andrross @reta @beanuwave @KarstenSchnitter @terryquigleysas please have a look and let me know if I got something wrong or forgot something.

[terryquigleysas]

@nibix Thanks for putting this together. It is a fair representation of the discussion. From first reading I would make the following comments.

Change the title and Intro to target FIPS 140-3

I would relax the language on the Security plugin. We (the team/company I work for) have seen many of the features work, in fact all those that we use, when enforcing FIPS settings in a FIPS environment. Where there are known gaps (i.e. in specific features we don't personally use) we and the wider community should make a best effort to document them and aim to address in future.

Also I would like to stress again that it is important that user-provided Java Security files should be honored as compliance, and accepted syntax, can differ between platforms (e.g. RHEL with OpenJDK).

[reta]

Thanks @nibix a few data points:

Automated (smoke) tests for OpenSearch min in FIPS approved only mode. (Do we already have any tests?)

Yes, we do see please [1] this could be done on distribution level or per plugin (let say security)

Also I would like to stress again that it is important that user-provided Java Security files should be honored as compliance, and accepted syntax, can differ between platforms (e.g. RHEL with OpenJDK).

These checks could be bundled into the Bootstrap [2] using BootstrapChecks [3], for example making sure the JVM options (FIPS mode) matches classpath (BC FIPS libs are present), + security policies,

[1] https://github.com/opensearch-project/opensearch-build/blob/main/manifests/3.6.0/opensearch-3.6.0-test.yml#L30
[2] https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/bootstrap/BootstrapChecks.java
[3] https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/bootstrap/BootstrapChecks.java

[nibix]

@terryquigleysas

I would relax the language on the Security plugin. We (the team/company I work for) have seen many of the features work, in fact all those that we use, when enforcing FIPS settings in a FIPS environment. Where there are known gaps (i.e. in specific features we don't personally use) we and the wider community should make a best effort to document them and aim to address in future.

Would it be possible to compile a list of what works and what does not work? That would make it more easy to find the correct language :)

A related question: Even if things work, it does not necessarily mean that they work in a compliant way, right?

@reta

These checks could be bundled into the Bootstrap [2] using BootstrapChecks [3], for example making sure the JVM options (FIPS mode) matches classpath (BC FIPS libs are present), + security policies,

Thank you, that's interesting! However, I was rather referring to any integration tests which verify that the product in FIPS enforced mode actually works at all. Do we have something like this?

A sorry, got this wrong on first read. Now understood.

[cwperks]

@nibix its quite limited atm with a single qa module in core that runs with the gradle check: https://github.com/opensearch-project/OpenSearch/tree/main/qa/fips-compliance

Its possible to run the gradle check with fips enforced mode, but only locally and is not done on the jenkins runners with every PR.

[beanuwave]

@nibix Thank you for starting this thread - some decisions made in the past are already legacy, so it’s great to have an up-to-date summary.
BTW, user documentation does exist under https://docs.opensearch.org/latest/security/configuration/fips/

Its possible to run the gradle check with fips enforced mode, but only locally and is not done on the jenkins runners with every PR.

@cwperks We definitely need to ensure that the FIPS test pipeline gets executed at least before packaging a release distribution, or via a manual trigger. Every crypto-related code change has the potential to break something, so I’d strongly advocate working on this ASAP.
At the current state, FIPS smoke tests are mostly groundwork for what’s coming next. However, 99% of the tests already exist under the patterns **/*FipsTests.class and **/*FipsIT.class, and they need to be executed with the Pcrypto.standard=FIPS-140-3 parameter.

Thanks to our previous conversation, I finally managed to grasp how recent changes influence the FIPS parameters. Here’s my current understanding - please correct me if I’m wrong:

Everything ultimately goes through the org.bouncycastle.fips.approved_only=true flag. Atm this is the only hard indicator of whether FIPS mode is active or not. Based on this, we also have secondary indicators for different use cases, such as:

• CryptoServicesRegistrar.isInApprovedOnlyMode() -> BC's own class
• FipsBuildParams.isInFipsMode() -> custom gradle info parameter
• BuildParams.isInFipsJvm() -> inherits FipsBuildParams.isInFipsMode()
• FipsMode.CHECK.isFipsEnabled() -> shortcut for tests only

In addition, the JVM flag -Dorg.bouncycastle.fips.approved_only=true can also be set indirectly via:

• tests.fips.enabled=true -> deprecated gradle parameter
• crypto.standard=FIPS-140-3 -> build gradle parameter
• OPENSEARCH_FIPS_MODE=true -> env parameter to run the application or CLIs

Important note: this is not a final state, but rather a snapshot of the current work. I expect many things to change in the near future, and the FIPS installer, in particular, may become significantly more important going forward.
https://github.com/opensearch-project/OpenSearch/blob/main/distribution/src/bin/opensearch-fips-demo-installer

Finally, I’d like to re-open the discussion around a low-effort, universal approach to FIPS integration for plugins. My current workaround is to enhance the relevant build jobs with FIPS-related environment parameters, but this leads to a lot of redundant code across repositories.