Upgrade to Opensearch 3.0 - #3694
Draft
koetsier wants to merge 34 commits into
Draft
Conversation
Elasticsearch 7+ does not allow nested 'should' queries
This filter is deprecated, as it no longer provides any functional behavior. https://stackoverflow.com/questions/76108163/getting-error-the-standard-token-filter-has-been-removed-when-running-a-que
…7.10 client. The Elasticsearch client gem has been updated and now issues POST requests for search operations instead of GET. # Conflicts: # spec/unit/elasticsearch_index_spec.rb
… client.
When performing a scroll search, the Elasticsearch client may now use
either of the following endpoint formats:
$HOSTNAME/_search/scroll/{scroll_id}?scroll=1m
$HOSTNAME/_search/scroll?scroll=1m&scroll_id={scroll_id}
Both URL patterns are supported by Elasticsearch 6.8.
Although _uid is still present in Elasticsearch 6.8, it is deprecated. All usages have been updated to _id to ensure forward compatibility with future Elasticsearch versions. Reference: https://www.elastic.co/guide/en/elasticsearch/reference/6.8/mapping-uid-field.html # Conflicts: # lib/tasks/export.rake
Use a fallback when doc['popularity'] is empty by applying POPULARITY_OFFSET directly; otherwise add the offset to the existing value. This is necessary because Elasticsearch 7+ requires the property to exist if accessed popularity data is missing.
Introduce an ElasticsearchResponse helper to normalize the hits.total
field across Elasticsearch 6.x and 7.x, and update callers to use the
shared helper instead of accessing the response structure directly.
In Elasticsearch 6.x, the hits.total field is returned as a simple integer:
{
"hits": {
"total": 123,
"hits": [
...
]
}
}
In Elasticsearch 7.x, hits.total became an object that includes both
the count and information about whether it’s exact:
{
"hits": {
"total": {
"value": 123,
"relation": "eq"
},
"hits": [
...
]
}
}
…GET requests The _all parameter is optional in Elasticsearch 6 but is no longer supported in Elasticsearch 7. Remove _all from all GET requests to maintain compatibility with the upcoming Elasticsearch 7 upgrade.
Introduce an ElasticsearchClient helper to detect whether the application is connected to Elasticsearch 6 or 7. Support overriding the detected version via environment variables
Configure RSpec to directly use :integration tags for specs under spec/integration,
Unit tests do not allow connecting to a live Elasticsearch instance which means the compatibility class ElasticsearchClient cannot use it to determine which version to use Furthermore, unit tests use mocks to abstract away the Elasticsearch server and they cannot easily be made compatible for both Since we're going to migrate to Elasticsearch 7, it makes sense to ensure they are compatible with that version and so we ensure the ElasticsearchClient class assumes Elasticsearch 7 is present
The unit tests assume Elasticsearch 7 is used. Elasticsearch 7 no longer supports the _type parameter and so these need to be removed.
Introduce ElasticsearchClient.search so we can seamlessly migrate from Elasticsearch 6 to 7. The wrapper conditionally adds the legacy _type parameter required by ES6 and omits it for ES7, where _type is no longer supported. Update Index#raw_search to use the wrapper instead of passing _type directly.
Introduce ElasticsearchClient.search so we can seamlessly migrate from Elasticsearch 6 to 7. The wrapper conditionally adds the legacy _type parameter required by ES6 and omits it for ES7, where _type is no longer supported.
Introduce ElasticsearchClient.compatible_mappings to generate index mappings that work across Elasticsearch versions. For ES6, mappings are wrapped under the legacy "generic-document" type, while ES7 uses the updated format with "properties" at the root level. Update IndexSchema to use this helper instead of hardcoding ES6-style type mappings, ensuring consistent behavior during the migration.
Introduce ElasticsearchClient.put_mapping to handle differences in mapping APIs between Elasticsearch 6 and 7. The helper conditionally includes the legacy type: "generic-document" parameter for ES6 and omits it for ES7, where types are no longer supported. Update SchemaSynchroniser to use the compatibility layer instead of calling the Elasticsearch client directly. Adjust integration test expectations to account for the different mapping structure in ES7 versus ES6.
Introduce ElasticsearchClient.delete so we can seamlessly migrate from Elasticsearch 6 to 7. The wrapper conditionally adds the legacy _type parameter required by ES6 and omits it for ES7, where _type is no longer supported.
Add ElasticsearchClient.mappings_properties to normalize mapping format differences between Elasticsearch versions. For ES6, mappings are nested under "generic-document", while ES7 exposes "properties" at the top level. Update the indices rake task to extract the correct mapping structure via this helper before passing it to SchemaSynchroniser, ensuring schema synchronization works across both Elasticsearch 6 and 7 during migration.
…migration Update bulk indexing and deletion flows to remove _type from document identifiers, since Elasticsearch 7 no longer supports types. Introduce ElasticsearchClient.compatible_identifier to conditionally include _type only for ES6, ensuring bulk index and delete actions remain compatible across both ES versions during migration. This affects bulk processor actions, indexers, and related presenters that build Elasticsearch bulk request payloads.
GET _alias without a name resolves across all indices, including system indices (.kibana*, .security*, etc.), which triggers a deprecation warning in Elasticsearch 7. Restrict the request to a named alias to avoid scanning system indices and align with Elasticsearch system index protection behavior. Docs: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/migrating-7.10.html#breaking_710_indices_changes
…tion Everywhere we use the ElasticsearchProcessor for bulk actions. This ensures consistency and compatibility between ES version 6 and 7
Ordering by _id is going to be deprecated. Use another field instead.
The boost (0.3) is applied equally to service_manual_guide and service_manual_topic so the order of those two is not certain. The test should only test that the cma_case comes first
expect_document_is_in_rummager does not work when there are spaces in the id field
Deleted documents can continue to influence Lucene scoring statistics, so simply deleting documents is not sufficient for test isolation. Recreating indices ensures a clean state and stable search scores.
We are moving to Opensearch 3, so the gem will have to be updated accordingly
As we upgraded the client gem from Elasticsearch to Opensearch, we need to change all calls using the 'Elasticsearch::' namespace to 'OpenSearch::'
This endpoint returns the version of opensearch and other information of the cluster. The Opensearch client queries this, so it needs to be stubbed for Unit tests.
As part of upgrading from Elasticsearch to Opensearch, the query
fetching the aggregate examples needs to be tweaked.
The old query used 'query: { bool: { must: nil } }' which is no
longer considered valid. Instead the query section should simply
be omitted
Multiple types per index have been deprecated and so the _type parameter needs to be removed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR upgrades the code to Opensearch 3.0. It is compatible with Elasticsearch 7.10
Need to hold off until we have upgraded to ES7.10