Remove compatibility helpers for Elasticsearch 6.8 to 7.10 - #3693
Draft
koetsier wants to merge 27 commits into
Draft
Remove compatibility helpers for Elasticsearch 6.8 to 7.10#3693koetsier wants to merge 27 commits into
koetsier wants to merge 27 commits into
Conversation
Elasticsearch 7+ does not allow nested 'should' queries
This filter is deprecated, as it no longer provides any functional behavior. https://stackoverflow.com/questions/76108163/getting-error-the-standard-token-filter-has-been-removed-when-running-a-que
…7.10 client. The Elasticsearch client gem has been updated and now issues POST requests for search operations instead of GET. # Conflicts: # spec/unit/elasticsearch_index_spec.rb
… client.
When performing a scroll search, the Elasticsearch client may now use
either of the following endpoint formats:
$HOSTNAME/_search/scroll/{scroll_id}?scroll=1m
$HOSTNAME/_search/scroll?scroll=1m&scroll_id={scroll_id}
Both URL patterns are supported by Elasticsearch 6.8.
Although _uid is still present in Elasticsearch 6.8, it is deprecated. All usages have been updated to _id to ensure forward compatibility with future Elasticsearch versions. Reference: https://www.elastic.co/guide/en/elasticsearch/reference/6.8/mapping-uid-field.html # Conflicts: # lib/tasks/export.rake
Use a fallback when doc['popularity'] is empty by applying POPULARITY_OFFSET directly; otherwise add the offset to the existing value. This is necessary because Elasticsearch 7+ requires the property to exist if accessed popularity data is missing.
Introduce an ElasticsearchResponse helper to normalize the hits.total
field across Elasticsearch 6.x and 7.x, and update callers to use the
shared helper instead of accessing the response structure directly.
In Elasticsearch 6.x, the hits.total field is returned as a simple integer:
{
"hits": {
"total": 123,
"hits": [
...
]
}
}
In Elasticsearch 7.x, hits.total became an object that includes both
the count and information about whether it’s exact:
{
"hits": {
"total": {
"value": 123,
"relation": "eq"
},
"hits": [
...
]
}
}
…GET requests The _all parameter is optional in Elasticsearch 6 but is no longer supported in Elasticsearch 7. Remove _all from all GET requests to maintain compatibility with the upcoming Elasticsearch 7 upgrade.
Introduce an ElasticsearchClient helper to detect whether the application is connected to Elasticsearch 6 or 7. Support overriding the detected version via environment variables
Configure RSpec to directly use :integration tags for specs under spec/integration,
Unit tests do not allow connecting to a live Elasticsearch instance which means the compatibility class ElasticsearchClient cannot use it to determine which version to use Furthermore, unit tests use mocks to abstract away the Elasticsearch server and they cannot easily be made compatible for both Since we're going to migrate to Elasticsearch 7, it makes sense to ensure they are compatible with that version and so we ensure the ElasticsearchClient class assumes Elasticsearch 7 is present
The unit tests assume Elasticsearch 7 is used. Elasticsearch 7 no longer supports the _type parameter and so these need to be removed.
Introduce ElasticsearchClient.search so we can seamlessly migrate from Elasticsearch 6 to 7. The wrapper conditionally adds the legacy _type parameter required by ES6 and omits it for ES7, where _type is no longer supported. Update Index#raw_search to use the wrapper instead of passing _type directly.
Introduce ElasticsearchClient.search so we can seamlessly migrate from Elasticsearch 6 to 7. The wrapper conditionally adds the legacy _type parameter required by ES6 and omits it for ES7, where _type is no longer supported.
Introduce ElasticsearchClient.compatible_mappings to generate index mappings that work across Elasticsearch versions. For ES6, mappings are wrapped under the legacy "generic-document" type, while ES7 uses the updated format with "properties" at the root level. Update IndexSchema to use this helper instead of hardcoding ES6-style type mappings, ensuring consistent behavior during the migration.
Introduce ElasticsearchClient.put_mapping to handle differences in mapping APIs between Elasticsearch 6 and 7. The helper conditionally includes the legacy type: "generic-document" parameter for ES6 and omits it for ES7, where types are no longer supported. Update SchemaSynchroniser to use the compatibility layer instead of calling the Elasticsearch client directly. Adjust integration test expectations to account for the different mapping structure in ES7 versus ES6.
Introduce ElasticsearchClient.delete so we can seamlessly migrate from Elasticsearch 6 to 7. The wrapper conditionally adds the legacy _type parameter required by ES6 and omits it for ES7, where _type is no longer supported.
Add ElasticsearchClient.mappings_properties to normalize mapping format differences between Elasticsearch versions. For ES6, mappings are nested under "generic-document", while ES7 exposes "properties" at the top level. Update the indices rake task to extract the correct mapping structure via this helper before passing it to SchemaSynchroniser, ensuring schema synchronization works across both Elasticsearch 6 and 7 during migration.
…migration Update bulk indexing and deletion flows to remove _type from document identifiers, since Elasticsearch 7 no longer supports types. Introduce ElasticsearchClient.compatible_identifier to conditionally include _type only for ES6, ensuring bulk index and delete actions remain compatible across both ES versions during migration. This affects bulk processor actions, indexers, and related presenters that build Elasticsearch bulk request payloads.
GET _alias without a name resolves across all indices, including system indices (.kibana*, .security*, etc.), which triggers a deprecation warning in Elasticsearch 7. Restrict the request to a named alias to avoid scanning system indices and align with Elasticsearch system index protection behavior. Docs: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/migrating-7.10.html#breaking_710_indices_changes
…tion Everywhere we use the ElasticsearchProcessor for bulk actions. This ensures consistency and compatibility between ES version 6 and 7
Ordering by _id is going to be deprecated. Use another field instead.
The boost (0.3) is applied equally to service_manual_guide and service_manual_topic so the order of those two is not certain. The test should only test that the cma_case comes first
expect_document_is_in_rummager does not work when there are spaces in the id field
Deleted documents can continue to influence Lucene scoring statistics, so simply deleting documents is not sufficient for test isolation. Recreating indices ensures a clean state and stable search scores.
Contributor
Author
|
To be merged after we finish the upgrade to Elasticsearch 7.10. The tests are failing because they are running against Elasticsearch 6.8 for now. |
Previously, we introduced the ElasticsearchClient class to ensure our code is compatible with both Elasticsearch version 6.8 and 7.10 Having completed the upgrade to Elasticsearch 7.10, we can remove this compatibility code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously, we introduced the ElasticsearchClient class to ensure our code
is compatible with both Elasticsearch version 6.8 and 7.10
Having completed the upgrade to Elasticsearch 7.10, we can remove this
compatibility code.