ElasticCommitter - Restrict operation based on action verb (add, update, delete)

[stejacob]

Hi,

We would like to restrict specific actions for Elastic committers based on the type of operation (add, update, or delete).

We have three committers, and we're aiming to implement the following restrictions:

1. One committer should be restricted to only handle English documents based on language field containing value en.
2. Another committer should be restricted to handle only French documents based on language field containing value fr.
3. The third committer should be restricted to perform delete operations only.

However, we're unsure if this can be achieved with the current configuration. At the moment, using the restrictTo option based on a field causes delete commands to be rejected, as the field doesn't exist in the document being deleted.

If this is not yet supported, would it be possible to open an enhancement request?

Any guidance on how to achieve this setup would be greatly appreciated.

Thank you.

Stephen Jacob

[essiembre]

That could be useful indeed. I am marking it as a feature request. As a workaround for now, if you know a given field is always there except when deleted, you can add extra committers that are restricted on such a field being empty. You can likely do so by matching the regular expression $^ and making sure matchEmpty is true.

[stejacob]

Hi Pascal,

We configured below but we are getting "Attribute 'matchEmpty' is not allowed to appear in element 'valueMatcher'".

<committer class="JSONFileCommitter"> <restrictTo> <fieldMatcher method="regex" ignoreCase="true" ignoreDiacritic="true" partial="true"> collections.....new </fieldMatcher> <valueMatcher method="regex" matchEmpty="true"> $^ </valueMatcher> </restrictTo> <directory> ${'./jsonExportAddUpdate/'}</directory> <indent>4</indent> </committer>

Thanks.

Stephen Jacob

[essiembre]

You are correct. The matchEmpty option is part of the upcoming version 4.

To deal with using restrictTo in your committer, a workaround can be to set a predefined keyword that means "empty" in your importer config and filter on that keyword.

Let's say your document has a field that is usually populated but would like to apply different restrictions when empty. Let's call it my_field.

<importer>
  <postParseHandlers>
    <handler class="ConstantTagger" onSet="OPTIONAL">
      <constant name="my_field">EMPTY</constant>
    </handler>
  </postParseHandlers>
</importer>

The onSet="OPTIONAL" will ensure the value will only be set if no value currently exists for the field (which includes if the field is not present at all in the document).

Then in your committer, you can restrict on the EMPTY value like this:

<committer class="JSONFileCommitter">
  <restrictTo>
    <fieldMatcher>my_field</fieldMatcher>
    <valueMatcher>EMPTY</valueMatcher>
  </restrictTo>
  ...
</committer>

That way, you can identify empty fields for easy filtering.