Cloud Drive synchronization generates lot of indexation errors in 5.0.x

[vsellier]

During the synchronization, there is a lot of errors like this on the logs in 5.0.0-RC11 version :

2018-03-06 15:20:11,261 | ERROR | Error while indexing file fbaf0a669e4534520bdd52edca277631 [o.e.s.w.s.c.FileindexingConnector<ExoScheduler_Worker-13>]
javax.jcr.ItemNotFoundException: Node not found fbaf0a669e4534520bdd52edca277631 at collaboration
        at org.exoplatform.services.jcr.impl.core.SessionImpl.getNodeByIdentifier(SessionImpl.java:743) ~[exo.jcr.component.core-5.0.0-RC11.jar:5.0.0-RC11]
        at org.exoplatform.services.wcm.search.connector.FileindexingConnector.create(FileindexingConnector.java:124) ~[ecms-core-search-5.0.0-RC11.jar:5.0.0-RC11]
        at org.exoplatform.services.wcm.search.connector.FileindexingConnector.update(FileindexingConnector.java:176) ~[ecms-core-search-5.0.0-RC11.jar:5.0.0-RC11]
        at org.exoplatform.commons.search.es.client.ElasticContentRequestBuilder.getCreatePipelineDocumentRequestContent(ElasticContentRequestBuilder.java:159) ~[commons-search-5.0.0-RC11.jar:5.0.0-RC11]
        at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processCUD(ElasticIndexingOperationProcessor.java:320) [commons-search-5.0.0-RC11.jar:5.0.0-RC11]
        at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processBulk(ElasticIndexingOperationProcessor.java:212) [commons-search-5.0.0-RC11.jar:5.0.0-RC11]
        at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.process(ElasticIndexingOperationProcessor.java:150) [commons-search-5.0.0-RC11.jar:5.0.0-RC11]
        at org.exoplatform.commons.search.job.BulkIndexingJob.execute(BulkIndexingJob.java:41) [commons-search-5.0.0-RC11.jar:5.0.0-RC11]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.2.jar:na]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.2.jar:na]

There is a lot of exception like this one relative to different ids, but no other errors before.
I don't know if it's due to my drive content or a problem with some transactions concurrency.

[pnedonosko]

This problem can be a normal behaviour as indexer works asynchronously and some nodes may gone to reflect the remote changes.

[vsellier]

It doesn't seems so normal, for example, with an empty dropbox account (only the 2 default files), when I link it, 4 exception are logged :

2018-08-27 11:35:53,257 | INFO  | Dropbox - test vse connect started. [o.e.clouddrive.rest.ConnectService<http-nio-127.0.0.1-10180-exec-9>]
2018-08-27 11:35:56,321 | ERROR | Error while indexing file 7abd585495ca548f56d0ebb80f13c515 [o.e.s.w.s.c.FileindexingConnector<ExoScheduler_Worker-21>]
javax.jcr.ItemNotFoundException: Node not found 7abd585495ca548f56d0ebb80f13c515 at collaboration
	at org.exoplatform.services.jcr.impl.core.SessionImpl.getNodeByIdentifier(SessionImpl.java:743) ~[exo.jcr.component.core-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.create(FileindexingConnector.java:124) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.update(FileindexingConnector.java:176) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.es.client.ElasticContentRequestBuilder.getCreatePipelineDocumentRequestContent(ElasticContentRequestBuilder.java:163) ~[commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processCUD(ElasticIndexingOperationProcessor.java:323) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processBulk(ElasticIndexingOperationProcessor.java:212) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.process(ElasticIndexingOperationProcessor.java:150) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.job.BulkIndexingJob.execute(BulkIndexingJob.java:41) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.2.jar:na]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.2.jar:na]
2018-08-27 11:36:01,372 | ERROR | Error while indexing file 7abd638c95ca548f4cdfd5ce6490bc24 [o.e.s.w.s.c.FileindexingConnector<ExoScheduler_Worker-25>]
javax.jcr.ItemNotFoundException: Node not found 7abd638c95ca548f4cdfd5ce6490bc24 at collaboration
	at org.exoplatform.services.jcr.impl.core.SessionImpl.getNodeByIdentifier(SessionImpl.java:743) ~[exo.jcr.component.core-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.create(FileindexingConnector.java:124) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.update(FileindexingConnector.java:176) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.es.client.ElasticContentRequestBuilder.getCreatePipelineDocumentRequestContent(ElasticContentRequestBuilder.java:163) ~[commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processCUD(ElasticIndexingOperationProcessor.java:323) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processBulk(ElasticIndexingOperationProcessor.java:212) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.process(ElasticIndexingOperationProcessor.java:150) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.job.BulkIndexingJob.execute(BulkIndexingJob.java:41) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.2.jar:na]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.2.jar:na]
2018-08-27 11:36:01,372 | ERROR | Error while indexing file 7abd585495ca548f56d0ebb80f13c515 [o.e.s.w.s.c.FileindexingConnector<ExoScheduler_Worker-25>]
javax.jcr.ItemNotFoundException: Node not found 7abd585495ca548f56d0ebb80f13c515 at collaboration
	at org.exoplatform.services.jcr.impl.core.SessionImpl.getNodeByIdentifier(SessionImpl.java:743) ~[exo.jcr.component.core-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.create(FileindexingConnector.java:124) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.update(FileindexingConnector.java:176) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.es.client.ElasticContentRequestBuilder.getCreatePipelineDocumentRequestContent(ElasticContentRequestBuilder.java:163) ~[commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processCUD(ElasticIndexingOperationProcessor.java:377) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processBulk(ElasticIndexingOperationProcessor.java:212) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.process(ElasticIndexingOperationProcessor.java:150) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.job.BulkIndexingJob.execute(BulkIndexingJob.java:41) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.2.jar:na]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.2.jar:na]
2018-08-27 11:36:01,373 | ERROR | Error while indexing file 7abd585495ca548f56d0ebb80f13c515 [o.e.s.w.s.c.FileindexingConnector<ExoScheduler_Worker-25>]
javax.jcr.ItemNotFoundException: Node not found 7abd585495ca548f56d0ebb80f13c515 at collaboration
	at org.exoplatform.services.jcr.impl.core.SessionImpl.getNodeByIdentifier(SessionImpl.java:743) ~[exo.jcr.component.core-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.create(FileindexingConnector.java:124) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.update(FileindexingConnector.java:176) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.es.client.ElasticContentRequestBuilder.getCreatePipelineDocumentRequestContent(ElasticContentRequestBuilder.java:163) ~[commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processCUD(ElasticIndexingOperationProcessor.java:377) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processBulk(ElasticIndexingOperationProcessor.java:212) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.process(ElasticIndexingOperationProcessor.java:150) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.job.BulkIndexingJob.execute(BulkIndexingJob.java:41) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.2.jar:na]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.2.jar:na]
2018-08-27 11:36:01,581 | INFO  | Dropbox - test vse successfully connected. [o.e.clouddrive.rest.ConnectService<clouddrive-thread-1>]

It seems at least these exceptions should be catched correctly to avoid to generate such noise or at worst, this ES indexation does not support clouddrive documents as if I'm not wrong they are not present on the plf storage.

[pnedonosko]

Several things should be taken in account: connected files have content with zero length, in some cases (like Dropbox) file originally uploaded will be renamed by a connector to follow hierarchy rules specific to the provider.
As we see in the stacktraces, it's not the add-on code but a job of the indexer - it's where we need a fix of an empty docs and file rename (e.g. ignore not found and rely on a next indexing job for a renamed).