From b82ec884128e1b28a46884bf9c2b6b7f7d55489b Mon Sep 17 00:00:00 2001 From: Giuseppe Iossa Date: Wed, 17 Jul 2019 11:15:58 +0200 Subject: [PATCH 1/2] added README files and updated dataset-ingest.xml --- README_kylo_template.md | 31 + README_logs.md | 35 + README_new.md | 233 + README_opendata.md | 105 + .../dataset_ingestion/dataset-ingest.xml | 8727 ++++++++--------- 5 files changed, 4763 insertions(+), 4368 deletions(-) create mode 100644 README_kylo_template.md create mode 100644 README_logs.md create mode 100644 README_new.md create mode 100644 README_opendata.md diff --git a/README_kylo_template.md b/README_kylo_template.md new file mode 100644 index 0000000..43d0e18 --- /dev/null +++ b/README_kylo_template.md @@ -0,0 +1,31 @@ +**How to register a new Kylo template** +=================== +This document describes how to update a NiFi template using Kylo in production environment. + +---------- +First of all, check if there's an active thread. If so, wait for the active threads to finish and stop every started processor, in order to avoid pipeline errors throwing. Now download the old template on your device, in order to have a backup copy in case of any need, and save the new template on NiFi with the same name of the old one following the steps listed below: + + 1. Copy the new template you decided to register on Kylo and paste it on the NiFi main page in a free room far from anything else; + 2. Create a new template on NiFi with the last one; + 3. Download the new template on your device; + 4. Delete the template you've just pasted on NiFi main page. + +After that, you can finally register the new template on Kylo. In order to do that, follow the steps listed below: + + 1. Stop every FetchSFTP processor because password are not stored, in order to avoid errors generated by queued flowfiles; + 2. Click on this [link](https://kylo.daf.teamdigitale.it/index.html#!/dashboard) to redirect Kylo home; + 3. On the left bar, click on Admin tab and then on Templates tab; + 4. Click on the orange button on the bottom of the page, in order to add a new template; + 5. Choose the "Import from a file" option; + 6. Click on "CHOOSE FILE" and select the xml file that contains the new NiFi template you've just downloaded. + 7. Check every box appearing at the bottom of the page and go on; + 8. Check that the registration has not produced errors and that everything has been done correctly. + +Before you start to use your new registered template, don't forget to update Github repository, typing in sequence +``` +git status +git add +git commit -m "" +git push +``` +If the user is not a repository contributor, he can execute this operations on a fork of the repository and then perform a pull request to the main repository. \ No newline at end of file diff --git a/README_logs.md b/README_logs.md new file mode 100644 index 0000000..d377ca1 --- /dev/null +++ b/README_logs.md @@ -0,0 +1,35 @@ +**How to view logs** +=================== + + +In this document there are instruction on how to view logs in production environment. + +---------- + +## **How to view Kylo and Nifi logs** + +### **Kylo logs** +In order to view Kylo logs, just enter in the path you saved daf-kylo projects and the go in the folder scripts and run `kylologs.sh`. You can enter in your folder typing +``` +cd /daf-kylo/scripts +``` +and run `kylologs.sh` typing +``` +./kylologs.sh +``` +### **NiFi logs** +NiFi logs are saved in the file named `nifi-app.log`, located in the logs folder under the folder you installed NiFi on your device. As instance, if you installed it in `/usr/nifi`, you can view the logs of the node named `tba-nifi-0` typing +``` +kubectl exec -it tba-nifi-0 -- tail -1000f /usr/nifi/logs/nifi-app.log +``` +## **How to view Spark logs** + + +First perform Kerberos initialization by typing `kinit -k -t daf@DAF.GOV.IT` on the command line, where `` stands for the local path daf.keytab file is saved, useful for the Kerberos initialization in production environment. +Then click on this [link](https://master-2.platform.daf.gov.it:8090/cluster/) on Firefox browser, but not in a private window. +Once redirected on the link, you can see a list of application id and everyone of them refers to the corresponding Spark job. Alongside them there are all the corresponding properties. Clicking on application-id, you are redirected to another web page where you can see a resume of the most important informations of the job. Clicking on the bottom right button, you can see Spark logs. + +Another way to view Spark jobs is opening the command-line of the node where yarn CLI is installed and type +``` +yarn logs -applicationId +``` \ No newline at end of file diff --git a/README_new.md b/README_new.md new file mode 100644 index 0000000..7cde8e4 --- /dev/null +++ b/README_new.md @@ -0,0 +1,233 @@ +# Daf-Kylo for PDND (Piattaforma Digitale Nazionale Dati), previously DAF (Data & Analytics Framework) + + + +>Daf-Kylo repository contains the set of tools used for the PDND data ingestion. + +> In order to install and use this repo you may deploy all the components into a _cCloudera_ shared edge node. + +## What is the PDND (previously DAF)? + +PDND stands for "Piattaforma Digitale Nazionale Dati" (Italian Digital Data Platform), previously known as Data & Analytics Framework (DAF). + +In brief, is an attempt to establish a central Chief Data Officer (CDO) for the Government and Public Administration. Its main goal is to promote data exchange among Italian Public Administrations (PAs), to support the diffusion of open data, and to enable data-driven policies. You can find more about the PDND on the official [Digital Transformation Team website](https://teamdigitale.governo.it/it/projects/daf.htm). + +## What is Daf-Kylo? + + + Daf-Kylo is a data lake platform built on Apache Hadoop and Spark. Daf-Kylo provides a data lake solution enabling self-service data ingest, data preparation, and data discovery. Kylo integrates best practices around metadata capture, security, and data quality. Apache Nifi provides a flexible data processing framework for building batch or streaming pipeline templates, and for enabling self-service features. + + +> Daf-Kylo repository contains the set of components used to deploy and manage the PDND data ingestion process. + +> Folder /docker contains all the docker files for build images of daf-kylo components. + +> Folder /kubernetes contains all the yaml files for deploy pods and services on kubernetes. + +> Folder /kylo contains all the kylo stuff such as api documentation for the integration with PDND Portal, kylo templates, kylo patch. + +> Folder /nifi contains all the nifi templates and customized processors used in ingestion process. + +> Folder /scripts contains utils scripts for manage pods, log and other kubernetes stuff. + +### Prerequisites + +* [Kubernetes](https://kubernetes.io/) +* [Cloudera](https://www.cloudera.com/) + +### Project dependencies +Project dependencies can be find by clicking on this [link](https://kylo.readthedocs.io/en/v0.8.3/installation/Dependencies.html#kylo-stack-dependencies). + +### Project components +Project Daf-Kylo depends by the following components. + +* **ActiveMQ** version 5.15.1, available [here]([https://activemq.apache.org/activemq-5151-release.html](https://activemq.apache.org/activemq-5151-release.html)); + +* **Elasticsearch** version 5.6.4, available [here]([https://www.elastic.co/downloads/past-releases/elasticsearch-5-6-4](https://www.elastic.co/downloads/past-releases/elasticsearch-5-6-4)); +* **MariaDB** version 10.3, available [here]([https://mariadb.com/kb/en/library/changes-improvements-in-mariadb-103/](https://mariadb.com/kb/en/library/changes-improvements-in-mariadb-103/)); +* **Spark** version 2.2.0, available [here]([http://spark.apache.org/downloads.html](http://spark.apache.org/downloads.html)); +* **Kylo-Services** version 9.1.0, available [here]([https://kylo.readthedocs.io/en/v0.9.1/about/Downloads.html](https://kylo.readthedocs.io/en/v0.9.1/about/Downloads.html)); +* **Kylo-UI** version 9.1.0, available [here]([https://kylo.readthedocs.io/en/v0.9.1/about/Downloads.html](https://kylo.readthedocs.io/en/v0.9.1/about/Downloads.html)); +* **NiFi** version 1.7.0, available [here]([https://nifi.apache.org/download.html](https://nifi.apache.org/download.html)). + +## How to install and use Daf-Kylo + + Insert here a brief documentation to use this project as an end-user (not a developer) if applicable, including pre-requisites and internal and external dependencies. Insert a link to an extended documentation (user manual) if present.--> + +### MacOS and Linux +Installing Daf-Kylo on Unix-like systems requires a package manager such as Homebrew. You can download and install Homebrew following the instructions given in the [Homebrew official website]([https://brew.sh/](https://brew.sh/)). Once you have installed Homebrew, you can follow some steps to complete the setup. First step is Homebrew cask installation. Open a terminal and type the following command to install Homebrew cask: +``` +brew tap caskroom/cask +``` +Then, update all formulas and Homebrew itself by typing +``` +brew update +``` +Last, install kube-controller-manager, RPM, make and Git by typing +``` +brew install kubectl rpm make git +``` + +## How to build Daf-Kylo + + Insert here a brief documentation for the developer to build, test and contribute. Insert a link to an extended documentation (developer manual) if present.--> +To build most of Docker images, kylo code is required (source and compiled). To get it run, you have to download and compile it, using `Makefile`, by typing the following commands (production and test environment): + + - #### Production + ``` make -f Makefile daf-kylo``` +```make -f Makefile build-kylo``` + + + - #### Test + ```make -f Makefile.test daf-kylo``` +```make -f Makefile.test build-kylo``` + +### Login to nexus repository +``` +docker login nexus.daf.teamdigitale.it +``` +### Build Docker images of the components +Once this is completed, you can build every image (production and test environment), by typing the following comands: + + + #### Production +``` +make activemq +``` +``` +make mysql +``` +``` +make kylo-services +``` +``` +make kylo-ui +``` +``` +make nifi +``` + #### Test +``` +make -f Makefile.test activemq +``` +``` +make -f Makefile.test mysql +``` +``` +make -f Makefile.test kylo-services +``` +``` +make -f Makefile.test kylo-ui +``` +``` +make -f Makefile.test nifi +``` + +### Push Docker images to local artifactory repository +Please ensure previously configuration of docker client as well as correct tagging the image has been performed. 'How to' can be found in: + [TeamDigitale onboarding 'Setup Docker '](https://docs.google.com/document/d/1KqeaZ2yj7rofslqzklYTCLb3AxPnV1mzOgSXOuTHTyw/edit?ts=59faf23f&pli=1#heading=h.ubxuumcef218) + [TeamDigitale onboarding 'Push Docker Image'](https://docs.google.com/document/d/1KqeaZ2yj7rofslqzklYTCLb3AxPnV1mzOgSXOuTHTyw/edit?ts=59faf23f&pli=1#heading=h.47zm3aqq5wip) + +After config and proper tagging has been done, push can be performed typing: +`docker push [repositoryurl:repositoryport/artifact:version]` . For instance ([namespace] is optional): + + - #### Production + ``` ./nexus_push.sh prod [namespace] ``` + + - #### Test + ``` ./nexus_push.sh test [namespace] ``` + + +### Deploy components in kubernetes cluster +Please ensure previously configuration of kubectl has been done. 'How to' can be found in: + -[TeamDigitale onboarding , 'Setup Kubernetes'](https://docs.google.com/document/d/1KqeaZ2yj7rofslqzklYTCLb3AxPnV1mzOgSXOuTHTyw/edit?ts=59faf23f&pli=1#heading=h.vvi8emze7m35) + +After configuration is completed, **pod deployment** into kubernetes cluster can be performed typing `./playbook.sh [component]` . As instance, ``` ./playbook.sh prod activemq [namespace] ```. +**Pod deletion** can be performed running the command `./cleanup.sh [environment] [component]`. As instance, ``` ./cleanup.sh prod activemq [namespace] ```. These instructions are valid in production environment, but to perform the same things in test environment, just replace `prod` with `test`. + + +### Mysql Configuration +By default the kylo database is not created in mysql container, so you have to create it. + +### Ldap Configuration +To configure Ldap authentication: + Edit the config-maps kylo-services.yaml & kylo-ui.yaml as follows: + + + + - `config-map/kylo-services.yaml` shoud be: + + - ##### Production + ```security.auth.ldap.server.uri=ldap://idm.daf.gov.it:389/cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` ```security.auth.ldap.server.authDn=uid=admin,cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` + ```security.auth.ldap.server.password=xxxxx``` + + - ##### Test + ```security.auth.ldap.server.uri=ldap://idm.teamdigitale.test:389/cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` +```security.auth.ldap.server.authDn=uid=application,cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` ``` security.auth.ldap.server.password=xxxxx``` + - ```config-map/kylo-ui.yaml``` + - ##### Production + ```security.auth.ldap.server.uri=ldap://idm.daf.gov.it:389/cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` +```security.auth.ldap.server.authDn=uid=admin,cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` +```security.auth.ldap.server.password=xxxxxx``` + - ##### Test + ```security.auth.ldap.server.uri=ldap://idm.teamdigitale.test:389/cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` +```security.auth.ldap.server.authDn=uid=application,cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` +```security.auth.ldap.server.password=xxxxxx``` + + +Once these two changes are done, first delete the pods, by typing: +``` +kubectl delete -f config-map/kylo-services.yaml +kubectl delete -f config-map/kylo-ui.yaml +``` +then redeploy as follows: +``` +kubectl apply -f config-map/kylo-services.yaml +kubectl apply -f config-map/kylo-ui.yaml +``` + +In the above example, it is not take in account the [namespace], so go on idm.teamdigitale.test and create an user such as *dladmin* with a password *password*. + +Now, you are able to login into kylo ui! + +As pointed out above, once this is done *ldap login* will be substituted by *default login* , this will allow to log in with default user `dladmin/thinkbig`. This has to be done to create users with the same name that those exist in ldap in order to grant them permissions (same functionality but for groups [is currently being fixed by R&D](https://kylo-io.atlassian.net/browse/KYLO-496)) . Once user(s) or group(s) is(are) created, change back `config-map/kylo-services.yaml` and `config-map/kylo-ui.yaml` and redeploy again. Ldap is now ready to start. + +### Bootstrap note +When kylo starts for the first time it need liquibase for creating Kylo DB, make sure that in the application.properties in kylo-service's config map: + +``` +liquibase.enabled=true +``` + + + +## Custom Processors + +[Here](./nifi/extensions/processors/Readme.md) you can find additional information about custom processors created for the [DAF](https://teamdigitale.governo.it/it/projects/daf.htm). + +## Further resources +- [PDND open data pipeline documentation](./README_opendata.md) +- [How to view logs](./README_logs.md) +- [How to register a new Kylo template](./README_kylo_template.md) + +## How to contribute + +Contributions are welcome. Feel free to [open issues](./issues) and submit a [pull request](./pulls) at any time, but please read [our handbook](https://github.com/teamdigitale/daf-handbook) first. + +## License + +Copyright (c) 2019 Presidenza del Consiglio dei Ministri + +This program is a free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License along with this program. If not, see . \ No newline at end of file diff --git a/README_opendata.md b/README_opendata.md new file mode 100644 index 0000000..d34d492 --- /dev/null +++ b/README_opendata.md @@ -0,0 +1,105 @@ +**PDND open data pipeline documentation** +=================== + +> **Note:** + +>At the time this documentation is written, PDND open data pipeline is still in test environment, and is not deployed yet on production cluster. Some things need to be changed from test environment, such as file path from which tables that are created on Hadoop refers. + + + + +---------- + + +**Open data feed pipeline operation** +---------------- + +After the harvesting from ckan portal, data are transferred through a kafka producer into the pipeline , in order to start ETL process that ends with the creation of the table on Superset. + +Pipeline operation can be divided four parts: + + - File saving on HDFS and table creation on Hadoop; + - Superset request payload construction; + - Kerberos initialization and ACL's setting; + - Creation of the table on Superset. + + +#### **File saving on HDFS and table creation on Hadoop** + +Open data feed pipeline starts with data retrieval through a Kafka consumer, which sends all required data to create the table on Hadoop to a python script. This one connects to Impala engine using `impala_admin` user, and then it provides to retrieve all necessary metadata from ElasticSearch through a post request. Once all necessary data are retrieved, the script performs the creation of the tables whose each of them refers to the files persisting in the directory `/usr/daf/open_data`, different from the one to use in production environment, that is `/daf/opendata`. Files persisting in `/user/daf/open_data` are saved under the path `/organization_name/database_name/table_name` and named with a progressive number, such as `/usr/daf/open_data/parco_salento/AGRI_policy/attivitacommerciali/256.csv`. + +This way, the script performs the database creation (if it doesn't exist), with the same name as `database_name` but with lower case letters, as instance `agri_policy`, and the table creation (if it doesn't exist), with the same name as `table_name`, concatenated with the same progressive number the file is named, as instance `attivitacommerciali_256`. + + +#### **Superset request payload construction** + +The tables on Superset are created through a post request, whose payload is a json with the same schema of the following ⧸⧸one: +``` +{ + "schema":"agri__policy", + "dbName":"default_org-db", + "tableName":"attivitacommerciali_256" +} +``` +Field "schema" refers to the database name where tables are saved on Hadoop through Impala engine; field "dbName" refers to the database name where Superset tables are saved; field "tableName" refers to the name the table is saved on Hadoop through Impala engine. + +>**Note:** +>String used to create the table to view Superset must be no longer than one hundred characters, otherwise an error is generated and the table cannot be created. + +#### **Kerberos initialization and ACL's setting** +Before performing the post request to create the table that will be viewed on Superset, permission are set as stated in the following table: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
PATH_HDFSOWNER:GROUPPERMISSIONSDEFAULT ACLCUSTOM ACL
/dafdaf:dafrwx:rwx:--x-NO
/opendatadaf:dafrwx:rwx:--xhive:rwx
impala:rwx		NO
/[org_name]daf:[org_name]rwx:rwx:--xhive:rwx
impala:rwx		NO
/[domain]__[subdomain]daf:[org_name]rwx:rwx:--xhive:rwx
impala:rwx		NO
/[dataset_name][default user]:
[default user group]		rwx:rwx:---hive:rwx
impala:rwx		open_data_group:rwx
/[files].csv[default user]:
[default user group]		rwx:rwx:---hive:rwx
impala:rwx		open_data_group:rwx
+ +#### **Creation of the table on Superset** +This is the last step and involves the performing of the post request, whose aim is to create the table to view on Superset. \ No newline at end of file diff --git a/nifi/templates/dataset_ingestion/dataset-ingest.xml b/nifi/templates/dataset_ingestion/dataset-ingest.xml index 59b9496..675c4c2 100644 --- a/nifi/templates/dataset_ingestion/dataset-ingest.xml +++ b/nifi/templates/dataset_ingestion/dataset-ingest.xml @@ -5,33 +5,35 @@ dataset-ingest - 0282040a-fc22-3dbf-0000-000000000000 + 00273300-3dfe-3ef9-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 6dec3299-8888-3a98-0000-000000000000 + 063409f8-4bfa-3b15-0000-000000000000 PROCESSOR 0 sec 1 + json + unmatched 23ce9c70-cc0f-3f5f-0000-000000000000 - d7a5a90b-6c3d-3c77-0000-000000000000 - FUNNEL + 860476bb-5a9d-3035-0000-000000000000 + PROCESSOR 0 - 0516a01b-b69a-34ae-0000-000000000000 + 037827eb-de13-396c-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - bcd13e12-5e54-33a3-0000-000000000000 + 64fafa08-776a-3827-0000-000000000000 PROCESSOR 0 sec @@ -40,216 +42,228 @@ success 23ce9c70-cc0f-3f5f-0000-000000000000 - ed4f807c-fa46-3b95-0000-000000000000 + 191f4f5b-95fd-3e6e-0000-000000000000 PROCESSOR 0 - 08f1889c-365a-327c-0000-000000000000 + 04141c52-cdda-34f3-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0 MB - 0 + 1 GB + 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 4ccf55a8-080b-3ecb-0000-000000000000 + 68b1e54b-b6cf-3a3d-0000-000000000000 PROCESSOR 0 sec 1 - failure + success 23ce9c70-cc0f-3f5f-0000-000000000000 - cc4a30e3-41ef-31cb-0000-000000000000 + 4ba1246f-19cc-306d-0000-000000000000 PROCESSOR 0 - 0aee0ecb-5007-3b6d-0000-000000000000 + 0622c426-f73d-3486-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 + + 92b2a711-0b0e-334c-0000-000000000000 + 5bc434a1-b153-1af7-8e62-9bb1cfa1d614 + REMOTE_INPUT_PORT + + 0 sec + 1 + + + 23ce9c70-cc0f-3f5f-0000-000000000000 + 033e4789-942e-3048-0000-000000000000 + INPUT_PORT + + 0 + + + 07f99d99-dcdf-3084-0000-000000000000 + 23ce9c70-cc0f-3f5f-0000-000000000000 + 0 MB + 0 23ce9c70-cc0f-3f5f-0000-000000000000 - 1e97c692-ca3d-3d8d-0000-000000000000 + d9c9739c-3cca-3ef5-0000-000000000000 PROCESSOR 0 sec 1 - Success + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 060348e3-8321-3a62-0000-000000000000 + 0a9fe7ec-7074-3ccc-0000-000000000000 PROCESSOR 0 - 0cac0045-12a3-321c-0000-000000000000 + 0a80f0ac-f880-3575-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1 GB - 10000 + 0 MB + 0 23ce9c70-cc0f-3f5f-0000-000000000000 - 1f4f614a-1dac-388d-0000-000000000000 - PROCESSOR + 704eaa32-d8fa-359e-0000-000000000000 + FUNNEL 0 sec 1 - success + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 34d2c912-9fbe-30fd-0000-000000000000 + 0a9fe7ec-7074-3ccc-0000-000000000000 PROCESSOR 0 - 0cbe985f-0037-3b11-0000-000000000000 + 0b12e166-0e1a-3cc4-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 4ccf55a8-080b-3ecb-0000-000000000000 + 4ba1246f-19cc-306d-0000-000000000000 PROCESSOR 0 sec 1 - nonzero status + csv 23ce9c70-cc0f-3f5f-0000-000000000000 - 4beca4ca-6d13-3ecf-0000-000000000000 + 236835a4-aae7-3c21-0000-000000000000 PROCESSOR 0 - 123d2a24-e5aa-336f-0000-000000000000 + 0b3df984-3044-3406-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 4ccf55a8-080b-3ecb-0000-000000000000 + c6f7d1f4-ed23-36a6-0000-000000000000 PROCESSOR 0 sec 1 - failure + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 8bc905b9-5af8-3dac-0000-000000000000 + 47844730-7389-3b61-0000-000000000000 PROCESSOR 0 - 124d7c8b-5a3f-3849-0000-000000000000 + 0b8b83d5-ac27-3408-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - b0c9e12b-361f-3016-0000-000000000000 + c919821c-8cfb-364d-0000-000000000000 PROCESSOR 0 sec 1 - unmatched + original 23ce9c70-cc0f-3f5f-0000-000000000000 - 348c9bd4-a8e3-38d6-0000-000000000000 + cca574f2-5a7c-39b4-0000-000000000000 PROCESSOR 0 - 14acb823-d1d0-31c8-0000-000000000000 + 0bd777c2-5bd9-3883-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 04765b20-01cd-3390-0000-000000000000 + 44bff5ad-533b-3fe1-0000-000000000000 PROCESSOR 0 sec 1 - success + Initialize 23ce9c70-cc0f-3f5f-0000-000000000000 - 780f1975-2e31-39bf-0000-000000000000 + 10c53230-14db-3f70-0000-000000000000 PROCESSOR 0 - 14d2602e-9796-3a83-0000-000000000000 + 0deba792-51ea-367a-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 4ccf55a8-080b-3ecb-0000-000000000000 + fe750c37-e414-3eb3-0000-000000000000 PROCESSOR 0 sec 1 - nonzero status + success 23ce9c70-cc0f-3f5f-0000-000000000000 - bcd13e12-5e54-33a3-0000-000000000000 + b56903b6-2b2e-393d-0000-000000000000 PROCESSOR 0 - 155069cf-9250-3bea-0000-000000000000 + 0fe6f90b-89d3-3ef4-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - ba9884ad-8316-3bde-0000-000000000000 - FUNNEL + 34a53e9b-ca69-3ff7-0000-000000000000 + PROCESSOR 0 sec 1 - nonzero status + success 23ce9c70-cc0f-3f5f-0000-000000000000 - a2ef77b3-5ef8-3aca-0000-000000000000 + 8f9e2045-8a96-35d6-0000-000000000000 PROCESSOR 0 - 16c446ad-25c1-399e-0000-000000000000 + 104a5862-235f-36cb-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1 GB - 10000 - - 3796.0943603515625 - 1264.7039562111813 - - - 3582.2972127343382 - 1562.308027426613 - + 0 MB + 0 23ce9c70-cc0f-3f5f-0000-000000000000 - ba9884ad-8316-3bde-0000-000000000000 + 704eaa32-d8fa-359e-0000-000000000000 FUNNEL 0 sec @@ -258,90 +272,82 @@ failure 23ce9c70-cc0f-3f5f-0000-000000000000 - f7e910c3-4d84-3a05-0000-000000000000 + e5da1cdd-8a89-31ee-0000-000000000000 PROCESSOR 0 - 16ea3ea8-e03f-3735-0000-000000000000 + 14bb57a8-dc1d-37ca-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 617e3f6f-79cb-3e23-0000-000000000000 + 60a96755-2f94-362f-0000-000000000000 PROCESSOR 0 sec 1 - Original + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 74996cd7-67b3-3444-0000-000000000000 + fe750c37-e414-3eb3-0000-000000000000 PROCESSOR 0 - 178dab65-776c-3fb8-0000-000000000000 + 159fa231-b09b-3ff8-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 - - 3131.654291244151 - 0.0 - - - 2979.6292057949317 - 3.372772216796875 - 23ce9c70-cc0f-3f5f-0000-000000000000 - 780f1975-2e31-39bf-0000-000000000000 + 8067c255-8025-3e08-0000-000000000000 PROCESSOR 0 sec 1 - failure + sync_strategy 23ce9c70-cc0f-3f5f-0000-000000000000 - 780f1975-2e31-39bf-0000-000000000000 + 1e248f57-ce77-3335-0000-000000000000 PROCESSOR 0 - 17a3cf96-722a-3831-0000-000000000000 + 19a56d78-aae1-3d51-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 348c9bd4-a8e3-38d6-0000-000000000000 + afd730ff-bc4f-362d-0000-000000000000 PROCESSOR 0 sec 1 - success + unmatched 23ce9c70-cc0f-3f5f-0000-000000000000 - 363c1650-b960-364e-0000-000000000000 + 6986a9d1-d207-379a-0000-000000000000 PROCESSOR 0 - 1885e8fb-1294-3ba3-0000-000000000000 + 2082a225-cb72-3549-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 3e9f9601-cbe6-311e-0000-000000000000 + 69cc1f3e-1675-3433-0000-000000000000 PROCESSOR 0 sec @@ -350,252 +356,231 @@ success 23ce9c70-cc0f-3f5f-0000-000000000000 - 4c8c3d92-9cd3-3176-0000-000000000000 + fe750c37-e414-3eb3-0000-000000000000 PROCESSOR 0 - 197361b9-1d91-39b8-0000-000000000000 + 23a66215-3e87-3625-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 8bc905b9-5af8-3dac-0000-000000000000 + 3f944c7c-955c-358c-0000-000000000000 PROCESSOR 0 sec 1 - original + fetch 23ce9c70-cc0f-3f5f-0000-000000000000 - 54858faa-cb0a-316c-0000-000000000000 + ab452d14-4cee-36ec-0000-000000000000 PROCESSOR 0 - 1a796d70-40f2-37bc-0000-000000000000 + 240b8129-683b-3e7f-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 6c6627cd-f171-3d99-0000-000000000000 + 379f734d-d44d-3da0-0000-000000000000 PROCESSOR 0 sec 1 - split + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 6df01890-8e1b-3c57-0000-000000000000 + 3a2b6768-96e1-3b29-0000-000000000000 PROCESSOR 0 - 1b211de8-f0b9-36f4-0000-000000000000 + 247cfbc1-6dc8-36a5-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 05f3c225-18e7-378f-0000-000000000000 + 1e248f57-ce77-3335-0000-000000000000 PROCESSOR 0 sec 1 - success + original 23ce9c70-cc0f-3f5f-0000-000000000000 - 6c525f77-bb00-3af2-0000-000000000000 + 37670014-d158-3260-0000-000000000000 PROCESSOR 0 - 1cb2be7f-3c45-35c7-0000-000000000000 + 24aed664-4384-3457-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 - - 3143.7561971093382 - 1078.1088697117693 - 23ce9c70-cc0f-3f5f-0000-000000000000 - d7a5a90b-6c3d-3c77-0000-000000000000 - FUNNEL + cca574f2-5a7c-39b4-0000-000000000000 + PROCESSOR 0 sec 1 - failure + original 23ce9c70-cc0f-3f5f-0000-000000000000 - f89770bd-6d42-3b62-0000-000000000000 + be3b6cae-4b6f-3eae-0000-000000000000 PROCESSOR 0 - 1d8d6fa9-6141-3744-0000-000000000000 + 25b4615a-e569-30b0-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 - - 3124.3299275780882 - 975.319257895364 - 23ce9c70-cc0f-3f5f-0000-000000000000 - d7a5a90b-6c3d-3c77-0000-000000000000 - FUNNEL + d966765c-2aa7-318c-0000-000000000000 + PROCESSOR 0 sec 1 - failure + original 23ce9c70-cc0f-3f5f-0000-000000000000 - 334a39b3-d4e6-36a9-0000-000000000000 + acaeab44-15cd-3ec7-0000-000000000000 PROCESSOR 0 - 1f1cd3a4-0306-34a7-0000-000000000000 + 26a7e410-5a18-3510-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 488adf06-3023-38cc-0000-000000000000 - PROCESSOR + 7c9db424-1c20-39e2-0000-000000000000 + FUNNEL 0 sec 1 - csv + comms.failure + permission.denied + not.found 23ce9c70-cc0f-3f5f-0000-000000000000 - 348c9bd4-a8e3-38d6-0000-000000000000 + 3f944c7c-955c-358c-0000-000000000000 PROCESSOR 0 - 206fcb74-e11e-37ea-0000-000000000000 + 2d22dd71-ee5e-3061-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 - - 3498.2490321750374 - 461.93743611679474 - - - 3498.2490321750374 - 511.93743611679474 - 23ce9c70-cc0f-3f5f-0000-000000000000 - 3613a651-2682-38c4-0000-000000000000 + c001362b-0826-312a-0000-000000000000 PROCESSOR 0 sec 1 - failure + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 3613a651-2682-38c4-0000-000000000000 + 364bcfc8-7a9e-3455-0000-000000000000 PROCESSOR 0 - 20e2d60e-88d6-32ed-0000-000000000000 + 2f96eff4-fe22-3e30-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 - - 1257.1100117909155 - 2301.272106080799 - - - 1203.887004588767 - 2306.107311158924 - 23ce9c70-cc0f-3f5f-0000-000000000000 - 74996cd7-67b3-3444-0000-000000000000 - PROCESSOR + 704eaa32-d8fa-359e-0000-000000000000 + FUNNEL 0 sec 1 - No Retry + nonzero status 23ce9c70-cc0f-3f5f-0000-000000000000 - 74996cd7-67b3-3444-0000-000000000000 + 37670014-d158-3260-0000-000000000000 PROCESSOR 0 - 219e34f2-7180-3616-0000-000000000000 + 3077ca6c-8b0e-3cc5-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 6dec3299-8888-3a98-0000-000000000000 + 5b7cab9c-bdbc-38cd-0000-000000000000 PROCESSOR 0 sec 1 + original 23ce9c70-cc0f-3f5f-0000-000000000000 - ba9884ad-8316-3bde-0000-000000000000 - FUNNEL + 34a53e9b-ca69-3ff7-0000-000000000000 + PROCESSOR 0 - 21ce9f48-1b27-34c6-0000-000000000000 + 323ad995-1f03-3377-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - ed4f807c-fa46-3b95-0000-000000000000 + 493467f3-142d-3841-0000-000000000000 PROCESSOR 0 sec 1 - success + original 23ce9c70-cc0f-3f5f-0000-000000000000 - 8bc905b9-5af8-3dac-0000-000000000000 + d966765c-2aa7-318c-0000-000000000000 PROCESSOR 0 - 281a954c-ff10-3526-0000-000000000000 + 340e55b3-fbcc-3564-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 0 MB 0 23ce9c70-cc0f-3f5f-0000-000000000000 - cb1b1572-cc36-3613-0000-000000000000 + 85693d1b-e1fb-3cc3-0000-000000000000 PROCESSOR 0 sec @@ -604,69 +589,69 @@ success 23ce9c70-cc0f-3f5f-0000-000000000000 - 08cadf18-eb10-31e5-0000-000000000000 + e5da1cdd-8a89-31ee-0000-000000000000 PROCESSOR 0 - 287ab694-2b6b-3769-0000-000000000000 + 3425b1dd-2345-3fe0-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 + + 762.7746074864081 + 2124.231237599186 + + + 762.7746074864081 + 2174.231237599186 + 23ce9c70-cc0f-3f5f-0000-000000000000 - 4c8c3d92-9cd3-3176-0000-000000000000 + 47844730-7389-3b61-0000-000000000000 PROCESSOR 0 sec 1 - sync_strategy + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 1f477628-082a-3744-0000-000000000000 + 47844730-7389-3b61-0000-000000000000 PROCESSOR 0 - 292396ab-9f5d-37ba-0000-000000000000 + 3512f795-c8fe-3d62-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 - - 2615.0511890397024 - 2932.9367243493134 - - - 2587.8810230240774 - 3006.3725153649384 - 23ce9c70-cc0f-3f5f-0000-000000000000 - 34d2c912-9fbe-30fd-0000-000000000000 + 364bcfc8-7a9e-3455-0000-000000000000 PROCESSOR 0 sec 1 - failure + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 34d2c912-9fbe-30fd-0000-000000000000 + 8067c255-8025-3e08-0000-000000000000 PROCESSOR 0 - 294056ea-68fb-3b0c-0000-000000000000 + 38fe381b-09d0-389b-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1 GB - 10000 + 0 MB + 0 23ce9c70-cc0f-3f5f-0000-000000000000 - f89770bd-6d42-3b62-0000-000000000000 + e5da1cdd-8a89-31ee-0000-000000000000 PROCESSOR 0 sec @@ -675,81 +660,90 @@ success 23ce9c70-cc0f-3f5f-0000-000000000000 - 334a39b3-d4e6-36a9-0000-000000000000 + 54a62b8a-df1a-352e-0000-000000000000 PROCESSOR 0 - 2b64fd66-3a27-3e7a-0000-000000000000 + 3eee5d7b-79ad-34dd-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 2 GB + 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - a8bfa77e-a933-390d-0000-000000000000 + 992aa566-a4f0-32a1-0000-000000000000 PROCESSOR 0 sec 1 - success + csv 23ce9c70-cc0f-3f5f-0000-000000000000 - 04765b20-01cd-3390-0000-000000000000 + afd730ff-bc4f-362d-0000-000000000000 PROCESSOR 0 - 2d0403c4-0fa5-3452-0000-000000000000 + 4481233b-e830-3849-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 + + 1233.1129926328977 + 2124.231237599186 + + + 1233.1129926328977 + 2174.231237599186 + - 3b19898c-9c2c-3471-0000-000000000000 - 53023c6c-639a-1b69-bb0d-2e24bc2fa40b - REMOTE_INPUT_PORT + 23ce9c70-cc0f-3f5f-0000-000000000000 + 379f734d-d44d-3da0-0000-000000000000 + PROCESSOR 0 sec 1 + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - f5d690fa-e073-3556-0000-000000000000 - INPUT_PORT + 379f734d-d44d-3da0-0000-000000000000 + PROCESSOR 0 - 2fed0ebd-8799-322d-0000-000000000000 + 48bedc02-9587-382f-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1 GB - 10000 + 0 MB + 0 23ce9c70-cc0f-3f5f-0000-000000000000 - 512978db-e788-3bbb-0000-000000000000 + 60a96755-2f94-362f-0000-000000000000 PROCESSOR 0 sec 1 - zip + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 41b0429d-f239-36cc-0000-000000000000 + b56903b6-2b2e-393d-0000-000000000000 PROCESSOR 0 - 3199d74d-221a-31c1-0000-000000000000 + 48ed9857-b167-35f4-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - b0c9e12b-361f-3016-0000-000000000000 + acaeab44-15cd-3ec7-0000-000000000000 PROCESSOR 0 sec @@ -758,19 +752,19 @@ success 23ce9c70-cc0f-3f5f-0000-000000000000 - b4b5a0b5-e5a1-3321-0000-000000000000 + caefa30d-a8dd-3b99-0000-000000000000 PROCESSOR 0 - 350806ca-9cfd-3c51-0000-000000000000 + 4b5721ea-6aed-3258-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 0 MB 0 23ce9c70-cc0f-3f5f-0000-000000000000 - 060348e3-8321-3a62-0000-000000000000 + be494f54-1626-3bb3-0000-000000000000 PROCESSOR 0 sec @@ -778,48 +772,40 @@ 23ce9c70-cc0f-3f5f-0000-000000000000 - 337a79be-4658-3348-0000-000000000000 + 0bbc7065-1162-30c9-0000-000000000000 INPUT_PORT 0 - 3a6999a0-9fba-31e4-0000-000000000000 + 4bd26e69-ecc1-3c4d-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - b0c9e12b-361f-3016-0000-000000000000 + 860476bb-5a9d-3035-0000-000000000000 PROCESSOR 0 sec 1 - success + original 23ce9c70-cc0f-3f5f-0000-000000000000 - f7e910c3-4d84-3a05-0000-000000000000 + 64fafa08-776a-3827-0000-000000000000 PROCESSOR 0 - 3b352cc3-bb51-305f-0000-000000000000 + 4be5d149-b623-3482-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1 GB - 10000 - - 304.5983420484372 - 2212.707897096424 - - - 306.18766699960906 - 2180.060680299549 - + 0 MB + 0 23ce9c70-cc0f-3f5f-0000-000000000000 - 6df01890-8e1b-3c57-0000-000000000000 + 60a96755-2f94-362f-0000-000000000000 PROCESSOR 0 sec @@ -828,107 +814,120 @@ failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 6df01890-8e1b-3c57-0000-000000000000 + 44bff5ad-533b-3fe1-0000-000000000000 PROCESSOR 0 - 3b673d0f-be9e-3419-0000-000000000000 + 50813d2a-dfc0-3a5f-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1 GB - 10000 + 0 MB + 0 23ce9c70-cc0f-3f5f-0000-000000000000 - 4ccf55a8-080b-3ecb-0000-000000000000 + 37670014-d158-3260-0000-000000000000 PROCESSOR 0 sec 1 - failure + success 23ce9c70-cc0f-3f5f-0000-000000000000 - b19e39a4-c7cf-3f4b-0000-000000000000 + d9c9739c-3cca-3ef5-0000-000000000000 PROCESSOR 0 - 427d3425-1193-3143-0000-000000000000 + 515d3a46-f425-319a-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - b19e39a4-c7cf-3f4b-0000-000000000000 + 69f7de56-b526-3b01-0000-000000000000 PROCESSOR 0 sec 1 + failure success 23ce9c70-cc0f-3f5f-0000-000000000000 - cc4a30e3-41ef-31cb-0000-000000000000 + d78fda84-0be8-3ebb-0000-000000000000 PROCESSOR 0 - 470503de-cb61-33c4-0000-000000000000 + 57074ea0-1c06-3a86-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 + + 1572.2858441953977 + 1732.0501108030876 + + + 1572.2858441953977 + 1782.0501108030876 + 23ce9c70-cc0f-3f5f-0000-000000000000 - ba9884ad-8316-3bde-0000-000000000000 - FUNNEL + 773eeee6-f37f-381e-0000-000000000000 + PROCESSOR 0 sec 1 - nonzero status + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - aa223d35-d465-3bf0-0000-000000000000 + 773eeee6-f37f-381e-0000-000000000000 PROCESSOR 0 - 4bf7e388-53b6-3dad-0000-000000000000 + 58bfb2ee-6827-3e7a-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0 MB - 0 + 1 GB + 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - ba9884ad-8316-3bde-0000-000000000000 - FUNNEL + 60a96755-2f94-362f-0000-000000000000 + PROCESSOR 0 sec 1 - failure + nonzero status 23ce9c70-cc0f-3f5f-0000-000000000000 - 9dcf860a-1f57-3ca5-0000-000000000000 + 34a53e9b-ca69-3ff7-0000-000000000000 PROCESSOR 0 - 4f2015c6-a861-3b15-0000-000000000000 + 59af81c1-a9f6-3a14-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 - 3132.0418701171875 - 632.4123721962897 + 3917.1161849023047 + 1475.4369904567138 + + + 3703.3190372850804 + 1773.0410616721456 23ce9c70-cc0f-3f5f-0000-000000000000 - d7a5a90b-6c3d-3c77-0000-000000000000 + 704eaa32-d8fa-359e-0000-000000000000 FUNNEL 0 sec @@ -937,187 +936,195 @@ failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 363c1650-b960-364e-0000-000000000000 + 68b1e54b-b6cf-3a3d-0000-000000000000 PROCESSOR 0 - 52b6c3a7-4a09-3edd-0000-000000000000 + 5a4304a1-9541-3da2-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0 MB - 0 + 1 GB + 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 4ccf55a8-080b-3ecb-0000-000000000000 + 236835a4-aae7-3c21-0000-000000000000 PROCESSOR 0 sec 1 - failure + success 23ce9c70-cc0f-3f5f-0000-000000000000 - ccab579f-ab04-3c4d-0000-000000000000 + 10c53230-14db-3f70-0000-000000000000 PROCESSOR 0 - 5767dd32-67a9-3fe4-0000-000000000000 + 5b1bcaa2-4418-3e69-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - ba9884ad-8316-3bde-0000-000000000000 - FUNNEL + 6986a9d1-d207-379a-0000-000000000000 + PROCESSOR 0 sec 1 - failure + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 9918077d-f872-3034-0000-000000000000 + 3f944c7c-955c-358c-0000-000000000000 PROCESSOR 0 - 5794d028-c54a-3730-0000-000000000000 + 60fb1fa7-bcd8-37eb-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0 MB - 0 + 1 GB + 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 4ccf55a8-080b-3ecb-0000-000000000000 + 47844730-7389-3b61-0000-000000000000 PROCESSOR 0 sec 1 - failure + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 6c525f77-bb00-3af2-0000-000000000000 + 2ceb4542-7611-3043-0000-000000000000 PROCESSOR 0 - 579516a8-94ca-35c3-0000-000000000000 + 613ddf1a-f013-3fbe-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1 GB - 10000 + 0 MB + 0 23ce9c70-cc0f-3f5f-0000-000000000000 - 6c525f77-bb00-3af2-0000-000000000000 + b56903b6-2b2e-393d-0000-000000000000 PROCESSOR 0 sec 1 - Initialize + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 363c1650-b960-364e-0000-000000000000 + db5d4b49-9b90-349b-0000-000000000000 PROCESSOR 0 - 59e7ad85-66cc-3b83-0000-000000000000 + 66e0c47e-773c-352e-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 4ccf55a8-080b-3ecb-0000-000000000000 + e57145c1-e445-3043-0000-000000000000 PROCESSOR 0 sec 1 - nonzero status + unmatched 23ce9c70-cc0f-3f5f-0000-000000000000 - 54858faa-cb0a-316c-0000-000000000000 + 1e248f57-ce77-3335-0000-000000000000 PROCESSOR 0 - 59e800ff-4700-3194-0000-000000000000 + 67e5c5fe-4b17-3a74-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - b4b5a0b5-e5a1-3321-0000-000000000000 - PROCESSOR + 704eaa32-d8fa-359e-0000-000000000000 + FUNNEL 0 sec 1 - success + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - f89770bd-6d42-3b62-0000-000000000000 + 493467f3-142d-3841-0000-000000000000 PROCESSOR 0 - 5f892c8b-66ec-38d2-0000-000000000000 + 68c53729-045e-33e8-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 + + 2057.588957591768 + 0.0 + + + 1905.5638721425494 + 3.372772216796875 + 23ce9c70-cc0f-3f5f-0000-000000000000 - b917516c-bb0d-3cec-0000-000000000000 + fc934a2f-cd5e-3adc-0000-000000000000 PROCESSOR 0 sec 1 - success + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 7278d10f-35a9-3248-0000-000000000000 + fc934a2f-cd5e-3adc-0000-000000000000 PROCESSOR 0 - 5ffe7340-add6-3af3-0000-000000000000 + 69214130-daaa-3753-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - dfcd2870-bb2d-3422-0000-000000000000 + 60a96755-2f94-362f-0000-000000000000 PROCESSOR 0 sec 1 - original + nonzero status 23ce9c70-cc0f-3f5f-0000-000000000000 - aa223d35-d465-3bf0-0000-000000000000 + be3b6cae-4b6f-3eae-0000-000000000000 PROCESSOR 0 - 662126e5-7cb3-3f5e-0000-000000000000 + 6a0e77b5-1c0c-321f-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - aa223d35-d465-3bf0-0000-000000000000 + c232c661-b279-3d2a-0000-000000000000 PROCESSOR 0 sec @@ -1126,90 +1133,83 @@ success 23ce9c70-cc0f-3f5f-0000-000000000000 - 524f04f2-dc07-324c-0000-000000000000 + c6f7d1f4-ed23-36a6-0000-000000000000 PROCESSOR 0 - 66cfee2b-9aeb-37b8-0000-000000000000 + 6a81ec4a-6896-3ba7-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - e6a77bb8-8cba-3da1-0000-000000000000 + 54a62b8a-df1a-352e-0000-000000000000 PROCESSOR 0 sec 1 + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 69ecdec3-8981-3326-0000-000000000000 - INPUT_PORT + 68b1e54b-b6cf-3a3d-0000-000000000000 + PROCESSOR 0 - 673b0b67-edee-379f-0000-000000000000 + 6b49f8fd-a8d6-32a4-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - ba9884ad-8316-3bde-0000-000000000000 - FUNNEL + 54a62b8a-df1a-352e-0000-000000000000 + PROCESSOR 0 sec 1 - nonzero status + unmatched 23ce9c70-cc0f-3f5f-0000-000000000000 - dfcd2870-bb2d-3422-0000-000000000000 + 236835a4-aae7-3c21-0000-000000000000 PROCESSOR 0 - 67cfac8e-a2ca-33eb-0000-000000000000 + 724a3b51-cee3-308d-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 - - 3729.7866626725154 - 2971.7350162155917 - - - 3729.7866626725154 - 3021.7350162155917 - 23ce9c70-cc0f-3f5f-0000-000000000000 - 3e9f9601-cbe6-311e-0000-000000000000 + 8f9e2045-8a96-35d6-0000-000000000000 PROCESSOR 0 sec 1 - failure + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 3e9f9601-cbe6-311e-0000-000000000000 + c919821c-8cfb-364d-0000-000000000000 PROCESSOR 0 - 6c1564f2-b54c-3d17-0000-000000000000 + 745e413b-55ad-3b6b-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0 MB - 0 + 1 GB + 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - ba9884ad-8316-3bde-0000-000000000000 - FUNNEL + 69f7de56-b526-3b01-0000-000000000000 + PROCESSOR 0 sec 1 @@ -1217,106 +1217,103 @@ failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 9e3481f4-73b6-324d-0000-000000000000 + 8c875558-9915-35da-0000-000000000000 PROCESSOR 0 - 6ce830c6-4110-3f96-0000-000000000000 + 751e295c-180a-388d-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 784f65ed-b041-39ce-0000-000000000000 - FUNNEL + f3a2b29e-94e9-3428-0000-000000000000 + PROCESSOR 0 sec 1 - comms.failure - permission.denied - not.found + zip 23ce9c70-cc0f-3f5f-0000-000000000000 - 915d89df-849f-3bd7-0000-000000000000 + 6986a9d1-d207-379a-0000-000000000000 PROCESSOR 0 - 70cb9c6e-a34a-3e01-0000-000000000000 + 79cce043-e33c-39bf-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 4114b3af-1de6-3868-0000-000000000000 + 60a96755-2f94-362f-0000-000000000000 PROCESSOR 0 sec 1 - content + nonzero status 23ce9c70-cc0f-3f5f-0000-000000000000 - a8bfa77e-a933-390d-0000-000000000000 + 5b7cab9c-bdbc-38cd-0000-000000000000 PROCESSOR 0 - 7324e21a-2bbe-348f-0000-000000000000 + 7bb6f93c-cfba-3254-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0 MB - 0 + 1 GB + 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0c9cf0d7-a93b-342f-0000-000000000000 + b482b334-667f-38d7-0000-000000000000 PROCESSOR 0 sec 1 - failure success 23ce9c70-cc0f-3f5f-0000-000000000000 - 6dec3299-8888-3a98-0000-000000000000 + db350530-8fef-3bab-0000-000000000000 PROCESSOR 0 - 7b9745bf-7065-3f4f-0000-000000000000 + 80782b77-6ede-34b0-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - effed46e-3491-3f02-0000-000000000000 + 8c875558-9915-35da-0000-000000000000 PROCESSOR 0 sec 1 - original + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 512978db-e788-3bbb-0000-000000000000 + 493467f3-142d-3841-0000-000000000000 PROCESSOR 0 - 7bf56ce0-dba1-3bb5-0000-000000000000 + 840f2017-1c46-3f92-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - aa223d35-d465-3bf0-0000-000000000000 + 773eeee6-f37f-381e-0000-000000000000 PROCESSOR 0 sec @@ -1325,70 +1322,82 @@ success 23ce9c70-cc0f-3f5f-0000-000000000000 - 1f4f614a-1dac-388d-0000-000000000000 + c232c661-b279-3d2a-0000-000000000000 PROCESSOR 0 - 7e029bbe-9414-31f0-0000-000000000000 + 87345be9-9016-3922-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 - - 2639.7091480240774 - 2686.549395247751 - - - 2620.7462573990774 - 2725.171953841501 - 23ce9c70-cc0f-3f5f-0000-000000000000 - 1f4f614a-1dac-388d-0000-000000000000 + 0c2eee64-f3a2-351d-0000-000000000000 PROCESSOR 0 sec 1 - failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 1f4f614a-1dac-388d-0000-000000000000 - PROCESSOR + fa9aba2b-6957-357e-0000-000000000000 + INPUT_PORT 0 - 84f7c4c6-7fc0-3ba2-0000-000000000000 + 87d87802-a2a6-3fe0-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 524f04f2-dc07-324c-0000-000000000000 + afd730ff-bc4f-362d-0000-000000000000 PROCESSOR 0 sec 1 - unmatched + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 1f477628-082a-3744-0000-000000000000 + f3a2b29e-94e9-3428-0000-000000000000 PROCESSOR 0 - 85ebca27-e917-39f3-0000-000000000000 + 87e3c7e1-3e12-3ae5-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1 GB - 10000 + 0 MB + 0 23ce9c70-cc0f-3f5f-0000-000000000000 - 4ccf55a8-080b-3ecb-0000-000000000000 + 60a96755-2f94-362f-0000-000000000000 + PROCESSOR + + 0 sec + 1 + + failure + + 23ce9c70-cc0f-3f5f-0000-000000000000 + db5d4b49-9b90-349b-0000-000000000000 PROCESSOR + + 0 + + + 8b053f3b-8e80-3002-0000-000000000000 + 23ce9c70-cc0f-3f5f-0000-000000000000 + 0 MB + 0 + + 23ce9c70-cc0f-3f5f-0000-000000000000 + 704eaa32-d8fa-359e-0000-000000000000 + FUNNEL 0 sec 1 @@ -1396,166 +1405,170 @@ failure 23ce9c70-cc0f-3f5f-0000-000000000000 - b19e39a4-c7cf-3f4b-0000-000000000000 + d9c9739c-3cca-3ef5-0000-000000000000 PROCESSOR 0 - 88799428-4a7e-39ec-0000-000000000000 + 8d8bb51c-6dfd-3c88-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 9918077d-f872-3034-0000-000000000000 + acaeab44-15cd-3ec7-0000-000000000000 PROCESSOR 0 sec 1 - original + success 23ce9c70-cc0f-3f5f-0000-000000000000 - dfcd2870-bb2d-3422-0000-000000000000 + e57145c1-e445-3043-0000-000000000000 PROCESSOR 0 - 89120d81-8e58-3159-0000-000000000000 + 8e246efd-f269-39ec-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 0 MB 0 23ce9c70-cc0f-3f5f-0000-000000000000 - ba9884ad-8316-3bde-0000-000000000000 - FUNNEL + 0a9fe7ec-7074-3ccc-0000-000000000000 + PROCESSOR 0 sec 1 - failure + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 524f04f2-dc07-324c-0000-000000000000 + 85693d1b-e1fb-3cc3-0000-000000000000 PROCESSOR 0 - 89a72301-4c90-38ae-0000-000000000000 + 9590061d-e75b-318f-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 + + 3245.3517521288304 + 1186.0522921408956 + 23ce9c70-cc0f-3f5f-0000-000000000000 - 1080a605-3847-395b-0000-000000000000 - PROCESSOR + cca0ec94-1c50-36a1-0000-000000000000 + FUNNEL 0 sec 1 - success + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - b19e39a4-c7cf-3f4b-0000-000000000000 + db350530-8fef-3bab-0000-000000000000 PROCESSOR 0 - 8acfd99a-656b-3684-0000-000000000000 + 966ea063-d567-3968-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 0 MB 0 23ce9c70-cc0f-3f5f-0000-000000000000 - 50a45cc2-f153-3cd6-0000-000000000000 - PROCESSOR + 704eaa32-d8fa-359e-0000-000000000000 + FUNNEL 0 sec 1 - success + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - b0c9e12b-361f-3016-0000-000000000000 + e57145c1-e445-3043-0000-000000000000 PROCESSOR 0 - 94941ab1-0e7d-3462-0000-000000000000 + 9bc04898-029b-3533-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - b4c25172-cd77-3227-0000-000000000000 - PROCESSOR + 7c9db424-1c20-39e2-0000-000000000000 + FUNNEL 0 sec 1 - success + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 512978db-e788-3bbb-0000-000000000000 + f3a2b29e-94e9-3428-0000-000000000000 PROCESSOR 0 - 94979521-bf19-3a26-0000-000000000000 + 9c9af385-cb74-3fe9-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 784f65ed-b041-39ce-0000-000000000000 - FUNNEL + db350530-8fef-3bab-0000-000000000000 + PROCESSOR 0 sec 1 - failure + json 23ce9c70-cc0f-3f5f-0000-000000000000 - 512978db-e788-3bbb-0000-000000000000 + 236835a4-aae7-3c21-0000-000000000000 PROCESSOR 0 - 970238ab-da7c-3ec3-0000-000000000000 + 9e19c52b-0247-3377-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1 GB - 10000 + 0 MB + 0 23ce9c70-cc0f-3f5f-0000-000000000000 - 4ccf55a8-080b-3ecb-0000-000000000000 + 87f8eff2-a6e3-31c6-0000-000000000000 PROCESSOR 0 sec 1 - nonzero status + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 05f3c225-18e7-378f-0000-000000000000 + 62898293-e3dc-3c0b-0000-000000000000 PROCESSOR 0 - a22b0d71-3596-3367-0000-000000000000 + 9e23d67b-5dbd-35f2-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 34d2c912-9fbe-30fd-0000-000000000000 + be3b6cae-4b6f-3eae-0000-000000000000 PROCESSOR 0 sec @@ -1564,104 +1577,108 @@ success 23ce9c70-cc0f-3f5f-0000-000000000000 - 3e9f9601-cbe6-311e-0000-000000000000 + 44bff5ad-533b-3fe1-0000-000000000000 PROCESSOR 0 - a2351b0c-e8c6-3bb0-0000-000000000000 + 9ebfcca4-12b1-343a-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 + + 3264.7780216600804 + 1288.8419039573018 + 23ce9c70-cc0f-3f5f-0000-000000000000 - 363c1650-b960-364e-0000-000000000000 - PROCESSOR + cca0ec94-1c50-36a1-0000-000000000000 + FUNNEL 0 sec 1 - merged + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 4114b3af-1de6-3868-0000-000000000000 + b482b334-667f-38d7-0000-000000000000 PROCESSOR 0 - a2e32f46-bffe-33f5-0000-000000000000 + 9fc7ad7a-ffc0-36ee-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 4beca4ca-6d13-3ecf-0000-000000000000 - PROCESSOR + 704eaa32-d8fa-359e-0000-000000000000 + FUNNEL 0 sec 1 - original + nonzero status 23ce9c70-cc0f-3f5f-0000-000000000000 - bcd13e12-5e54-33a3-0000-000000000000 + d966765c-2aa7-318c-0000-000000000000 PROCESSOR 0 - a3d403f5-d1c5-359b-0000-000000000000 + a2e8d095-1826-3c51-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1 GB - 10000 + 0 MB + 0 23ce9c70-cc0f-3f5f-0000-000000000000 - ba30bbc8-3d35-39f2-0000-000000000000 - PROCESSOR + 704eaa32-d8fa-359e-0000-000000000000 + FUNNEL 0 sec 1 - matched + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 6c6627cd-f171-3d99-0000-000000000000 + 85693d1b-e1fb-3cc3-0000-000000000000 PROCESSOR 0 - a46860c1-4488-3774-0000-000000000000 + a30878e6-434a-3f02-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 915d89df-849f-3bd7-0000-000000000000 + 2ceb4542-7611-3043-0000-000000000000 PROCESSOR 0 sec 1 - fetch + InvalidSessionHandle 23ce9c70-cc0f-3f5f-0000-000000000000 - b917516c-bb0d-3cec-0000-000000000000 + 063409f8-4bfa-3b15-0000-000000000000 PROCESSOR 0 - a62c8685-552c-35fd-0000-000000000000 + a78b7d4f-54aa-3eb5-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0 MB - 0 + 1 GB + 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - ba9884ad-8316-3bde-0000-000000000000 - FUNNEL + 60a96755-2f94-362f-0000-000000000000 + PROCESSOR 0 sec 1 @@ -1669,40 +1686,40 @@ failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 53ee0c1a-6ddc-3e9e-0000-000000000000 + c919821c-8cfb-364d-0000-000000000000 PROCESSOR 0 - a8907e23-f645-31be-0000-000000000000 + af81e693-3d7b-3153-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 4ccf55a8-080b-3ecb-0000-000000000000 + b36c6afa-6946-307b-0000-000000000000 PROCESSOR 0 sec 1 - failure + success 23ce9c70-cc0f-3f5f-0000-000000000000 - ed4f807c-fa46-3b95-0000-000000000000 + b482b334-667f-38d7-0000-000000000000 PROCESSOR 0 - aaa92cb2-069a-3bf7-0000-000000000000 + b12f94b5-68de-3ae0-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0 MB - 0 + 1 GB + 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 9dcf860a-1f57-3ca5-0000-000000000000 + 632de85c-953c-3834-0000-000000000000 PROCESSOR 0 sec @@ -1711,40 +1728,40 @@ success 23ce9c70-cc0f-3f5f-0000-000000000000 - 53ee0c1a-6ddc-3e9e-0000-000000000000 + 992aa566-a4f0-32a1-0000-000000000000 PROCESSOR 0 - ae567a90-6ec1-3f2a-0000-000000000000 + b2ac8e07-6532-33c9-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - d7a5a90b-6c3d-3c77-0000-000000000000 + 704eaa32-d8fa-359e-0000-000000000000 FUNNEL 0 sec 1 - failure + nonzero status 23ce9c70-cc0f-3f5f-0000-000000000000 - b4b5a0b5-e5a1-3321-0000-000000000000 + acaeab44-15cd-3ec7-0000-000000000000 PROCESSOR 0 - ae79580a-3621-3267-0000-000000000000 + b3f03165-4979-39f0-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 780f1975-2e31-39bf-0000-000000000000 + 8b8ed360-feb6-30b4-0000-000000000000 PROCESSOR 0 sec @@ -1753,27 +1770,27 @@ success 23ce9c70-cc0f-3f5f-0000-000000000000 - b4c25172-cd77-3227-0000-000000000000 + 632de85c-953c-3834-0000-000000000000 PROCESSOR 0 - b1eea50b-58a2-31db-0000-000000000000 + b4843d0a-e0d0-30a0-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 - 3749.3923755631404 - 2669.6554743493134 + 3870.4142001138825 + 2880.388508594846 - 3754.7229419693904 - 2733.5109430993134 + 3875.7447665201325 + 2944.243977344846 23ce9c70-cc0f-3f5f-0000-000000000000 - 4c8c3d92-9cd3-3176-0000-000000000000 + 8067c255-8025-3e08-0000-000000000000 PROCESSOR 0 sec @@ -1782,19 +1799,19 @@ failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 4c8c3d92-9cd3-3176-0000-000000000000 + 8067c255-8025-3e08-0000-000000000000 PROCESSOR 0 - b301a94b-e41c-30bd-0000-000000000000 + ba1106da-97c1-3ba9-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1 GB - 10000 + 0 MB + 0 23ce9c70-cc0f-3f5f-0000-000000000000 - f79f9e04-f93d-3609-0000-000000000000 + 6cffd4ca-db07-3169-0000-000000000000 PROCESSOR 0 sec @@ -1803,82 +1820,93 @@ success 23ce9c70-cc0f-3f5f-0000-000000000000 - 1e97c692-ca3d-3d8d-0000-000000000000 + 87f8eff2-a6e3-31c6-0000-000000000000 PROCESSOR 0 - b59f197b-c41f-30a1-0000-000000000000 + bd8ff991-fa04-33e4-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0 MB - 0 + 1 GB + 10000 + + 3850.8084872232575 + 3182.468050461124 + + + 3850.8084872232575 + 3232.468050461124 + 23ce9c70-cc0f-3f5f-0000-000000000000 - cc4a30e3-41ef-31cb-0000-000000000000 + 364bcfc8-7a9e-3455-0000-000000000000 PROCESSOR 0 sec 1 - success + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - ccab579f-ab04-3c4d-0000-000000000000 + 364bcfc8-7a9e-3455-0000-000000000000 PROCESSOR 0 - b6af1b4c-98ef-38ed-0000-000000000000 + c1ae6c8a-9069-31ca-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 00b9a736-83c4-30f5-0000-000000000000 + d78fda84-0be8-3ebb-0000-000000000000 PROCESSOR 0 sec 1 - success 23ce9c70-cc0f-3f5f-0000-000000000000 - 0c1a61dd-e748-36ba-0000-000000000000 - PROCESSOR + cca0ec94-1c50-36a1-0000-000000000000 + FUNNEL 0 - b72ccc90-d79f-3664-0000-000000000000 + c5e23ad6-764b-3a8e-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 + + 3253.0636946679297 + 843.1454064418213 + 23ce9c70-cc0f-3f5f-0000-000000000000 - f7e910c3-4d84-3a05-0000-000000000000 - PROCESSOR + cca0ec94-1c50-36a1-0000-000000000000 + FUNNEL 0 sec 1 - success + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 488adf06-3023-38cc-0000-000000000000 + 10c53230-14db-3f70-0000-000000000000 PROCESSOR 0 - b8de8315-d188-3160-0000-000000000000 + c6e22b96-8738-30b4-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 74996cd7-67b3-3444-0000-000000000000 + 62898293-e3dc-3c0b-0000-000000000000 PROCESSOR 0 sec @@ -1887,160 +1915,146 @@ success 23ce9c70-cc0f-3f5f-0000-000000000000 - 49a50f1b-67f0-354c-0000-000000000000 + 71d77886-d680-32b6-0000-000000000000 PROCESSOR 0 - ba66b3d2-bb44-3da2-0000-000000000000 + ca309727-2bb5-39dd-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0 MB - 0 + 1 GB + 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - a2ef77b3-5ef8-3aca-0000-000000000000 + 60a96755-2f94-362f-0000-000000000000 PROCESSOR 0 sec 1 - success + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 9dcf860a-1f57-3ca5-0000-000000000000 + fe750c37-e414-3eb3-0000-000000000000 PROCESSOR 0 - bdddf46b-4415-345c-0000-000000000000 + cdce8d13-d578-321b-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - ccab579f-ab04-3c4d-0000-000000000000 + 10c53230-14db-3f70-0000-000000000000 PROCESSOR 0 sec 1 - original + merged 23ce9c70-cc0f-3f5f-0000-000000000000 - 4beca4ca-6d13-3ecf-0000-000000000000 + 8b8ed360-feb6-30b4-0000-000000000000 PROCESSOR 0 - bed2e875-71ac-308d-0000-000000000000 + ceb68ccb-d317-3a14-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 6dec3299-8888-3a98-0000-000000000000 + 632de85c-953c-3834-0000-000000000000 PROCESSOR 0 sec 1 + json + unmatched 23ce9c70-cc0f-3f5f-0000-000000000000 - 784f65ed-b041-39ce-0000-000000000000 - FUNNEL + afd730ff-bc4f-362d-0000-000000000000 + PROCESSOR 0 - bf7516f8-e95c-327d-0000-000000000000 + cec642be-6776-35e3-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 - - 1432.0246159779272 - 2306.480602174549 - - - 1506.7249333607397 - 2299.263072877674 - 23ce9c70-cc0f-3f5f-0000-000000000000 - 74996cd7-67b3-3444-0000-000000000000 + fc934a2f-cd5e-3adc-0000-000000000000 PROCESSOR 0 sec 1 - Retry + zip + unmatched 23ce9c70-cc0f-3f5f-0000-000000000000 - 74996cd7-67b3-3444-0000-000000000000 + 6986a9d1-d207-379a-0000-000000000000 PROCESSOR 0 - bf78f4ad-c834-389a-0000-000000000000 + d02e5fab-bc65-315d-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 - - 1041.1151540028295 - 2313.822643190174 - - - 980.6523167835912 - 2304.605358033924 - 23ce9c70-cc0f-3f5f-0000-000000000000 - 74996cd7-67b3-3444-0000-000000000000 + d78fda84-0be8-3ebb-0000-000000000000 PROCESSOR 0 sec 1 - Failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 74996cd7-67b3-3444-0000-000000000000 - PROCESSOR + 704eaa32-d8fa-359e-0000-000000000000 + FUNNEL 0 - c8026a77-c092-369a-0000-000000000000 + d5d127a5-df2a-329d-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - b4c25172-cd77-3227-0000-000000000000 + db5d4b49-9b90-349b-0000-000000000000 PROCESSOR 0 sec 1 - unmatched + original 23ce9c70-cc0f-3f5f-0000-000000000000 - 41b0429d-f239-36cc-0000-000000000000 + 5b7cab9c-bdbc-38cd-0000-000000000000 PROCESSOR 0 - c80ed89d-5481-35b3-0000-000000000000 + d62a3023-8c3a-3419-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 3613a651-2682-38c4-0000-000000000000 + ab452d14-4cee-36ec-0000-000000000000 PROCESSOR 0 sec @@ -2049,104 +2063,107 @@ success 23ce9c70-cc0f-3f5f-0000-000000000000 - effed46e-3491-3f02-0000-000000000000 + f8446518-3873-33e9-0000-000000000000 PROCESSOR 0 - ce728056-b312-3b40-0000-000000000000 + d7d2a294-3763-3c83-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 41b0429d-f239-36cc-0000-000000000000 + 3a2b6768-96e1-3b29-0000-000000000000 PROCESSOR 0 sec 1 - not_to_fetch unmatched 23ce9c70-cc0f-3f5f-0000-000000000000 - b917516c-bb0d-3cec-0000-000000000000 + 063409f8-4bfa-3b15-0000-000000000000 PROCESSOR 0 - cff9cdc6-d57c-3e21-0000-000000000000 + d9d7fee0-61b3-3897-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 49a50f1b-67f0-354c-0000-000000000000 + 6986a9d1-d207-379a-0000-000000000000 PROCESSOR 0 sec 1 - success + not_to_fetch + unmatched 23ce9c70-cc0f-3f5f-0000-000000000000 - 0c9cf0d7-a93b-342f-0000-000000000000 + ab452d14-4cee-36ec-0000-000000000000 PROCESSOR 0 - d8e23f4d-8a75-319b-0000-000000000000 + dab1a2b9-b3aa-334c-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 41b0429d-f239-36cc-0000-000000000000 + f8446518-3873-33e9-0000-000000000000 PROCESSOR 0 sec 1 - success + Success 23ce9c70-cc0f-3f5f-0000-000000000000 - 915d89df-849f-3bd7-0000-000000000000 + 0c2eee64-f3a2-351d-0000-000000000000 PROCESSOR 0 - d9795f2e-547d-37e9-0000-000000000000 + daea293a-305b-3a67-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 6df01890-8e1b-3c57-0000-000000000000 + d78fda84-0be8-3ebb-0000-000000000000 PROCESSOR 0 sec 1 - Response 23ce9c70-cc0f-3f5f-0000-000000000000 - 74996cd7-67b3-3444-0000-000000000000 - PROCESSOR + 7c9db424-1c20-39e2-0000-000000000000 + FUNNEL 0 - e016604b-f7f8-38ed-0000-000000000000 + dc885e1a-0cd9-3130-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0 MB - 0 + 1 GB + 10000 + + 1796.2734715014558 + 2050.3522029743544 + 23ce9c70-cc0f-3f5f-0000-000000000000 - 53ee0c1a-6ddc-3e9e-0000-000000000000 + 191f4f5b-95fd-3e6e-0000-000000000000 PROCESSOR 0 sec @@ -2155,132 +2172,132 @@ success 23ce9c70-cc0f-3f5f-0000-000000000000 - 9e3481f4-73b6-324d-0000-000000000000 + 69f7de56-b526-3b01-0000-000000000000 PROCESSOR 0 - e3cd8741-4292-39e9-0000-000000000000 + e0b54f17-112b-3008-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 7278d10f-35a9-3248-0000-000000000000 + 54a62b8a-df1a-352e-0000-000000000000 PROCESSOR 0 sec 1 - Success + success 23ce9c70-cc0f-3f5f-0000-000000000000 - e6a77bb8-8cba-3da1-0000-000000000000 + b36c6afa-6946-307b-0000-000000000000 PROCESSOR 0 - e701a216-bc9b-331d-0000-000000000000 + e6b234fb-d663-35c7-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 - - 304.9787589185544 - 2005.430971421044 - - - 306.87931739023406 - 1962.332094467919 - 23ce9c70-cc0f-3f5f-0000-000000000000 - 6c6627cd-f171-3d99-0000-000000000000 + ec68082e-ecf5-39e3-0000-000000000000 PROCESSOR 0 sec 1 - failure + csv 23ce9c70-cc0f-3f5f-0000-000000000000 - 6c6627cd-f171-3d99-0000-000000000000 + 860476bb-5a9d-3035-0000-000000000000 PROCESSOR 0 - e9566585-abd0-35fb-0000-000000000000 + e8ff32a1-2029-3b5b-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 + + 2736.073013590445 + 3143.669758594846 + + + 2708.90284757482 + 3217.105549610471 + 23ce9c70-cc0f-3f5f-0000-000000000000 - 0c1a61dd-e748-36ba-0000-000000000000 + c001362b-0826-312a-0000-000000000000 PROCESSOR 0 sec 1 - uuid_matched_message + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - ba30bbc8-3d35-39f2-0000-000000000000 + c001362b-0826-312a-0000-000000000000 PROCESSOR 0 - e97f4229-1ec2-341e-0000-000000000000 + ef6faddc-0d8c-31f0-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0 MB - 0 + 1 GB + 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 9e3481f4-73b6-324d-0000-000000000000 - PROCESSOR + cca0ec94-1c50-36a1-0000-000000000000 + FUNNEL 0 sec 1 - success + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 50a45cc2-f153-3cd6-0000-000000000000 + b36c6afa-6946-307b-0000-000000000000 PROCESSOR 0 - ec0a719c-ffad-3fd8-0000-000000000000 + f112ec13-8281-379d-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - e29b1073-0e21-3bec-0000-000000000000 + 60a96755-2f94-362f-0000-000000000000 PROCESSOR 0 sec 1 - success + nonzero status 23ce9c70-cc0f-3f5f-0000-000000000000 - 9918077d-f872-3034-0000-000000000000 + cca574f2-5a7c-39b4-0000-000000000000 PROCESSOR 0 - edb078d1-0d1b-3772-0000-000000000000 + f2489647-02e0-343f-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0 MB - 0 + 1 GB + 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - ba9884ad-8316-3bde-0000-000000000000 + 7c9db424-1c20-39e2-0000-000000000000 FUNNEL 0 sec @@ -2289,69 +2306,69 @@ failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 50a45cc2-f153-3cd6-0000-000000000000 + 992aa566-a4f0-32a1-0000-000000000000 PROCESSOR 0 - ef2cbf44-bb2c-3117-0000-000000000000 + f45fb3ae-4b1e-32f3-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0 MB - 0 + 1 GB + 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 08cadf18-eb10-31e5-0000-000000000000 + 71d77886-d680-32b6-0000-000000000000 PROCESSOR 0 sec 1 - success + Success 23ce9c70-cc0f-3f5f-0000-000000000000 - f79f9e04-f93d-3609-0000-000000000000 + be494f54-1626-3bb3-0000-000000000000 PROCESSOR 0 - ef449b92-a24c-3cb5-0000-000000000000 + f90113af-05f9-3c38-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 334a39b3-d4e6-36a9-0000-000000000000 + 60a96755-2f94-362f-0000-000000000000 PROCESSOR 0 sec 1 - json + failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 348c9bd4-a8e3-38d6-0000-000000000000 + 8f9e2045-8a96-35d6-0000-000000000000 PROCESSOR 0 - f1c15946-89fa-346c-0000-000000000000 + fabf7bb7-7620-37c6-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 - 1559.1740300404272 - 1523.9300962988536 + 2760.73097257482 + 2897.2824294932834 - 1559.1740300404272 - 1573.9300962988536 + 2741.76808194982 + 2935.9049880870334 23ce9c70-cc0f-3f5f-0000-000000000000 - 00b9a736-83c4-30f5-0000-000000000000 + caefa30d-a8dd-3b99-0000-000000000000 PROCESSOR 0 sec @@ -2360,99 +2377,70 @@ failure 23ce9c70-cc0f-3f5f-0000-000000000000 - 00b9a736-83c4-30f5-0000-000000000000 + caefa30d-a8dd-3b99-0000-000000000000 PROCESSOR 0 - f27cf5f9-9fd6-343e-0000-000000000000 + fe5e781d-744c-3562-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 54858faa-cb0a-316c-0000-000000000000 + 063409f8-4bfa-3b15-0000-000000000000 PROCESSOR 0 sec 1 - original + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 05f3c225-18e7-378f-0000-000000000000 + ec68082e-ecf5-39e3-0000-000000000000 PROCESSOR 0 - f8bb6ff8-fb5c-34ba-0000-000000000000 + fe74d89d-798e-3094-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1f477628-082a-3744-0000-000000000000 - PROCESSOR - - 0 sec - 1 - - original - - 23ce9c70-cc0f-3f5f-0000-000000000000 - a2ef77b3-5ef8-3aca-0000-000000000000 - PROCESSOR - - 0 - - - fa3d0e08-8fbb-39bd-0000-000000000000 - 23ce9c70-cc0f-3f5f-0000-000000000000 - 0 MB - 0 - - 23ce9c70-cc0f-3f5f-0000-000000000000 - 0c9cf0d7-a93b-342f-0000-000000000000 + caefa30d-a8dd-3b99-0000-000000000000 PROCESSOR 0 sec 1 - failure + success 23ce9c70-cc0f-3f5f-0000-000000000000 - e29b1073-0e21-3bec-0000-000000000000 + c001362b-0826-312a-0000-000000000000 PROCESSOR 0 - fbbfad86-7cc3-3f7b-0000-000000000000 + ff055418-f29a-3523-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 1 GB 10000 - - 1004.9215275990209 - 1871.6822247136497 - - - 1007.6209904896459 - 1834.5665020573997 - 23ce9c70-cc0f-3f5f-0000-000000000000 - 617e3f6f-79cb-3e23-0000-000000000000 + c6f7d1f4-ed23-36a6-0000-000000000000 PROCESSOR 0 sec 1 - failure + success 23ce9c70-cc0f-3f5f-0000-000000000000 - 617e3f6f-79cb-3e23-0000-000000000000 + 379f734d-d44d-3da0-0000-000000000000 PROCESSOR 0 @@ -2682,81 +2670,81 @@ com.thinkbiganalytics.nifi.v2.core.metadata.MetadataProviderSelectorService - 784f65ed-b041-39ce-0000-000000000000 + 704eaa32-d8fa-359e-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 2174.975081108852 - 900.6857825020525 + 3179.543191584373 + 1900.3650211971744 - ba9884ad-8316-3bde-0000-000000000000 + 7c9db424-1c20-39e2-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 3058.5213670336307 - 1689.631986951642 + 2092.2940248002196 + 1089.8139583491466 - d7a5a90b-6c3d-3c77-0000-000000000000 + cca0ec94-1c50-36a1-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 2889.251280449083 - 1071.4367026833816 + 3010.2731049998256 + 1282.169736928914 - 337a79be-4658-3348-0000-000000000000 + 033e4789-942e-3048-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 90.33578173624119 - 720.5680289130132 + 0.0 + 644.8829032236554 1 - dataset-ingest-cleanup + dataset-ingest STOPPED false INPUT_PORT - 69ecdec3-8981-3326-0000-000000000000 + 0bbc7065-1162-30c9-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 379.28938481716614 - 159.7399444580078 + 211.35760628698335 + 931.3010631585457 1 - RemotePort + dataset-ingest-cleanup STOPPED false INPUT_PORT - f5d690fa-e073-3556-0000-000000000000 + fa9aba2b-6957-357e-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 362.05434732425783 - 461.93743611679474 + 5.879202531970805 + 374.5411367416382 1 - dataset-ingest + RemotePort STOPPED false INPUT_PORT - 0b1642b0-5882-370f-0000-000000000000 + 7de4d3b8-c120-339f-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 2708.971999838053 - 2321.8661012688326 + 2829.9938243887955 + 2532.599135514365 223.0777130126953 + - 1447.1922607421875 + 1196.8462829589844 - 9f174d4f-34ed-3d21-0000-000000000000 + b0b734eb-593f-3d96-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 0.0 - 684.772004635749 + 3396.1681644812015 + 973.3268178606195 - 574.7320556640625 - + 631.3086853027344 + - 1500.92626953125 + 1417.4462127685547 - a8aba586-ed34-3f2a-0000-000000000000 + bcc9748c-7cab-3c9f-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 3275.1463399304594 - 762.593783615087 + 121.02182455074217 + 895.5050388812815 - 631.3086853027344 - + 574.7320556640625 + - 1417.4462127685547 + 1500.92626953125 - d7a8f4c6-3f64-3d3e-0000-000000000000 + e34e5063-45b3-39c0-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 199.07283584350444 - 1398.2389954566288 + 267.7745615172935 + 1621.0854330224738 - 978.0617828369141 - + 1428.479019165039 + - 1774.8998413085938 + 1884.1813354492188 - de64c1f9-4d0d-3c03-0000-000000000000 + f5c3ac1e-b4e6-39f9-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 2592.740263429819 - 2589.6038858857255 + 5063.222019863242 + 716.1473839586397 - 563.1233062744141 - + 1265.4043579101562 + - 1196.8462829589844 + 1447.1922607421875 - 00b9a736-83c4-30f5-0000-000000000000 + 063409f8-4bfa-3b15-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1104.1740300404272 - 1483.9300962988536 + 552.9921430235227 + 2649.8595664089753 - nifi-hadoop-nar + nifi-standard-nar org.apache.nifi 1.7.0 WARN - - 10 + Capta l'errore riscontrato dal flusso e definisce la gestione dello specifico errore (ES. per l'errore InvalidSessionHandle, il flusso viene rilanciato da capo) + 1 - Permissions umask - - Permissions umask - - - - Hadoop Configuration Resources - - Hadoop Configuration Resources - - - - Remote Owner - - Remote Owner - - - - Additional Classpath Resources - - Additional Classpath Resources - - - - Compression codec - - Compression codec - - - - IO Buffer Size - - IO Buffer Size - - - - Remote Group - - Remote Group - - - - Block Size + InvalidSessionHandle - Block Size + InvalidSessionHandle - kerberos-credentials-service + Routing Strategy - org.apache.nifi.kerberos.KerberosCredentialsService - kerberos-credentials-service + Routing Strategy + + ALL + false + 30 sec + - Kerberos Keytab - - Kerberos Keytab - + InvalidSessionHandle + ${Message:toLower():contains('invalid sessionhandle')} - Kerberos Principal - - Kerberos Principal - + Routing Strategy + Route to Property name + + 0 + 0 sec + TIMER_DRIVEN + 1 sec + + false + Failure Handling + + false + InvalidSessionHandle + + + false + unmatched + + STOPPED + + org.apache.nifi.processors.standard.RouteOnAttribute + + + 0a9fe7ec-7074-3ccc-0000-000000000000 + 23ce9c70-cc0f-3f5f-0000-000000000000 + + 3908.1616103958004 + 2194.9063947164504 + + + nifi-scripting-nar + org.apache.nifi + 1.7.0 + + + WARN + + 1 + - Replication + Script File - Replication + Script File - Kerberos Relogin Period + Script Engine - Kerberos Relogin Period + Script Engine - Directory + Script Body - Directory + Script Body - Conflict Resolution Strategy + Module Directory - Conflict Resolution Strategy + Module Directory @@ -2981,59 +2969,35 @@ 30 sec - Permissions umask - 077 - - - Hadoop Configuration Resources - /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml - - - Remote Owner - ${daf.owner} - - - Additional Classpath Resources - - - Compression codec - NONE - - - IO Buffer Size - - - Remote Group - ${daf.owner} - - - Block Size - - - kerberos-credentials-service - - - Kerberos Keytab - /usr/nifi/daf.keytab - - - Kerberos Principal - daf@DAF.GOV.IT - - - Replication + Script File - Kerberos Relogin Period - 4 hours + Script Engine + Groovy - Directory - /daf/${daf.type}/${daf.organization}/kylo/${daf.domain}__${daf.subdomain}/${feed}/log + Script Body + def flowFile = session.get() + if(!flowFile) return + def json = flowFile.getAttribute("metadata.table.fieldPoliciesJson"); + def inputFolder = flowFile.getAttribute("spark.input_folder") + def feed = flowFile.getAttribute("feed") + def category = flowFile.getAttribute("category") + def feedts = flowFile.getAttribute("feedts") + def folder = new File(inputFolder + "/"+category+"/"+feed+"/"+feedts) + // If it doesn't exist + if( !folder.exists() ) { + // Create all folders + folder.mkdirs() + } + def jsonFile = new File(folder,feed+"_field_policy.json") + jsonFile.write(json) + flowFile = session.putAttribute(flowFile,"table_field_policy_json_file",jsonFile.getCanonicalPath()) + session.transfer(flowFile, REL_SUCCESS) + - Conflict Resolution Strategy - fail + Module Directory 0 @@ -3042,35 +3006,35 @@ 1 sec false - Logs to Recovery Area + Fetch field policies false failure - true + false success STOPPED - org.apache.nifi.processors.hadoop.PutHDFS + org.apache.nifi.processors.script.ExecuteScript - 04765b20-01cd-3390-0000-000000000000 + 0c2eee64-f3a2-351d-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 3320.126164687026 - 119.73994445800781 + 291.62232208251953 + 334.5411367416382 - nifi-update-attribute-nar - org.apache.nifi - 1.7.0 + kylo-nifi-core-v1.2-nar + default + unversioned WARN @@ -3078,27 +3042,28 @@ 1 - Delete Attributes Expression + Metadata Provider Service - Delete Attributes Expression + com.thinkbiganalytics.nifi.core.api.metadata.MetadataProviderService + Metadata Provider Service - filename + System feed category - filename + System feed category - Store State + Cache Expiry Duration - Store State + Cache Expiry Duration - Stateful Variables Initial Value + System feed name - Stateful Variables Initial Value + System feed name @@ -3107,18 +3072,20 @@ 30 sec - Delete Attributes Expression + Metadata Provider Service + ad55fab7-e0c2-3b0f-0000-000000000000 - filename - ${filename:substringBefore('.bz2')} + System feed category + ${category} - Store State - Do not store state + Cache Expiry Duration + 1 secs - Stateful Variables Initial Value + System feed name + ${feed} 0 @@ -3127,31 +3094,26 @@ 1 sec false - Remove .bz2 Extension + GetFeedMetadata false - success + Success STOPPED - - org.apache.nifi.processors.attributes.UpdateAttribute + - org.apache.nifi.processors.standard.ExecuteStreamCommand + com.thinkbiganalytics.nifi.v2.init.InitializeFeed - 060348e3-8321-3a62-0000-000000000000 + 191f4f5b-95fd-3e6e-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 35.33578173624119 - 836.9224897761242 + 1622.954747283562 + 2084.231237599186 - kylo-nifi-core-v1.2-nar - default - unversioned + nifi-update-attribute-nar + org.apache.nifi + 1.7.0 WARN @@ -3278,28 +3267,39 @@ 1 - Metadata Provider Service + str_length - com.thinkbiganalytics.nifi.core.api.metadata.MetadataProviderService - Metadata Provider Service + str_length - System feed category + Delete Attributes Expression - System feed category + Delete Attributes Expression - Cache Expiry Duration + filename - Cache Expiry Duration + filename - System feed name + Store State - System feed name + Store State + + + + uuid.toMatch + + uuid.toMatch + + + + Stateful Variables Initial Value + + Stateful Variables Initial Value @@ -3308,20 +3308,26 @@ 30 sec - Metadata Provider Service - ad55fab7-e0c2-3b0f-0000-000000000000 + str_length + 2000 - System feed category - ${category} + Delete Attributes Expression - Cache Expiry Duration - 1 secs + filename + ${now():toNumber()}_${filename} - System feed name - ${feed} + Store State + Do not store state + + + uuid.toMatch + ${uuid} + + + Stateful Variables Initial Value 0 @@ -3330,26 +3336,31 @@ 1 sec false - GetFeedMetadata + Prepend TS and Save UUID false - Success + success STOPPED - + org.apache.nifi.processors.attributes.UpdateAttribute - 08cadf18-eb10-31e5-0000-000000000000 + 1e248f57-ce77-3335-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1138.667866352378 - 1084.611032452206 + 3395.8084872232575 + 2582.1659832268597 - kylo-nifi-core-v1.2-nar - default - unversioned + nifi-standard-nar + org.apache.nifi + 1.7.0 WARN @@ -3357,34 +3368,15 @@ 1 - Table Type - - Table Type - - - - Additional Tables - - Additional Tables - - - - System feed category - - System feed category - - - - Database Connection Pooling Service + Routing Strategy - com.thinkbiganalytics.nifi.v2.thrift.ThriftService - Database Connection Pooling Service + Routing Strategy - System feed name + sync_strategy - System feed name + sync_strategy @@ -3393,23 +3385,12 @@ 30 sec - Table Type - ALL - - - Additional Tables - - - System feed category - ${daf.domain}__${daf.subdomain} - - - Database Connection Pooling Service - a11778f5-3976-3513-0000-000000000000 + Routing Strategy + Route to Property name - System feed name - ${feed} + sync_strategy + ${metadata.table.targetMergeStrategy:contains("SYNC")} 0 @@ -3418,33 +3399,33 @@ 1 sec false - Drop Feed Tables + Route If Merge Strategy is Sync - true - failure + false + sync_strategy false - success + unmatched STOPPED - com.thinkbiganalytics.nifi.v2.ingest.DropFeedTables + org.apache.nifi.processors.standard.RouteOnAttribute - 0c1a61dd-e748-36ba-0000-000000000000 + 236835a4-aae7-3c21-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 399.51945410898406 - 1483.9300962988536 + 3908.1616103958004 + 999.0533824479025 - nifi-update-attribute-nar + nifi-standard-nar org.apache.nifi 1.7.0 @@ -3454,27 +3435,21 @@ 1 - Delete Attributes Expression - - Delete Attributes Expression - - - - filename + Routing Strategy - filename + Routing Strategy - Store State + csv - Store State + csv - Stateful Variables Initial Value + json - Stateful Variables Initial Value + json @@ -3483,26 +3458,16 @@ 30 sec - Delete Attributes Expression - - - filename - ${filename:contains('.csv'):ifElse( - ${filename:replace('.csv','.log')}, -${filename:contains('.json'):ifElse( - ${filename:replace('.json','.log')}, -${filename:contains('.zip'):ifElse( - ${filename:replace('.zip','.log')},${filename:append('.log')} -)} -)} -)} + Routing Strategy + Route to Property name - Store State - Do not store state + csv + ${daf.format:equals('csv')} - Stateful Variables Initial Value + json + ${daf.format:equals('json')} 0 @@ -3511,83 +3476,73 @@ ${filename:contains('.zip'):ifElse( 1 sec false - Replace .csv|.zip|.json with .log + RouteOnAttribute false - success + csv + + + false + json + + + false + unmatched STOPPED - org.apache.nifi.processors.attributes.UpdateAttribute + org.apache.nifi.processors.standard.RouteOnAttribute - 0c9cf0d7-a93b-342f-0000-000000000000 + 2ceb4542-7611-3043-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1603.1673771368914 - 1777.1364483464622 + 310.7281597325018 + 2302.71222861523 - nifi-standard-nar + nifi-update-attribute-nar org.apache.nifi 1.7.0 WARN - - 10 + Imposta l'attributo RestartFromBeginning a true + 1 - Group - - Group - - - - Owner - - Owner - - - - Create Missing Directories - - Create Missing Directories - - - - Permissions + Delete Attributes Expression - Permissions + Delete Attributes Expression - Maximum File Count + Message - Maximum File Count + Message - Last Modified Time + Store State - Last Modified Time + Store State - Directory + RestartFromBeginning - Directory + RestartFromBeginning - Conflict Resolution Strategy + Stateful Variables Initial Value - Conflict Resolution Strategy + Stateful Variables Initial Value @@ -3596,31 +3551,22 @@ ${filename:contains('.zip'):ifElse( 30 sec - Group - - - Owner - - - Create Missing Directories - true - - - Permissions + Delete Attributes Expression - Maximum File Count + Message + ${Message:replace('\n',' ')} - Last Modified Time + Store State + Do not store state - Directory - ${kylo.tmp.baseFolder:replaceEmpty('/tmp/kylo-nifi')}/failed-flows/${category}/${feed}/${feedts} + RestartFromBeginning + true - Conflict Resolution Strategy - ignore + Stateful Variables Initial Value 0 @@ -3629,35 +3575,26 @@ ${filename:contains('.zip'):ifElse( 1 sec false - Failed Flow - - true - failure - + RestartAttribute false success STOPPED - - org.apache.nifi.processors.standard.PutFile + - com.thinkbiganalytics.nifi.v2.init.CompleteInitializeFeed + org.apache.nifi.processors.standard.ExecuteStreamCommand - 1e97c692-ca3d-3d8d-0000-000000000000 + 364bcfc8-7a9e-3455-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 35.33578173624119 - 1084.611032452206 + 3395.8084872232575 + 3142.468050461124 - nifi-update-attribute-nar - org.apache.nifi - 1.7.0 + kylo-nifi-core-v1.2-nar + default + unversioned WARN @@ -3753,228 +3721,138 @@ ${filename:contains('.zip'):ifElse( 1 - daf.type + Feed partition value - daf.type + Feed partition value - daf.domain + Merge Strategy - daf.domain + Merge Strategy - daf.standard.dataset + Hive Configurations - daf.standard.dataset + Hive Configurations - hive.profile.root + Database Connection Pooling Service - hive.profile.root + com.thinkbiganalytics.nifi.v2.thrift.ThriftService + Database Connection Pooling Service - Store State + Source table - Store State + Source table - daf.format + Hive Connection Pooling Service - daf.format + org.apache.nifi.dbcp.hive.HiveDBCPService + Hive Connection Pooling Service - daf.external.target + Field specification - daf.external.target + Field specification - merge.correlation + Target schema - merge.correlation + Target schema - nifi.home + Source schema - nifi.home + Source schema - skipHeader + Reset hive on connection - skipHeader + Reset hive on connection - Stateful Variables Initial Value + Target table - Stateful Variables Initial Value + Target table - daf.separator.char + Partition Specification - daf.separator.char + Partition Specification - hdfs.archive.root + blocking-id - hdfs.archive.root + blocking-id + + ALL + false + 30 sec + - daf.quote.char - - daf.quote.char - + Feed partition value + ${feedts} - daf.subdomain - - daf.subdomain - + Merge Strategy + ${metadata.table.targetMergeStrategy} - hdfs.ingest.root - - hdfs.ingest.root - + Hive Configurations - Delete Attributes Expression - - Delete Attributes Expression - + Database Connection Pooling Service + a11778f5-3976-3513-0000-000000000000 - daf.escape.char - - daf.escape.char - + Source table + ${feed}_valid - daf.organization - - daf.organization - - - - metadata.table.targetTblProperties.parquet - - metadata.table.targetTblProperties.parquet - - - - hive.ingest.root - - hive.ingest.root - - - - hive.master.root - - hive.master.root - - - - ALL - false - 30 sec - - - daf.type - ${feedJson:jsonPath('$.userProperties.dafType')} - - - daf.domain - ${feedJson:jsonPath('$.userProperties.dafDomain')} - - - daf.standard.dataset - ${feedJson:jsonPath('$.userProperties.dafStandardDataset')} - - - hive.profile.root - ${hive.profile.root:replaceEmpty('/model.db')} - - - Store State - Store state locally - - - daf.format - ${feedJson:jsonPath('$.userProperties.dafFormat')} - - - daf.external.target - ${feedJson:jsonPath('$.userProperties.dafTarget')} - - - merge.correlation - ${category}.${feed}.correlation - - - nifi.home - ${nifi.home:replaceEmpty("/opt/nifi")} - - - skipHeader - ${skipHeader:replaceEmpty('false')} - - - Stateful Variables Initial Value - - - - daf.separator.char - ${metadata.table.feedFormat:substringAfterLast("separatorChar' = '"):substringBeforeLast("' ,'escapeChar'")} - - - hdfs.archive.root - ${hdfs.archive.root:replaceEmpty('/archive')} - - - daf.quote.char - ${metadata.table.feedFormat:substringAfterLast("quoteChar' = '"):substringBeforeLast("') STORED AS TEXTFILE")} - - - daf.subdomain - ${feedJson:jsonPath('$.userProperties.dafSubdomain')} + Hive Connection Pooling Service - hdfs.ingest.root - ${hdfs.ingest.root:replaceEmpty('/etl')} + Field specification + ${metadata.table.fieldStructure} - Delete Attributes Expression + Target schema + ${daf.domain}__${daf.subdomain} - daf.escape.char - ${metadata.table.feedFormat:substringAfterLast("escapeChar' = '"):substringBeforeLast("' ,'quoteChar' = '")} + Source schema + ${daf.domain}__${daf.subdomain} - daf.organization - ${category} + Reset hive on connection + true - metadata.table.targetTblProperties.parquet - tblproperties("parquet.compression"="SNAPPY","EXTERNAL"="TRUE") + Target table + temp_${feed} - hive.ingest.root - ${hive.ingest.root:replaceEmpty('/model.db')} + Partition Specification + ${metadata.table.partitionSpecs} - hive.master.root - ${hive.master.root:replaceEmpty('/app/warehouse')} + blocking-id 0 @@ -3983,10 +3861,10 @@ ${filename:contains('.zip'):ifElse( 1 sec false - Set Feed Defaults + Merge Table into Temp_table - true - set state fail + false + failure false @@ -3996,17 +3874,17 @@ ${filename:contains('.zip'):ifElse( - org.apache.nifi.processors.attributes.UpdateAttribute + com.thinkbiganalytics.nifi.v2.ingest.MergeTable - 1f477628-082a-3744-0000-000000000000 + 37670014-d158-3260-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 3274.7866626725154 - 2371.432948981327 + 3908.1616103958004 + 2582.1659832268597 nifi-standard-nar @@ -4016,18 +3894,48 @@ ${filename:contains('.zip'):ifElse( WARN - 1 + 0 - Routing Strategy + Max Attribute Length - Routing Strategy + Max Attribute Length - sync_strategy + Command Path - sync_strategy + Command Path + + + + Working Directory + + Working Directory + + + + Output Destination Attribute + + Output Destination Attribute + + + + Argument Delimiter + + Argument Delimiter + + + + Ignore STDIN + + Ignore STDIN + + + + Command Arguments + + Command Arguments @@ -4036,330 +3944,100 @@ ${filename:contains('.zip'):ifElse( 30 sec - Routing Strategy - Route to Property name + Max Attribute Length + 256 - sync_strategy - ${metadata.table.targetMergeStrategy:contains("SYNC")} + Command Path + /usr/bin/kinit + + + Working Directory + + + Output Destination Attribute + + + Argument Delimiter + ; + + + Ignore STDIN + false + + + Command Arguments + -k;-t;/usr/nifi/daf.keytab;daf@DAF.GOV.IT 0 0 sec - TIMER_DRIVEN + EVENT_DRIVEN 1 sec false - Route If Merge Strategy is Sync + Kerberos Initialization false - sync_strategy + nonzero status false - unmatched + original + + + true + output stream STOPPED - org.apache.nifi.processors.standard.RouteOnAttribute + org.apache.nifi.processors.standard.ExecuteStreamCommand - 1f4f614a-1dac-388d-0000-000000000000 + 379f734d-d44d-3da0-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 2742.2140308365774 - 2621.265703841501 + 778.1129926328977 + 2084.231237599186 - kylo-nifi-core-v1.2-nar - default - unversioned + nifi-hadoop-nar + org.apache.nifi + 1.7.0 WARN - - 1 + Trasporta il flowfile nell'area dei files falliti che si trova nel path /daf/${daf.type}/${daf.organization}/kylo/${daf.domain}__${daf.subdomain}/${feed}/failed + 10 - Statement + Permissions umask - Statement + Permissions umask - Database Connection Pooling Service + Hadoop Configuration Resources - com.thinkbiganalytics.nifi.v2.thrift.ThriftService - Database Connection Pooling Service + Hadoop Configuration Resources - - ALL - false - 30 sec - - Statement - DROP TABLE ${daf.domain}__${daf.subdomain}.temp_${feed}; + Remote Owner + + Remote Owner + - Database Connection Pooling Service - a11778f5-3976-3513-0000-000000000000 - - - 0 - 0 sec - TIMER_DRIVEN - 1 sec - - false - Drop Temp_table - - false - failure - - - false - success - - STOPPED - - it.gov.daf.nifi.processors.DafJSONCleansing - - - 348c9bd4-a8e3-38d6-0000-000000000000 - 23ce9c70-cc0f-3f5f-0000-000000000000 - - 3787.139785845058 - 788.320348202371 - - - nifi-standard-nar - org.apache.nifi - 1.7.0 - - - WARN - - 1 - - - Routing Strategy - - Routing Strategy - - - - csv - - csv - - - - json - - json - - - - ALL - false - 30 sec - - - Routing Strategy - Route to Property name - - - csv - ${daf.format:equals('csv')} - - - json - ${daf.format:equals('json')} - - - 0 - 0 sec - TIMER_DRIVEN - 1 sec - - false - RouteOnAttribute - - false - csv - - - false - json - - - false - unmatched - - STOPPED - - org.apache.nifi.processors.standard.RouteOnAttribute - - - 34d2c912-9fbe-30fd-0000-000000000000 - 23ce9c70-cc0f-3f5f-0000-000000000000 - - 2742.2140308365774 - 2936.1418024743134 - - - kylo-nifi-core-v1.2-nar - default - unversioned - - - WARN - - 1 - - - Statement - - Statement - - - - Database Connection Pooling Service - - com.thinkbiganalytics.nifi.v2.thrift.ThriftService - Database Connection Pooling Service - - - - ALL - false - 30 sec - - - Statement - INSERT OVERWRITE TABLE ${daf.domain}__${daf.subdomain}.${feed} SELECT * FROM ${daf.domain}__${daf.subdomain}.temp_${feed}; - - - Database Connection Pooling Service - a11778f5-3976-3513-0000-000000000000 - - - 0 - 0 sec - TIMER_DRIVEN - 1 sec - - false - Insert Overwrite TargetTable - - false - failure - - - false - success - - STOPPED - org.apache.nifi.processors.hadoop.PutHDFS - 363c1650-b960-364e-0000-000000000000 + 3a2b6768-96e1-3b29-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 3787.139785845058 - 563.821286214451 + 781.0665448789914 + 2302.71222861523 - kylo-nifi-core-v1.2-nar - default - unversioned + nifi-update-attribute-nar + org.apache.nifi + 1.7.0 WARN - + Imposta l'attributo RestartFromBeginning a false 1 - System feed category + Delete Attributes Expression - System feed category + Delete Attributes Expression - Initialization Retry Delay (seconds) + Message - Initialization Retry Delay (seconds) + Message - Clone initialization flowfile + Store State - Clone initialization flowfile + Store State - Max Flow File Count + RestartFromBeginning - Max Flow File Count + RestartFromBeginning - Max Initialization Attempts + Stateful Variables Initial Value - Max Initialization Attempts - - - - Initialization Failure Strategy - - Initialization Failure Strategy - - - - System feed name - - System feed name - - - - Max Flow Files size - - Max Flow Files size - - - - Metadata Service - - com.thinkbiganalytics.nifi.core.api.metadata.MetadataProviderService - Metadata Service + Stateful Variables Initial Value ALL false - 5 sec + 30 sec - System feed category - ${category} - - - Initialization Retry Delay (seconds) - 15 - - - Clone initialization flowfile - true - - - Max Flow File Count - 1000 - - - Max Initialization Attempts - 5 + Delete Attributes Expression - Initialization Failure Strategy - RETRY + Message + ${Message:replace('\n',' ')} - System feed name - ${feed} + Store State + Do not store state - Max Flow Files size - 1000000000 + RestartFromBeginning + false - Metadata Service - ad55fab7-e0c2-3b0f-0000-000000000000 + Stateful Variables Initial Value 0 - 1 sec + 0 sec TIMER_DRIVEN 1 sec false - Initialize Feed? - - false - failure - - - false - Initialize - - - true - Re-Initialize - + RestartAttribute false success STOPPED - - com.thinkbiganalytics.nifi.v2.init.InitializeFeed + - com.thinkbiganalytics.nifi.v2.ingest.MergeTable + - org.apache.nifi.processors.standard.MergeContent + com.thinkbiganalytics.nifi.v2.hdfs.CreateHDFSFolder - 41b0429d-f239-36cc-0000-000000000000 + 47844730-7389-3b61-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1930.3129093914467 - 119.73994445800781 + 307.77460748640806 + 2084.231237599186 - nifi-standard-nar + nifi-hadoop-nar org.apache.nifi 1.7.0 WARN - - 1 + Inserisce il flowfile nella cartella /daf/${daf.type}/${daf.organization}/kylo/${daf.domain}__${daf.subdomain}/${feed}/failed/restart, dalla quale kylo prende i files che devono ricominciare il loro flusso. + 10 - zip + Permissions umask - zip + Permissions umask - Routing Strategy + Hadoop Configuration Resources - Routing Strategy + Hadoop Configuration Resources - - ALL + + Remote Owner + + Remote Owner + + + + Additional Classpath Resources + + Additional Classpath Resources + + + + Compression codec + + Compression codec + + + + IO Buffer Size + + IO Buffer Size + + + + Remote Group + + Remote Group + + + + Block Size + + Block Size + + + + kerberos-credentials-service + + org.apache.nifi.kerberos.KerberosCredentialsService + kerberos-credentials-service + + + + Kerberos Keytab + + Kerberos Keytab + + + + Kerberos Principal + + Kerberos Principal + + + + Replication + + Replication + + + + Kerberos Relogin Period + + Kerberos Relogin Period + + + + Directory + + Directory + + + + Conflict Resolution Strategy + + Conflict Resolution Strategy + + + + ALL false 30 sec - zip - ${filename:endsWith('.zip')} + Permissions umask + 077 - Routing Strategy - Route to Property name + Hadoop Configuration Resources + /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml + + + Remote Owner + ${daf.owner} + + + Additional Classpath Resources + + + Compression codec + NONE + + + IO Buffer Size + + + Remote Group + ${daf.owner} + + + Block Size + + + kerberos-credentials-service + + + Kerberos Keytab + /usr/nifi/daf.keytab + + + Kerberos Principal + daf@DAF.GOV.IT + + + Replication + + + Kerberos Relogin Period + 4 hours + + + Directory + /daf/${daf.type}/${daf.organization}/kylo/${daf.domain}__${daf.subdomain}/${feed}/failed/restart + + + Conflict Resolution Strategy + ignore 0 @@ -5112,33 +4852,33 @@ ${filename:contains('.zip'):ifElse( 1 sec false - RouteOnAttribute + Restart From Beginning false - unmatched + failure false - zip + success STOPPED - org.apache.nifi.processors.standard.RouteOnAttribute + org.apache.nifi.processors.hadoop.PutHDFS - 488adf06-3023-38cc-0000-000000000000 + 493467f3-142d-3841-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 4292.45285794037 - 932.9602290196017 + 2269.635736331169 + 2189.579355283109 - nifi-update-attribute-nar + nifi-scripting-nar org.apache.nifi 1.7.0 @@ -5148,39 +4888,39 @@ ${filename:contains('.zip'):ifElse( 1 - daf.separator.char + Script File - daf.separator.char + Script File - daf.quote.char + Script Engine - daf.quote.char + Script Engine - Delete Attributes Expression + invalidateMetadata - Delete Attributes Expression + invalidateMetadata - daf.escape.char + Script Body - daf.escape.char + Script Body - Store State + Module Directory - Store State + Module Directory - Stateful Variables Initial Value + impalaThriftServiceName - Stateful Variables Initial Value + impalaThriftServiceName @@ -5189,27 +4929,57 @@ ${filename:contains('.zip'):ifElse( 30 sec - daf.separator.char - ${metadata.table.feedFormat:getDelimitedField(6,"'")} + Script File - daf.quote.char - ${metadata.table.feedFormat:getDelimitedField(14,"'")} + Script Engine + Groovy - Delete Attributes Expression + invalidateMetadata + invalidate metadata - daf.escape.char - ${metadata.table.feedFormat:getDelimitedField(10,"'")} + Script Body + import org.apache.nifi.controller.ControllerService + import org.apache.nifi.processor.ProcessSession + + session = (ProcessSession) session + + flowFile = session.get() + if (!flowFile) return + + lookup = context.controllerServiceLookup + impalaThriftServiceName = impalaThriftServiceName.evaluateAttributeExpressions(flowFile).value + invalidateMetadata = invalidateMetadata.evaluateAttributeExpressions(flowFile).value + + def dbcpServiceCS = lookup.getControllerServiceIdentifiers(ControllerService).find { + cs -> lookup.getControllerServiceName(cs) == impalaThriftServiceName + } + def impalaConnection = lookup.getControllerService(dbcpServiceCS)?.getConnection() + def statement = impalaConnection.prepareCall(invalidateMetadata) + + try { + statement.execute() + } catch (Exception ex) { + log.error(ex.message) + flowFile = session.putAttribute(flowFile, context.getName() + " Exit code", "FAILED") + flowFile = session.putAttribute(flowFile, context.getName() + " Exit description", ex.message) + flowFile = session.putAttribute(flowFile, "kylo.jobExitDescription", ex.message) + impalaConnection?.close() + session.transfer(flowFile, REL_FAILURE) + return -1 + } + + impalaConnection?.close() + session.transfer(flowFile, REL_SUCCESS) - Store State - Store state locally + Module Directory - Stateful Variables Initial Value - + impalaThriftServiceName + Impala 0 @@ -5218,10 +4988,10 @@ ${filename:contains('.zip'):ifElse( 1 sec false - Set Feed Defaults + Invalidate Metadata - true - set state fail + false + failure false @@ -5231,17 +5001,17 @@ ${filename:contains('.zip'):ifElse( - org.apache.nifi.processors.attributes.UpdateAttribute + org.apache.nifi.processors.script.ExecuteScript - 49a50f1b-67f0-354c-0000-000000000000 + 4ba1246f-19cc-306d-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1603.1673771368914 - 2126.6162222917364 + 4413.474682491113 + 1143.6932632651333 nifi-update-attribute-nar @@ -5254,27 +5024,33 @@ ${filename:contains('.zip'):ifElse( 1 - Delete Attributes Expression + daf.separator.char - Delete Attributes Expression + daf.separator.char - filename + daf.quote.char - filename + daf.quote.char - Store State + Delete Attributes Expression - Store State + Delete Attributes Expression - uuid.toMatch + daf.escape.char - uuid.toMatch + daf.escape.char + + + + Store State + + Store State @@ -5289,22 +5065,27 @@ ${filename:contains('.zip'):ifElse( 30 sec - Delete Attributes Expression + daf.separator.char + ${metadata.table.feedFormat:getDelimitedField(6,"'")} - filename - ${now():toNumber()}_${filename} + daf.quote.char + ${metadata.table.feedFormat:getDelimitedField(14,"'")} - Store State - Do not store state + Delete Attributes Expression - uuid.toMatch - ${uuid} + daf.escape.char + ${metadata.table.feedFormat:getDelimitedField(10,"'")} + + + Store State + Store state locally Stateful Variables Initial Value + 0 @@ -5313,7 +5094,11 @@ ${filename:contains('.zip'):ifElse( 1 sec false - Prepend TS and Save UUID + Set Feed Defaults + + true + set state fail + false success @@ -5322,17 +5107,103 @@ ${filename:contains('.zip'):ifElse( org.apache.nifi.processors.attributes.UpdateAttribute - 4beca4ca-6d13-3ecf-0000-000000000000 + 54a62b8a-df1a-352e-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 6001.094374462176 - 1575.470411352354 + 3908.1616103958004 + 1659.4557305006929 + + + nifi-update-attribute-nar + org.apache.nifi + 1.7.0 + + + WARN + + 1 + + + feedts + + feedts + + + + Delete Attributes Expression + + Delete Attributes Expression + + + + Store State + + Store State + + + + Stateful Variables Initial Value + + Stateful Variables Initial Value + + + + ALL + false + 30 sec + + + feedts + ${now():toNumber()} + + + Delete Attributes Expression + + + Store State + Store state locally + + + Stateful Variables Initial Value + + + + 0 + 0 sec + TIMER_DRIVEN + 1 sec + + false + UpdateTimestamp + + true + set state fail + + + false + success + + STOPPED + + org.apache.nifi.processors.attributes.UpdateAttribute + + + 5b7cab9c-bdbc-38cd-0000-000000000000 + 23ce9c70-cc0f-3f5f-0000-000000000000 + + 6122.1161990129185 + 1786.2034455978865 nifi-standard-nar @@ -5447,11 +5318,11 @@ ${filename:contains('.zip'):ifElse( org.apache.nifi.processors.standard.ExecuteStreamCommand - 4c8c3d92-9cd3-3176-0000-000000000000 + 60a96755-2f94-362f-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 3274.7866626725154 - 2620.691729232126 + 5616.819008706458 + 1246.7257237783197 kylo-nifi-core-v1.2-nar @@ -5464,16 +5335,28 @@ ${filename:contains('.zip'):ifElse( 1 - Statement + System feed category - Statement + System feed category - Database Connection Pooling Service + Initialization Result - com.thinkbiganalytics.nifi.v2.thrift.ThriftService - Database Connection Pooling Service + Initialization Result + + + + System feed name + + System feed name + + + + Metadata Service + + com.thinkbiganalytics.nifi.core.api.metadata.MetadataProviderService + Metadata Service @@ -5482,12 +5365,20 @@ ${filename:contains('.zip'):ifElse( 30 sec - Statement - create table ${daf.domain}__${daf.subdomain}.temp_${feed} LIKE ${daf.domain}__${daf.subdomain}.${feed}; + System feed category + ${category} - Database Connection Pooling Service - a11778f5-3976-3513-0000-000000000000 + Initialization Result + FAILURE + + + System feed name + ${feed} + + + Metadata Service + ad55fab7-e0c2-3b0f-0000-000000000000 0 @@ -5496,30 +5387,35 @@ ${filename:contains('.zip'):ifElse( 1 sec false - Create Temp_table + Initialization Failure - false + true failure - false + true success STOPPED - + com.thinkbiganalytics.nifi.v2.init.CompleteInitializeFeed - 4ccf55a8-080b-3ecb-0000-000000000000 + 62898293-e3dc-3c0b-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 5495.7971841557155 - 1035.9926895327872 + 677.1791547488974 + 1295.3440666977385 - kylo-nifi-core-v1.2-nar - default - unversioned + nifi-update-attribute-nar + org.apache.nifi + 1.7.0 WARN @@ -5527,28 +5423,27 @@ ${filename:contains('.zip'):ifElse( 1 - System feed category + Delete Attributes Expression - System feed category + Delete Attributes Expression - Initialization Result + Store State - Initialization Result + Store State - System feed name + daf.category.folder - System feed name + daf.category.folder - Metadata Service + Stateful Variables Initial Value - com.thinkbiganalytics.nifi.core.api.metadata.MetadataProviderService - Metadata Service + Stateful Variables Initial Value @@ -5557,20 +5452,19 @@ ${filename:contains('.zip'):ifElse( 30 sec - System feed category - ${category} + Delete Attributes Expression - Initialization Result - FAILURE + Store State + Store state locally - System feed name - ${feed} + daf.category.folder + /daf/${daf.type}/${category} - Metadata Service - ad55fab7-e0c2-3b0f-0000-000000000000 + Stateful Variables Initial Value + 0 @@ -5579,130 +5473,174 @@ ${filename:contains('.zip'):ifElse( 1 sec false - Initialization Failure + Set Category Folder true - failure + set state fail - true + false success STOPPED - com.thinkbiganalytics.nifi.v2.init.CompleteInitializeFeed + org.apache.nifi.processors.attributes.UpdateAttribute - 50a45cc2-f153-3cd6-0000-000000000000 + 632de85c-953c-3834-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 3787.139785845058 - 1646.0604252902513 + 3193.1136889222253 + 334.25436305999756 - nifi-hadoop-nar + nifi-update-attribute-nar org.apache.nifi 1.7.0 WARN - 10 + 1 - Permissions umask - - Permissions umask - - - - Hadoop Configuration Resources - - Hadoop Configuration Resources - - - - Remote Owner + Delete Attributes Expression - Remote Owner + Delete Attributes Expression - Additional Classpath Resources + daf.type.folder - Additional Classpath Resources + daf.type.folder - Compression codec + Store State - Compression codec + Store State - IO Buffer Size + daf.owner.role - IO Buffer Size + daf.owner.role - Remote Group + Stateful Variables Initial Value - Remote Group + Stateful Variables Initial Value + + ALL + false + 30 sec + - Block Size - - Block Size - + Delete Attributes Expression - kerberos-credentials-service - - org.apache.nifi.kerberos.KerberosCredentialsService - kerberos-credentials-service - + daf.type.folder + ${hive.master.root}/${daf.type}/${daf.organization} - Kerberos Keytab - - Kerberos Keytab - + Store State + Do not store state - Kerberos Principal - - Kerberos Principal - + daf.owner.role + ${daf.owner}_user_role - Replication - - Replication - + Stateful Variables Initial Value - - Kerberos Relogin Period + + 0 + 0 sec + TIMER_DRIVEN + 1 sec + + false + Set daf.type.folder for ORDINARY + + false + success + + STOPPED + + org.apache.nifi.processors.attributes.UpdateAttribute + + + 64fafa08-776a-3827-0000-000000000000 + 23ce9c70-cc0f-3f5f-0000-000000000000 + + 1622.954747283562 + 2302.71222861523 + + + nifi-standard-nar + org.apache.nifi + 1.7.0 + + + WARN + Get al bullettin board di nifi per ottenere il motivo del fallimento. La response viene inserita nell'attributo Message + + + 1 + + + Max Attribute Length - Kerberos Relogin Period + Max Attribute Length - Directory + Command Path - Directory + Command Path - Conflict Resolution Strategy + Working Directory - Conflict Resolution Strategy + Working Directory + + + + Output Destination Attribute + + Output Destination Attribute + + + + Argument Delimiter + + Argument Delimiter + + + + Ignore STDIN + + Ignore STDIN + + + + Command Arguments + + Command Arguments @@ -5711,59 +5649,105 @@ ${filename:contains('.zip'):ifElse( 30 sec - Permissions umask - 007 - - - Hadoop Configuration Resources - /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml - - - Remote Owner - daf + Max Attribute Length + 2000 - Additional Classpath Resources + Command Path + /usr/bin/python3 - Compression codec - NONE + Working Directory + /usr/nifi/scripts - IO Buffer Size + Output Destination Attribute + ErrorMessage - Remote Group - daf + Argument Delimiter + - Block Size + Ignore STDIN + true - kerberos-credentials-service + Command Arguments + nifi_bulletin_request.py ${str_length} ${uuid.toMatch} + + 0 + 0 sec + TIMER_DRIVEN + 1 sec + + false + NifiBulletinRequest + + false + original + + STOPPED + + org.apache.nifi.processors.standard.ExecuteStreamCommand + + + 68b1e54b-b6cf-3a3d-0000-000000000000 + 23ce9c70-cc0f-3f5f-0000-000000000000 + + 4413.474682491113 + 1412.0207138323676 + + + teamdigitale-daf-nifi-processors + default + unversioned + + + WARN + + 1 + - Kerberos Keytab - /usr/nifi/daf.keytab + Separator Char + + Separator Char + - Kerberos Principal - daf@DAF.GOV.IT + Escape Char + + Escape Char + - Replication + Quote Char + + Quote Char + + + ALL + false + 30 sec + - Kerberos Relogin Period - 4 hours + Separator Char + ${daf.separator.char} - Directory - ${daf.type.folder}/kylo/${daf.domain}__${daf.subdomain}/${feed}/feed/${feedts} + Escape Char + ${daf.escape.char} - Conflict Resolution Strategy - replace + Quote Char + ${daf.quote.char} 0 @@ -5772,7 +5756,7 @@ ${filename:contains('.zip'):ifElse( 1 sec false - Upload to HDFS + DafCSVCleansing false failure @@ -5785,17 +5769,17 @@ ${filename:contains('.zip'):ifElse( - org.apache.nifi.processors.hadoop.PutHDFS + it.gov.daf.nifi.processors.DafCSVCleansing - 512978db-e788-3bbb-0000-000000000000 + 6986a9d1-d207-379a-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 2118.642769650381 - 421.93743611679474 + 1772.767106989064 + 331.42774868011475 nifi-standard-nar @@ -5808,15 +5792,15 @@ ${filename:contains('.zip'):ifElse( 1 - Packaging Format + zip - Packaging Format + zip - File Filter + Routing Strategy - File Filter + Routing Strategy @@ -5825,12 +5809,12 @@ ${filename:contains('.zip'):ifElse( 30 sec - Packaging Format - zip + zip + ${filename:endsWith('.zip')} - File Filter - .* + Routing Strategy + Route to Property name 0 @@ -5839,29 +5823,30 @@ ${filename:contains('.zip'):ifElse( 1 sec false - UnpackContent - - false - failure - + Is Zip? false - original + unmatched false - success + zip STOPPED - + org.apache.nifi.processors.standard.RouteOnAttribute - 524f04f2-dc07-324c-0000-000000000000 + 69cc1f3e-1675-3433-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 2742.2140308365774 - 2371.432948981327 + 6122.1161990129185 + 758.7802224671832 kylo-nifi-core-v1.2-nar @@ -5874,138 +5859,172 @@ ${filename:contains('.zip'):ifElse( 1 - Feed partition value + System feed category - Feed partition value + System feed category - Merge Strategy + Initialization Result - Merge Strategy + Initialization Result - Hive Configurations + System feed name - Hive Configurations + System feed name - Database Connection Pooling Service + Metadata Service - com.thinkbiganalytics.nifi.v2.thrift.ThriftService - Database Connection Pooling Service + com.thinkbiganalytics.nifi.core.api.metadata.MetadataProviderService + Metadata Service + + ALL + false + 30 sec + - Source table - - Source table - + System feed category + ${category} - Hive Connection Pooling Service - - org.apache.nifi.dbcp.hive.HiveDBCPService - Hive Connection Pooling Service - + Initialization Result + SUCCESSFUL - Field specification - - Field specification - + System feed name + ${feed} - Target schema - - Target schema - + Metadata Service + ad55fab7-e0c2-3b0f-0000-000000000000 - - Source schema + + 0 + 0 sec + TIMER_DRIVEN + 1 sec + + false + Record Initialization + + true + failure + + + true + success + + STOPPED + + com.thinkbiganalytics.nifi.v2.init.CompleteInitializeFeed + + + 69f7de56-b526-3b01-0000-000000000000 + 23ce9c70-cc0f-3f5f-0000-000000000000 + + 1622.954747283562 + 1888.94619865787 + + + nifi-standard-nar + org.apache.nifi + 1.7.0 + + + WARN + + 10 + + + Group - Source schema + Group - Reset hive on connection + Owner - Reset hive on connection + Owner - Target table + Create Missing Directories - Target table + Create Missing Directories - Partition Specification + Permissions - Partition Specification + Permissions - blocking-id + Maximum File Count - blocking-id + Maximum File Count - - ALL - false - 30 sec - - - Feed partition value - ${feedts} - - - Merge Strategy - ${metadata.table.targetMergeStrategy} - - Hive Configurations + Last Modified Time + + Last Modified Time + - Database Connection Pooling Service - a11778f5-3976-3513-0000-000000000000 + Directory + + Directory + - Source table - ${feed}_valid + Conflict Resolution Strategy + + Conflict Resolution Strategy + + + ALL + false + 30 sec + - Hive Connection Pooling Service + Group - Field specification - ${metadata.table.fieldStructure} + Owner - Target schema - ${daf.domain}__${daf.subdomain} + Create Missing Directories + true - Source schema - ${daf.domain}__${daf.subdomain} + Permissions - Reset hive on connection - true + Maximum File Count - Target table - ${feed} + Last Modified Time - Partition Specification - ${metadata.table.partitionSpecs} + Directory + ${kylo.tmp.baseFolder:replaceEmpty('/tmp/kylo-nifi')}/failed-flows/${category}/${feed}/${feedts} - blocking-id + Conflict Resolution Strategy + ignore 0 @@ -6014,9 +6033,9 @@ ${filename:contains('.zip'):ifElse( 1 sec false - Merge Table + Failed Flow - false + true failure @@ -6027,22 +6046,22 @@ ${filename:contains('.zip'):ifElse( - com.thinkbiganalytics.nifi.v2.ingest.MergeTable + org.apache.nifi.processors.standard.PutFile - 53ee0c1a-6ddc-3e9e-0000-000000000000 + 6cffd4ca-db07-3169-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 3787.139785845058 - 1984.173360470918 + 1259.6896909031202 + 1047.6555240216567 - nifi-scripting-nar - org.apache.nifi - 1.7.0 + kylo-nifi-hadoop-v1.2-nar + default + unversioned WARN @@ -6050,27 +6069,33 @@ ${filename:contains('.zip'):ifElse( 1 - Script File + Hadoop Configuration Resources - Script File + Hadoop Configuration Resources - Script Engine + Kerberos Keytab - Script Engine + Kerberos Keytab - Script Body + Kerberos Principal - Script Body + Kerberos Principal - Module Directory + Kerberos Relogin Period - Module Directory + Kerberos Relogin Period + + + + Directory + + Directory @@ -6079,35 +6104,25 @@ ${filename:contains('.zip'):ifElse( 30 sec - Script File + Hadoop Configuration Resources + /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml - Script Engine - Groovy + Kerberos Keytab + /usr/nifi/daf.keytab - Script Body - def flowFile = session.get() - if(!flowFile) return - def json = flowFile.getAttribute("metadata.table.fieldPoliciesJson"); - def inputFolder = flowFile.getAttribute("spark.input_folder") - def feed = flowFile.getAttribute("feed") - def category = flowFile.getAttribute("category") - def feedts = flowFile.getAttribute("feedts") - def folder = new File(inputFolder + "/"+category+"/"+feed+"/"+feedts) - // If it doesn't exist - if( !folder.exists() ) { - // Create all folders - folder.mkdirs() - } - def jsonFile = new File(folder,feed+"_field_policy.json") - jsonFile.write(json) - flowFile = session.putAttribute(flowFile,"table_field_policy_json_file",jsonFile.getCanonicalPath()) - session.transfer(flowFile, REL_SUCCESS) - + Kerberos Principal + daf@DAF.GOV.IT - Module Directory + Kerberos Relogin Period + 4 hours + + + Directory + ${daf.category.folder}/${daf.domain}__${daf.subdomain}/${feed}/ + ${daf.category.folder}/kylo/${daf.domain}__${daf.subdomain}/${feed}/ 0 @@ -6116,149 +6131,296 @@ ${filename:contains('.zip'):ifElse( 1 sec false - Fetch field policies + Remove HDFS Folders - false + true failure - false + true success STOPPED - org.apache.nifi.processors.script.ExecuteScript + com.thinkbiganalytics.nifi.v2.hdfs.RemoveHDFSFolder - 54858faa-cb0a-316c-0000-000000000000 + 71d77886-d680-32b6-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 4979.764505602609 - 1013.8100723452872 + 156.35760628698335 + 1295.3440666977385 - nifi-standard-nar + nifi-update-attribute-nar org.apache.nifi 1.7.0 WARN - 0 + 1 - Max Attribute Length + daf.type - Max Attribute Length + daf.type - Command Path + daf.domain - Command Path + daf.domain - Working Directory + daf.standard.dataset - Working Directory + daf.standard.dataset - Output Destination Attribute + hive.profile.root - Output Destination Attribute + hive.profile.root - Argument Delimiter + Store State - Argument Delimiter + Store State - Ignore STDIN + daf.format - Ignore STDIN + daf.format - Command Arguments + daf.external.target - Command Arguments + daf.external.target - - ALL - false - 30 sec - - Max Attribute Length - 256 + merge.correlation + + merge.correlation + - Command Path - /opt/cloudera/parcels/CDH/bin/hdfs + nifi.home + + nifi.home + - Working Directory + skipHeader + + skipHeader + - Output Destination Attribute + Stateful Variables Initial Value + + Stateful Variables Initial Value + - Argument Delimiter - + daf.separator.char + + daf.separator.char + - Ignore STDIN - false + hdfs.archive.root + + hdfs.archive.root + - Command Arguments - dfs -setfacl -m user:hive:rwx,user:impala:rwx,group::rwx,other::--x,mask::rwx,default:user:hive:rwx,default:user:impala:rwx,default:group::rwx,default:other::--x,default:mask::rwx ${daf.type.folder} + daf.quote.char + + daf.quote.char + + + + daf.subdomain + + daf.subdomain + + + + hdfs.ingest.root + + hdfs.ingest.root + + + + Delete Attributes Expression + + Delete Attributes Expression + + + + daf.escape.char + + daf.escape.char + + + + daf.organization + + daf.organization + + + + metadata.table.targetTblProperties.parquet + + metadata.table.targetTblProperties.parquet + + + + hive.ingest.root + + hive.ingest.root + + + + hive.master.root + + hive.master.root + + + + ALL + false + 30 sec + + + daf.type + ${feedJson:jsonPath('$.userProperties.dafType')} + + + daf.domain + ${feedJson:jsonPath('$.userProperties.dafDomain')} + + + daf.standard.dataset + ${feedJson:jsonPath('$.userProperties.dafStandardDataset')} + + + hive.profile.root + ${hive.profile.root:replaceEmpty('/model.db')} + + + Store State + Store state locally + + + daf.format + ${feedJson:jsonPath('$.userProperties.dafFormat')} + + + daf.external.target + ${feedJson:jsonPath('$.userProperties.dafTarget')} + + + merge.correlation + ${category}.${feed}.correlation + + + nifi.home + ${nifi.home:replaceEmpty("/opt/nifi")} + + + skipHeader + ${skipHeader:replaceEmpty('false')} + + + Stateful Variables Initial Value + + + + daf.separator.char + ${metadata.table.feedFormat:substringAfterLast("separatorChar' = '"):substringBeforeLast("' ,'escapeChar'")} + + + hdfs.archive.root + ${hdfs.archive.root:replaceEmpty('/archive')} + + + daf.quote.char + ${metadata.table.feedFormat:substringAfterLast("quoteChar' = '"):substringBeforeLast("') STORED AS TEXTFILE")} + + + daf.subdomain + ${feedJson:jsonPath('$.userProperties.dafSubdomain')} + + + hdfs.ingest.root + ${hdfs.ingest.root:replaceEmpty('/etl')} + + + Delete Attributes Expression + + + daf.escape.char + ${metadata.table.feedFormat:substringAfterLast("escapeChar' = '"):substringBeforeLast("' ,'quoteChar' = '")} + + + daf.organization + ${category} + + + metadata.table.targetTblProperties.parquet + tblproperties("parquet.compression"="SNAPPY","EXTERNAL"="TRUE") + + + hive.ingest.root + ${hive.ingest.root:replaceEmpty('/model.db')} + + + hive.master.root + ${hive.master.root:replaceEmpty('/app/warehouse')} 0 0 sec - EVENT_DRIVEN + TIMER_DRIVEN 1 sec false - Set Organization Folder ACLs + Set Feed Defaults - false - nonzero status + true + set state fail false - original - - - true - output stream + success STOPPED - org.apache.nifi.processors.standard.ExecuteStreamCommand + org.apache.nifi.processors.attributes.UpdateAttribute - 617e3f6f-79cb-3e23-0000-000000000000 + 773eeee6-f37f-381e-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1104.1740300404272 - 1775.5469708073997 + 1117.2858441953977 + 1692.0501108030876 nifi-hadoop-nar @@ -6267,7 +6429,7 @@ ${filename:contains('.zip'):ifElse( WARN - + Mette il file in hdfs al path /daf/${daf.type}/${daf.organization}/kylo/${daf.domain}__${daf.subdomain}/${feed}/log 10 @@ -6415,7 +6577,7 @@ ${filename:contains('.zip'):ifElse( Directory - /daf/${daf.type}/${daf.organization}/kylo/${daf.domain}__${daf.subdomain}/${feed}/failed + /daf/${daf.type}/${daf.organization}/kylo/${daf.domain}__${daf.subdomain}/${feed}/log Conflict Resolution Strategy @@ -6428,7 +6590,7 @@ ${filename:contains('.zip'):ifElse( 1 sec false - Failed to Recovery Area + Logs to Recovery Area false failure @@ -6447,14 +6609,14 @@ ${filename:contains('.zip'):ifElse( org.apache.nifi.processors.hadoop.PutHDFS - 6c525f77-bb00-3af2-0000-000000000000 + 8067c255-8025-3e08-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 4979.764505602609 - 547.0452350966516 + 3395.8084872232575 + 2831.4247634776584 - kylo-nifi-hadoop-v1.2-nar + kylo-nifi-core-v1.2-nar default unversioned @@ -6464,51 +6626,16 @@ ${filename:contains('.zip'):ifElse( 1 - Permissions umask - - Permissions umask - - - - Hadoop Configuration Resources - - Hadoop Configuration Resources - - - - Kerberos Keytab - - Kerberos Keytab - - - - Remote Owner - - Remote Owner - - - - Kerberos Principal - - Kerberos Principal - - - - Kerberos Relogin Period - - Kerberos Relogin Period - - - - Remote Group + Statement - Remote Group + Statement - Directory + Database Connection Pooling Service - Directory + com.thinkbiganalytics.nifi.v2.thrift.ThriftService + Database Connection Pooling Service @@ -6517,37 +6644,12 @@ ${filename:contains('.zip'):ifElse( 30 sec - Permissions umask - 0026 - - - Hadoop Configuration Resources - /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml - - - Kerberos Keytab - /usr/nifi/daf.keytab - - - Remote Owner - daf - - - Kerberos Principal - daf@DAF.GOV.IT - - - Kerberos Relogin Period - 4 hours - - - Remote Group - ${daf.organization} + Statement + create table ${daf.domain}__${daf.subdomain}.temp_${feed} LIKE ${daf.domain}__${daf.subdomain}.${feed}; - Directory - ${daf.type.folder} -${daf.type.folder}/${daf.domain}__${daf.subdomain} + Database Connection Pooling Service + a11778f5-3976-3513-0000-000000000000 0 @@ -6556,7 +6658,7 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} 1 sec false - Register HDFS Folders + Create Temp_table false failure @@ -6566,59 +6668,37 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} success STOPPED - - com.thinkbiganalytics.nifi.v2.hdfs.CreateHDFSFolder - + + com.thinkbiganalytics.nifi.v2.thrift.ExecuteHQLStatement - 6dec3299-8888-3a98-0000-000000000000 + 860476bb-5a9d-3035-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1603.1673771368914 - 1485.5195738379161 + 1622.954747283562 + 2648.1119598031564 - kylo-nifi-core-v1.2-nar - default - unversioned + nifi-standard-nar + org.apache.nifi + 1.7.0 WARN @@ -6688,46 +6759,21 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} 1 - Release All - - Release All - - - - System feed category - - System feed category - - - - High-Water Mark - - High-Water Mark - - - - Mode - - Mode - - - - System feed name + Routing Strategy - System feed name + Routing Strategy - Metadata Service + csv - com.thinkbiganalytics.nifi.core.api.metadata.MetadataProviderService - Metadata Service + csv - High-Water Mark Value Property Name + json - High-Water Mark Value Property Name + json @@ -6736,32 +6782,16 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} 30 sec - Release All - true - - - System feed category - ${category} - - - High-Water Mark - highWaterMark - - - Mode - REJECT - - - System feed name - ${feed} + Routing Strategy + Route to Property name - Metadata Service - ad55fab7-e0c2-3b0f-0000-000000000000 + csv + ${daf.format:equals('csv')} - High-Water Mark Value Property Name - water.mark + json + ${daf.format:equals('json')} 0 @@ -6770,39 +6800,39 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} 1 sec false - Failed - Release Highwater Mark + CSV&JSONFilter - true - cancelledWaterMark + false + csv false - failure + json false - success + unmatched STOPPED - com.thinkbiganalytics.nifi.v2.core.watermark.ReleaseHighWaterMark + org.apache.nifi.processors.standard.RouteOnAttribute - 6df01890-8e1b-3c57-0000-000000000000 + 87f8eff2-a6e3-31c6-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 399.51945410898406 - 2125.026744752674 + 1259.6896909031202 + 1295.3440666977385 - nifi-standard-nar - org.apache.nifi - 1.7.0 + kylo-nifi-core-v1.2-nar + default + unversioned WARN @@ -6810,15 +6840,34 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} 1 - Null Value Representation + Table Type + + Table Type + + + + Additional Tables + + Additional Tables + + + + System feed category + + System feed category + + + + Database Connection Pooling Service - Null Value Representation + com.thinkbiganalytics.nifi.v2.thrift.ThriftService + Database Connection Pooling Service - JsonPath Expression + System feed name - JsonPath Expression + System feed name @@ -6827,12 +6876,23 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} 30 sec - Null Value Representation - empty string + Table Type + ALL + + + Additional Tables + + + System feed category + ${daf.domain}__${daf.subdomain} + + + Database Connection Pooling Service + a11778f5-3976-3513-0000-000000000000 - JsonPath Expression - $.bulletinBoard.bulletins + System feed name + ${feed} 0 @@ -6841,32 +6901,33 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} 1 sec false - Split Bulletins - - false - failure - + Drop Feed Tables true - original + failure false - split + success STOPPED - + com.thinkbiganalytics.nifi.v2.ingest.DropFeedTables - 7278d10f-35a9-3248-0000-000000000000 + 8b8ed360-feb6-30b4-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1053.335250417862 - 119.73994445800781 + 3901.9013564895504 + 334.5411367416382 - nifi-update-attribute-nar + nifi-standard-nar org.apache.nifi 1.7.0 @@ -6876,147 +6937,111 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} 1 - daf.type + Keep Path - daf.type + Keep Path - daf.domain + Maximum Group Size - daf.domain + Maximum Group Size - daf.standard.dataset + Merge Strategy - daf.standard.dataset + Merge Strategy - hive.profile.root + Attribute Strategy - hive.profile.root + Attribute Strategy - Store State + Compression Level - Store State + Compression Level - daf.format + Maximum Number of Entries - daf.format + Maximum Number of Entries - daf.external.target + Minimum Group Size - daf.external.target + Minimum Group Size - merge.correlation + Maximum number of Bins - merge.correlation + Maximum number of Bins - nifi.home + Tar Modified Time - nifi.home + Tar Modified Time - skipHeader + Delimiter Strategy - skipHeader + Delimiter Strategy - Stateful Variables Initial Value + Merge Format - Stateful Variables Initial Value + Merge Format - daf.separator.char + Footer File - daf.separator.char + Footer File - hdfs.archive.root + Max Bin Age - hdfs.archive.root + Max Bin Age - daf.quote.char - - daf.quote.char - - - - daf.subdomain - - daf.subdomain - - - - hdfs.ingest.root - - hdfs.ingest.root - - - - Delete Attributes Expression - - Delete Attributes Expression - - - - daf.escape.char - - daf.escape.char - - - - daf.organization - - daf.organization - - - - daf.owner + Demarcator File - daf.owner + Demarcator File - daf.opendata + Correlation Attribute Name - daf.opendata + Correlation Attribute Name - metadata.table.targetTblProperties.parquet + Header File - metadata.table.targetTblProperties.parquet + Header File - hive.ingest.root + mergecontent-metadata-strategy - hive.ingest.root + mergecontent-metadata-strategy - hive.master.root + Minimum Number of Entries - hive.master.root + Minimum Number of Entries @@ -7025,115 +7050,93 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} 30 sec - daf.type - ${feedJson:jsonPath('$.userProperties.dafType')} - - - daf.domain - ${feedJson:jsonPath('$.userProperties.dafDomain')} - - - daf.standard.dataset - ${feedJson:jsonPath('$.userProperties.dafStandardDataset')} - - - hive.profile.root - ${hive.profile.root:replaceEmpty('/model.db')} - - - Store State - Store state locally - - - daf.format - ${feedJson:jsonPath('$.userProperties.dafFormat')} - - - daf.external.target - ${feedJson:jsonPath('$.userProperties.dafTarget')} + Keep Path + false - merge.correlation - ${category}.${feed}.correlation + Maximum Group Size + 500 MB - nifi.home - ${nifi.home:replaceEmpty("/opt/nifi")} + Merge Strategy + Bin-Packing Algorithm - skipHeader - ${skipHeader:replaceEmpty('false')} + Attribute Strategy + Keep Only Common Attributes - Stateful Variables Initial Value - + Compression Level + 1 - daf.separator.char - ${metadata.table.feedFormat:substringAfterLast("separatorChar' = '"):substringBeforeLast("' ,'escapeChar'")} + Maximum Number of Entries + 1000 - hdfs.archive.root - ${hdfs.archive.root:replaceEmpty('/archive')} + Minimum Group Size + 0 B - daf.quote.char - ${metadata.table.feedFormat:substringAfterLast("quoteChar' = '"):substringBeforeLast("') STORED AS TEXTFILE")} + Maximum number of Bins + 100 - daf.subdomain - ${feedJson:jsonPath('$.userProperties.dafSubdomain')} + Tar Modified Time + ${file.lastModifiedTime} - hdfs.ingest.root - ${hdfs.ingest.root:replaceEmpty('/etl')} + Delimiter Strategy + Text - Delete Attributes Expression + Merge Format + Binary Concatenation - daf.escape.char - ${metadata.table.feedFormat:substringAfterLast("escapeChar' = '"):substringBeforeLast("' ,'quoteChar' = '")} + Footer File - daf.organization - ${category} + Max Bin Age + 45s - daf.owner - ${feedJson:jsonPath('$.userProperties.dafOwner')} + Demarcator File - daf.opendata - ${feedJson:jsonPath('$.userProperties.dafOpendata')} + Correlation Attribute Name + merge.correlation - metadata.table.targetTblProperties.parquet - tblproperties("parquet.compression"="SNAPPY","EXTERNAL"="TRUE") + Header File - hive.ingest.root - ${hive.ingest.root:replaceEmpty('/model.db')} + mergecontent-metadata-strategy + Do Not Merge Uncommon Metadata - hive.master.root - ${hive.master.root:replaceEmpty('/app/warehouse')} + Minimum Number of Entries + 1 0 - 100 ms + 10 sec TIMER_DRIVEN 1 sec false - Set Feed Defaults + Merge Content true - set state fail + failure false - success + merged + + + true + original STOPPED - org.apache.nifi.processors.attributes.UpdateAttribute + org.apache.nifi.processors.standard.MergeContent - 74996cd7-67b3-3444-0000-000000000000 + 8c875558-9915-35da-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1104.1740300404272 - 2125.026744752674 + 2270.468744143669 + 1987.8694825919938 - nifi-standard-nar - org.apache.nifi - 1.7.0 + kylo-nifi-core-v1.2-nar + default + unversioned WARN @@ -7162,280 +7165,80 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} 1 - Proxy Host - - Proxy Host - - - - Attributes to Send - - Attributes to Send - - - - Trusted Hostname - - Trusted Hostname - - - - use-etag - - use-etag - - - - proxy-configuration-service - - org.apache.nifi.proxy.ProxyConfigurationService - proxy-configuration-service - - - - Proxy Type - - Proxy Type - - - - Basic Authentication Password - - Basic Authentication Password - - - - Connection Timeout + Release All - Connection Timeout + Release All - send-message-body + System feed category - send-message-body + System feed category - Always Output Response + High-Water Mark - Always Output Response + High-Water Mark - etag-max-cache-size + Mode - etag-max-cache-size + Mode - Put Response Body In Attribute + System feed name - Put Response Body In Attribute + System feed name - Use Chunked Encoding + Metadata Service - Use Chunked Encoding + com.thinkbiganalytics.nifi.core.api.metadata.MetadataProviderService + Metadata Service - Follow Redirects + High-Water Mark Value Property Name - Follow Redirects + High-Water Mark Value Property Name + + ALL + false + 30 sec + - Remote URL - - Remote URL - + Release All + true - Content-Type - - Content-Type - + System feed category + ${category} - invokehttp-proxy-user - - invokehttp-proxy-user - + High-Water Mark + highWaterMark - Read Timeout - - Read Timeout - + Mode + COMMIT - Max Length To Put In Attribute - - Max Length To Put In Attribute - + System feed name + ${feed} - invokehttp-proxy-password - - invokehttp-proxy-password - + Metadata Service + ad55fab7-e0c2-3b0f-0000-000000000000 - Proxy Port - - Proxy Port - - - - SSL Context Service - - org.apache.nifi.ssl.SSLContextService - SSL Context Service - - - - Digest Authentication - - Digest Authentication - - - - HTTP Method - - HTTP Method - - - - Add Response Headers to Request - - Add Response Headers to Request - - - - Basic Authentication Username - - Basic Authentication Username - - - - Include Date Header - - Include Date Header - - - - Penalize on "No Retry" - - Penalize on "No Retry" - - - - ALL - false - 30 sec - - - Proxy Host - - - Attributes to Send - - - Trusted Hostname - - - use-etag - false - - - proxy-configuration-service - - - Proxy Type - http - - - Basic Authentication Password - - - Connection Timeout - 5 secs - - - send-message-body - true - - - Always Output Response - false - - - etag-max-cache-size - 10MB - - - Put Response Body In Attribute - - - Use Chunked Encoding - false - - - Follow Redirects - True - - - Remote URL - http://tba-nifi.default.svc.cluster.local:8080/nifi-api/flow/bulletin-board - - - Content-Type - application/json - - - invokehttp-proxy-user - - - Read Timeout - 15 secs - - - Max Length To Put In Attribute - 256 - - - invokehttp-proxy-password - - - Proxy Port - - - SSL Context Service - - - Digest Authentication - false - - - HTTP Method - GET - - - Add Response Headers to Request - false - - - Basic Authentication Username - - - Include Date Header - True - - - Penalize on "No Retry" - false + High-Water Mark Value Property Name + water.mark 0 @@ -7444,52 +7247,44 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} 1 sec false - Get Bulletin Board - - false - Failure - - - false - No Retry - + Commit Highwater Mark - false - Original + true + cancelledWaterMark false - Response + failure - false - Retry + true + success STOPPED - org.apache.nifi.processors.standard.InvokeHTTP + com.thinkbiganalytics.nifi.v2.core.watermark.ReleaseHighWaterMark - 780f1975-2e31-39bf-0000-000000000000 + 8f9e2045-8a96-35d6-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 2841.9414616543067 - 119.73994445800781 + 5100.786330153351 + 1786.2034455978865 - nifi-hadoop-nar - org.apache.nifi - 1.7.0 + kylo-nifi-hadoop-v1.2-nar + default + unversioned WARN - 10 + 1 Permissions umask @@ -7503,49 +7298,6 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} Hadoop Configuration Resources - - Remote Owner - - Remote Owner - - - - Additional Classpath Resources - - Additional Classpath Resources - - - - Compression codec - - Compression codec - - - - IO Buffer Size - - IO Buffer Size - - - - Remote Group - - Remote Group - - - - Block Size - - Block Size - - - - kerberos-credentials-service - - org.apache.nifi.kerberos.KerberosCredentialsService - kerberos-credentials-service - - Kerberos Keytab @@ -7553,15 +7305,15 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} - Kerberos Principal + Remote Owner - Kerberos Principal + Remote Owner - Replication + Kerberos Principal - Replication + Kerberos Principal @@ -7571,15 +7323,15 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} - Directory + Remote Group - Directory + Remote Group - Conflict Resolution Strategy + Directory - Conflict Resolution Strategy + Directory @@ -7589,58 +7341,35 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} Permissions umask - 0277 + 0077 Hadoop Configuration Resources /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml - Remote Owner - ${daf.owner} - - - Additional Classpath Resources - - - Compression codec - BZIP - - - IO Buffer Size + Kerberos Keytab + /usr/nifi/daf.keytab - Remote Group + Remote Owner ${daf.owner} - - Block Size - - - kerberos-credentials-service - - - Kerberos Keytab - /usr/nifi/daf.keytab - Kerberos Principal daf@DAF.GOV.IT - - Replication - Kerberos Relogin Period 4 hours - Directory - /user/daf/archive/${daf.type}/${daf.domain}__${daf.subdomain}/${feed}/${archiveId}_${now():toNumber()} + Remote Group + ${daf.owner} - Conflict Resolution Strategy - fail + Directory + ${daf.type.folder}/${daf.domain}__${daf.subdomain}/${feed} 0 @@ -7649,7 +7378,7 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} 1 sec false - Archive Originals + Register HDFS Table Folder false failure @@ -7662,22 +7391,22 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} - org.apache.nifi.processors.hadoop.PutHDFS + com.thinkbiganalytics.nifi.v2.hdfs.CreateHDFSFolder - 8bc905b9-5af8-3dac-0000-000000000000 + 992aa566-a4f0-32a1-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 4979.764505602609 - 1313.2853627812647 + 2652.949909313549 + 606.14365625145 - kylo-nifi-hadoop-v1.2-nar - default - unversioned + nifi-scripting-nar + org.apache.nifi + 1.7.0 WARN @@ -7685,51 +7414,120 @@ ${daf.type.folder}/${daf.domain}__${daf.subdomain} 1 - Permissions umask + Script File - Permissions umask + Script File - Hadoop Configuration Resources + Script Engine - Hadoop Configuration Resources + Script Engine - Kerberos Keytab + Script Body - Kerberos Keytab + Script Body - Remote Owner + Module Directory - Remote Owner + Module Directory + + ALL + false + 30 sec + - Kerberos Principal - - Kerberos Principal - + Script File - Kerberos Relogin Period + Script Engine + Groovy + + + Script Body + def ff = session.get() + if(!ff)return + String header = '' + ff = session.write(ff, {rawIn, rawOut-> + // ## transform streams into reader and writer + rawIn.withReader("UTF-8"){reader-> + rawOut.withWriter("UTF-8"){writer-> + reader.eachLine{line, lineNum-> + if(lineNum == 1){ + header = line + } + if(lineNum>1) { // # skip the first line + writer << line + '\n' + } + } + } + } + } as StreamCallback) + session.putAttribute(ff, 'header', header) + session.transfer(ff, REL_SUCCESS) + + + Module Directory + + + 0 + 0 sec + TIMER_DRIVEN + 1 sec + + false + Strip Header + + false + failure + + + false + success + + STOPPED + - com.thinkbiganalytics.nifi.v2.hdfs.CreateHDFSFolder + org.apache.nifi.processors.standard.RouteOnAttribute - 915d89df-849f-3bd7-0000-000000000000 + acaeab44-15cd-3ec7-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 1647.1827561560494 - 421.93743611679474 + 2270.468744143669 + 2582.1659832268597 nifi-standard-nar @@ -7815,272 +7591,245 @@ ${daf.type.folder}/kylo/${daf.domain}__${daf.subdomain}/${feed}/profile WARN - 1 + 0 - Proxy Host + Max Attribute Length - Proxy Host + Max Attribute Length - Http Proxy Password + Command Path - Http Proxy Password + Command Path - Port + Working Directory - Port + Working Directory - Remote File + Output Destination Attribute - Remote File + Output Destination Attribute - proxy-configuration-service + Argument Delimiter - org.apache.nifi.proxy.ProxyConfigurationService - proxy-configuration-service + Argument Delimiter - Http Proxy Username + Ignore STDIN - Http Proxy Username + Ignore STDIN - Hostname + Command Arguments - Hostname + Command Arguments + + ALL + false + 30 sec + - Create Directory - - Create Directory - + Max Attribute Length + 256 - Host Key File - - Host Key File - + Command Path + /opt/cloudera/parcels/CDH/bin/hdfs - Use Compression - - Use Compression - + Working Directory - Disable Directory Listing - - Disable Directory Listing - + Output Destination Attribute - Send Keep Alive On Timeout - - Send Keep Alive On Timeout - + Argument Delimiter + ; - Proxy Type - - Proxy Type - + Ignore STDIN + false - Completion Strategy - - Completion Strategy - + Command Arguments + dfs;-chown;${daf.owner}:${daf.owner};${daf.type.folder}/${daf.domain}__${daf.subdomain}/${feed}/0* + + 0 + 0 sec + EVENT_DRIVEN + 1 sec + + false + Change Owner + + false + nonzero status + + + false + original + + + true + output stream + + STOPPED + + org.apache.nifi.processors.standard.ExecuteStreamCommand + + + afd730ff-bc4f-362d-0000-000000000000 + 23ce9c70-cc0f-3f5f-0000-000000000000 + + 2417.602838331386 + 340.7820653915405 + + + nifi-standard-nar + org.apache.nifi + 1.7.0 + + + WARN + + 1 + - Proxy Port + Routing Strategy - Proxy Port + Routing Strategy - Private Key Passphrase + csv - Private Key Passphrase + csv - Connection Timeout + json - Connection Timeout + json + + ALL + false + 30 sec + - Data Timeout - - Data Timeout - + Routing Strategy + Route to Property name - Private Key Path - - Private Key Path - + csv + ${daf.format:equals('csv')} - Username - - Username - - - - Move Destination Directory - - Move Destination Directory - - - - Strict Host Key Checking - - Strict Host Key Checking - - - - Password - - Password - - - - ALL - false - 30 sec - - - Proxy Host - - - Http Proxy Password - - - Port - ${sftp.remote.port} - - - Remote File - ${path}/${filename} - - - proxy-configuration-service - - - Http Proxy Username - - - Hostname - ${sftp.remote.host} - - - Create Directory - false - - - Host Key File - - - Use Compression - false - - - Disable Directory Listing - false - - - Send Keep Alive On Timeout - true - - - Proxy Type - DIRECT - - - Completion Strategy - Delete File - - - Proxy Port - - - Private Key Passphrase - - - Connection Timeout - 30 sec - - - Data Timeout - 30 sec - - - Private Key Path - /etc/security/nifi_sftp - - - Username - nifi_sftp - - - Move Destination Directory - - - Strict Host Key Checking - false - - - Password + json + ${daf.format:equals('json')} 0 - 3 sec + 0 sec TIMER_DRIVEN 1 sec false - FetchSFTP + CSV or JSON? false - comms.failure + csv false - not.found + json false - permission.denied + unmatched + + STOPPED + + org.apache.nifi.processors.standard.RouteOnAttribute + + + b36c6afa-6946-307b-0000-000000000000 + 23ce9c70-cc0f-3f5f-0000-000000000000 + + 3445.3577029802254 + 1453.6156231848872 + + + teamdigitale-daf-nifi-processors + default + unversioned + + + WARN + + 1 + + ALL + false + 30 sec + + 0 + 0 sec + TIMER_DRIVEN + 1 sec + + false + DafJSONArrayToMultiline + + false + failure false success STOPPED - + it.gov.daf.nifi.processors.DafJSONArrayToMultiline - 9918077d-f872-3034-0000-000000000000 + b482b334-667f-38d7-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 2149.4469195929264 - 1987.1713942797642 + 3445.3577029802254 + 1276.332948761763 - nifi-scripting-nar + nifi-standard-nar org.apache.nifi 1.7.0 @@ -8090,39 +7839,39 @@ ${daf.type.folder}/kylo/${daf.domain}__${daf.subdomain}/${feed}/profile 1 - Script File + Regular Expression - Script File + Regular Expression - Script Engine + Replacement Value - Script Engine + Replacement Value - invalidateMetadata + Evaluation Mode - invalidateMetadata + Evaluation Mode - Script Body + Character Set - Script Body + Character Set - Module Directory + Maximum Buffer Size - Module Directory + Maximum Buffer Size - impalaThriftServiceName + Replacement Strategy - impalaThriftServiceName + Replacement Strategy @@ -8131,57 +7880,28 @@ ${daf.type.folder}/kylo/${daf.domain}__${daf.subdomain}/${feed}/profile 30 sec - Script File + Regular Expression + ( *\n *) - Script Engine - Groovy + Replacement Value + - invalidateMetadata - invalidate metadata + Evaluation Mode + Line-by-Line - Script Body - import org.apache.nifi.controller.ControllerService -import org.apache.nifi.processor.ProcessSession - -session = (ProcessSession) session - -flowFile = session.get() -if (!flowFile) return - -lookup = context.controllerServiceLookup -impalaThriftServiceName = impalaThriftServiceName.evaluateAttributeExpressions(flowFile).value -invalidateMetadata = invalidateMetadata.evaluateAttributeExpressions(flowFile).value - -def dbcpServiceCS = lookup.getControllerServiceIdentifiers(ControllerService).find { - cs -> lookup.getControllerServiceName(cs) == impalaThriftServiceName -} -def impalaConnection = lookup.getControllerService(dbcpServiceCS)?.getConnection() -def statement = impalaConnection.prepareCall(invalidateMetadata) - -try { - statement.execute() -} catch (Exception ex) { - log.error(ex.message) - flowFile = session.putAttribute(flowFile, context.getName() + " Exit code", "FAILED") - flowFile = session.putAttribute(flowFile, context.getName() + " Exit description", ex.message) - flowFile = session.putAttribute(flowFile, "kylo.jobExitDescription", ex.message) - impalaConnection?.close() - session.transfer(flowFile, REL_FAILURE) - return -1 -} - -impalaConnection?.close() -session.transfer(flowFile, REL_SUCCESS) + Character Set + UTF-8 - Module Directory + Maximum Buffer Size + 1 MB - impalaThriftServiceName - Impala + Replacement Strategy + Regex Replace 0 @@ -8190,7 +7910,7 @@ session.transfer(flowFile, REL_SUCCESS) 1 sec false - Invalidate Metadata + Remove Newlines false failure @@ -8203,20 +7923,20 @@ session.transfer(flowFile, REL_SUCCESS) - org.apache.nifi.processors.script.ExecuteScript + org.apache.nifi.processors.standard.ReplaceText - 9dcf860a-1f57-3ca5-0000-000000000000 + b56903b6-2b2e-393d-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 3787.139785845058 - 2167.6415245760863 + 6122.1161990129185 + 1223.0991368642572 - kylo-nifi-spark-v1.2-nar + kylo-nifi-core-v1.2-nar default unversioned @@ -8226,136 +7946,46 @@ session.transfer(flowFile, REL_SUCCESS) 1 - Spark YARN Deploy Mode + Metadata Provider Service - Spark YARN Deploy Mode + com.thinkbiganalytics.nifi.core.api.metadata.MetadataProviderService + Metadata Provider Service - Hadoop Configuration Resources + Feed Name - Hadoop Configuration Resources + Feed Name - SparkHome + hiveTableNames - SparkHome + hiveTableNames - Executor Memory + hiveSchema - Executor Memory + hiveSchema - Yarn Queue + Namespace - Yarn Queue + Namespace - ApplicationJAR + Feed Category - ApplicationJAR + Feed Category - Data Sources + hdfsFolders - Data Sources - - - - MainArgs - - MainArgs - - - - Spark Application Name - - Spark Application Name - - - - Spark Process Timeout - - Spark Process Timeout - - - - Extra JARs - - Extra JARs - - - - MainClass - - MainClass - - - - Executor Cores - - Executor Cores - - - - Kerberos Keytab - - Kerberos Keytab - - - - Spark Configurations - - Spark Configurations - - - - SparkMaster - - SparkMaster - - - - Kerberos Principal - - Kerberos Principal - - - - Number of Executors - - Number of Executors - - - - Extra Files - - Extra Files - - - - Driver Memory - - Driver Memory - - - - Network Timeout - - Network Timeout - - - - Metadata Service - - com.thinkbiganalytics.nifi.core.api.metadata.MetadataProviderService - Metadata Service + hdfsFolders @@ -8364,90 +7994,36 @@ session.transfer(flowFile, REL_SUCCESS) 30 sec - Spark YARN Deploy Mode - cluster - - - Hadoop Configuration Resources - /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml - - - SparkHome - /opt/cloudera/parcels/CDH/lib/spark - - - Executor Memory - 5120m - - - Yarn Queue - - - ApplicationJAR - /usr/nifi/kylo/kylo-spark-validate-cleanse-spark-v1-0.9.1.3-jar-with-dependencies.jar - - - Data Sources - - - MainArgs - ${daf.domain}__${daf.subdomain},${feed},${feedts},${table_field_policy_json_file} - - - Spark Application Name - Validator - - - Spark Process Timeout - 1 hr - - - Extra JARs - /opt/cloudera/parcels/CDH/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core.jar,/usr/nifi/drivers/mariadb-java-client-2.1.0.jar - - - MainClass - com.thinkbiganalytics.spark.datavalidator.Validator - - - Executor Cores - 4 - - - Kerberos Keytab - /usr/nifi/daf.keytab - - - Spark Configurations - spark.default.parallelism=4 - - - SparkMaster - yarn + Metadata Provider Service + ad55fab7-e0c2-3b0f-0000-000000000000 - Kerberos Principal - daf@DAF.GOV.IT + Feed Name + ${feed} - Number of Executors - 4 + hiveTableNames + ${feed}_feed + ${feed}_valid + ${feed}_invalid + ${feed}_profile + ${feed} - Extra Files - /etc/hive/conf/hive-site.xml,${table_field_policy_json_file} + hiveSchema + ${daf.domain}__${daf.subdomain} - Driver Memory - 2048m + Namespace + registration - Network Timeout - 120s + Feed Category + ${daf.organization} - Metadata Service - ad55fab7-e0c2-3b0f-0000-000000000000 + hdfsFolders + ${daf.type.folder} 0 @@ -8456,77 +8032,7 @@ session.transfer(flowFile, REL_SUCCESS) 1 sec false - Validate And Split Records - - false - failure - - - false - success - - STOPPED - - com.thinkbiganalytics.nifi.v2.spark.ExecuteSparkJob - - - 9e3481f4-73b6-324d-0000-000000000000 - 23ce9c70-cc0f-3f5f-0000-000000000000 - - 3787.139785845058 - 1819.494253048978 - - - kylo-nifi-core-v1.2-nar - default - unversioned - - - WARN - - 1 - - - Statement - - Statement - - - - Database Connection Pooling Service - - com.thinkbiganalytics.nifi.v2.thrift.ThriftService - Database Connection Pooling Service - - - - ALL - false - 30 sec - - - Statement - alter table ${daf.domain}__${daf.subdomain}.${feed}_feed add - if not exists partition (processing_dttm=${feedts}) - location '${daf.type.folder}/kylo/${daf.domain}__${daf.subdomain}/${feed}/feed/${feedts}/' - - - Database Connection Pooling Service - a11778f5-3976-3513-0000-000000000000 - - - 0 - 1 sec - TIMER_DRIVEN - 1 sec - - false - Create Feed Partition + Record Metadata false failure @@ -8539,17 +8045,17 @@ session.transfer(flowFile, REL_SUCCESS) - com.thinkbiganalytics.nifi.v2.thrift.ExecuteHQLStatement + com.thinkbiganalytics.nifi.v2.metadata.PutFeedMetadata - a2ef77b3-5ef8-3aca-0000-000000000000 + be3b6cae-4b6f-3eae-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 3787.139785845058 - 2371.432948981327 + 5100.786330153351 + 1015.9776133724881 nifi-standard-nar @@ -8559,7 +8065,7 @@ session.transfer(flowFile, REL_SUCCESS) WARN - 0 + 1 Max Attribute Length @@ -8637,7 +8143,7 @@ session.transfer(flowFile, REL_SUCCESS) 0 0 sec - EVENT_DRIVEN + TIMER_DRIVEN 1 sec false @@ -8664,11 +8170,11 @@ session.transfer(flowFile, REL_SUCCESS) org.apache.nifi.processors.standard.ExecuteStreamCommand - a8bfa77e-a933-390d-0000-000000000000 + be494f54-1626-3bb3-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 3787.139785845058 - 123.75086975097656 + 156.35760628698335 + 1047.6555240216567 kylo-nifi-core-v1.2-nar @@ -8681,15 +8187,28 @@ session.transfer(flowFile, REL_SUCCESS) 1 - Header Line Count + Metadata Provider Service - Header Line Count + com.thinkbiganalytics.nifi.core.api.metadata.MetadataProviderService + Metadata Provider Service - Enable processing + System feed category - Enable processing + System feed category + + + + Cache Expiry Duration + + Cache Expiry Duration + + + + System feed name + + System feed name @@ -8698,12 +8217,20 @@ session.transfer(flowFile, REL_SUCCESS) 30 sec - Header Line Count - 1 + Metadata Provider Service + ad55fab7-e0c2-3b0f-0000-000000000000 + + + System feed category + ${category} + + + Cache Expiry Duration + 1 secs - Enable processing - ${skipHeader} + System feed name + ${feed} 0 @@ -8712,89 +8239,43 @@ session.transfer(flowFile, REL_SUCCESS) 1 sec false - Strip Header + GetFeedMetadata false - content - - - true - failure - - - true - header - - - true - original + Success STOPPED - - com.thinkbiganalytics.nifi.v2.ingest.StripHeader + - org.apache.nifi.processors.standard.ExecuteStreamCommand + org.apache.nifi.processors.attributes.UpdateAttribute - b19e39a4-c7cf-3f4b-0000-000000000000 + c6f7d1f4-ed23-36a6-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 6001.094374462176 - 776.4210161681431 + 568.7648481016477 + 1888.94619865787 - nifi-scripting-nar + nifi-standard-nar org.apache.nifi 1.7.0 WARN - + Porta il log posto nell'attributo ErrorMessage nel flowfile e inserisce un campo indicante se il file ha ricominciato il flusso o meno 1 - Script File - - Script File - - - - grantRoleQuery + attributes-to-json-regex - grantRoleQuery + attributes-to-json-regex - Script Engine + Destination - Script Engine + Destination - hiveThriftServiceName + Null Value - hiveThriftServiceName + Null Value - Script Body + Attributes List - Script Body + Attributes List - Module Directory + Include Core Attributes - Module Directory + Include Core Attributes @@ -9002,61 +8446,23 @@ session.transfer(flowFile, REL_SUCCESS) 30 sec - Script File - - - grantRoleQuery - GRANT ALL ON TABLE ${daf.domain}__${daf.subdomain}.${feed} TO ROLE ${daf.owner.role} WITH GRANT OPTION + attributes-to-json-regex - Script Engine - Groovy + Destination + flowfile-content - hiveThriftServiceName - Hive Thrift Service + Null Value + false - Script Body - import groovy.sql.Sql -import org.apache.nifi.controller.ControllerService -import org.apache.nifi.processor.ProcessSession - -session = (ProcessSession) session - -flowFile = session.get() -if (!flowFile) return - -lookup = context.controllerServiceLookup -hiveThriftServiceName = hiveThriftServiceName.evaluateAttributeExpressions(flowFile).value -grantRoleQueryList = grantRoleQuery.evaluateAttributeExpressions(flowFile).value - -def dbcpServiceCS = lookup.getControllerServiceIdentifiers(ControllerService).find { - cs -> lookup.getControllerServiceName(cs) == hiveThriftServiceName -} -def hiveConnection = lookup.getControllerService(dbcpServiceCS)?.getConnection() -def sql = new Sql(hiveConnection) - -try { - // this should not fail - grantRoleQueryList.split(";").each { - sql.execute(it) - } -} catch (Exception ex) { - log.error(ex.message) - flowFile = session.putAttribute(flowFile, context.getName() + " Exit code", "FAILED") - flowFile = session.putAttribute(flowFile, context.getName() + " Exit description", ex.message) - flowFile = session.putAttribute(flowFile, "kylo.jobExitDescription", ex.message) - hiveConnection?.close() - session.transfer(flowFile, REL_FAILURE) - return -1 -} - -hiveConnection?.close() -session.transfer(flowFile, REL_SUCCESS) + Attributes List + ErrorMessage, RestartFromBeginning - Module Directory + Include Core Attributes + true 0 @@ -9065,9 +8471,9 @@ session.transfer(flowFile, REL_SUCCESS) 1 sec false - Grant ALL on Table To Owner Role + LogFileCreation - false + true failure @@ -9076,177 +8482,71 @@ session.transfer(flowFile, REL_SUCCESS) STOPPED - it.gov.daf.nifi.processors.DafJSONArrayToMultiline - - - b4c25172-cd77-3227-0000-000000000000 - 23ce9c70-cc0f-3f5f-0000-000000000000 - - 2391.8570010902326 - 119.73994445800781 - - - nifi-update-attribute-nar - org.apache.nifi - 1.7.0 - WARN 1 - Delete Attributes Expression + Permissions umask - Delete Attributes Expression + Permissions umask - daf.type.folder + Hadoop Configuration Resources - daf.type.folder + Hadoop Configuration Resources - Store State + Kerberos Keytab - Store State + Kerberos Keytab - daf.owner.role + Remote Owner - daf.owner.role + Remote Owner - Stateful Variables Initial Value + Kerberos Principal - Stateful Variables Initial Value + Kerberos Principal - - ALL - false - 30 sec - - - Delete Attributes Expression - - - daf.type.folder - ${hive.master.root}/${daf.type}/${daf.organization} - - - Store State - Do not store state - - - daf.owner.role - ${daf.owner}_user_role - - - Stateful Variables Initial Value - - - 0 - 0 sec - TIMER_DRIVEN - 1 sec - - false - Set daf.type.folder for ORDINARY - - false - success - - STOPPED - - org.apache.nifi.processors.attributes.UpdateAttribute - - - b917516c-bb0d-3cec-0000-000000000000 - 23ce9c70-cc0f-3f5f-0000-000000000000 - - 1471.4457371502358 - 119.73994445800781 - - - nifi-standard-nar - org.apache.nifi - 1.7.0 - - - WARN - - 10 - - not_to_fetch + Kerberos Relogin Period - not_to_fetch + Kerberos Relogin Period - Routing Strategy + Remote Group - Routing Strategy + Remote Group - fetch + Directory - fetch + Directory @@ -9255,16 +8555,42 @@ session.transfer(flowFile, REL_SUCCESS) 30 sec - not_to_fetch - ${toFetch:contains('no')} + Permissions umask + 0007 - Routing Strategy - Route to Property name + Hadoop Configuration Resources + /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml - fetch - ${toFetch:contains('yes')} + Kerberos Keytab + /usr/nifi/daf.keytab + + + Remote Owner + daf + + + Kerberos Principal + daf@DAF.GOV.IT + + + Kerberos Relogin Period + 4 hours + + + Remote Group + daf + + + Directory + ${daf.type.folder}/kylo + ${daf.type.folder}/kylo/${daf.domain}__${daf.subdomain} + ${daf.type.folder}/kylo/${daf.domain}__${daf.subdomain}/${feed} + ${daf.type.folder}/kylo/${daf.domain}__${daf.subdomain}/${feed}/valid + ${daf.type.folder}/kylo/${daf.domain}__${daf.subdomain}/${feed}/invalid + ${daf.type.folder}/kylo/${daf.domain}__${daf.subdomain}/${feed}/feed + ${daf.type.folder}/kylo/${daf.domain}__${daf.subdomain}/${feed}/profile 0 @@ -9273,67 +8599,52 @@ session.transfer(flowFile, REL_SUCCESS) 1 sec false - RouteOnAttribute - - false - fetch - + Register HDFS Kylo Folders false - not_to_fetch + failure false - unmatched + success STOPPED - org.apache.nifi.processors.standard.RouteOnAttribute + com.thinkbiganalytics.nifi.v2.hdfs.CreateHDFSFolder - ba30bbc8-3d35-39f2-0000-000000000000 + caefa30d-a8dd-3b99-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 399.51945410898406 - 1687.722963608544 + 2863.23585538732 + 2831.9987380870334 - nifi-standard-nar - org.apache.nifi - 1.7.0 + kylo-nifi-core-v1.2-nar + default + unversioned WARN - 50 + 1 - Content Buffer Size - - Content Buffer Size - - - - Match Requirement - - Match Requirement - - - - Character Set + Statement - Character Set + Statement - uuid_matched_message + Database Connection Pooling Service - uuid_matched_message + com.thinkbiganalytics.nifi.v2.thrift.ThriftService + Database Connection Pooling Service @@ -9342,47 +8653,39 @@ session.transfer(flowFile, REL_SUCCESS) 30 sec - Content Buffer Size - 10 MB - - - Match Requirement - content must contain match - - - Character Set - UTF-8 + Statement + DROP TABLE ${daf.domain}__${daf.subdomain}.temp_${feed}; - uuid_matched_message - ${uuid.toMatch} + Database Connection Pooling Service + a11778f5-3976-3513-0000-000000000000 0 - 1 sec + 0 sec TIMER_DRIVEN 1 sec false - Filter Message By uuid To Match + Drop Temp_table - true - unmatched + false + failure false - uuid_matched_message + success STOPPED - com.thinkbiganalytics.nifi.v2.hdfs.RemoveHDFSFolder + com.thinkbiganalytics.nifi.v2.core.watermark.ReleaseHighWaterMark - cc4a30e3-41ef-31cb-0000-000000000000 + d966765c-2aa7-318c-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 6001.094374462176 - 1012.3661026187247 + 2270.468744143669 + 2391.287318172297 - kylo-nifi-core-v1.2-nar - default - unversioned + nifi-standard-nar + org.apache.nifi + 1.7.0 WARN - 1 + 0 - Metadata Provider Service + Max Attribute Length - com.thinkbiganalytics.nifi.core.api.metadata.MetadataProviderService - Metadata Provider Service + Max Attribute Length - Feed Name + Command Path - Feed Name + Command Path - hiveTableNames + Working Directory - hiveTableNames + Working Directory - hiveSchema + Output Destination Attribute - hiveSchema + Output Destination Attribute - Namespace + Argument Delimiter - Namespace + Argument Delimiter - Feed Category + Ignore STDIN - Feed Category + Ignore STDIN - hdfsFolders + Command Arguments - hdfsFolders + Command Arguments @@ -9660,168 +8986,208 @@ ${daf.category.folder}/kylo/${daf.domain}__${daf.subdomain}/${feed}/ 30 sec - Metadata Provider Service - ad55fab7-e0c2-3b0f-0000-000000000000 + Max Attribute Length + 256 - Feed Name - ${feed} + Command Path + /opt/cloudera/parcels/CDH/bin/hdfs - hiveTableNames - ${feed}_feed -${feed}_valid -${feed}_invalid -${feed}_profile -${feed} + Working Directory - hiveSchema - ${daf.domain}__${daf.subdomain} + Output Destination Attribute - Namespace - registration + Argument Delimiter + - Feed Category - ${daf.organization} + Ignore STDIN + false - hdfsFolders - ${daf.type.folder} + Command Arguments + dfs -setfacl -m mask::rwx ${daf.type.folder}/${daf.domain}__${daf.subdomain}/${feed}/0* 0 0 sec - TIMER_DRIVEN + EVENT_DRIVEN 1 sec false - Record Metadata + Set Acl's false - failure + nonzero status false - success + original + + + true + output stream STOPPED - com.thinkbiganalytics.nifi.v2.metadata.PutFeedMetadata + org.apache.nifi.processors.standard.ExecuteStreamCommand - ccab579f-ab04-3c4d-0000-000000000000 + d9c9739c-3cca-3ef5-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 6001.094374462176 - 1242.9083350968313 + 3908.1616103958004 + 2378.374558821619 - kylo-nifi-core-v1.2-nar + kylo-nifi-spark-v1.2-nar default unversioned WARN - 2 + 1 - Target Table Properties + Spark YARN Deploy Mode - Target Table Properties + Spark YARN Deploy Mode - Database Connection Pooling Service + Hadoop Configuration Resources - com.thinkbiganalytics.nifi.v2.thrift.ThriftService - Database Connection Pooling Service + Hadoop Configuration Resources - System feed name + SparkHome - System feed name + SparkHome - Feed Root Path + Executor Memory - Feed Root Path + Executor Memory - Profile Root Path + Yarn Queue - Profile Root Path + Yarn Queue - Field specification + ApplicationJAR - Field specification + ApplicationJAR - Feed Table Properties + Data Sources - Feed Table Properties + Data Sources - Feed Table Override + MainArgs - Feed Table Override + MainArgs - Table Type + Spark Application Name - Table Type + Spark Application Name - Feed Field specification + Spark Process Timeout - Feed Field specification + Spark Process Timeout - System feed category + Extra JARs - System feed category + Extra JARs - Master Root Path + MainClass - Master Root Path + MainClass - Feed Table Storage Format + Executor Cores - Feed Table Storage Format + Executor Cores - Partition specification + Kerberos Keytab - Partition specification + Kerberos Keytab - Target Table Format + Spark Configurations - Target Table Format + Spark Configurations + + + + SparkMaster + + SparkMaster + + + + Kerberos Principal + + Kerberos Principal + + + + Number of Executors + + Number of Executors + + + + Extra Files + + Extra Files + + + + Driver Memory + + Driver Memory + + + + Network Timeout + + Network Timeout + + + + Metadata Service + + com.thinkbiganalytics.nifi.core.api.metadata.MetadataProviderService + Metadata Service @@ -9830,73 +9196,99 @@ ${feed} 30 sec - Target Table Properties - ${metadata.table.targetTblProperties.parquet} + Spark YARN Deploy Mode + cluster - Database Connection Pooling Service - a11778f5-3976-3513-0000-000000000000 + Hadoop Configuration Resources + /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml - System feed name - ${feed} + SparkHome + /opt/cloudera/parcels/CDH/lib/spark - Feed Root Path - ${daf.type.folder}/kylo + Executor Memory + 5120m - Profile Root Path - ${daf.type.folder}/kylo + Yarn Queue - Field specification - ${metadata.table.fieldStructure} + ApplicationJAR + /usr/nifi/kylo/kylo-spark-validate-cleanse-spark-v1-0.9.1.3-jar-with-dependencies.jar - Feed Table Properties - ${metadata.table.feedTblProperties} + Data Sources - Feed Table Override - ${metadata.table.feedTableOverride} + MainArgs + ${daf.domain}__${daf.subdomain},${feed},${feedts},${table_field_policy_json_file} - Table Type - ALL + Spark Application Name + Validator - Feed Field specification - ${metadata.table.feedFieldStructure} + Spark Process Timeout + 1 hr - System feed category - ${daf.domain}__${daf.subdomain} + Extra JARs + /opt/cloudera/parcels/CDH/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core.jar,/usr/nifi/drivers/mariadb-java-client-2.1.0.jar - Master Root Path - ${daf.type.folder} + MainClass + com.thinkbiganalytics.spark.datavalidator.Validator - Feed Table Storage Format - ${metadata.table.feedFormat} + Executor Cores + 1 - Partition specification - ${metadata.table.partitionStructure} + Kerberos Keytab + /usr/nifi/daf.keytab - Target Table Format - ${metadata.table.targetFormat} + Spark Configurations + spark.default.parallelism=4 - - 0 - 0 sec - TIMER_DRIVEN - 1 sec + + SparkMaster + yarn + + + Kerberos Principal + daf@DAF.GOV.IT + + + Number of Executors + 1 + + + Extra Files + /etc/hive/conf/hive-site.xml,${table_field_policy_json_file} + + + Driver Memory + 2048m + + + Network Timeout + 120s + + + Metadata Service + ad55fab7-e0c2-3b0f-0000-000000000000 + + + 0 + 0 sec + TIMER_DRIVEN + 1 sec false - Register Tables + Validate And Split Records false failure @@ -9909,68 +9301,162 @@ ${feed} - com.thinkbiganalytics.nifi.v2.ingest.RegisterFeedTables + com.thinkbiganalytics.nifi.v2.spark.ExecuteSparkJob - dfcd2870-bb2d-3422-0000-000000000000 + db350530-8fef-3bab-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 2149.4469195929264 - 2180.5542839267646 + 3443.5261600114754 + 1099.119350474466 - nifi-standard-nar - org.apache.nifi - 1.7.0 + teamdigitale-daf-nifi-processors + default + unversioned WARN - 0 + 1 + + ALL + false + 30 sec + + 0 + 0 sec + TIMER_DRIVEN + 1 sec + + false + DafJSONCleansing + + false + failure + + + false + success + + STOPPED + + it.gov.daf.nifi.processors.DafJSONCleansing + + + db5d4b49-9b90-349b-0000-000000000000 + 23ce9c70-cc0f-3f5f-0000-000000000000 + + 6122.1161990129185 + 1453.6413693423638 + + + kylo-nifi-core-v1.2-nar + default + unversioned + + + WARN + + 2 - Max Attribute Length + Target Table Properties - Max Attribute Length + Target Table Properties - Command Path + Database Connection Pooling Service - Command Path + com.thinkbiganalytics.nifi.v2.thrift.ThriftService + Database Connection Pooling Service - Working Directory + System feed name - Working Directory + System feed name - Output Destination Attribute + Feed Root Path - Output Destination Attribute + Feed Root Path - Argument Delimiter + Profile Root Path - Argument Delimiter + Profile Root Path - Ignore STDIN + Field specification - Ignore STDIN + Field specification - Command Arguments + Feed Table Properties - Command Arguments + Feed Table Properties + + + + Feed Table Override + + Feed Table Override + + + + Table Type + + Table Type + + + + Feed Field specification + + Feed Field specification + + + + System feed category + + System feed category + + + + Master Root Path + + Master Root Path + + + + Feed Table Storage Format + + Feed Table Storage Format + + + + Partition specification + + Partition specification + + + + Target Table Format + + Target Table Format @@ -9979,66 +9465,96 @@ ${feed} 30 sec - Max Attribute Length - 256 + Target Table Properties + ${metadata.table.targetTblProperties.parquet} - Command Path - /opt/cloudera/parcels/CDH/bin/hdfs + Database Connection Pooling Service + a11778f5-3976-3513-0000-000000000000 - Working Directory + System feed name + ${feed} - Output Destination Attribute + Feed Root Path + ${daf.type.folder}/kylo - Argument Delimiter - + Profile Root Path + ${daf.type.folder}/kylo - Ignore STDIN - false + Field specification + ${metadata.table.fieldStructure} - Command Arguments - dfs -setfacl -m mask::rwx ${daf.type.folder}/${daf.domain}__${daf.subdomain}/${feed}/0* + Feed Table Properties + ${metadata.table.feedTblProperties} + + + Feed Table Override + ${metadata.table.feedTableOverride} + + + Table Type + ALL + + + Feed Field specification + ${metadata.table.feedFieldStructure} + + + System feed category + ${daf.domain}__${daf.subdomain} + + + Master Root Path + ${daf.type.folder} + + + Feed Table Storage Format + ${metadata.table.feedFormat} + + + Partition specification + ${metadata.table.partitionStructure} + + + Target Table Format + ${metadata.table.targetFormat} 0 0 sec - EVENT_DRIVEN + TIMER_DRIVEN 1 sec false - Set Acl's + Register Tables false - nonzero status + failure false - original - - - true - output stream + success STOPPED - org.apache.nifi.processors.standard.ExecuteStreamCommand + com.thinkbiganalytics.nifi.v2.ingest.RegisterFeedTables - e29b1073-0e21-3bec-0000-000000000000 + e57145c1-e445-3043-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 2149.4469195929264 - 1777.1364483464622 + 2863.23585538732 + 2582.1659832268597 kylo-nifi-core-v1.2-nar @@ -10051,46 +9567,83 @@ ${feed} 1 - Release All + Feed partition value - Release All + Feed partition value - System feed category + Merge Strategy - System feed category + Merge Strategy - High-Water Mark + Hive Configurations - High-Water Mark + Hive Configurations - Mode + Database Connection Pooling Service - Mode - + com.thinkbiganalytics.nifi.v2.thrift.ThriftService + Database Connection Pooling Service + - System feed name + Source table - System feed name + Source table - Metadata Service + Hive Connection Pooling Service - com.thinkbiganalytics.nifi.core.api.metadata.MetadataProviderService - Metadata Service + org.apache.nifi.dbcp.hive.HiveDBCPService + Hive Connection Pooling Service - High-Water Mark Value Property Name + Field specification - High-Water Mark Value Property Name + Field specification + + + + Target schema + + Target schema + + + + Source schema + + Source schema + + + + Reset hive on connection + + Reset hive on connection + + + + Target table + + Target table + + + + Partition Specification + + Partition Specification + + + + blocking-id + + blocking-id @@ -10099,32 +9652,53 @@ ${feed} 30 sec - Release All - true + Feed partition value + ${feedts} - System feed category - ${category} + Merge Strategy + ${metadata.table.targetMergeStrategy} - High-Water Mark - highWaterMark + Hive Configurations - Mode - COMMIT + Database Connection Pooling Service + a11778f5-3976-3513-0000-000000000000 - System feed name + Source table + ${feed}_valid + + + Hive Connection Pooling Service + + + Field specification + ${metadata.table.fieldStructure} + + + Target schema + ${daf.domain}__${daf.subdomain} + + + Source schema + ${daf.domain}__${daf.subdomain} + + + Reset hive on connection + true + + + Target table ${feed} - Metadata Service - ad55fab7-e0c2-3b0f-0000-000000000000 + Partition Specification + ${metadata.table.partitionSpecs} - High-Water Mark Value Property Name - water.mark + blocking-id 0 @@ -10133,134 +9707,94 @@ ${feed} 1 sec false - Commit Highwater Mark - - true - cancelledWaterMark - + Merge Table false failure - true + false success STOPPED - com.thinkbiganalytics.nifi.v2.core.watermark.ReleaseHighWaterMark + com.thinkbiganalytics.nifi.v2.ingest.MergeTable - e6a77bb8-8cba-3da1-0000-000000000000 + e5da1cdd-8a89-31ee-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 665.0325043677149 - 119.73994445800781 + 3908.1616103958004 + 1856.7934595357838 - kylo-nifi-core-v1.2-nar - default - unversioned + nifi-hadoop-nar + org.apache.nifi + 1.7.0 WARN - 1 + 10 - Metadata Provider Service + Permissions umask - com.thinkbiganalytics.nifi.core.api.metadata.MetadataProviderService - Metadata Provider Service + Permissions umask - System feed category + Hadoop Configuration Resources - System feed category + Hadoop Configuration Resources - Cache Expiry Duration + Remote Owner - Cache Expiry Duration + Remote Owner - System feed name + Additional Classpath Resources - System feed name + Additional Classpath Resources - - ALL - false - 30 sec - - - Metadata Provider Service - ad55fab7-e0c2-3b0f-0000-000000000000 - - System feed category - ${category} + Compression codec + + Compression codec + - Cache Expiry Duration - 1 secs + IO Buffer Size + + IO Buffer Size + - System feed name - ${feed} + Remote Group + + Remote Group + - - 0 - 0 sec - TIMER_DRIVEN - 1 sec - - false - GetFeedMetadata - - false - Success - - STOPPED - - com.thinkbiganalytics.nifi.v2.hdfs.CreateHDFSFolder + org.apache.nifi.processors.hadoop.PutHDFS - effed46e-3491-3f02-0000-000000000000 + ec68082e-ecf5-39e3-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 2601.996527511757 - 421.93743611679474 + 1101.7382555843433 + 2849.1693808621003 - nifi-update-attribute-nar + nifi-scripting-nar org.apache.nifi 1.7.0 WARN - + Reinserisce l'header al flowfile recuperandolo dall'attributo header + 1 - Delete Attributes Expression + Script File - Delete Attributes Expression + Script File - filename + Script Engine - filename + Script Engine - Store State + Script Body - Store State + Script Body - Stateful Variables Initial Value + Module Directory - Stateful Variables Initial Value + Module Directory @@ -10408,18 +9966,99 @@ ${feed} 30 sec - Delete Attributes Expression + Script File - filename - ${filename}.read + Script Engine + Groovy - Store State - Do not store state + Script Body + def ff = session.get() + if(!ff)return + header = ff.getAttribute('header') + ff = session.write(ff, {rawIn, rawOut-> + // ## transform streams into reader and writer + rawIn.withReader("UTF-8"){reader-> + rawOut.withWriter("UTF-8"){writer-> + writer << header + '\n' + reader.eachLine{line-> + writer << line + '\n' + } + } + } + } as StreamCallback) + session.transfer(ff, REL_SUCCESS) - Stateful Variables Initial Value + Module Directory + + + 0 + 0 sec + TIMER_DRIVEN + 1 sec + + false + Merge With Header + + true + failure + + + false + success + + STOPPED + + org.apache.nifi.processors.script.ExecuteScript + + + f3a2b29e-94e9-3428-0000-000000000000 + 23ce9c70-cc0f-3f5f-0000-000000000000 + + 2067.539960412061 + 609.053568360825 + + + nifi-standard-nar + org.apache.nifi + 1.7.0 + + + WARN + + 1 + + + Packaging Format + + Packaging Format + + + + File Filter + + File Filter + + + + ALL + false + 30 sec + + + Packaging Format + zip + + + File Filter + .* 0 @@ -10428,21 +10067,29 @@ ${feed} 1 sec false - Set read filename + UnpackContent + + false + failure + + + true + original + false success STOPPED org.apache.nifi.processors.attributes.UpdateAttribute - f7e910c3-4d84-3a05-0000-000000000000 + fc934a2f-cd5e-3adc-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 4292.45285794037 - 1201.287679586835 + 1767.8761280019244 + 119.73994445800781 - teamdigitale-daf-nifi-processors - default - unversioned + nifi-hadoop-nar + org.apache.nifi + 1.7.0 WARN - 1 + 10 - Separator Char + Permissions umask - Separator Char + Permissions umask - Escape Char + Hadoop Configuration Resources - Escape Char + Hadoop Configuration Resources - Quote Char + Remote Owner - Quote Char + Remote Owner + + + + Additional Classpath Resources + + Additional Classpath Resources + + + + Compression codec + + Compression codec + + + + IO Buffer Size + + IO Buffer Size + + + + Remote Group + + Remote Group + + + + Block Size + + Block Size + + + + kerberos-credentials-service + + org.apache.nifi.kerberos.KerberosCredentialsService + kerberos-credentials-service + + + + Kerberos Keytab + + Kerberos Keytab + + + + Kerberos Principal + + Kerberos Principal + + + + Replication + + Replication + + + + Kerberos Relogin Period + + Kerberos Relogin Period + + + + Directory + + Directory + + + + Conflict Resolution Strategy + + Conflict Resolution Strategy @@ -10564,16 +10484,59 @@ ${feed} 30 sec - Separator Char - ${daf.separator.char} + Permissions umask + 0277 - Escape Char - ${daf.escape.char} + Hadoop Configuration Resources + /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml - Quote Char - ${daf.quote.char} + Remote Owner + ${daf.owner} + + + Additional Classpath Resources + + + Compression codec + BZIP + + + IO Buffer Size + + + Remote Group + ${daf.owner} + + + Block Size + + + kerberos-credentials-service + + + Kerberos Keytab + /usr/nifi/daf.keytab + + + Kerberos Principal + daf@DAF.GOV.IT + + + Replication + + + Kerberos Relogin Period + 4 hours + + + Directory + /user/daf/archive/${daf.type}/${daf.domain}__${daf.subdomain}/${feed}/${archiveId}_${now():toNumber()} + + + Conflict Resolution Strategy + fail 0 @@ -10582,33 +10545,33 @@ ${feed} 1 sec false - DafCSVCleansing + Archive Originals false failure - false + true success STOPPED - it.gov.daf.nifi.processors.DafCSVCleansing + org.apache.nifi.processors.hadoop.PutHDFS - f89770bd-6d42-3b62-0000-000000000000 + fe750c37-e414-3eb3-0000-000000000000 23ce9c70-cc0f-3f5f-0000-000000000000 - 3324.3358784294833 - 1065.5999145162314 + 6122.1161990129185 + 987.1540504136756 - nifi-standard-nar + nifi-scripting-nar org.apache.nifi 1.7.0 @@ -10618,39 +10581,39 @@ ${feed} 1 - Regular Expression + Script File - Regular Expression + Script File - Replacement Value + grantRoleQuery - Replacement Value + grantRoleQuery - Evaluation Mode + Script Engine - Evaluation Mode + Script Engine - Character Set + hiveThriftServiceName - Character Set + hiveThriftServiceName - Maximum Buffer Size + Script Body - Maximum Buffer Size + Script Body - Replacement Strategy + Module Directory - Replacement Strategy + Module Directory @@ -10659,28 +10622,61 @@ ${feed} 30 sec - Regular Expression - ( *\n *) + Script File - Replacement Value - + grantRoleQuery + GRANT ALL ON TABLE ${daf.domain}__${daf.subdomain}.${feed} TO ROLE ${daf.owner.role} WITH GRANT OPTION - Evaluation Mode - Line-by-Line + Script Engine + Groovy - Character Set - UTF-8 + hiveThriftServiceName + Hive Thrift Service - Maximum Buffer Size - 1 MB + Script Body + import groovy.sql.Sql + import org.apache.nifi.controller.ControllerService + import org.apache.nifi.processor.ProcessSession + + session = (ProcessSession) session + + flowFile = session.get() + if (!flowFile) return + + lookup = context.controllerServiceLookup + hiveThriftServiceName = hiveThriftServiceName.evaluateAttributeExpressions(flowFile).value + grantRoleQueryList = grantRoleQuery.evaluateAttributeExpressions(flowFile).value + + def dbcpServiceCS = lookup.getControllerServiceIdentifiers(ControllerService).find { + cs -> lookup.getControllerServiceName(cs) == hiveThriftServiceName + } + def hiveConnection = lookup.getControllerService(dbcpServiceCS)?.getConnection() + def sql = new Sql(hiveConnection) + + try { + // this should not fail + grantRoleQueryList.split(";").each { + sql.execute(it) + } + } catch (Exception ex) { + log.error(ex.message) + flowFile = session.putAttribute(flowFile, context.getName() + " Exit code", "FAILED") + flowFile = session.putAttribute(flowFile, context.getName() + " Exit description", ex.message) + flowFile = session.putAttribute(flowFile, "kylo.jobExitDescription", ex.message) + hiveConnection?.close() + session.transfer(flowFile, REL_FAILURE) + return -1 + } + + hiveConnection?.close() + session.transfer(flowFile, REL_SUCCESS) - Replacement Strategy - Regex Replace + Module Directory 0 @@ -10689,7 +10685,7 @@ ${feed} 1 sec false - Remove Newlines + Grant ALL on Table To Owner Role false failure @@ -10699,20 +10695,15 @@ ${feed} success STOPPED - - org.apache.nifi.processors.standard.ReplaceText + - org.apache.nifi.processors.attributes.UpdateAttribute - 4be90f6d-7b62-3005-0000-000000000000 b265aac8-443c-33de-0000-000000000000 - 1775.804400851553 - 791.3043427706891 + 1500.4966755140986 + 0.0 nifi-standard-nar @@ -721,8 +655,8 @@ 7587b7b0-46e9-39db-0000-000000000000 b265aac8-443c-33de-0000-000000000000 - 441.0632414437292 - 242.7092943347792 + 163.44190216064453 + 537.5202111597212 nifi-update-attribute-nar @@ -802,8 +736,8 @@ a941e102-9197-3026-0000-000000000000 b265aac8-443c-33de-0000-000000000000 - 441.0632414437292 - 444.9732841037961 + 163.44190216064453 + 743.4980681162381 nifi-hadoop-nar @@ -989,8 +923,8 @@ af1d4292-8e18-311f-0000-000000000000 b265aac8-443c-33de-0000-000000000000 - 2373.628811542234 - 790.326966001533 + 2097.187537414178 + 0.0 nifi-update-attribute-nar @@ -1055,8 +989,8 @@ b50712fc-affa-3536-0000-000000000000 b265aac8-443c-33de-0000-000000000000 - 1273.9295091617605 - 786.6700149856769 + 998.94670066072 + 0.0 nifi-kafka-0-9-nar @@ -1201,8 +1135,8 @@ bc65b52e-f156-3d1a-0000-000000000000 b265aac8-443c-33de-0000-000000000000 - 1423.8910876325872 - 504.7801608180812 + 1003.9436522819506 + 257.1757849506839 nifi-update-attribute-nar @@ -1283,336 +1217,12 @@