diff --git a/README_kylo_template.md b/README_kylo_template.md new file mode 100644 index 0000000..43d0e18 --- /dev/null +++ b/README_kylo_template.md @@ -0,0 +1,31 @@ +**How to register a new Kylo template** +=================== +This document describes how to update a NiFi template using Kylo in production environment. + +---------- +First of all, check if there's an active thread. If so, wait for the active threads to finish and stop every started processor, in order to avoid pipeline errors throwing. Now download the old template on your device, in order to have a backup copy in case of any need, and save the new template on NiFi with the same name of the old one following the steps listed below: + + 1. Copy the new template you decided to register on Kylo and paste it on the NiFi main page in a free room far from anything else; + 2. Create a new template on NiFi with the last one; + 3. Download the new template on your device; + 4. Delete the template you've just pasted on NiFi main page. + +After that, you can finally register the new template on Kylo. In order to do that, follow the steps listed below: + + 1. Stop every FetchSFTP processor because password are not stored, in order to avoid errors generated by queued flowfiles; + 2. Click on this [link](https://kylo.daf.teamdigitale.it/index.html#!/dashboard) to redirect Kylo home; + 3. On the left bar, click on Admin tab and then on Templates tab; + 4. Click on the orange button on the bottom of the page, in order to add a new template; + 5. Choose the "Import from a file" option; + 6. Click on "CHOOSE FILE" and select the xml file that contains the new NiFi template you've just downloaded. + 7. Check every box appearing at the bottom of the page and go on; + 8. Check that the registration has not produced errors and that everything has been done correctly. + +Before you start to use your new registered template, don't forget to update Github repository, typing in sequence +``` +git status +git add +git commit -m "" +git push +``` +If the user is not a repository contributor, he can execute this operations on a fork of the repository and then perform a pull request to the main repository. \ No newline at end of file diff --git a/README_logs.md b/README_logs.md new file mode 100644 index 0000000..d377ca1 --- /dev/null +++ b/README_logs.md @@ -0,0 +1,35 @@ +**How to view logs** +=================== + + +In this document there are instruction on how to view logs in production environment. + +---------- + +## **How to view Kylo and Nifi logs** + +### **Kylo logs** +In order to view Kylo logs, just enter in the path you saved daf-kylo projects and the go in the folder scripts and run `kylologs.sh`. You can enter in your folder typing +``` +cd /daf-kylo/scripts +``` +and run `kylologs.sh` typing +``` +./kylologs.sh +``` +### **NiFi logs** +NiFi logs are saved in the file named `nifi-app.log`, located in the logs folder under the folder you installed NiFi on your device. As instance, if you installed it in `/usr/nifi`, you can view the logs of the node named `tba-nifi-0` typing +``` +kubectl exec -it tba-nifi-0 -- tail -1000f /usr/nifi/logs/nifi-app.log +``` +## **How to view Spark logs** + + +First perform Kerberos initialization by typing `kinit -k -t daf@DAF.GOV.IT` on the command line, where `` stands for the local path daf.keytab file is saved, useful for the Kerberos initialization in production environment. +Then click on this [link](https://master-2.platform.daf.gov.it:8090/cluster/) on Firefox browser, but not in a private window. +Once redirected on the link, you can see a list of application id and everyone of them refers to the corresponding Spark job. Alongside them there are all the corresponding properties. Clicking on application-id, you are redirected to another web page where you can see a resume of the most important informations of the job. Clicking on the bottom right button, you can see Spark logs. + +Another way to view Spark jobs is opening the command-line of the node where yarn CLI is installed and type +``` +yarn logs -applicationId +``` \ No newline at end of file diff --git a/README_new.md b/README_new.md new file mode 100644 index 0000000..7cde8e4 --- /dev/null +++ b/README_new.md @@ -0,0 +1,233 @@ +# Daf-Kylo for PDND (Piattaforma Digitale Nazionale Dati), previously DAF (Data & Analytics Framework) + + + +>Daf-Kylo repository contains the set of tools used for the PDND data ingestion. + +> In order to install and use this repo you may deploy all the components into a _cCloudera_ shared edge node. + +## What is the PDND (previously DAF)? + +PDND stands for "Piattaforma Digitale Nazionale Dati" (Italian Digital Data Platform), previously known as Data & Analytics Framework (DAF). + +In brief, is an attempt to establish a central Chief Data Officer (CDO) for the Government and Public Administration. Its main goal is to promote data exchange among Italian Public Administrations (PAs), to support the diffusion of open data, and to enable data-driven policies. You can find more about the PDND on the official [Digital Transformation Team website](https://teamdigitale.governo.it/it/projects/daf.htm). + +## What is Daf-Kylo? + + + Daf-Kylo is a data lake platform built on Apache Hadoop and Spark. Daf-Kylo provides a data lake solution enabling self-service data ingest, data preparation, and data discovery. Kylo integrates best practices around metadata capture, security, and data quality. Apache Nifi provides a flexible data processing framework for building batch or streaming pipeline templates, and for enabling self-service features. + + +> Daf-Kylo repository contains the set of components used to deploy and manage the PDND data ingestion process. + +> Folder /docker contains all the docker files for build images of daf-kylo components. + +> Folder /kubernetes contains all the yaml files for deploy pods and services on kubernetes. + +> Folder /kylo contains all the kylo stuff such as api documentation for the integration with PDND Portal, kylo templates, kylo patch. + +> Folder /nifi contains all the nifi templates and customized processors used in ingestion process. + +> Folder /scripts contains utils scripts for manage pods, log and other kubernetes stuff. + +### Prerequisites + +* [Kubernetes](https://kubernetes.io/) +* [Cloudera](https://www.cloudera.com/) + +### Project dependencies +Project dependencies can be find by clicking on this [link](https://kylo.readthedocs.io/en/v0.8.3/installation/Dependencies.html#kylo-stack-dependencies). + +### Project components +Project Daf-Kylo depends by the following components. + +* **ActiveMQ** version 5.15.1, available [here]([https://activemq.apache.org/activemq-5151-release.html](https://activemq.apache.org/activemq-5151-release.html)); + +* **Elasticsearch** version 5.6.4, available [here]([https://www.elastic.co/downloads/past-releases/elasticsearch-5-6-4](https://www.elastic.co/downloads/past-releases/elasticsearch-5-6-4)); +* **MariaDB** version 10.3, available [here]([https://mariadb.com/kb/en/library/changes-improvements-in-mariadb-103/](https://mariadb.com/kb/en/library/changes-improvements-in-mariadb-103/)); +* **Spark** version 2.2.0, available [here]([http://spark.apache.org/downloads.html](http://spark.apache.org/downloads.html)); +* **Kylo-Services** version 9.1.0, available [here]([https://kylo.readthedocs.io/en/v0.9.1/about/Downloads.html](https://kylo.readthedocs.io/en/v0.9.1/about/Downloads.html)); +* **Kylo-UI** version 9.1.0, available [here]([https://kylo.readthedocs.io/en/v0.9.1/about/Downloads.html](https://kylo.readthedocs.io/en/v0.9.1/about/Downloads.html)); +* **NiFi** version 1.7.0, available [here]([https://nifi.apache.org/download.html](https://nifi.apache.org/download.html)). + +## How to install and use Daf-Kylo + + Insert here a brief documentation to use this project as an end-user (not a developer) if applicable, including pre-requisites and internal and external dependencies. Insert a link to an extended documentation (user manual) if present.--> + +### MacOS and Linux +Installing Daf-Kylo on Unix-like systems requires a package manager such as Homebrew. You can download and install Homebrew following the instructions given in the [Homebrew official website]([https://brew.sh/](https://brew.sh/)). Once you have installed Homebrew, you can follow some steps to complete the setup. First step is Homebrew cask installation. Open a terminal and type the following command to install Homebrew cask: +``` +brew tap caskroom/cask +``` +Then, update all formulas and Homebrew itself by typing +``` +brew update +``` +Last, install kube-controller-manager, RPM, make and Git by typing +``` +brew install kubectl rpm make git +``` + +## How to build Daf-Kylo + + Insert here a brief documentation for the developer to build, test and contribute. Insert a link to an extended documentation (developer manual) if present.--> +To build most of Docker images, kylo code is required (source and compiled). To get it run, you have to download and compile it, using `Makefile`, by typing the following commands (production and test environment): + + - #### Production + ``` make -f Makefile daf-kylo``` +```make -f Makefile build-kylo``` + + + - #### Test + ```make -f Makefile.test daf-kylo``` +```make -f Makefile.test build-kylo``` + +### Login to nexus repository +``` +docker login nexus.daf.teamdigitale.it +``` +### Build Docker images of the components +Once this is completed, you can build every image (production and test environment), by typing the following comands: + + + #### Production +``` +make activemq +``` +``` +make mysql +``` +``` +make kylo-services +``` +``` +make kylo-ui +``` +``` +make nifi +``` + #### Test +``` +make -f Makefile.test activemq +``` +``` +make -f Makefile.test mysql +``` +``` +make -f Makefile.test kylo-services +``` +``` +make -f Makefile.test kylo-ui +``` +``` +make -f Makefile.test nifi +``` + +### Push Docker images to local artifactory repository +Please ensure previously configuration of docker client as well as correct tagging the image has been performed. 'How to' can be found in: + [TeamDigitale onboarding 'Setup Docker '](https://docs.google.com/document/d/1KqeaZ2yj7rofslqzklYTCLb3AxPnV1mzOgSXOuTHTyw/edit?ts=59faf23f&pli=1#heading=h.ubxuumcef218) + [TeamDigitale onboarding 'Push Docker Image'](https://docs.google.com/document/d/1KqeaZ2yj7rofslqzklYTCLb3AxPnV1mzOgSXOuTHTyw/edit?ts=59faf23f&pli=1#heading=h.47zm3aqq5wip) + +After config and proper tagging has been done, push can be performed typing: +`docker push [repositoryurl:repositoryport/artifact:version]` . For instance ([namespace] is optional): + + - #### Production + ``` ./nexus_push.sh prod [namespace] ``` + + - #### Test + ``` ./nexus_push.sh test [namespace] ``` + + +### Deploy components in kubernetes cluster +Please ensure previously configuration of kubectl has been done. 'How to' can be found in: + -[TeamDigitale onboarding , 'Setup Kubernetes'](https://docs.google.com/document/d/1KqeaZ2yj7rofslqzklYTCLb3AxPnV1mzOgSXOuTHTyw/edit?ts=59faf23f&pli=1#heading=h.vvi8emze7m35) + +After configuration is completed, **pod deployment** into kubernetes cluster can be performed typing `./playbook.sh [component]` . As instance, ``` ./playbook.sh prod activemq [namespace] ```. +**Pod deletion** can be performed running the command `./cleanup.sh [environment] [component]`. As instance, ``` ./cleanup.sh prod activemq [namespace] ```. These instructions are valid in production environment, but to perform the same things in test environment, just replace `prod` with `test`. + + +### Mysql Configuration +By default the kylo database is not created in mysql container, so you have to create it. + +### Ldap Configuration +To configure Ldap authentication: + Edit the config-maps kylo-services.yaml & kylo-ui.yaml as follows: + + + + - `config-map/kylo-services.yaml` shoud be: + + - ##### Production + ```security.auth.ldap.server.uri=ldap://idm.daf.gov.it:389/cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` ```security.auth.ldap.server.authDn=uid=admin,cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` + ```security.auth.ldap.server.password=xxxxx``` + + - ##### Test + ```security.auth.ldap.server.uri=ldap://idm.teamdigitale.test:389/cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` +```security.auth.ldap.server.authDn=uid=application,cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` ``` security.auth.ldap.server.password=xxxxx``` + - ```config-map/kylo-ui.yaml``` + - ##### Production + ```security.auth.ldap.server.uri=ldap://idm.daf.gov.it:389/cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` +```security.auth.ldap.server.authDn=uid=admin,cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` +```security.auth.ldap.server.password=xxxxxx``` + - ##### Test + ```security.auth.ldap.server.uri=ldap://idm.teamdigitale.test:389/cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` +```security.auth.ldap.server.authDn=uid=application,cn=users,cn=accounts,dc=daf,dc=gov,dc=it``` +```security.auth.ldap.server.password=xxxxxx``` + + +Once these two changes are done, first delete the pods, by typing: +``` +kubectl delete -f config-map/kylo-services.yaml +kubectl delete -f config-map/kylo-ui.yaml +``` +then redeploy as follows: +``` +kubectl apply -f config-map/kylo-services.yaml +kubectl apply -f config-map/kylo-ui.yaml +``` + +In the above example, it is not take in account the [namespace], so go on idm.teamdigitale.test and create an user such as *dladmin* with a password *password*. + +Now, you are able to login into kylo ui! + +As pointed out above, once this is done *ldap login* will be substituted by *default login* , this will allow to log in with default user `dladmin/thinkbig`. This has to be done to create users with the same name that those exist in ldap in order to grant them permissions (same functionality but for groups [is currently being fixed by R&D](https://kylo-io.atlassian.net/browse/KYLO-496)) . Once user(s) or group(s) is(are) created, change back `config-map/kylo-services.yaml` and `config-map/kylo-ui.yaml` and redeploy again. Ldap is now ready to start. + +### Bootstrap note +When kylo starts for the first time it need liquibase for creating Kylo DB, make sure that in the application.properties in kylo-service's config map: + +``` +liquibase.enabled=true +``` + + + +## Custom Processors + +[Here](./nifi/extensions/processors/Readme.md) you can find additional information about custom processors created for the [DAF](https://teamdigitale.governo.it/it/projects/daf.htm). + +## Further resources +- [PDND open data pipeline documentation](./README_opendata.md) +- [How to view logs](./README_logs.md) +- [How to register a new Kylo template](./README_kylo_template.md) + +## How to contribute + +Contributions are welcome. Feel free to [open issues](./issues) and submit a [pull request](./pulls) at any time, but please read [our handbook](https://github.com/teamdigitale/daf-handbook) first. + +## License + +Copyright (c) 2019 Presidenza del Consiglio dei Ministri + +This program is a free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. + +This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License along with this program. If not, see . \ No newline at end of file diff --git a/README_opendata.md b/README_opendata.md new file mode 100644 index 0000000..d34d492 --- /dev/null +++ b/README_opendata.md @@ -0,0 +1,105 @@ +**PDND open data pipeline documentation** +=================== + +> **Note:** + +>At the time this documentation is written, PDND open data pipeline is still in test environment, and is not deployed yet on production cluster. Some things need to be changed from test environment, such as file path from which tables that are created on Hadoop refers. + + + + +---------- + + +**Open data feed pipeline operation** +---------------- + +After the harvesting from ckan portal, data are transferred through a kafka producer into the pipeline , in order to start ETL process that ends with the creation of the table on Superset. + +Pipeline operation can be divided four parts: + + - File saving on HDFS and table creation on Hadoop; + - Superset request payload construction; + - Kerberos initialization and ACL's setting; + - Creation of the table on Superset. + + +#### **File saving on HDFS and table creation on Hadoop** + +Open data feed pipeline starts with data retrieval through a Kafka consumer, which sends all required data to create the table on Hadoop to a python script. This one connects to Impala engine using `impala_admin` user, and then it provides to retrieve all necessary metadata from ElasticSearch through a post request. Once all necessary data are retrieved, the script performs the creation of the tables whose each of them refers to the files persisting in the directory `/usr/daf/open_data`, different from the one to use in production environment, that is `/daf/opendata`. Files persisting in `/user/daf/open_data` are saved under the path `/organization_name/database_name/table_name` and named with a progressive number, such as `/usr/daf/open_data/parco_salento/AGRI_policy/attivitacommerciali/256.csv`. + +This way, the script performs the database creation (if it doesn't exist), with the same name as `database_name` but with lower case letters, as instance `agri_policy`, and the table creation (if it doesn't exist), with the same name as `table_name`, concatenated with the same progressive number the file is named, as instance `attivitacommerciali_256`. + + +#### **Superset request payload construction** + +The tables on Superset are created through a post request, whose payload is a json with the same schema of the following ⧸⧸one: +``` +{ + "schema":"agri__policy", + "dbName":"default_org-db", + "tableName":"attivitacommerciali_256" +} +``` +Field "schema" refers to the database name where tables are saved on Hadoop through Impala engine; field "dbName" refers to the database name where Superset tables are saved; field "tableName" refers to the name the table is saved on Hadoop through Impala engine. + +>**Note:** +>String used to create the table to view Superset must be no longer than one hundred characters, otherwise an error is generated and the table cannot be created. + +#### **Kerberos initialization and ACL's setting** +Before performing the post request to create the table that will be viewed on Superset, permission are set as stated in the following table: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
PATH_HDFSOWNER:GROUPPERMISSIONSDEFAULT ACLCUSTOM ACL
/dafdaf:dafrwx:rwx:--x-NO
/opendatadaf:dafrwx:rwx:--xhive:rwx
impala:rwx		NO
/[org_name]daf:[org_name]rwx:rwx:--xhive:rwx
impala:rwx		NO
/[domain]__[subdomain]daf:[org_name]rwx:rwx:--xhive:rwx
impala:rwx		NO
/[dataset_name][default user]:
[default user group]		rwx:rwx:---hive:rwx
impala:rwx		open_data_group:rwx
/[files].csv[default user]:
[default user group]		rwx:rwx:---hive:rwx
impala:rwx		open_data_group:rwx
+ +#### **Creation of the table on Superset** +This is the last step and involves the performing of the post request, whose aim is to create the table to view on Superset. \ No newline at end of file diff --git a/nifi/opendata-templates/delete_opendata.xml b/nifi/opendata-templates/delete_opendata.xml index 8039c46..c3318d0 100644 --- a/nifi/opendata-templates/delete_opendata.xml +++ b/nifi/opendata-templates/delete_opendata.xml @@ -1,6 +1,6 @@