Deep-Learning-Prediction-Engine-on-Big-Data

Motivation:

To mitigate the housing business in a specific region, the proposed model helps to predict the best feasible pricing(real-estate).

Requirements: Python3, TensorFlow, Keras, Hadoop 3.2.1 or above.

Project pipeline

1. Data loading into Hadoop Distributed File System(HDFS).
2. The integration of python environment with Hadoop ecosystem.
3. Data pre-processing (Filling the NA value, looking for inconsistency, data binning, outlier detection and analysis, data transformation and normalization).
4. Feature engineering (creation of new features: dummies variables, interactions between variables and so on).
5. Deep Learning modeling (Keras-TensorFlow).
6. Load the results into Hadoop Distributed File System(HDFS).

Why Big Data for Official Statistics:(BDOS)

1. Big Data can keep official statistics relevant–because the private sector moves fast.
2. Big Data is part of modernization of statistical systems – new production processes and partnerships.
3. Big Data is needed for agile statistics –for emergency issues.

Why Deep Learning for BDOS:

1. Deep Learning algorithms extract high-level, complex abstractions as data representations through a hierarchical learning process.
2. A key benefit of Deep Learning is the analysis and learning of massive amounts of unsupervised data, making it a valuable tool for Big Data Analytics where raw data is largely unlabeled and un-categorized.
3. In the present study, how Deep Learning can be utilized for addressing some important problems in Big Data Analytics, including extracting complex patterns from massive volumes of data, semantic indexing, data tagging, fast information retrieval, simplifying discriminative tasks, streaming data, dealing with the high dimensionality of data, scalability of models, and distributed and parallel computing.

Future Enhancement

1. Blockwise reading of data(Chunkwise)
2. Direct pre-processing/engineering and modelling on HDFS.