Flight Stat is an Alexa app that speaks with the user, gathers flight information and predicts flight delays. I presented this project at re:Invent 2017.
Input (X): [Origin City, Destination City, Flight Date, Airline]
Output (Y): Flight Delay prediction in seconds
The dataset came from Bureau of Transportation Statistics. It contains all domestic flight which is about 6,000,000 datapoints per year in CSV format.
To better visualize the data and be able to query the dataset, an ETL job imported the data into an SQLite database. Then, the data processed and normalized into the HDF5 format to be used for ML models.
Fig 1. Flight Stat Data Pipeline
All ETL jobs are implemented in Python.
I created and evaluated three ML models for this project:
The first model is a simple linear regression model with all the inputs. I was able to quickly implement this model and train it. I used this as a baseline for other models in this project.
Fig 2. Flight Stat Regression Model
The model is implemented by AWS Lambda and Amazon Machine Learning.
The second model is a simple decision tree plus four different linear regression models. The decision tree will determine the season of the prediction (e.g. Spring) and then uses the corresponding linear model for prediction. Four different models were trained independently based on their appropriate seasonal datapoints.
Fig 3. Flight Stat Decision Tree + Regression Model
The model is implemented by AWS Lambda and Amazon Machine Learning.
The last model was a 5-layer MLP Neural Network that was trained on all the data.
Fig 4. Flight Stat MLP Model
The model is implemented in TensorFlow.
The dataset was sliced into 99% training and 1% testing. I trained the MLP model on one P3.16xlarge with Deep Learning AMI. The model is saved after each epoch.
The inference is done with a saved TensorFlow model in a single lambda function.
The final models were all deployed in a serverless architecture using AWS Lambda:
Fig 5. Flight Stat Deployment Architecture