This is a project completed in three weeks during the Insight Data Engineering program in Seattle, Fall 2019. Insight helps recent grads learn the latest big data technologies by building a platform to handle large datasets.
Project slides
Dash frontend (screencast)
When earthquakes occur their locations are calculated very quickly in order to issue real-time alerts. More accurate locations can be determined afterwards and used to infer fault locations and make hazard assessments. But how can new location techniques be tested efficiently on a large, historical dataset? I have provided a pipeline to do this using distributed processing, which will allow researchers and policy makers to make more informed hazard determinations.
I used earthquake arrival time data from the Pacific Northwest Seismic Network, PNSN. In order to test Spark's functionality I wrote a script to augment the dataset to an arbitrarily large size, in this case ~80 GB.
Amazon AWS S3: This is a common storage option for long-term data of an arbitrarily large size.
Apache Spark: Spark is an analytics engine to work with large datasets in a distributed manner. I used the python version, PySpark.
MySQL: The summary results were stored in a MySQL relational database.
Dash: Summary results can be accessed and explored by the user through a Dash frontend.
Challenge #1: Implementing Pandas user-defined functions (UDFs)
- Pandas UDFs allow the user to perform transformations on the dataset using Pandas dataframes, which will be returned into the overall Spark dataframes.
- In my case I split up the earthquake arrival times for each earthquake and operate on them separately with
df.groupby('Event_id').apply(<pandas_function>).pandas_functionis a separate function that converts that portion of the Spark dataframe into a pandas dataframe using Apache Arrow, operates on it, then returns a new dataframe. - This would allow researchers to write and test code using only python/pandas, without worrying about Spark. The tradeoff is that this could take more time than a function implemented entirely within Spark. Testing a naive location algorithm (average recording station location) on a smaller dataset showed ~4x faster performance with Spark native functions than with a Pandas UDF.
Challenge #2: Spark tuning
- Spark uses multiple nodes to perform distributed computation. The number of nodes, the number of executors, the number of cores per executor, and the RAM per executor, can all be adjusted, among many other parameters.
- I tested multiple configurations to see how the pipeline could be made more efficient. The difference between the worst and best configurations (with the same total number of cores, worker nodes) was about ~60% in speed.
- Dash: Contains the frontend code to display the website that summarizes the new location data.
- Parsing: Contains the scripts to parse the initial dataset into json format and upload it to an S3 bucket.
- Processing: Contains the scripts to read the data from S3, calculate new earthquake locations, and write to a MySQL database, all using Spark.