My workplaces current CRM is poorly managed and myself and some coworkers wanted to create our own dashboards so that we could understand our data from working with clients and stores much better. So I set off on building the data pipeline myself.
Each batch is then sent to the designated Amazon S3 bucket to update the preexisting table.
I am webscraping certain tables that i choose from the ShareCRM that my workplace uses. I am extracting the data using selenium in python to automate this scraping.
Then I wrote python code that transformed the data response from json to . I deployed and scheduled that code using Apache Airflow running on an Amazon EC2 Ubuntu machine.
After airflow runs successfully, we will have a new updated object in our S3 bucket. Now we can take this data and load it into any data warehouse you enjoy!