This project was realised with the purpose of finding the underlying technologies that were used in websites mentioned in a parquet file provided by the user.
It analyzes their HTTP headers, meta tags and outputs a detailed CSV containing the detected technologies.
Import the project to your pc with git clone
To run the program, we have to create a virtual environment in the project folder with the command python -m venv <project_name>.
Then, we will activate the virtual env by running the command source <project_name>/bin/activate.
After that, we will install all the packages that are required by the program, as follows:
pip install pandas pyarrow requests beautifulsoup4 python-Wappalyzer
Finally, we run by typing : python3 main.py
The project was realized using Python.
The network requests were handled using the requests package. Requests are sent to webpages, and after that we create a WebPage object(Wappalyzer package). The technologies that are used will be mentioned afterwards in the results.csv file.
The extraction is further proved by analysing the headers of the webpage.
File extraction is done using pandas package, and concurrent for parallel thread computation.
Some websites block HTTP requests, resulting in timeouts or error 403). Also the requests package only downloads the static HTML.
The solution would be either using specialized APIs like ScraperAPI or ZenRows.
A single local script cannot handle millions of domains efficiently. I would re-architect this into a distributed system:
Using a database system instead of a CSV file like firebase
DNS caching
Push all domains in a queue and have many machines working in parallel
Human in the loop - an engineer can flag anomalies
Market monitoring - identifying frameworks that gain market share
Automating anomalies - put a flag on distinct technologies