shoppalyzer-tool

The code to analyze archived pages from Common Crawl assisted by live crawls.

Setting up the project on cloud server

Cloning the project

Open terminal and log in via ssh -i keyfile root@188.245......

Then run the following commands to get GitHub connected
sudo apt update and apt install gh.

After some authentication magic via tokens from GitHub and gh auth login we can clone the project.

gh repo clone Chaosheld/shoppalyzer-tool

Installing requirements

Move to project: cd shoppalyzer-tool

Rename the distribution file for the credentials and fill accordingly using nano credentials.py.dist.

Then setting up venv and install the requirements:

apt install python3.12-venv

python3 -m venv venv

source venv/bin/activate

pip install -r requirements.txt

Start the first data job

First make sure to have a recent version: git pull

Now create folder "files > input" and provide a simple csv containing domains to be queried and crawled.