Collecty

An Intelligent, Agentic Web Scraping Framework for Automated Schema Discovery and Structural Analysis.

About

Collecty is a Python-based framework designed to intelligently scrape websites. Its core innovation lies in its "agentic" approach, meaning it can autonomously discover and analyze the structure of the data it encounters, rather than just extracting based on pre-defined, static rules. This makes it particularly powerful for projects requiring automated schema discovery from web pages.

Key Features

• Intelligent Scraping: Leverages logic to navigate and extract data from complex web structures.
• Automated Schema Discovery: Automatically identifies and maps the underlying data schema of web content.
• Structural Analysis: Analyzes the layout and patterns within HTML to inform the extraction process.
• Python-Based: Built entirely with Python, ensuring ease of use and integration into existing data science or automation workflows.

Getting Started

Follow these instructions to get a copy of the project up and running on your local machine for development and testing purposes.

Prerequisites

• Python 3 (The project is 100% Python)
• pip (Python package installer)
• A Chrome browser (required for the included chromedriver.exe)

Installation

1. Clone the repository

   git clone https://github.com/loueylahwel/Collecty.git
   cd Collecty

2. Install dependencies The project dependencies are listed in requirements.txt. Install them using pip:

   pip install -r requirements.txt

3. ChromeDriver The repository includes a chromedriver.exe file. Ensure this driver is compatible with your installed version of the Chrome browser. If you encounter issues, you may need to download the correct version from the official ChromeDriver site.

Usage

The project is structured into several core Python scripts. While the exact user interface is not detailed in the repository, the primary logic is contained within these files:

• main.py: Likely the main entry point for running the framework.
• scrape.py: Handles the core web scraping functionality.
• parse.py: Responsible for parsing scraped HTML and performing schema discovery.

A typical workflow might involve running main.py with a target URL. Refer to the comments within the scripts for more specific details on their functions and parameters.

Contributing

This is an active project. Feel free to fork the repository and submit pull requests.

License

This project does not currently specify a license. You may need to contact the repository owner for permission to use or distribute this code.

Contact

For questions or feedback, please open an issue on the GitHub repository.