An advanced AI-powered web scraper that extracts content from a given URL and answers user queries based on the scraped data. Built using Streamlit for the UI, LangChain_Ollama for parsing and understanding the content, and Selenium for web scraping.
- URL-Based Scraping – Enter a website URL, and the scraper extracts relevant content.
- AI-Powered Answers – Uses LangChain_Ollama to process the scraped data and generate intelligent responses to user queries.
- Interactive UI – Built with Streamlit for a user-friendly experience.
- Dynamic Web Scraping – Leverages Selenium to extract content from static and JavaScript-rendered websites.
- Efficient & Scalable – Optimized for fast and responsible web scraping.
- Frontend/UI: Streamlit
- Web Scraping: Selenium
- AI & Parsing: LangChain_Ollama
- Backend: Python
git clone https://github.com/TGSK07/AI_WebScraper.git
cd AI_WebScraperEnsure you have Python installed (recommended Python 3.8+), then run:
pip install -r requirements.txt- Download the WebDriver compatible with your browser (ChromeDriver recommended).
- Place it in your system's PATH or specify its location in the script.
streamlit run app.py- Input URL:
https://example.com - Scraped Content: Extracts text and relevant data.
- User Query: "What is the main topic of this page?"
- AI Response: "The page discusses example-based learning methods."
- Improve content extraction for more complex page structures.
- Enhance AI responses with more advanced NLP models.
- Implement caching to speed up repeated queries.
- Support for scraping multiple pages at once.
Contributions are welcome! Feel free to open issues or submit pull requests.
Developed by Sonu Kumar 🚀