🟡 Yellow Journalism Detector

An asynchronous HTTP API built with Python to evaluate the "jaundice rate" (sensationalism and clickbait density) of online news articles. Leveraging aiohttp and anyio, the service processes batches of URLs concurrently without blocking the main event loop and caches successfully processed URLs in Redis. It strips complex HTML structures down to meaningful text, isolates heavy CPU-bound morphological analysis, and scores emotional keyword density against dynamically loaded charged words dictionaries.

Currently, only one news site is supported: INOSMI.RU. A dedicated adapter has been developed for it, capable of extracting the article text from the surrounding HTML markup. New adapters will be required for other news sites, and all of them will be located in the adapters directory. The code for the INOSMI.RU site is also placed there: adapters/inosmi_ru.py.

📌 Table of Contents

• ⚙️ Tech Stack
• 📁 Project Structure
• 🛠️ Installation and Setup
• 🚀 Quick Start Guide
  • Development Server Launch
• 🔍 Inspecting Redis Data via Docker

⚙️ Tech Stack

• Operating System: Linux, macOS, or Windows
• Language: Python 3.11+
• Database: Redis (via Docker)
• Configuration: pydantic-settings & pydantic
• Async HTTP Framework: aiohttp
• Asynchronous Framework: asyncio & anyio
• HTML Parser: BeautifulSoup4
• Morphological Analysis: pymorphy3
• Unit Testing: pytest
• Containerization & Orchestration: Docker & Docker Compose

📁 Project Structure

.
├── adapters/
│   ├── __init__.py            # Registry mapping domains to dedicated sanitizers
│   ├── exceptions.py          # Centralized error declarations
│   ├── html_tools.py          # Core BeautifulSoup utilities to prune tags and attributes
│   └── inosmi_ru.py           # Specialized scraper logic tailored for inosmi.ru
├── tests/                     # Decoupled test suite powered by pytest
├── logs/                      # Dynamically generated application logs folder
├── charged_words/             # Charged words dictionaries
├── config.py                  # Central application settings mapper
├── database.py                # Redis connection pool setup and CRUD operations
├── logging_config.py          # Non-blocking async queue logger
├── text_tools.py              # Word splitting engine and jaundice formula calculation
├── analyzer.py                # The core analytical engine
├── server.py                  # The main entry point of the web application
├── .env.example               # Example of environment variable configuration
├── .editorconfig              # System-wide formatting and text indentation guidelines
├── Dockerfile.dev             # Python application container blueprint
├── docker-compose-dev.yaml    # Docker services orchestration
└── requirements.txt           # Python dependencies

🛠️ Installation and Setup

Basic Setup:

1. Clone the repository:

git clone https://github.com/...
cd project-directory

2. Set up a virtual environment:

python -m venv venv
venv\Scripts\activate # on Windows
source venv/bin/activate # on Linux / macOS

3. Install Python dependencies:

pip install -r requirements.txt

4. Configure environment variables:

Create a .env file in the root directory based on .env.example:

# Redis
REDIS_HOST=redis  # or '127.0.0.1' if you don't plan to run BOTH services via Docker
REDIS_PORT=6379
REDIS_PASSWORD=your_redis_password

# Server
SERVER_HOST=0.0.0.0  # or '127.0.0.1' if you don't plan to run BOTH services via Docker
SERVER_PORT=8080

# Logging
LOG_LEVEL=INFO

# Detector
CHARGED_WORDS_DIR=charged_words

🚀 Quick Start Guide

Development Server Launch

1. Run the test suite.

Verify that everything is set up perfectly by running the localized tests. pytest is used for testing; test coverage extends to code segments that are difficult to debug: text_tools.py, analyzer.py and adapters. Commands to run the tests:

python -m pytest tests/adapters/test_inosmi_ru.py -v
python -m pytest tests/test_text_tools.py -v
python -m pytest tests/test_analyzer.py -v

2. Build and run the app:

docker compose -f docker-compose-dev.yaml up --build

The server will start on port 8080; to verify it is running, navigate to http://localhost:8080/ in your browser.

3. Send a request.

To analyze an article, simply send a request in the following format, listing the URLs separated by commas:

http://localhost:8080/?urls=https://domain.com,https://domain.com

In response, you will receive data in JSON format:

[
 {"url": "http://example.com", "status": "PARSING_ERROR", "rate": null, "words_count": null},
 {"url": "https://inosmi.ru/...", "status": "OK", "rate": 0.39, "words_count": 1452},
 {"url": "https://inosmi.ru/...", "status": "OK", "rate": 0.54, "words_count": 1727}
]

4. Useful Docker commands:

• docker compose -f docker-compose-dev.yaml down - Stop and remove all containers and networks defined in the dev configuration.
• docker compose -f docker-compose-dev.yaml up -d - Start all containers in detached background mode.
• docker compose -f docker-compose-dev.yaml restart - Restart all containers.
• docker compose -f docker-compose-dev.yaml logs -f server - View and follow real-time logs from the web server.

🔍 Inspecting Redis Data via Docker

1. Access the Redis container CLI.

Run the following command to open the interactive Redis CLI inside your running container:

docker compose -f docker-compose-dev.yaml exec redis redis-cli
127.0.0.1:6379> AUTH <your_redis_password>

Or you can log in via the CLI:

docker compose -f docker-compose-dev.yaml exec redis redis-cli -a <your_redis_password>

⚠️ Use this command only for local development.

2. Useful Redis commands.

Once inside the CLI, you can use these basic commands to inspect the data:

• KEYS * - List all keys currently stored in the database.
• GET <key> - View the content of a specific text key.
• TTL <key> - Check the remaining Time-To-Live for temporary keys.
• DEL <key> or DEL <key1> <key2> - Remove specific keys from the database.
• UNLINK <huge_key> - Asynchronously delete huge keys without blocking the main thread.
• FLUSHDB - Clear all data from current database.
• FLUSHALL - Clear all data from all databases.

3. Exit the CLI.

Type exit or press Ctrl + C to return to your local terminal.