Feat narrative analysis indicators command

[ftello0403]

Overview

This Pull Request adds a new process to identify the virality status from a narrative based on the increasing statistics related to their videos. The goal is to make this an automated process by executing it with a cronjob every day.

1. run_narrative_analysis_indicators_pipeline command

To achieve this, the command line run_narrative_analysis_indicators_pipeline was created to run a calculation process divided on three phases:

1. Calculate per-narrative virality scores (engagement, reach, velocity) in batches.
2. Compute composite virality and acceleration rate indicators for the day.
3. Classify and persist alert levels for every narrative.

This iterates all over the active prevalent narratives, which are the ones that their videos were updated on the last 24 hours, and persists each calculation on the database to store the results for future analysis.

The command allows the following parameters:

• batch-size (optional, default=100): Number of narratives to process per batch.
• hours (optional, default=24): Time window in hours for narrative analysis indicators calculation.

2. GET /api/narratives/{narrative_id:uuid}/indicators endpoint

The calculated results and the alert level classification will be available on the narrative's detail view on the frontend side by consuming a new endpoint GET /api/narratives/{narrative_id:uuid}/indicators.

This endpoint accepts a date parameter to obtain the results from a specific day. By default, it returns the last analysis indicators stored on the database; this means that if a narrative has not received an update on the last 24 hours, the dashboard will render the last stored result.

[JamesMcMinn]

Good fix.

[JamesMcMinn]

I think this mostly looks good, but some of the more complicated queries make the assumption that video_stats data gets updated for every video at least once per day, which isn't the case.

Once these are fixed I think it's good to go.

[JamesMcMinn]

There's no guarantee that every video in a narrative gets scraped every day, which I think breaks this query and a few others added in this PR.

We scrape the last 200 videos per channel / keyword approximately every hour, but once they drop out of the 200 most recent videos, we move to a dynamic approach where we scrape each video based on the expected new views. This means that an older, low view count video could only be scraped once every few days (or even weeks) since we're limited by how many videos we can re-scrape data for.

Additionally, if the data hasn't changed since the last scrape, we don't update it, so a video that's not getting any new views wouldn't be included in the count for a day.

I think this query, and any other query that uses dates as a filter or join on the video_stats table needs an update to use the last value for a video, rather than assuming there will be an entry for every date.

Claude is suggesting the following which I think would correct for this:

WITH narrative_videos AS (
    SELECT DISTINCT v.id
    FROM videos v
    JOIN video_claims vc ON v.id = vc.video_id
    JOIN claim_narratives cn ON vc.id = cn.claim_id
    WHERE cn.narrative_id = %(narrative_id)s
),
daily_latest AS (
    -- Latest snapshot per (video, day) — unchanged.
    SELECT DISTINCT ON (vs.video_id, vs.recorded_at::date)
        vs.video_id,
        vs.recorded_at::date AS day,
        vs.views, vs.likes, vs.comments
    FROM video_stats vs
    JOIN narrative_videos nv ON vs.video_id = nv.id
    ORDER BY vs.video_id, vs.recorded_at::date, vs.recorded_at DESC
),
change_days AS (
    -- The days on which the narrative's cumulative state actually changed.
    SELECT DISTINCT day FROM daily_latest
),
carried AS (
    -- For each change-day, carry forward each video's latest snapshot
    -- recorded on or before that day. This is the real end-of-day state:
    -- a video keeps contributing its last-known counts on days when it
    -- wasn't re-snapshotted.
    SELECT cd.day, latest.video_id, latest.views, latest.likes, latest.comments
    FROM change_days cd
    JOIN LATERAL (
        SELECT DISTINCT ON (dl.video_id)
            dl.video_id, dl.views, dl.likes, dl.comments
        FROM daily_latest dl
        WHERE dl.day <= cd.day
        ORDER BY dl.video_id, dl.day DESC
    ) latest ON TRUE
),
per_day AS (
    SELECT
        day,
        COUNT(DISTINCT video_id) AS videos_with_stats,
        COALESCE(SUM(views), 0)    AS cum_views,
        COALESCE(SUM(likes), 0)    AS cum_likes,
        COALESCE(SUM(comments), 0) AS cum_comments
    FROM carried
    GROUP BY day
)
SELECT
    day AS date,
    videos_with_stats AS video_count,
    GREATEST(cum_views    - LAG(cum_views,    1, 0::bigint) OVER (ORDER BY day), 0) AS views,
    GREATEST(cum_likes    - LAG(cum_likes,    1, 0::bigint) OVER (ORDER BY day), 0) AS likes,
    GREATEST(cum_comments - LAG(cum_comments, 1, 0::bigint) OVER (ORDER BY day), 0) AS comments,
    videos_with_stats AS cumulative_video_count,
    cum_views    AS cumulative_views,
    cum_likes    AS cumulative_likes,
    cum_comments AS cumulative_comments
FROM per_day
ORDER BY day

basically, new change_days and carried CTEs to correct for "missing" days. Otherwise I think the query is fine.

[JamesMcMinn]

This needs a fix similar to get_narrative_stats.

[MiguelAzores]

@JamesMcMinn @Naroh091 That's right, I've already made the changes to the three queries to account for the fact that we won't always have daily video statistics. I've also included a test that simulates this situation.
I've made the changes to the following functions:

• get_narrative_stats
• get_bulk_narrative_stats_comparison
• get_narrative_stats_delta_for_period