Skip to content

The PostgreSQL knowledge base query cache is not working, resulting in a significant drop in EPS. #301

Description

@datasc31

Description

In WarpParse V0.23.4, runtime performance drops significantly after enabling PostgreSQL knowledge base enrichment logic.

The input data used in this test is a single repeated record. In theory, the PostgreSQL query result should be cached after the first lookup.
However, the observed EPS still drops significantly, which suggests that the PostgreSQL query cache may not be working as expected.

Evidence

Based on the runtime statistics:

Scenario EPS
Without PostgreSQL knowledge base lookup ~170,000
With PostgreSQL knowledge base lookup ~86,000

Because the input data is a single repeated record, this scenario should be highly cache-friendly.

Expected Result

When the input data is highly repetitive, PostgreSQL knowledge base lookup should hit the cache after the first query.

If the cache works correctly, EPS should not drop from around 170k to around 86k, which is close to a 50% performance degradation.

Actual Result

After enabling PostgreSQL knowledge base lookup, EPS drops from around 170,000 to around 86,000.

Version

  • WarpParse: V0.23.4
  • Related feature: PostgreSQL knowledge base enrichment / lookup logic

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions