evidence-fetcher is an interactive evidence-search and relevance-feedback experiment.
The current vertical slice fetches candidate web evidence from Brave Search, normalizes the provider response, removes duplicate URLs, applies a lightweight local relevance ranker, and exposes the results through a Python API and a small command-line interface.
The feedback-aware reranking is based on a Rocchio-style relevance feedback algorithm over TF-IDF vectors.
Because Interactive Information Retrieval can amplify the feedback you provide, it should be used with care. The goal is to condense useful evidence from search results, not to cherry-pick sources that only support a preferred conclusion. Reviewers should keep contrary, ambiguous, and low-confidence evidence visible when it matters to the question.
Install the core package:
pip install evidence-fetcherCreate a Brave Search API key and expose it as BRAVE_SEARCH_API_KEY:
export BRAVE_SEARCH_API_KEY="..."from evidence_fetcher import fetch_evidence
results = fetch_evidence("retrieval augmented generation evaluation")
for evidence in results:
print(evidence.title)
print(evidence.url)
print(evidence.snippet)fetch_evidence() returns typed Evidence objects with normalized title,
url, snippet, source, and rank fields.
Readable text output:
evidence-fetcher "retrieval augmented generation evaluation"JSON output:
evidence-fetcher "retrieval augmented generation evaluation" --format jsonUse --limit to control the maximum number of results.
Install the optional UI extra:
pip install "evidence-fetcher[ui]"Launch the Streamlit review app:
evidence-fetcher-uiThe UI uses the same Brave-backed retrieval, local reranking, relevance feedback, and explicit expansion workflow as the CLI.
Interactive relevance feedback:
evidence-fetcher "retrieval augmented generation evaluation" --interactiveInteractive mode performs the same initial search, then lets you mark displayed
results, request a local rerank, or explicitly expand retrieval with a refined
query. Ordinary text and JSON commands are unchanged and remain non-interactive.
Interactive mode is text-only; combining --interactive with --format json is
rejected.
Supported interactive commands:
more 1 3
less 4
irrelevant 5
unsure 2
show # or rerank
expand
help # or ?
done # or quit, q, exit
Result numbers are presentation-only positions in the currently displayed list.
Feedback is stored against each evidence URL, and after show the same number
may refer to a different result.
The interactive prompt shows the number of displayed results and accumulated
feedback items, for example [10 results, 2 feedback items] >.
show reranks only the current candidate set using accumulated feedback.
expand builds a deterministic lexical query from the original query and
feedback-marked evidence, performs one additional Brave retrieval, merges any
new URLs into the session, preserves accumulated feedback, and reranks the
expanded candidate set. Expansion is always user-triggered; feedback commands do
not automatically run new searches.
Example session:
$ evidence-fetcher "retrieval augmented generation evaluation" --interactive
Interactive relevance feedback. Type 'help' for commands.
1. Result title
https://example.com/page
Short snippet...
[10 results, 0 feedback items] > more 1
recorded more: 1 (1 feedback item total)
[10 results, 1 feedback item] > irrelevant 3
recorded irrelevant: 3 (2 feedback items total)
[10 results, 2 feedback items] > show
Updated ranking from accumulated feedback:
1. Result title
https://example.com/page
Short snippet...
[10 results, 2 feedback items] > expand
Searching with refined query:
retrieval augmented generation evaluation benchmark faithfulness
Added 4 new results.
Updated ranking:
1. Result title
https://example.com/page
Short snippet...
[14 results, 2 feedback items] > done
The public API is intentionally small: fetch_evidence() accepts a query and
returns normalized, deduplicated Evidence results. Provider-specific HTTP
requests, credential handling, and response parsing live behind an internal
Brave Search provider. The provider depends on a small HTTP client protocol, so
tests and future integrations can inject a mock client instead of using the
network.
Candidate retrieval and local relevance ranking are separate steps. Brave Search retrieves candidate evidence from the web; a replaceable local ranker then orders those candidates against the original query. The default ranker is a small standard-library TF-IDF implementation over evidence titles and snippets. This is the first foundation for Interactive Information Retrieval: the code now has a boundary where a workflow can collect user relevance feedback and rerank results.
Feedback-aware reranking is available through the Python/domain layer. A
SearchSession retains the original query, candidate evidence, current ranked
results, and accumulated RelevanceFeedback entries keyed by evidence URL. The
TF-IDF ranker applies a lightweight Rocchio-style query update: it keeps the
original query vector, moves it toward evidence marked MORE or RELEVANT,
moves it away from evidence marked LESS or IRRELEVANT, and records UNSURE
without changing ranking.
Feedback-informed retrieval expansion is separate from local reranking. The
default query refiner tokenizes the original query and feedback-marked evidence,
removes small stop words, scores candidate expansion terms lexically with TF-IDF,
avoids duplicating original query terms, suppresses terms associated mainly with
irrelevant evidence, and appends up to four useful terms. The refined query is
printed before each expand retrieval. Expansion does not persist a session
across invocations, run in the background, use an LLM, or send feedback anywhere
except the explicit refined Brave query.
Package-specific exceptions distinguish configuration errors, request failures, and malformed provider responses.
Only Brave Search is implemented in this slice. No persistence or additional search providers are included yet.
Interactive feedback can make evidence review faster, but it can also drift toward cherry-picking if the reviewer marks only convenient sources as useful. Treat reranking and expansion as prioritization aids. They do not replace balanced judgment, source quality checks, or deliberate review of evidence that complicates the initial hypothesis.
This project uses uv for environment and dependency management.
uv sync
uv run pytest
uv run ruff check .
uv run mypy src
uv build
uv run sphinx-build -b html docs/source docs/builtCommon shortcuts are available through make:
make check
make docs
make cleanDocumentation is built with Sphinx and Furo from docs/source/ into docs/built/. Read the Docs should build the documentation through its normal repository integration.
Releases are managed with Python Semantic Release and Conventional Commits. The GitHub Actions release workflow is intended to create the release commit, tag, GitHub release, wheel, sdist, and PyPI publication through trusted publishing.