Skip to content

README.md

Latest commit

 

History

History
65 lines (42 loc) · 1.23 KB

README.md

File metadata and controls

65 lines (42 loc) · 1.23 KB
Raw

WikiFind

WikiFind is a command-line search engine for Wikipedia XML dumps. It allows you to index large Wikipedia datasets and perform fast searches on them.

Prerequisites

Installation

Build the project:

go build -o wikifind ./cmd

Usage

Indexing

To index a Wikipedia XML dump:

./wikifind index <xml_file> <index_path>
  • <xml_file>: Path to the Wikipedia XML dump file
  • <index_path>: Directory where the index will be stored

Example:

./wikifind index enwiki-20231201-pages-articles.xml index/

Searching

To search the indexed data:

./wikifind search <index_path>
  • <index_path>: Directory containing the index

This will start an interactive search prompt. Enter your queries and get results.

Example:

./wikifind search index/
> apple
Found 5 results:
1. DocID: Apple (Score: 0.95)
...

Architecture

The project is organized into several packages:

  • cmd/: Main application entry point
  • indexer/: Indexing logic, including XML parsing, text processing, and inverted index creation
  • search/: Search engine implementation with compression and query processing