feat: test document chunking & embedding strategies for LLM context

[iamanishsrivastava]

Experiment with different document chunking and embedding strategies to improve LLM’s understanding of context. Evaluate the impact of different chunking techniques (e.g., sentence-level vs. sliding windows) on model output.

• Scope:

  • Test chunking strategies: sentence-level, sliding window, paragraph-based.
  • Experiment with embedding methods: word2vec, GloVe, etc.
  • Assess the impact on the accuracy and relevance of output.

• Outcome:

  • Results of chunking and embedding experiments.
  • Recommendations for the most effective strategy.