Skip to content

fixed add-docs for documents and website#73

Open
DonTizi wants to merge 1 commit into
mainfrom
bugfix/web-crawler
Open

fixed add-docs for documents and website#73
DonTizi wants to merge 1 commit into
mainfrom
bugfix/web-crawler

Conversation

@DonTizi

@DonTizi DonTizi commented Mar 23, 2025

Copy link
Copy Markdown
Owner

This pull request includes several enhancements and new features to the cmd/wizard.go file, focusing on improving the configuration and setup process for web crawling and directory watching. Additionally, there is a bug fix in internal/crawler/crawler.go to prevent a potential panic.

Enhancements and new features:

  • cmd/wizard.go: Added options for single URL crawling and specific URLs list. This includes prompts to configure these options and display the crawling configuration. [1] [2] [3]
  • cmd/wizard.go: Implemented directory watching setup for local folder RAGs and website watching setup for web crawler RAGs, including prompts for user confirmation and interval configuration.

Bug fix:

  • internal/crawler/crawler.go: Fixed an issue where attempting to calculate URL depth could cause a panic if the URL length was shorter than the base URL length.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant