增加错误处理：当“访问频次异常”时自动停止继续爬取#61
Open
onkej wants to merge 2 commits into
Open
Conversation
…rch results. Also rename output dir and add empty lines for readability.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
博主你好，感谢分享！
我尝试使用关键词搜索功能的时候偶尔会有
msg: 访问频次异常，请勿频繁操作或重启试试的 log 信息，从这时候开始程序实际上已经无法访问和爬取但仍会继续做无谓的重复尝试直到最后一个 URL……所以我在main.py加了无法访问之后自动终止爬取的功能以及相应的 CLI 提醒。希望对项目有所帮助！（PS：其他文件的更改不重要可以全部忽略）