Refactor duplicate detection logic for better maintainability#24

saurabhsharma2u merged 2 commits into

mainCrawlith/crawlith:mainfrom

refactor-duplicate-detection-8265135193601837981Crawlith/crawlith:refactor-duplicate-detection-8265135193601837981Copy head branch name to clipboard