This is the web crawler I coded and my team used for Information Retrieval's class final project at Concordia University in Canada. The code was written in Python 2 and will not work on Python 3. This was due to the fact that the computers in Concordia's labs run Python 2 and I wanted the code to work on those machines. Having Python 2 installed, run the crawler by typing in "python main.py" into your terminal while in the source code's directory. You'll have to specify what root webpage(s) will be used for the crawling as well as how many web pages the crawler will crawl until it terminates. If you want multiple starting websites, pass a list of links to the first data input. Typing in 0 will have the crawler run "forever", or until your computer runs out of disk space to hold the text documents. Note, the given links must be absolute, e.g. http://www.google.com and not google.com.
I am not responsible for any legal issues you may encounter when using the crawler.