Moved URL's aren't crawled

[n3wjack]

A url that returns a 301 (moved permanently) doesn't get crawled afterwards.
I seem to be having a bunch of those where url's not ending in a slash give a 301 to the page containing the trailing slash.

I'm guessing this is not intended.
If not, let me know and I have some code ready that fixes this which needs some cleanup and perhaps some unit tests.

[leglock]

This is also happening with 302 responses.

[tdwright]

Hi @n3wjack.

Sorry about the delay in getting back to you - didn't spot this issue.

If you've got code that's "nearly there", perhaps we could start a feature branch and work on it there?

Let me know and I'll create a branch for you.

Tom