NTLM Auth with IIS from Windows Server 2025

[onixterry]

I'm trying to crawl a web site that uses IIS on Windows Server 2016. We have V2 of the HTTP Collector running without any issues.

We have converted the configuration file over to version 3 and tested it on a public site and it works fine. When configuring it for the Intranet site, it is receiving a 401 error on the home page and failing to authenticate.

We are using the same username, password and domain information from the V2 implementation.

We have tried the username in these formats:

username
DOMAIN\username
username@domain.fqdn.example.com

We've tried the "preemptive" setting as both true and false.

In all cases, we see the same results, it never authenticates.

The V3 version is running on a different workstation. We have used curl -I --ntlm -u user:passs and confirmed that we see the intiital 401 error followed by a successfuly NTLM negotiation and a 200 response.

I tested the configuration against a public NTLM endpoint (https://authenticationtest.com/HTTPAuth/), which worked fine if "preemptive" was set to true but not false.

Attached are the logs from V2 and V3 as well as the configuration file for V3.

Any suggestions as to what might be happening here?

iis-ntlm-authentication-issue.zip

[essiembre]

Silly question maybe: Have you tried with V2 on the d same "different workstation" as V3?

[onixterry]

We just tried that. We had to install a newer version of Java on the old workstation, since it was using Java 8. The new HTTP Collector has the same issue from the old workstation. It receives the 401 and cannot crawl any pages.

[onixterry]

Just a quick note that I realized I mentioned we were using Windows Server 2016 IIS v10 where the Norconex Collector that not working. The machine running Norconex is running Windows Server 2025 and we tried it on the old machine that was running Windows Server 2019. Again, in all cases, IIS is v10 hosted on Windows Server 2016.

Maybe there is a different version/flavor of NTLM in use that you hadn't come across before? I see there is an NTLM v2 that exists, not sure if your code supports it and/or was tested with it.

[akshaysalokheonix]

Any Updates on last message, just reminder behalf of @onixterry
We stuck on this issue so hope will get that soon
Thanks

[essiembre]

Thanks for the detailed logs and config.

The root cause: GenericHttpFetcher was silently skipping all authentication setup (NTLM, Basic, or Digest) whenever the <host> element under <authentication> wasn't explicitly filled in with a host name and port. Your config had <host/> (present but empty), which is why the crawler got a 401 and gave up immediately — it never actually attempted the NTLM handshake at all. That matches your logs exactly: a single 401 with no Type1/Type2/Type3 negotiation.

This wasn't a requirement in v2, and it isn't meant to be one in v3 either — the Javadoc on that setting has always said an unset host means "apply to any host." The v3 just failed to honor that when wiring up the credentials provider.

I've fixed this so that leaving <host> unset now correctly applies credentials to any host, as documented, instead of disabling authentication.

Separately: preemptive="true" currently sends a hardcoded HTTP Basic auth header regardless of the configured <method>, which is why it "worked" against authenticationtest.com (which happens to accept Basic too) but wouldn't help against an IIS server restricted to NTLM. That's a separate quirk we'll track independently — for NTLM, leave preemptive as false.

A new snapshot of the Web Crawler was made. Give it a try.