"Cannot fetch logs" error in 2.0.3 when trying to view streaming logs for service/container from job page

[rmatte]

Nomad version

2.0.3

Operating system and Environment details

Rocky 8.9

Issue

I had upgraded from version 2.0.2 to version 2.0.3 today in hopes of receiving a fix for #28062. The UI ended up being even more broken than it was before. Instead of just having the log flashing I was now receiving an error like the following...

Cannot fetch logs
The logs for this task are inaccessible. Check the condition of the node the allocation is on.

Then when I looked in the dev console, I could see messages like...

ec2dfe71ef90d13e6a9e8f65d10dd58d.js:3869 LOG FETCH: Couldn't connect to /v1/client/fs/logs/c7eb4502-820e-bb7a-5573-2abaa0e97cb4?follow=true&offset=50000&origin=end&task=rabbitmq&type=stdout

But if I visit that URL directly in my web browser it would work fine. I tried clearing my browser cache several times to no avail. Another thing I noticed is that if I navigated to Jobs -> clicked on one of the jobs -> clicked to view logs next to one of the containers I would get the "Cannot fetch logs" error. If I copied the URL out of the URL bar, opened a new tab, and just pasted that URL back in the log would load without erroring when the page loaded. For instance when I loaded...

http://lab-nomad-01/ui/jobs/my-service-job@rmatte-test-cluster?activeTask=c7eb4502-820e-bb7a-5573-2abaa0e97cb4-rabbitmq

It would load the logs fine. This issue reproduces when hitting the nomad UI directly without any kind of proxy in front of it.

I also noticed one other UI oddity, which was that from the Clients page I couldn't navigate to the Jobs page by clicking on the link, yet I could navigate to the Jobs page from any other page other than the Clients page. This issue only occurs for me in 2.0.3, it is not present in 2.0.2.

I ended up having to roll back to 2.0.2 because it's better to have flashing logs than no logs loading at all.

Another thing to note is that in both 2.0.2 and 2.0.3 if I click on the link to go to the Task Group that one of the services belongs to and then I view the logs for one of the containers/services from there it loads and doesn't flash. Not sure what the difference is between how the same log files are being displayed within the task group view vs how they are being displayed within the jobs views, but it is working better for me under task group for whatever reason, so I've been using that as a workaround.

This appears to be javascript related. If I visit the http://lab-nomad-01/v1/client/fs/logs/c7eb4502-820e-bb7a-5573-2abaa0e97cb4?follow=true&offset=50000&origin=end&task=rabbitmq&type=stdout URL directly I can see the initial data and the data streaming in via the browser dev tools. It also works if I change follow flag from true to false, then I just see the raw log output in the browser. It seems like something in the javascript code is erroring out when trying to handle the above URL in 2.0.3 and causing the log pull to fail when trying to view service/container logs from job pages. This problem doesn't appear to occur in 2.0.2, though as mentioned before, the logs flash there (#28062).

Reproduction steps

Install version 2.0.3 and try to view logs for a service/container under one of the jobs pages.

Expected Result

Logs should load without errors and without flashing.

Actual Result

In 2.0.2 the logs flash after loading (#28062) and in 2.0.3 they fail to load with a "Cannot fetch logs" error.

Job file (if appropriate)

Nomad Server logs (if appropriate)

Unfortunately there is nothing in the server logs or the client logs related to this issue.

Nomad Client logs (if appropriate)

Unfortunately there is nothing in the server logs or the client logs related to this issue.

[rmatte]

As an aside, I'd really like to see a config option added to be able to disable the log streaming completely and statically display whatever is in the log up to that point, maybe with a manual refresh button at the top or something. I don't need streaming, I just need to be able to view the logs without having them flash and without errors while retrieving them.

[mismithhisler]

Hi @rmatte, I think this issue has been reported in #27821, so I'm curious if there's anything different you are seeing here?

As for disabling log streaming completely, we do have a long standing feature request for this in #14056

I'm taking a look at the Cannot fetch logs error to see if we can get a quick fix up for this.

[rmatte]

Hi @rmatte, I think this issue has been reported in #27821, so I'm curious if there's anything different you are seeing here?

As for disabling log streaming completely, we do have a long standing feature request for this in #14056

I'm taking a look at the Cannot fetch logs error to see if we can get a quick fix up for this.

The difference is that #27821 was filed for v1.11.3, but in my case the problem does not reproduce at all in v2.0.2 but reproduces in v2.0.3. So it would seem that it's the same symptom but probably a different cause that cropped up due to some change that was made between v2.0.2 and v2.0.3.

[mismithhisler]

@rmatte Maybe we are reproducing this differently, but I can reproduce this just fine in 2.0.2 by refreshing the page displaying the logs.

[rmatte]

@rmatte Maybe we are reproducing this differently, but I can reproduce this just fine in 2.0.2 by refreshing the page displaying the logs.

Strange, refreshing the page displaying the logs in 2.0.2 doesn't cause this error at all for me, but this error happens 100% of the time when I first click into the logs in 2.0.3. The log flickering from #28062 happens in 2.0.2 for me, but not the "Cannot fetch logs" error popup (the error popup only happens in 2.0.3 for me).

@mismithhisler Are you going to the UI directly in 2.0.2 when you're seeing that behavior reproduce for you, or are you hitting it via nginx?

[mismithhisler]

@rmatte I am going directly to the UI via Firefox and I'm on a MacOS build of Nomad.

[mismithhisler]

I've tagged this issue in a PR as potentially fixing this issue. It certainly solves at least some valid cases for this same error that I can reliably reproduce. The issue goes all the way back to 1.9.6, so I'll be curious if this resolves your logging problem or not.