Heatmap lines corresponding with detected/matched line segments?

[sunshineinsandiego]

Hi - I am using the SOLD2 model for line detection and matching based on the kornia tutorial here.. I asked this on the kornia github, but it seems like this is a better avenue for questions given your development of the model.

I would like to retrieve the lines on outputs['line_heatmap] that correspond with the final detected / matched line segments. For example: If outputs['line_segments'] returns a line segment with endpoints [(y1, x1), (y2, x2)], how can I find the lines on outputs['line_heatmap] that correspond with that line segment?

My understanding of the returned information from outputs is:

• outputs['line_heatmap] may include detected curves/edges that are not necessarily completely straight line segments.
• outputs['line_heatmap] values are used in conjunction with the outputs['junction_heatmap] values to determine potential line segments.
• outputs['line_segments] are straight line segments parameterized by two endpoints. These straight line segments may not reflect the lines detected in line_heatmap

So given two reported endpoints of a straight line segment from outputs['line_segments'], can I retrieve the (potentially not straight) line from outputs['line_heatmap'] that corresponds to that line segment?

My first thought is simply to use outputs['line_segments'] to find a straight line segment. Then search in a vicinity of the endpoints of the straight line segment as well as in a region along that straight line segment for the pixels in outputs['line_heatmap'] that have the highest intensity. My assumptions are:

• The two reported endpoints of a line segment from outputs['line_segments'] will also be on a line in the line_heatmap (i.e. the coordinates of the endpoints will have a high pixel intensity in the heatmap).
• There will be lines on the 'line_heatmap' that correspond with every reported line segment in outputs['line_segments'].
• The highest intensity pixels in 'line_heatmap' that are close to the endpoints from 'line_segments' and close to the straight line parameterized by 'line_segments' are used to determine the final output line segment.

Ideally, for a single reported line segment in outputs['line_segments'], I would like to return a list of pixel coordinates [(x1, y1), (x2, y2), (x3, y3),...,(xn, yn)] that represents the pixels on the detected line in outputs['line_heatmap'] that is associated with the that single reported line segment.

Is there a way to do this using kornia? Thanks!

[rpautrat]

Hi, it is indeed a good idea to ask your question here.

Your understanding in the first three bullet points, as well as your three assumptions are absolutely correct.

I would do the following to obtain what you want:

• rasterize the line you are looking for to obtain a binary mask. In kornia, you can start with an image full of zeros, and use the draw_line function of kornia to draw the line on your image. By casting the resulting tensor to torch.bool, you can obtain a binary mask of the pixels that are on the line.
• then simply apply the mask to outputs['line_heatmap] and this will return you the pixels that you are looking for.

I hope this answer your question, let me know if not.

[sunshineinsandiego]

Thanks, I think the problem with that approach is that I will simply get the pixels that correspond to the straight line between the endpoints. Instead, I would like the heatmap detected pixels that are used to determine the final reported straight line segments.

Another example to try to clarify:

• Let's assume that in the heatmap, there are a number of pixels [(x1, y1), (x2, y2),...,(xn, yn)] that have a high intensity and "approximate a line" but don't actually define a line.
• These pixels are not on a straight line due to noise in the image, camera calibration, etc so they are an "approximate line."
• SOLD2 then uses these pixel intensities from the approximate line, de noises the approximate line, detects potential end points, and reports an actual straight line segment defined by endpoints [(xa, ya), (xb, yb)] in outputs['line_segments'].
• So [(x1, y1), (x2, y2),...,(xn, yn)] are not necessarily on the line segment defined by [(xa, ya), (xb, yb)].

Therefore, drawing a line through the reported [(xa, ya), (xb, yb)] and getting the pixels on this reported line segment, will not actually get me the heatmap pixels [(x1, y1), (x2, y2),...,(xn, yn)] that approximate the line used to calculate the final reported line segment.

Any ideas? Thanks!

[rpautrat]

Well, if I would summarize what SOLD2 is doing, I would say that it generates the junctions, check all pairs of junctions, extract the rasterized pixels between the two junctions from the line heatmap, and compute a score to decide whether it is a line or not (it is a bit different in practice, but that is the high-level idea). So in practice, the rasterized pixels are the ones that are used to compute the final SOLD2 line segment. The noisy pixels around it are not taken into account.

If you want to extract something like a Line Segment Region (LSR), as was done in previous works such as the LSD line detector (https://pdfs.semanticscholar.org/3dce/b913ccb8394e1a5abec051708f86deed5483.pdf), then you could start from the rasterized pixels that are above a certain threshold in the line heatmap, and do a region growing algorithm to the neighboring pixels that have also a high activation in the line heatmap and that have a similar orientation as the line segment. But this boils down to applying LSD on the learned line heatmap, and this is what we did in a following work that you might want to check out: https://github.com/cvg/DeepLSD

Overall, I am not sure why you want these noisy activated pixels, but LSD is already detecting them internally in a sense.

[sunshineinsandiego]

Thanks Remi. I need the heatmap lines because the images we are using have significant vignetting. The quality of the images deteriorates significantly at the corners of the frame due to the pinhole lens we use. This means that the heatmap lines correctly identify all the lines in the image, but some of these lines are dropped from the final result. I think this is because the junction detection doesn't correctly associate endpoints that are at the center of the image (the high quality region) with endpoints that are at the edges of the image (the low quality region). My guess is that the feature space around the endpoints in the center is different enough from the feature space around the endpoints at the edges that they are not identified as coming from the same line despite the fact that they are indeed part of the same line.

So if I do want to analyze the pixels on the heatmap lines that are discarded / not reported in the final output due to this image artifact, I need to go to the line heatmap because the lines do appear on the heatmap (while not appearing on the final resulting line segments output).

And thanks for clarifying with your comment. It seems like taking the final reported endpoints and drawing a straight line between the endpoints on the heatmap will result in some combination of high/low intensity pixels along that straight line. SOLD2 then weighs that combination of high/low intensity pixels on that straight line to decide if it qualifies as a line segment.

[rpautrat]

Ok, I see your issue. Indeed, SOLD2 was trained and meant to be used with non distorted images.

If you have access to the calibration of your camera, you could of course first undistort the image and then run SOLD2. But otherwise, it is tricky to directly use the output of SOLD2... Except for the line heatmap which does contain some raw edge information that could be used in your case.

[sunshineinsandiego]

Thanks Remi. Quick hack solution is simply to crop the image so the distortion / quality at the edges of the image is removed. This seems to work quite well as the junction heatmap now shows line segment endpoints at the end of the image.

As for recovering the heatmap lines, it sounds like the best move is to search in a region around the endpoints + along the straight line defined by the final endpoints to get the pixels with highest line heatmap intensity and use that as the noisy "detected line".

[rpautrat]

Yes, that's also a quick hack that should give decent result!