Non-occluded tracks being swapped over

[LiellPlane]

firstly, thanks for sharing this great library

I have issues whereby the sports players we track are becoming mixed up, and this is before Re-ID kicks in. Two players momentarily run into each other then each change direction, which swaps the tracks in Norfair

Is there a way to force the tracker to use embeddings when a collision is detected? What other solutions can mitigate this. Currently I send Norfair embeddings for all detections every 4 frames

[facundo-lezama]

Hi @LiellPlane!

This is a common issue when doing multiple object tracking. If you want to use embeddings when matching detections and tracked objects, I recommend using a custom distance_function and implementing your logic inside, combining the embeddings and the spatial distance. There isn't a way to handle collisions specifically since the definition of collision differs for each scenario.

Let me know if you need any help building the distance_function.

[facundo-lezama]

Sure! Can you share your distance_function and reid_distance_function (if you use any) definitions? Also, can you share how you're instantiating the Tracker object?

If data isn't sensitive, you could also share a short video where we can see the problem to better understand the situation.

[LiellPlane]

@facundo-lezama I instance the tracking as so:

    self._tracker = Tracker(
        initialization_delay=2,
        distance_function="euclidean",
        hit_counter_max=10,
        distance_threshold=distance_threshold,
        past_detections_length=5,
        reid_distance_function=embedding_distance,
        reid_distance_threshold=0.5,
        reid_hit_counter_max=50,
    )

The reid_distance_function is the default one in the documentation. We have players on a pitch but we transform their positions from a spectator perspective to a "birds-eye" view. The issue we are getting (unfortunately I cannot share videos) are:

Player 1 track (top down view):

player 2 track:
<

so simultaneously it looks like this:

<

but the tracker believes player one to continue diagonally:

and vice-versa for player 2:
/

Is there anything we can do to mitigate this? At the intersection I assume no reID is used

thanks for your help, it is much appreciated!

[facundo-lezama]

Thanks for sharing this information! Is the explanation shown as expected? Because it seems to be truncated or having some formatting issues.

Anyway, there are a couple of things you could try:

• if latency is not an issue, you could replace the distance_function with a custom one that takes into account the visual aspect (using some visual embeddings)
  • this way, you could have more precision in the matching
• you could also check what's happening with the detections when the occlusion occurs
  • are you having detections for both players?
  • what happens with detections after the occlusion?
  • with this information, you could make better decisions on how to match

[LiellPlane]

@facundo-lezama Hi, yes it seems the formatting did mess up! Well I think you got the idea, the tale as old as time - that two players crossing over get mixed up

Using the custom distance_function with embedding seems best tactic - I have simple histograms used currently, is there any example of how to modify the distance_function using the last few embeddings? Or are we limited to the most recent? I am not sure how the distance_function gets access to any embeddings

many thanks for your help and patience!

[facundo-lezama]

@LiellPlane Within the custom distance_function, you can access tracked_object.past_detections, and for each of the past_detections (conditioned by past_detections_length), you can use the embedding field to get the embeddings. Be aware that Norfair doesn't store the last past_detections_length detections; instead, it tries to distribute the detections homogeneously over time, and so past_detections represent in some way the whole history of the object.