Questions about keypoints

[ChengkaiYang]

Dear @cocoon2wong :
What a great job in trajectory prediction! But I still have some questions. In AKL loss, we supervise coarse trajectory with keypoints. How do we get ground truth keypoints used for loss function? Are ground truth keypoints part of ground truth trajectory or ground truth keypoints are constructed artificially based on ground truth of trajectory? If keypoints are constructed artificially based on ground truth, how can we guarantee it will bring more efficient or extra information to model to learn? (Imagine if we don't use such method, ground truth may be the only thing to supervise.If anything irrelevant or not such useful brought in model to supervise, does the performance of model drop?) If keypoints are part of ground truth, my question will be solved. Looking forward for your reply! Thank you very much!

[cocoon2wong]

Hi @ChengkaiYang !
Thank you for your questions!

These keypoints are part of groundtruth, we just need to determine their corresponding time indices.
In the average keypoint loss, we have

Vertical/silverballers/agents/__baseAgent.py

Lines 67 to 74 in 178866c

	def l2_loss(self, outputs: list[tf.Tensor],
	labels: tf.Tensor,
	*args, **kwargs) -> tf.Tensor:
	"""
	L2 distance between predictions and labels on predicted key points
	"""
	labels_pickled = tf.gather(labels, self.p_index, axis=1)
	return C.training.loss.ADE(outputs[0], labels_pickled)

Here, self.p_index is the indices of temporal keypoints. In our experiments, it will be set to [3, 7, 11] when forecasting $t_f = 12$ future points. Meanwhile, len(self.p_index) is the number of keypoints, i.e., the $N_{key}$ in our paper.

These different values were actually determined by ablation experiments as well. Please note that these keypoints are only used during training and are not needed during testing. Therefore, no additional information is introduced for training in our model.

I hope these answers can help you!

[cocoon2wong]

Hi @ChengkaiYang !

I noticed that you haven't replied to this question for a long time. If you have further questions, please feel free to ask! If you have no further questions, I will close this question later!

Thank you again for your question!