Calculating attention mechanism

[ans92]

Hi,
Thank you for this great work and code. I want to ask could you please refer me the lines in the code where you are calculating the attention as shown in equation 1 of the paper? Actually I want to know what are W, K and q in equation 1. So if you can refer these things in you code then that would be great. Thank you.

[daoyuan98]

Hi, Thank you for your attention!
Please refer to the following lines for attention computation.

Relation-CZSL/model/SepMask.py

Line 240 in ea76ed8

A_cpl_att_ = self.att_key_transformer(self.lin_att_query[att_idx]) @ self.att_query_transformer(cat_key).t() # / np.sqrt(self.att_query.shape[1])

Relation-CZSL/model/SepMask.py

Line 241 in ea76ed8

A_cpl_obj_ = self.obj_key_transformer(self.lin_obj_query[obj_idx]) @ self.obj_query_transformer(cat_key).t() # / np.sqrt(self.obj_query.shape[1])

[ans92]

Hi @daoyuan98
Thank you for your response. Yes I have seen that. I have observed that you have instantiated your key, query and values by normal distribution. So they are getting semantic information of the concept like Red Apple? Normally what I observe is people use word2vec or similar pre-train language models to get the concept embedding and then train it. So you get semantic information by normal distribution values. Please correct me if I understood wrong. Thank you.