How to benchmark our own model on all the datasets?

[Karn3003]

Hi,

Thanks for the awesome work. I wanted to know how to add our model and benchmark it. For model what should be take input and output format to benchmark. It would be great if you can share some dummy code for benchmarking. Looking forward for your response.

[sebastiandro]

@Karn3003 I created a Gist for how to get started with the Electronic Circuits benchmark:
https://gist.github.com/sebastiandro/051b152cb8fc1a7c14b6dc960b6962cf

[Timo-SH]

Hi,
thanks for the feedback. For now I would provide pseudocode to give a rough guide at first on how to use your own models with our data: 



import graphbench
from torch_geometric.loader import DataLoader

datasets = graphbench.Loader(root_dir, dataset_names, solver)

#The list of dataset names can be found in the datasets.csv file or the shortcuts can be used as stated in the readme. The solver has to be specified for the epm datasets in the SAT solving dataset as outlined in the paper. 

#get train, valid, test data for each dataset at position i 
train_data, valid_data, test_data = datasets[i]['train'], datasets[i]['valid'], datasets[i]['test]

#load the data into a torch_geoemetric dataloader
train_dataloader = DataLoader(train_data, batch_size)
valid_dataloader = DataLoader(valid_data, batch_size)
test_dataloader = DataLoader(test_data, batch_size)

For data in train_dataloader:
	#add your model forward pass here 
    true_values = data.y
	predicted_values = model(data) 
	#also do the parameter update and loss calculations here 
	

#depending on your model you can evaluate on the validation and test dataset
pred = []
true_data = []
For test_data in test_dataloader:
	pred.append(model(test_data))
	true_data.append(test_data.y)

eval_metrics = graphbench.Evaluator(task_name).evaluate(torch.cat(pred), torch.cat(true_data))

Of course the specific code depends on the selected model. Additional information regarding shape and contents of data.x, data.edge_index and data.y can be found at the datasets tab on graphbench.io. The evaluation task names can be found in the master.csv file.

In general this pseudocode holds for most of the datasets with weatherforecasting, unsupervised combinatorial optimization and chip design tasks being the exemptions.

For unsupervised combinatorial optimization tasks the corresponding predictions should be given for nodes in the graphs. The evaluator further expects the corresponding data.batch object but no data.y since there does not exist a target value.

For weather forecasting data the true values to be predicted are given by data.grid_x for a data object of the sequence at a later point in time. Since weather forecasting can be done for different time frames the dataset has no set true values for given input. Then data.y would correspond to data.grid_x for sequence element i+x in the dataloader using element i for data.x. Furthermore, data.x contains both the grid information as well as the intermediate format as outlined in the GraphCast paper. data.grid_x does only contain the grid information. 



In case of chip design we have a generation task at hand which requires the generation of a new edge index and node/edge features. However, training data results are still provided in data.y as solution circuits.

[Timo-SH]

Additional information regarding expected shapes and tensors for each dataset can be found in the following comment as well:
#6 (comment)

[Ani1211999]

Hi @Timo-SH I am trying to re-generate the results for GIN on edge classification task max matching. I have created this code-https://gist.github.com/Ani1211999/be677cdbdd09e697908400d27bb0ccb7 . Can you please take a look on the same and let me know whether the workflow is correct? I have not included the FFN, fully connected layers as I wanted to ensure I have a working codebase for the same. It would be great to have your feedback on the same. Thanks, in advance!