-
Notifications
You must be signed in to change notification settings - Fork 2
Tasks
To submit please zip the submission files and send the zip archive via regular e-mail to the task organizers Pål Halvorsen (paalh at simula.no), Steven Hicks (steven at simula.no), and Michael Riegler (Michael at simula.no).
An example of a submission can be found in this repository.
The tasks are the following:
-
Detection subtask - id:
detection. The detection task stems from the requirement of the high detection accuracy needed to be viable for use in a clinical setting. Participants are asked to develop algorithms that achieve high classification scores on the 23 different classes presneted in the development dataset. Each submission should consist of a single ".csv" file, which contains one line per image in the unlabeled dataset. Each line should start with the name of the predicted file, followed by the predicted label, and then end with the model's confidence of that prediction. -
Efficient detection - id:
speed. The efficient detection task focuses on the real-time analysis needed to deliver instant feedback to doctors performing endoscopies. To satisfy this requirement, the algorithm must achieve good classification scores while also being able to classify images as fast as they are put on screen, which is approximately 30 frames per second. Submissions to this task require the participants to submit their code in the form of a Docker image so that we, the organizers, can evaluate the submissions on the same hardware. The Docker image should produce a ".csv" file similar to that of the detection task, with the addition of the processing time being added to the end of each line. -
Segmentation - id:
segmentation. In the segmentation task, we ask participants to use test images provided in the dataset to generate segmentations of polyps automatically. For this task, the participants are asked to submit a zip file containing the predicted mask. The name of the mask image should be the same as the input image.
Submissions should be submitted in separated files with the following names, depending on the subtask:
endotect20_GroupName_SubmissionName_SubmissionId.txt
GroupName: the name of the participating group, maximum 12 symbols (example: colonwizards)
SubmissionName: The id of the submission, maximum 9 symbols, one of the following ids: detection, speed, or segmentation.
SubmissionId: the id of the submission, maximum 15 symbols (example: ver1)
Examples: endotect20_colonwizards_detection_ver1.txt
For all submissions regarding the detection, the files should report the multi-class classification of each image together with a confidence value/interestingness level/rank and (for the efficient detection subtask only) image processing time, according to the following format:
Submissions to the detection task should be a .CSV file containing the image name, classification label, and confidence value. The .CSV file should be generated using the test dataset.
imagename,[classification decision],[confidence value, floating point value]
0014d89e-5afa-4186-99f9-4400c8c58a18.jpg,polyp,0.831
ae68bef5-8ca4-42fd-a275-d34ee4ea8c14_00000074.jpg,dyed-resection-margins,0.1234
...
In all the submissions the confidence value value refers to rank orders, expressing an expectation that some classified instances are more likely to be positive / correctly classified than others. The rank value is expected to be a floating-point value within [0.0;1.0] interval deciding for any pair of instances whether the first is more likely (>), equally likely (=), or less likely (<) to be positive / correctly classified than the second. If your algorithm does not provide any ranking for the classified instances, you must use 1.0 as the rank value.
Submissions to the efficient detection task are similar to the detection task, but also include information about processing time. The .CSV file should contain the image name, classification label, confidence value, image processing time in seconds. The .CSV file should be generated by a Docker image containing your code and takes the test dataset as input.
imagename,[classification decision],[confidence value, floating point value],[image processing time in seconds, floating point value]
0014d89e-5afa-4186-99f9-4400c8c58a18.jpg,polyp,0.831,2.09
ae68bef5-8ca4-42fd-a275-d34ee4ea8c14_00000074.jpg,dyed-resection-margins,0.1234,0.529
...
In all the Efficient detection submissions the image processing time in seconds value refers to a time interval from the moment when image has been completely loaded into memory to the moment of the final decision about the image class made by the classification algorithm. In other words, the full amount of time required to classify a particular image minus time spent to image loading and classification results saving. This image processing time referrers to the classification of the images from the test dataset only, you do not need to include algorithm training time.
Submissions to the segmentation task should be a single zip file for all the predicted mask. The name of the predicted mask image should be the same as the input image.