---

RA-Lib is a benchmarking environment for Rank Aggregation (RA) algorithms. This website contains the current benchmarking results, which have 22 unsupervised RA methods, 9 supervised RA methods and 1 semi-supervised RA methods, these algorithms were tested on our preprocessed datasets. These datasets cover the areas of person re-identification (re-ID), recommendation system, bioinformatics and social choice. The code of tested methods includes both classical and state-of-the-art RA methods that can be funded in https://github.com/nercms-mmap. As well as having all the experimental details and settings on this website.

If you want to add your own algorithm to improve the benchmarking system, please send a package of your algorithm code and a link to the published paper to waii2022@whu.edu.cn.

Unsupervised	Supervised	Semi-supervised
$\textrm{CombMIN}$ [1] $\textrm{CombMAX}$ [1] $\textrm{CombSUM}$ [1] $\textrm{CombANZ}$ [1] $\textrm{CombMNZ}$ [1] $\textrm{MC1}$ [2] $\textrm{MC2}$ [2] $\textrm{MC3}$ [2] $\textrm{MC4}$ [2] $\textrm{Borda count}$ [3] $\textrm{Dowdall}$ [4] $\textrm{Median}$ [5] $\textrm{RRF}$ [6] $\textrm{iRANK}$ [7] $\textrm{Mean}$ [8] $\textrm{HPA}$ [9] $\textrm{PostNDCG}$ [9] $\textrm{ER}$ [10] $\textrm{Mork-H}$ [11] $\textrm{CG}$ [12] $\textrm{DIBRA}$ [13] $\textrm{Borda-Score}$ [14]	$\textrm{RankNet}$ [15] $\textrm{ListNet}$ [16] $\textrm{wBorda}$ [17] $\textrm{CRF}$ [18] $\textrm{CSRA}$ [19] $\textrm{AggRankDE}$ [20] $\textrm{IRA}_\textrm{R}$ [21] $\textrm{IRA}_\textrm{S}$ [21] $\textrm{QI-IRA}$ [22]	$\textrm{SSRA}$ [23]

Directory Structure

│  .gitignore
│  README.md
│ 
│  
├─datasets
│  ├─NSCLC      
│  ├─Re-ID  
│  ├─RecSys     
│  └─World University Ranking 2022 
├─examples
│      matlab_run_examples.m
│      py_run_examples.ipynb
├─results
├─src
│  │  requirements.txt
│  │  __init__.py
│  │  
│  ├─ramatlab
│  │  ├─common
│  │  │      InputType.m
│  │  │      IRAType.m
│  │  │      McType.m
│  │  │      
│  │  ├─semi
│  │  │      SSRA.m
│  │  │      
│  │  ├─supervised
│  │  │      RankNet.m
│  │  │      ListNet.m
│  │  │      AggRankDE.m
│  │  │      CRF.m
│  │  │      IRA.m
│  │  │      QI_IRA.m
│  │  │      WeightedBorda.m
│  │  │      
│  │  └─unsupervised
│  │          bordacount.m
│  │          borda_score.m
│  │          cg.m
│  │          combanz.m
│  │          combmax.m
│  │          combmed.m
│  │          combmin.m
│  │          combmnz.m
│  │          combsum.m
│  │          dibra.m
│  │          dowdall.m
│  │          er.m
│  │          hpa.m
│  │          irank.m
│  │          markovchainmethod.m
│  │          mean.m
│  │          median.m
│  │          mork_heuristic_maximum.m
│  │          postndcg.m
│  │          rrf.m
│  │          
│  └─rapython
│      │  __init__.py
│      │  
│      ├─common
│      │      constant.py
│      │      params.py
│      │      __init__.py
│      │      
│      ├─datatools
│      │      data_class.py
│      │      data_io.py
│      │      data_process.py
│      │      __init__.py
│      │      
│      ├─evaluation
│      │      evaluation.py
│      │      __init__.py
│      │      
│      ├─semi
│      │      ssra.py
│      │      __init__.py
│      │      
│      ├─supervised
│      │  │  ranknet.py
│      │  │  listnet.py
│      │  │  aggrankde.py
│      │  │  crf.py
│      │  │  ira.py
│      │  │  qi_ira.py
│      │  │  weighted_borda.py
│      │  │  __init__.py
│      │  └─CSRA
│      │          
│      └─unsupervised
│              bordacount.py
│              borda_score.py
│              cg.py
│              combanz.py
│              combmax.py
│              combmed.py
│              combmin.py
│              combmnz.py
│              combsum.py
│              dibra.py
│              dowdall.py
│              er.py
│              hpa.py
│              irank.py
│              markovchain.py
│              mean.py
│              median.py
│              mork_heuristic_maximum.py
│              postndcg.py
│              rrf.py
│              scorefunc.py
│              __init__.py
│              
└─test

Demo

3-re-id

Running

1. Run python plot.py to plot results.

Follow-up Plan

We will be updating and adding more RA methods for shared use.

References

[1] Fox, E., & Shaw, J. (1994). Combination of multiple searches. NIST special publication SP, 243-243.

[2] Dwork, C., Kumar, R., Naor, M., & Sivakumar, D. (2001, April). Rank aggregation methods for the web. In Proceedings of the 10th international conference on World Wide Web (pp. 613-622).

[3] Aslam, J. A., & Montague, M. (2001, September). Models for metasearch. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval (pp. 276-284).

[4] Reilly, B. (2002). Social choice in the south seas: Electoral innovation and the borda count in the pacific island countries. International Political Science Review, 23(4), 355-372.

[5] Fagin, R., Kumar, R., & Sivakumar, D. (2003, June). Efficient similarity search and classification via rank aggregation. In Proceedings of the 2003 ACM SIGMOD international conference on Management of data (pp. 301-312).

[6] Cormack, G. V., Clarke, C. L., & Buettcher, S. (2009, July). Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval (pp. 758-759).

[7] Wei, F., Li, W., & Liu, S. (2010). iRANK: A rank‐learn‐combine framework for unsupervised ensemble ranking. Journal of the American Society for Information Science and Technology, 61(6), 1232-1243.

[8] Burges, C., Svore, K., Bennett, P., Pastusiak, A., & Wu, Q. (2011, January). Learning to rank using an ensemble of lambda-gradient models. In Proceedings of the learning to rank Challenge (pp. 25-35). PMLR.

[9] Fujita, S., Kobayashi, H., & Okumura, M. (2020). Unsupervised Ensemble of Ranking Models for News Comments Using Pseudo Answers. In Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part II 42 (pp. 133-140). Springer International Publishing.

[10] Mohammadi, M., & Rezaei, J. (2020). Ensemble ranking: Aggregation of rankings produced by different multi-criteria decision-making methods. Omega, 96, 102254.

[11] Azzini, I., and Munda, G. (2020). A new approach for identifying the Kemeny median ranking. European Journal of Operational Research. Vol. 281. No. 2. 2020.

[12] Xiao, Y., Deng, H. Z., Lu, X., & Wu, J. (2021). Graph-based rank aggregation method for high-dimensional and partial rankings. Journal of the Operational Research Society, 72(1), 227-236.

[13] Akritidis, L., Fevgas, A., Bozanis, P., & Manolopoulos, Y. (2022). An unsupervised distance-based model for weighted rank aggregation with list pruning. Expert Systems with Applications, 202, 117435.

[14] Boehmer, Niclas, Robert Bredereck, and Dominik Peters. "Rank aggregation using scoring rules." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 5. 2023.

[15] Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., & Hullender, G. (2005). Learning to rank using gradient descent. In Proceedings of the 22nd International Conference on Machine Learning (pp. 89–96).

[16] Cao, Z., Qin, T., Liu, T.-Y., Tsai, M.-F., & Li, H. (2007). Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th International Conference on Machine Learning (pp. 129–136).

[17] Pujari, M., & Kanawati, R. (2012, November). Link prediction in complex networks by supervised rank aggregation. In 2012 IEEE 24th International Conference on Tools with Artificial Intelligence (Vol. 1, pp. 782-789). IEEE.

[18] Volkovs, M. N., & Zemel, R. S. (2014). New learning methods for supervised and unsupervised preference aggregation. The Journal of Machine Learning Research, 15(1), 1135-1176.

[19] Yu, Y., Liang, C., Ruan, W., & Jiang, L. (2020, May). Crowdsourcing-Based Ranking Aggregation for Person Re-Identification. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1933-1937). IEEE.

[20] Bałchanowski, M., & Boryczka, U. (2022). Aggregation of Rankings Using Metaheuristics in Recommendation Systems. Electronics, 11(3), 369.

[21] Huang, J., Liang, C., Zhang, Y., Wang, Z., & Zhang, C. (2022). Ranking Aggregation with Interactive Feedback for Collaborative Person Re-identification.

[22] Hu, C., Zhang, H., Liang, C., & Huang, H. (2024). QI-IRA: Quantum-inspired interactive ranking aggregation for person re-identification. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, pp. 1-9).

[23] Chen, S., Wang, F., Song, Y., & Zhang, C. (2008, October). Semi-supervised ranking aggregation. In Proceedings of the 17th ACM conference on Information and knowledge management (pp. 1427-1428).

Contacts

If you encounter any problems, you can contact us via email 2021302111226@whu.edu.cn