Document Type : Original/Review Paper
Authors
Department of the Computer Engineering, Shahrood University of Technology (SUT), Shahrood, Semnan, Iran.
Abstract
The exponential growth of digital information has increased the demand for robust and efficient Information Retrieval (IR) systems. Query Performance Prediction (QPP) is a critical task for identifying difficult queries and enhancing retrieval strategies. However, existing QPP methods suffer from several limitations: (1) score-based approaches fail to capture the structural relationships among retrieved documents, (2) supervised methods require labeled training data, making them costly and impractical for new domains, and (3) unsupervised post-retrieval predictors often rely solely on retrieval score dispersion, neglecting document clustering effects. To address these challenges, we propose a novel clustering-based post-retrieval QPP method. Specifically, we introduce three unsupervised predictors: Clustered Distinction, which measures query-specific separability of retrieved clusters; Clustered Query Drift, which estimates the deviation of top-ranked documents from query intent; and a hybrid approach combining both. By analyzing the clustering structure of retrieved documents, our method improves interpretability while eliminating the need for labeled data. We evaluate our approach on three standard datasets: the large-scale MS MARCO Passage Ranking dataset, TREC DL 2019, and TREC DL 2020. Experimental results demonstrate that our method significantly outperforms state-of-the-art score-based QPP models. These findings highlight the potential of cluster-aware QPP for enhancing IR systems and reducing the impact of difficult queries.
Keywords
Main Subjects
[1] N. Arabzadeh, C. Meng, M. Aliannejadi, and E. Bagheri, “Query Performance Prediction: Techniques and Applications in Modern Information Retrieval,” in Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, 2024, pp. 291–294. doi: 10.1145/3673791.369843.
[2] S. Datta, S. MacAvaney, D. Ganguly, and D. Greene, “A’Pointwise-Query, Listwise-Document’based Query Performance Prediction Approach,” in Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022, pp. 2148–2153. doi: 10.1145/3477495.353182.
[3] G. Faggioli, S. Lupart, S. Marchesin, N. Ferro, and B. Piwowarski, “Towards Query Performance Prediction for Neural Information Retrieval : Challenges and Opportunities,” pp. 51–63, doi: 10.1145/3578337.3605142.
[4] G. Faggioli, T. Formal, S. Marchesin, S. Clinchant, N. Ferro, and B. Piwowarski, “Query Performance Prediction for Neural IR: Are We There Yet?,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 13980 LNCS, pp. 232–248, 2023, doi: 10.1007/978-3-031-28244-7_15.
[5] A. Singh, D. Ganguly, S. Datta, and C. Macdonald, Unsupervised Query Performance Prediction for Neural Models with Pairwise Rank Preferences, vol. 1, no. 1. Association for Computing Machinery, 2023. doi: 10.1145/3539618.3592082.
[6] C. Meng, N. Arabzadeh, M. Aliannejadi, and M. de Rijke, “Query performance prediction: From ad-hoc to conversational search,” in Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023, pp. 2583–2593. doi: 10.1145/3539618.3591919.
[7] Z. Abbasiantaeb, C. Meng, D. Rau, A. Krasakis, H. A. Rahmani, and M. Aliannejadi, “LLM-based Retrieval and Generation Pipelines for TREC Interactive Knowledge Assistance Track (iKAT) 2023,” TREC, 2023.
[8] G. Faggioli, N. Ferro, C. I. Muntean, R. Perego, and N. Tonellotto, “A geometric framework for query performance prediction in conversational search,” in Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023, pp. 1355–1365. doi: 10.1145/3539618.3591625.
[9] M. Allhgholi, H. Rahmani, A. Derakhshan, and S. Mohammadi Raouf, “DOSTE: Document Similarity Matching considering Informative Name Entities,” J. AI Data Min., vol. 13, no. 1, pp. 85–94, 2025.
[10] H. Hashemi, H. Zamani, and W. B. Croft, “Performance prediction for non-factoid question answering,” in Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, 2019, pp. 55–58. doi: 10.1145/3341981.33442.
[11] M. Samadi and D. Rafiei, “Performance Prediction for Multi-hop Questions,” arXiv Prepr. arXiv2308.06431, 2023.
[12] E. Poesina, R. T. Ionescu, and J. Mothe, “iqpp: A benchmark for image query performance prediction,” in Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023, pp. 2953–2963. doi: 10.1145/3539618.359190.
[13] N. Arabzadeh, C. Meng, M. Aliannejadi, and E. Bagheri, “Query Performance Prediction: From Fundamentals to Advanced Techniques,” in European Conference on Information Retrieval, Springer, 2024, pp. 381–388. doi: 10.1007/978-3-031-56069-9_51.
[14] D. Carmel and E. Yom-Tov, Estimating the query difficulty for information retrieval. Morgan & Claypool Publishers, 2010.
[15] B. He and I. Ounis, Inferring Query Performance Using Pre-retrieval Predictors. Springer, 2004. doi: 10.1007/978-3-540-30213-1_5.
[16] A. Shtok, O. Kurland, and D. Carmel, “Predicting Query Performance by Query-Drift Estimation,” in TOIS, 2009. doi: 10.1007/978-3-642-04417-5_30.
[17] R. Cummins, M. Lalmas, C. O’Riordan, and J. M. Jose, “Navigating the user query space,” in String Processing and Information Retrieval: 18th International Symposium, SPIRE 2011, Pisa, Italy, October 17-21, 2011. Proceedings 18, Springer, 2011, pp. 380–385. doi: 10.1007/978-3-642-24583-1_37.
[18] S. Déjean, R. T. Ionescu, J. Mothe, and M. Z. Ullah, “Forward and backward feature selection for query performance prediction,” in Proceedings of the 35th annual ACM symposium on applied computing, 2020, pp. 690–697. doi: 10.1145/3341105.3373904.
[19] N. Arabzadeh, R. Hamidi Rad, M. Khodabakhsh, and E. Bagheri, “Noisy perturbations for estimating query difficulty in dense retrievers,” in Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, 2023, pp. 3722–3727. doi: 10.1145/3583780.361527.
[20] R. Cummins, J. Jose, and C. O’Riordan, “Improved query performance prediction using standard deviation,” in Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, 2011, pp. 1089–1090. doi: 10.1145/2009916.2010063.
[21] Y. T. and S. Wu, “Query performance prediction by considering score magnitude and variance together,” in in Proceedings of the 23rd ACM Inter_national Conference on Conference on Information and Knowledge Man_agemen, 2014, pp. 1891–1894. doi: 10.1145/2661829.2661906.
[22] Y. Z. and W. B. Croft, “Query performance prediction in web search environments,” in ” in Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, 2007, pp. 543–550. doi: 10.1145/1277741.1277835.
[23] N. Arabzadeh, M. Khodabakhsh, and E. Bagheri, “BERT-QPP: Contextualized Pre-trained transformers for Query Performance Prediction,” Int. Conf. Inf. Knowl. Manag. Proc., pp. 2857–2861, 2021, doi: 10.1145/3459637.3482063.
[24] M. Khodabakhsh and E. Bagheri, “Semantics-enabled query performance prediction for ad hoc table retrieval,” Inf. Process. Manag., vol. 58, no. 1, p. 102399, 2021, doi: 10.1016/j.ipm.2020.102399.
[25] H. Zamani, W. B. Croft, and J. S. Culpepper, “Neural query performance prediction using weak supervision from multiple signals,” in The 41st international ACM SIGIR conference on research & development in information retrieval, 2018, pp. 105–114. doi: 10.1145/3209978.3210041.
[26] S. Datta, D. Ganguly, D. Greene, and M. Mitra, “Deep-qpp: A pairwise interaction-based deep learning model for supervised query performance prediction,” in Proceedings of the fifteenth ACM international conference on web search and data mining, 2022, pp. 201–209. doi: 10.1145/3488560.3498491.
[27] F. Raiber and O. Kurland, “Query-performance prediction: Setting the expectations straight,” SIGIR 2014 - Proc. 37th Int. ACM SIGIR Conf. Res. Dev. Inf. Retr., pp. 13–22, 2014, doi: 10.1145/2600428.2609581.
[28] H. Scells, L. Azzopardi, G. Zuccon, and B. Koopman, “Query variation performance prediction for systematic reviews,” 41st Int. ACM SIGIR Conf. Res. Dev. Inf. Retrieval, SIGIR 2018, pp. 1089–1092, 2018, doi: 10.1145/3209978.3210078.
[29] S. Cronen-Townsend, Y. Zhou, and W. B. Croft, “Predicting query performance,” SIGIR Forum (ACM Spec. Interes. Gr. Inf. Retrieval), pp. 299–306, 2002, doi: 10.1145/564426.564429.
[30] G. Amati, C. Carpineto, and G. Romano, “Query difficulty, robustness, and selective application of query expansion,” in Advances in Information Retrieval: 26th European Conference on IR Research, ECIR 2004, Sunderland, UK, April 5-7, 2004. Proceedings 26, Springer, 2004, pp. 127–137. doi: 10.1007/978-3-540-24752-4_10.
[31] F. Diaz and R. Jones, “Using temporal profiles of queries for precision prediction,” in Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, 2004, pp. 18–24. doi: 10.1145/1008992.1008998.
[32] Y. Zhou and W. B. Croft, “Ranking robustness: a novel framework to predict query performance,” in Proceedings of the 15th ACM international conference on Information and knowledge management, 2006, pp. 567–574. doi: 10.1145/1183614.1183696.
[33] J. A. Aslam and V. Pavlu, “Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions,” in European conference on information retrieval, Springer, 2007, pp. 198–209. doi: 10.1007/978-3-540-71496-5_20.
[34] J. Pérez-Iglesias and L. Araujo, “Standard deviation as a query hardness estimator,” in International Symposium on String Processing and Information Retrieval, Springer, 2010, pp. 207–212. doi: 10.1007/978-3-642-16321-0_21.
[35] D. Carmel, E. Yom-Tov, A. Darlow, and D. Pelleg, “What makes a query difficult?,” in Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, 2006, pp. 390–397. doi: 10.1145/1148170.1148238.
[36] F. Diaz, “Performance prediction using spatial autocorrelation,” in Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, 2007, pp. 583–590. doi: 10.1145/1277741.1277841.
[37] H. Roitman, S. Erera, and B. Weiner, “Robust standard deviation estimation for query performance prediction,” ICTIR 2017 - Proc. 2017 ACM SIGIR Int. Conf. Theory Inf. Retr., pp. 245–248, 2017, doi: 10.1145/3121050.3121087.
[38] C. J. Van Rijsbergen, “Automatic information structuring and retrieval.” University of Cambridge, 1972.
[39] C. J. van Rijsbergen and K. SPARCK JONES, “A test for the separation of relevant and non‐relevant documents in experimental retrieval collections,” J. Doc., vol. 29, no. 3, pp. 251–257, 1973, doi: 10.1108/eb026557.
[40] N. Craswell, B. Mitra, E. Yilmaz, D. Campos, and E. M. Voorhees, “Overview of the TREC 2019 deep learning track,” arXiv Prepr. arXiv2003.07820, pp. 1–22, 2020, doi: 10.48550/arXiv.2003.07820.
[41] I. MacKie, J. Dalton, and A. Yates, “How Deep is your Learning: The DL-HARD Annotated Deep Learning Dataset,” SIGIR 2021 - Proc. 44th Int. ACM SIGIR Conf. Res. Dev. Inf. Retr., no. July, pp. 2335–2341, 2021, doi: 10.1145/3404835.3463262.
[42] T. Nguyen et al., “MS MARCO: A human generated MAchine reading COmprehension dataset,” CEUR Workshop Proc., vol. 1773, pp. 1–10, 2016.
[43] M. Khodabakhsh and E. Bagheri, “Learning to rank and predict: Multi-task learning for ad hoc retrieval and query performance prediction,” Inf. Sci. (Ny)., vol. 639, 2023, doi: 10.1016/j.ins.2023.119015.
[44] K. Sparck Jones, S. Walker, and S. E. Robertson, “Probabilistic model of information retrieval: Development and comparative experiments. Part 2,” Inf. Process. Manag., vol. 36, no. 6, pp. 809–840, 2000, doi: 10.1016/S0306-4573(00)00016-9.
[45] C. Hauff, “Predicting the effectiveness of queries and retrieval systems,” ACM SIGIR Forum, vol. 44, no. 1, pp. 88–88, 2010, doi: 10.1145/1842890.1842906.
[46] R. Mohemad, N. N. M. Muhait, N. M. M. Noor, and Z. A. Othman, “Performance analysis in text clustering using k-means and k-medoids algorithms for Malay crime documents,” Int. J. Electr. Comput. Eng., vol. 12, no. 5, pp. 5014–5026, 2022, doi: 10.11591/ijece.v12i5.pp5014-5026.
[47] A. M. Ikotun, A. E. Ezugwu, L. Abualigah, B. Abuhaija, and J. Heming, “K-means clustering algorithms: A comprehensive review, variants analysis, and advances in the era of big data,” Inf. Sci. (Ny)., vol. 622, pp. 178–210, 2023, doi: 10.1016/j.ins.2022.11.139.
[48] J. A. Hartigan and M. A. Wong, “A k-means clustering algorithm,” Appl. Stat., vol. 28, no. 1, pp. 100–108, 1979, doi: 10.2307/2346830.
[49] S. Miyamoto, Y. Kaizu, and Y. Endo, “Hierarchical and Non-hierarchical Medoid Clustering Using Asymmetric Similarity Measures,” in 2016 Joint 8th International Conference on Soft Computing and Intelligent Systems (SCIS) and 17th International Symposium on Advanced Intelligent Systems (ISIS), 2016, pp. 1–4. doi: 10.1109/SCIS-ISIS.2016.0091.
)