Date
Jul 30, 2026
slug
ranker-for-wayground
Status
Published
Subtitle
Logistic regression as a ranker
Tags
recommendation systems
My project was about building a ranker model for the current system which was only reliant on elastic search for Retrieval. There was no ranking stage , which is the norm in the industry for this kind of a problem statement. A colleague of mine had built a ranker model but it was not holding out in production over a significant period of time.
We already had search data (tabular format), to be specific , which userid hosted fired which query, what quizIds were shown to the user and in which order . Quiz's were served from elastic search in batches of 12, and we had the data that the user clicked on which quizzes (0.5) and then ended up hosting which quiz (+1). Based on this kind of data we can formulate the problem in multiple ways (Classification, regression , pointwise, listwise , pairwise etc) and then train a ranker model .
To gather clean data was of course a major problem and it involved writing a lot of sql queries, collating data from multiple databases, dropping data from unreliable experiments run by other teams, writing scripts to generate semantic tags etc.
My colleague was using close to 70 features in his ranking model, with multiple experiments i was able to drop the number of feature to 14 and deployed a simple logistic regression model in production, (MRR (for clicks and hosts both )as the base metric to track).
More complicated models were performing slightly better, but they were not as explainable (product manager ! ) as logistic regression. So i chose to go with them in my initial experiments.
Upon running experiments I also determined that the model performance would degrade after every 3 weeks, so i also built a pipeline to retrain the model once the performance drops below a certain threshold.
After running a lot of experiments, i was seeing that the MRR was doing worse than the baseline, In face it was actually closer to what the MRR would be if all the results were randomised. When i inspected the weights, I found that the multiplier for the position was actually quite high as compared to the others . Due to this the model was simply throwing items from the bottom to the top, giving no regard to whether they were relevant or not.
A Senior colleague and mentor of mine suggested that we should train with the Positions since it is an important signal but we should pass the position as 1 for the items at test/inference time. This stabilised the results and gave me a good baseline which i iterated upon and was able to build a good model.