Improving Search Relevance with Data Science: Key Lessons from Data Driven WV

This is a guest post from the Data Driven WV team, who worked on a search relevance project with me as part of the Data Driven WV project at West Virginia University in the Spring 2025 semester.

Introduction

At Data Driven WV, we teamed up with Mountain Fog to tackle a big question. How can we use AI to improve search results? We wanted to understand how well Large Language Models (LLMs) could judge search relevance and, more importantly, how they compared to human judgment.

Search engines are everywhere, but getting the right results isn’t as simple as it seems. What makes a search result “good” or “bad”? And could AI help us make those decisions faster and more accurately? To find out, we put both human reviewers and AI models to the test. But first, allow us to introduce our team!

The team

Karson Bastug, Lewis Fellow and Project Lead

Karson Bastug is one of two Lewis Fellows leading this project. A senior majoring in Management Information Systems, Karson has been with Data Driven WV for two years. His experience and leadership help guide the team through the project and support collaboration, innovation, and real-world impact.

Justin Davis, Lewis Fellow and Project Lead

Justin Davis is one of the two Lewis Fellows leading this project. A senior majoring in Management Information Systems and Accounting, with a minor in Business Data Analytics. Justin has been with Data Driven WV for two years.

Jake Clapper, Experiential Learner

Jake Clapper is a junior pursuing a B.S. in Computer Science and is now in his second semester with Data Driven WV. As an experiential learner on the Search Relevance project, he contributed to the development of code modules that enhance search accuracy and performance.

Robert Geyer, Experimental Learner

Robert Geyer is an experiential learner for this project. This was his first session with Data Driven WV. He was able to enhance his skills with coding and data analytics by being a part of this program.

Jeff Zemerick, Project Guide

Jeff Zemerick served as the team’s guide for this project.

1. Human judgment

Before testing ways to improve search results, we needed a baseline for comparison. The best approach was human judgment, manually scoring search results on a scale from 0 (least relevant) to 3 (most relevant). Using Quepid , a tool for rating movie search relevance, we tested queries like Star Wars, Avengers, and Pirates, evaluating results as either “good” or “bad.”

In our evaluations, we had disagreements and had to come together for a consensus on some of the ratings. What is relevant to one person may be irrelevant to somebody else. This taught us a key lesson. Search evaluation is an ongoing process, not a one-time fix. It also revealed to us just how demanding it can be to manually evaluate search results, which led us to utilize an LLM.

2. LLMs revealed our own biases

As we incorporated LLMs into our search evaluation, we found that their judgments didn’t just reflect search quality. They also exposed our own biases. By feeding prompts into an LLM and comparing its ratings to human judgments, we saw discrepancies between the LLM and our own rating. Our initial response was that the LLM was “wrong” and it needs to better align with our own judgements, but who’s to say that we are even right?

For example, we noticed that LLMs sometimes rated results higher or lower than we expected, often due to subtle differences in phrasing. This forced us to reflect on the criteria we were using. Were we prioritizing the right factors in our relevance judgments? Were we unconsciously biased toward certain types of results? In Figure 1, we can directly see the difference between the LLM ratings and our own. The graphs show the exact count of each relevancy score. The goal was to get the two counts for the LLM and human ratings as close as possible, as this would indicate both the LLM and human agrees on a score for a particular movie.

As you can see, the human judgements deemed almost twice the amount of movies as irrelevant than the LLM did. Rather than just using it as a tool to score search results, we learned to evaluate our own assumptions and biases, as sometimes the LLM only knows what we tell it to know. It did present an interesting thought to us. If the goal is to improve the relevance of search results for other humans, then maybe it is okay to be biased, after all, we are the target audience.

3. Small changes make a big difference

One of our most exciting findings came from testing LLMs as search relevance judges. By feeding prompts into an LLM and comparing its ratings to human judgments, we identified patterns in what makes a good search prompt.

In our initial runs of calling the LLM, we fed it the following prompt with the query being set to Star Wars:

Instruction: Provide a judgment on a scale of 0 to 3 indicating the relevance of the movie to the search query.

Funnily enough, some of the movies it got most wrong (as in the LLM rated the movie relevancy as 0) were movies set within the Star Wars universe. In an effort to better help the LLM understand the context, a data dictionary was created that allowed each evaluation to additionally read the Wikipedia entry for that movie. Afterwards, we saw significant improvement in the LLMs ability to effectively rate movies.

We discovered that specificity is key. Queries with clear context, like “Does this movie tell story about a strong conflict between good and evil within the Star Wars Universe?” performed better than plain ones like “Is this movie about Star Wars?” Prompts referencing established universes (e.g., Star Wars, Marvel) often yielded better responses. This became trickier with vaguer queries such as “Pirates”. However, this is when the ability to influence the decision making process as a human became invaluable, as we may have a better understanding of what is currently relevant within the online world in regards to something like Pirates.

Conclusion

At the end of the day, search relevance isn’t just about numbers. It’s about people. We learned that human judgment is messy but necessary, that LLMs can challenge our assumptions, and that even tiny changes in wording can make a huge difference. The process of improving search results never really ends, but neither does discovering something new, and that’s what makes it interesting.

Thanks

I want to thank the students who took on this project and did the hard work behind everything above. A special thank you to Hannah Bailey and Joshua Meadows for their support in making it all happen. Thank you as well to the supporting faculty and to West Virginia University for their participation and for making this collaboration possible. It was a pleasure working with all of you this semester, and I’m grateful for your time, curiosity, and effort.

If you’re working on search relevance and want to talk through how AI and human judgment can work together, please reach out.