From Queries to Results: Testing LLMs for Search Improvement

This is a guest post from the Data Driven WV team, who worked on a search relevance project with me as their guide as part of the Data Driven WV project at West Virginia University in the Summer 2025 semester.

Written by Jacob Auer, Elizabeth Lowe, Lorenzo Nunez, Jacob Sossamon, and Joseph Yesufu.

Introduction

Search engines play an indispensable role in helping users find the content or products that they are inquiring about. Search engines are most effective when they can match user queries to the actual information or products the user is looking for, even if that intent isn’t stated explicitly. This project explores diverse ways to evaluate and improve search relevance using Large Language Models (LLMs), manual judgements, and OpenSearch tools. The overall goal is to understand how well search systems perform for certain queries and how expansion terms and/or search settings can potentially improve it. Our evaluations focus on three primary case studies, The Movie Database (TMDB), E-commerce, and Sports, each designed to simulate real-world search scenarios with distinct user intents. We used Quepid, a tool designed for rating search relevance, to support our evaluations.

Case 1: The Movie Database (TMDB)

This case evaluates how well the TMDB search engine performs when handling popular sci-fi queries, with a focus on the “Star Wars” and “Star Trek” franchises. These queries are perfect test cases not only because of their volume and diversity of related content, but because they reveal deeper issues with search engines’ process results. When a user types in a query like “Star Wars”, they normally expect to see the main films and series pop up first. However, TMDB’s search results often include search results that are not related to the content that the user is inquiring about. These unrelated search results can include other sci-fi films that have the word “star” in the title as well.

We looked at the top 40 results for each query and manually reviewed them to see how well they matched what the user inquired about. There are a few issues in the search engine that involve ambiguous titles, naming collisions, and overlap.

This case helped us analyze the challenges that came with using a search engine to find the intended content especially when handling popular franchise based queries that could have different types of media/film. The case also highlighted the need for better tools that recognize the differences between franchises and also to filter out unrelated searches more effectively.

Case 2: E-commerce

This second case explores search relevance in the context of online shopping, where the goal is to connect shoppers with the exact products they have in mind. Unlike movie searches in Case 1, product searches have to deal with a much wider variety of possible meanings for the same term. In retail, even simple queries can be tricky. “notebook” might mean a laptop or a paper journal, and a “bag” could point to a laptop sleeve, a tote bag, luggage, etc. We looked at queries such as bag (computer bag) and notebook (laptop).

E-commerce search systems face two main challenges. First, they must guess the correct meaning of a term from minimal input. Second, they must rank the results so the most relevant options rise to the top. This is complicated by the fact that relevance is drawn from multiple parts of a listing, not just a title, but also its description, category, and other important attributes. The figure that goes along with this case shows just how much these challenges can affect what a shopper sees, and why improvements in ranking and context-awareness are essential for an ideal shopping experience.

Case 3: Sports

The sports case dives into search performance and query expansion across a dataset of professional teams from multiple leagues and sports. Many queries can be short or ambiguous, such as “Giants,” “Reds,” or “football,” which can point to several teams or leagues. This makes it an ideal scenario to test how query expansion can refine results. We examined what we can do to the data to help query expansion perform.

This case showed how important it is for a search engine to recognize context meaning and not just word matching. It also showed that adding more detail to the data helps the search engine better understand what users are searching for which in turn, leads to more accurate results.

Across these three cases, a total of 180 relevance judgments were made. Each search result was individually reviewed and rated based on how well it matched the user’s intended search. These scores highlighted strong and weak matches based on the user’s intention. Scoring gives an insight on how accurate the query was in providing the intended results for the user to improve their search experiences. After scoring these results, the judgements were then used to guide experiments in OpenSearch which tested changes in indexing, query construction, and relevance tuning. LLMs assisted in identifying the search intentions of the users and creating query expansion terms. With the use of these different approaches of LLMs, manual judgements, and OpenSearch tools, this analysis will help in improving search relevance issues. While this analysis is useful for TMDB, E-commerce, and Sports, it can also be used for other search inquiries in order to deliver accurate and meaningful results for boosting user satisfaction.

Case 1: TMDB and sci-fi search evaluation

This case study focuses on evaluating the search results for “Star Wars” and “Star Trek” using data from TMDB (The Movie Database), a publicly available, community-maintained platform that provides structured metadata for a wide range of media, including films, TV series, and documentaries. TMDB serves as a real-world dataset for testing search relevance across diverse content types. These two iconic sci-fi franchises present a useful challenge due to the volume of related content and the ambiguity introduced by overlapping keywords. For each query, the top 40 search results were manually evaluated for relevance, with each result scored based on how well it aligned with the presumed user intent.

While analyzing the results, there were a few challenges that arose, including keyword ambiguity with overlap, and naming collisions.

Keyword ambiguity and overlap

This search engine relies on basic keyword matching which produces a high rate of unintended results. Words like “Star”, “Wars”, and “Galaxy” are common terms that appear in many unrelated sci-fi films. In a lot of cases, when “Star Trek” terms were being searched, Star Wars results were returned due to Star Wars being an extremely popular franchise with the word “Star” in it.

Naming collisions

There are multiple film titles that share common words of a franchise name which causes issues in ranking and the relevance. There is another film that includes the word “Trek” named Trek: The Movie. Obviously, this movie does not have a connection to Star Trek, but it appears in the results of the search due to the name similarity. Because of these naming collisions, irrelevant results tend to appear first.

The first 10 results mostly captured the intended results of the search. However, in the results after, noise began increasing. This search engine relied heavily on keywords instead of franchise matching which led to these inconsistencies of relevant search results. Strong matches included “Star Wars film”, “Star Wars”, “Star Wars Jedi”, and “Star Wars Fantasy of the Galaxy”. Weak matches included “Star Trek”, “Interstellar Travel”, “Space Exploration”, and “Star Trek sci-fi Adventure”.

Overall, this TMDB case showed that while using search engines, there needs to be a sense of the context while also being more specific especially when there are popular franchises that share similar names to each other.

Case 2: E-commerce, laptops and bags

In the e-commerce case, we examined search performance for the queries “notebook” and “bag,” focusing on laptops and computer bags. Both terms presented notable ambiguity. “notebook” oftentimes surfaced paper products, while “bag” returned a wide mix including fashion tote bags and travel bags. Judging the top 40 results for each query showed how important it is to interpret user intent accurately in product search. Rating results for bags and notebooks is more difficult because product relevance isn’t immediately clear. Unlike movies, the items often require extra context or inspection to understand, which adds subjectivity and slows down evaluation. Unlike media searches where relevance is usually tied to a single field like the title, e-commerce relevance tends to span across multiple fields, thus making it harder to pinpoint strong matches. This just shows how careful prompt design and actual human review is essential. Overall, this case demonstrated both the complexity and potential of manipulating e-commerce search for better precision and relevance.

The figure above shows how we manually evaluated search results for the ambiguous query “notebook” in an e-commerce context. The top example shows a highly relevant match, a waterproof laptop sleeve designed for 15.6-inch notebooks. The bottom example, on the other hand, shows a poor match, returning a paper notebook product instead of a computer-related product.

This contrast makes it clear that small differences in interpretation can completely change the usefulness of results. It also shows how ranking plays a critical role. Even if the correct type of product is in the search index, it can be buried if the ranking model doesn’t prioritize it correctly. These errors point to opportunities for improvement, such as adding better category fillers, using product metadata more effectively, and training the system to pick up on subtle context cues. In practice, refining how results are ordered can be just as important as, if not more important than, matching the right keyword in the first place.

LLM-driven query expansion experiments

Given the challenges we faced with ambiguous queries and limited search relevance, we explored whether Large Language Models (LLMs) could help improve search results through a technique called query expansion. Query expansion involves adding extra, related terms to a user’s original search query to better capture their intent and retrieve more relevant documents.

Large Language Models, advanced AI systems trained on vast amounts of text data, are particularly skilled at understanding language patterns and relationships between words. We leveraged this capability by prompting an LLM to generate additional keywords and phrases related to our original queries. These generated expansion terms were then added to our search queries within Quepid, a tool used for evaluating search relevance.

To measure the impact of these expanded queries, we used relevance metrics such as nDCG (normalized Discounted Cumulative Gain). This metric quantifies how well the search results align with what users actually want by assigning higher scores to relevant items that appear near the top of the search results. By comparing nDCG scores before and after adding the LLM-generated terms, we could objectively assess whether query expansion improved the quality of search results.

Building on our movie-related queries, we experimented with prompts such as:

• “What are 10 key words related to ‘Star Wars’?”
• “What are 10 synonyms for ‘Star Trek’?”

Applying the same logic to our e-commerce case, we crafted prompts like:

• “What are 10 synonyms for ’notebook’ in the context of laptops?”
• “Provide key words that could improve the search for ‘bag’ in the context of a computer bag.”

The Gemma 3 1B LLM generated relevant expansion terms to improve search results. For example, it suggested terms like “Force” and “Han Solo” when prompted about Star Wars, and “tablet” and “digital pad” for notebook-related searches. These additional terms helped the search engine find content that might have been missed by relying solely on exact keyword matches.

Below is a table that has two example queries, LLM-generated expansion terms, and the reasoning provided by the LLM for choosing each term.

From there, we incorporated the LLM-generated expansion terms into our original Quepid queries to evaluate whether they improved our search results. Using the default OpenSearch query structure that Quepid runs, we compared the performance of the original queries against the expanded versions that included the additional terms.

Prompting the LLM proved effective in generating useful expansion terms that improved search recall, meaning the search engine was able to find more relevant documents that might have been missed by the original, narrower queries. However, some minor tuning was needed to reduce noise from overly broad or loosely related terms. Overall, this approach shows strong potential for enhancing search through query expansion. When the added terms stay closely aligned with the original query’s intent, we saw improvements in relevance scoring. For example, terms like fantasy of the galaxy were too far removed from the core context of a Star Wars query and introduced irrelevant results.

After reviewing the metrics from our initial expansion terms and LLM prompts, we refined our approach by providing the LLM with additional context. Specifically, we explained that our goal was to improve search relevance within an OpenSearch environment. This added context helped guide the LLM to generate more targeted and relevant expansion terms, allowing us to explore whether contextual prompting leads to better query optimization.

With the Star Wars and Star Trek queries, we noticed that the expansion terms generated were more focused, often referencing specific elements like Wookiees or original trilogy. These terms felt aligned with what users might actually search for, increasing the likelihood of relevant results. For example, a query like “Star Wars Wookiees” is likely to return more targeted content than a generic or overly broad phrase like “Star Wars fantasy.”

In contrast, the notebook and bag queries tended to produce broader expansion terms. For instance, a prompt around “notebook” returned terms like “MacBook Pro 14” and “high performance”, which cast a wider net. While this can be useful for improving recall, it risks introducing irrelevant results if the terms drift too far from the user’s intent.

This ties into an important observation about search performance:

• A high number of results can mean the query is too broad, retrieving many irrelevant documents and lowering precision (the proportion of relevant results).
• A low number of results may mean the query is too narrow, missing useful documents and lowering recall (the ability to find all relevant documents).

To test this further, we used two types of OpenSearch queries:

• Multi-match queries, which allow a search to be run across multiple fields (such as title, description, and tags) in a single query. These are useful when users might search using different types of information.
• Boolean queries, which allow precise control using logic operators like AND, OR, and NOT to include or exclude certain terms.

Our tests showed that expanded queries often returned more results, but sometimes at the cost of relevance. Other queries produced fewer hits but had a higher concentration of useful results. This confirmed the importance of carefully balancing precision and recall when using LLM-driven query expansion.

Search tuning and tool exploration

As part of our evaluation of search relevance tuning within Quepid, we tested several query structures to determine their impact on result quality, ranking behavior, and precision. This included comparing match types, exploring Boolean logic, and translating natural language prompts into structured Boolean syntax.

Match vs phrase search

The initial comparison involved match and match_phrase queries. These two approaches, while similar in syntax, produced noticeably different outcomes when applied to user-style queries. We used variations of the query “Star Wars and Star trek” to observe how results shifted depending on whether terms were loosely matched or required to appear in exact sequence. match queries produced broader results, allowing term combinations across fields and sentences. This was helpful in identifying more loosely related documents, improving overall recall. However, the lack of precision sometimes introduced irrelevant items. In contrast, match_phrase queries narrowed the scope considerably. Only documents containing the exact sequence were returned, increasing precision but potentially missing valuable results with slight variations in wording.

We found that the dis_max (disjunction maximum) query type worked particularly well in the movie domain, where user searches often focus on one key piece of information, like a movie’s title. In simple terms, dis_max looks at several different fields (like title, description, or tags), but instead of combining all their scores, it only uses the single best match to rank the result.

For example, queries like “Star Wars” benefit from dis_max prioritizing the strongest match, typically in the title field, leading to more relevant results and higher nDCG scores. It also handles longer queries gracefully by falling back on fields like description when appropriate. By focusing on the best single match, dis_max avoids score dilution from weaker matches across multiple fields.

In contrast, e-commerce queries often rely on combined relevance across fields such as title, brand, and description. A query like “notebook high performance” might have partial matches scattered across these fields. Because dis_max only rewards the strongest single field, it tends to miss this layered context. In these cases, bool queries with multiple should clauses or the cross_fields type performed better.

The cross_fields query treats multiple fields in a document, like title, description, and tags, as if they are one combined field. Instead of requiring all search terms to appear in the same field, it allows them to be matched across different fields. This approach was especially effective in our e-commerce case, where important information is often spread out across several fields rather than concentrated in just one. For example, a product listing might have “notebook” in the title and “high performance” in the description. A traditional query might not treat these as a strong match, but with cross_fields, the search engine recognizes that together they form a complete and relevant idea. This leads to better recall, finding more useful results, and improves the overall quality of the search experience by connecting related terms even when they appear in different parts of the data. We can see that the ecommerce expanded queries in Figure 5 are slightly better than the baseline metrics in Figure 3.

However, in the movie domain, where relevant terms are usually concentrated in one field, cross_fields was less helpful. It sometimes diluted scoring by spreading matches across less relevant fields instead of rewarding strong, focused matches like dis_max does. In summary, dis_max excels when there’s a dominant field (like titles in media), while cross_fields is better suited for multi-attribute domains like e-commerce, where query intent spans several fields.

Boolean exploration

Boolean query structures allowed for greater control over inclusion and exclusion logic. By structuring queries with must, should, and must_not clauses, we were able to define more specific criteria for what should appear in the results. Boolean queries proved especially useful in domains with overlapping content types. For example, to isolate Star Wars films while filtering out books or fan content, we used the structure:

("star wars" AND (movie OR film)) NOT (parody OR book OR game)

This led to improved result quality by reducing irrelevant documents and emphasizing those matching the intended context. Weighting fields such as title^4 or description^2 further refined scoring.

Using AI to convert natural language into Boolean queries

To streamline Boolean construction, we used language models to convert user-style queries into structured Boolean logic. A typical example included the query:

Find anything Star Wars related, but only movies, not books or games.

Which was translated into:

("star wars" AND (movie OR film)) NOT (parody OR book OR game)

The results were generally useful. However, this process also required human validation to ensure important terms were not omitted or misinterpreted.

Implications of Boolean query generation

Translating natural language into Boolean logic offers clear practical benefits. Structured queries help refine search behavior and produce more relevant results, particularly when dealing with ambiguous or noisy datasets. However, Boolean logic also introduces trade-offs. Overly strict filters may reduce recall, and generated queries must be reviewed carefully to avoid excluding important content. Structured prompts and domain-specific term knowledge are essential in creating effective Boolean expressions.

While automated query generation can reduce time spent building search logic, it should not replace subject matter review. Boolean queries remain a valuable method of enforcing logic and structure in search tuning, but their success depends on thoughtful construction and validation.

Case 3: Exploring the sports data

How query expansion interacts with the data

The Sports case investigates how query expansion can be improved through the quality and structure of the underlying data. Unlike the TMDB and E-commerce cases, where most of our focus was on crafting and refining the queries themselves, this case shifts attention to the dataset. Sports queries present a unique challenge because they are often short, broad, or ambiguous. A single word like “Giants,” “Reds,” or “football” can correspond to multiple teams, leagues, or even different sports entirely. This makes the role of effective query expansion crucial. We hypothesized that even before worrying about the ins and outs of query expansion, improving the data first with more rich, descriptive keywords would make the query expansion much more efficient.

Creating the dataset

After establishing the Sports case, our next step was to create said dataset. The original data contained team names, leagues, and basic attributes, which is a good start. Though, we found that short queries could only go so far without richer connections in the data. To address this, we expanded the dataset with additional descriptive fields such as team histories, notable players, cities, and mascot information. We also made sure to include these descriptive, unique keywords. We turned to ChatGPT to help generate this. In our prompt, we provided the original dataset, and the context that we are trying to enhance query expansion by improving the data with the characteristics mentioned above. It was ingested into Quepid and ready for testing.

Testing query expansion terms

As we worked with the enhanced data, we discovered that the most effective search scenarios emerged from a balance. The dataset needed to be rich with descriptive keywords while still maintaining a natural level of ambiguity. The keyword coverage gave the search engine more entry points to connect a user’s query to relevant teams, while the ambiguity created opportunities for query expansion to play a meaningful role. For example, a single-word query like “Giants” could point to multiple teams. The descriptive fields, combined with this ambiguity, allowed expansion terms like “MLB,” “NFL,” or “San Francisco” to narrow results in a way that felt intuitive. Chat GPT was able to generate many query expansion terms that fit our new data.

In Quepid, we tested the scenario of searching for the term “football” and expecting results about Premier League football teams. The results came back as expected, a mixed bag of both American Football and Premier League teams. Though, the simple expansion of the search term to “premier league football” gave us perfect results.

Simple, but effective, and combined with further research into the expansion terms themselves would garnish an extremely efficient system. By creating this balance, we built a dataset that both supported broader matches for initial queries and provided the context needed for expanded queries to refine results.

Key takeaways

This project explored the effectiveness of search systems by evaluating search relevance across three distinct domains: movies (TMDB), e-commerce, and sports. A key focus was understanding user intent and addressing challenges arising from ambiguous queries and overlapping keywords.

Using 180 manual relevance judgments, the team scored the quality of search results, which guided subsequent experiments within OpenSearch. These manual assessments helped identify gaps in existing rankings. Through iterative testing and group discussions, the team gained clearer insights into actual user needs and preferences.

Tuning search parameters such as phrase and match types led to noticeable improvements in ranking quality. Additionally, Large Language Models (LLMs) were employed to generate query expansion terms. When carefully aligned with user intent, these expansions enhanced both precision and recall, underscoring the importance of thoughtful prompt design in LLM-driven query improvements.

In the TMDB case study, franchise overlaps caused difficulties for title-based rankings. The e-commerce domain highlighted how relevance depends on multiple fields and varies by context, with product ratings often requiring additional inspection. Meanwhile, the sports case demonstrated that poor metadata diminished search performance, whereas improved metadata enabled more effective query expansion.

Broader and more diverse queries provided stronger signals for comparing search strategies, which helped refine scoring and indexing approaches. Overall, combining LLM-generated expansions with manual reviews and OpenSearch tuning, including adjustments to match types, Boolean logic, and query composition (cross-fields vs. dis_max), proved essential to enhancing accuracy and delivering more relevant, satisfying search experiences.

What’s next and recommendations to the community

To improve search relevance, we recommend pairing manual relevance judgments with LLM-assisted query expansion to surface hidden gaps in ranking and better capture user intent. Continued experimentation with prompt design can sharpen expansion terms while minimizing noise. The community should also prioritize metadata quality, especially in domains like e-commerce and sports where inconsistencies can quickly degrade results. Expanding evaluations to include broader, more diverse queries will stress-test systems and expose weaknesses that typical queries might miss. Most importantly, treat search relevance as an iterative, collaborative process. Human-in-the-loop evaluation, regular discussion, and thoughtful tuning are essential to building more accurate, satisfying search experiences.

References

• OpenAI. (2025). ChatGPT (Aug 6 version) [Large language model]. https://chat.openai.com/
• OpenSearch Documentation. (n.d.). Retrieved from https://opensearch.org/docs/latest/
• Quepid. (n.d.). Search relevance tuning tool. Retrieved August 4, 2025, from https://quepidapp.com/
• Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2023. Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search? In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ‘23). Association for Computing Machinery, New York, NY, USA, 1426 to 1436. https://doi.org/10.1145/3539618.3591703

Thanks

I want to thank the students who took on this project and did the hard work behind everything above. A special thank you to Hannah Bailey and Joshua Meadows for their support in making it all happen. Thank you as well to the supporting faculty and to West Virginia University for their participation and for making this collaboration possible. It was a pleasure working with all of you this semester, and I’m grateful for your time, curiosity, and effort.

If you’re working on search relevance and want to talk through query expansion, tuning, or measuring your results, please reach out.