Research for building better AI retrieval systems that resist duplicate information and preserve critical evidence.
This project aims to explore how duplicate information within search pipelines can cause loss of information. Within a RAG pipeline, information is retrieved before being used to answer questions. If there are multiple duplicates of information within the retrieved information, relevant evidence may be lost. By introducing duplicates in the retrieval results, I have shown that accuracy can be reduced significantly (by 26.4 percentage points in one experiment). I have also explored duplicate removal techniques, and found that some techniques could remove documents with slight differences in information that are important. To further this research, I aim to find improvements to handling duplicate information, while retaining useful information.
My aim is to reduce negative impacts duplicates may have on AI responses, whilst not removing useful information. I will explore different techniques for identifying duplicates within data, including documents with very similar information, but altered numbers/facts. Through varied datasets and models, I will perform experiments, and compare the accuracy to previous methods to determine which methods improve accuracy.
By comparing accuracy with and without duplicate removal, I will assess whether useful evidence was retained in the data. I will also assess how often different documents with differing facts were incorrectly grouped together. I aim for the method to be effective with a variety of datasets and models. However, even if this method does not improve accuracy, knowing in detail why it was ineffective in certain scenarios will be useful information.
This money would be used to provide me with a workstation with GPU, RAM, and storage resources so I can effectively perform AI experiments. Currently, resources are limiting me in how many and large experiments I can perform. Providing me with my own workstation would allow me to test additional models, provide consistency in experimentation, and reduce reliance on limited external computational resources.
This project is being undertaken by Ishaan Sharma (High School Student). I am supported in research by my mentor Leilani H. Gilpin at UC Santa Cruz. I have performed experimentation, analysis, and tested various methods of duplicate detection in this project. This has resulted in a research manuscript titled "Duplicate Crowding in Retrieval-Augmented Generation: How Repeated Documents Displace Unique Evidence". Experimental results have already been obtained in this project, and this money would allow me to develop this work further.
The methods may not be effective with different models and/or datasets. Removing duplicates may result in useful information being removed. Alternatively, not removing enough duplicates may still negatively impact the quality of retrieval. Computational resource limitations may also be faced with larger models. However, even if a more effective method is not created, I will be able to identify which methods did not work and why, as well as identifying limitations in current duplicate detection methods. This will therefore inform improvement in future research.
No funds have been raised specifically for this project in the last 12 months. This project has been undertaken using resources and research computing resources available.
There are no bids on this project.
Comments
No comments yet. Sign in to create one!