부산대학교 도서관

In today's digital age, higher education utilizes chatbots as virtual assistants to assist users, especially prospective students to access information easily. A chatbot is an application in natural language conversations to simulate intelligent interactions. Intelligent chatbots are needed to understand user needs and answer questions relevantly. We propose a chatbot with Retrieval Augmented Generation approach involving a retriever with cosine similarity search using OpenAI Ada embeddings to obtain relevant documents. The LLM OpenAI GPT-3.5- Turbo then generates the final answer. The chatbot mechanism begins with the retrieval module systematically identifying documents stored in the vector database that contain relevant information related to the user's query. The selected documents and query are provided to the LLM as part of the prompt to generate responses based on the knowledge provided in the relevant documents. The retrieval method is evaluated based on two criteria: the search method and the embedding model. The comparison method uses similarity search with Maximum Marginal Relevance (MMR) Search and the proposed embedding method against other models such as Google Embedding-001 and MPNet-Multilingual. The retrieval process is assessed using an evaluation dataset that incorporates Recall and Precision metrics, while answer generation is measured with BLEU and ROUGE Score. The observed disparity result between similarity search and MMR is not notably significant. Nonetheless, our chatbot holds an advantage in referencing past conversations due to its ability to store conversation history. Furthermore, potential enhancements are identified by augmenting the knowledge provided to the LLM in forthcoming iterations.