YohsinConnect: A Retrieval-Augmented Generation Chatbot for University Admissions

Problem & Approach

Traditional channels — a cluttered website, call centers, email — don't match how people now expect to get answers: instantly, and in their own words. YohsinConnect pairs a large language model with Retrieval-Augmented Generation (RAG) so answers stay grounded in real university content instead of the model's own, sometimes wrong, memory. The system has three parts: an embedding model that turns text into vectors, a vector database that stores and retrieves them, and an LLM that turns retrieved context into a natural-language answer — served through a Streamlit interface.

Building the Knowledge Base

We built a Python scraper (BeautifulSoup) to pull admissions-relevant content directly from the Habib University website, then manually reviewed and cleaned it — stripping out time-sensitive material like event announcements — before splitting it into chunks with LangChain's recursive text splitter. The final knowledge base held 97 chunks, averaging 232 tokens each. We also tried a semantic-splitting approach that used a language model to cluster related content, but found it lost context relative to our manually annotated approach, so we shipped the simpler method.

Choosing the Model Stack

We benchmarked 4 embedding models — OpenAI's text-embedding-3-small, Sentence-Transformers MiniLM v6, Llama2, and BAAI/bge-small-en — and found OpenAI's the strongest despite the added cost; notably, bge-small-en topped the public MTEB leaderboard but underperformed on our domain-specific data. For the vector store, we evaluated ChromaDB and FAISS; ChromaDB repeatedly crashed indexing our dataset, so we shipped FAISS.

For generation, we compared 3 LLMs paired with the same retrieval pipeline: Gemini as a commercial baseline, and two open-source 7B models — Llama2 and Mistral — served locally on Habib University's HPC cluster. We tested each open-source model with and without Chain-of-Thought (CoT) prompting, for 5 experiment configurations total.

Evaluation

We built a 43-question benchmark spanning the project's full scope (provided by the university) and scored every configuration on BLEU, ROUGE, BERTScore, RAGAS Faithfulness, RAGAS Answer Relevancy, and a custom Answer-Correctness metric implemented as an LLM-as-judge (Gemini, graded against a step-by-step grading prompt).

Mistral with CoT prompting was the best-performing configuration, beating the Gemini baseline on correctness (0.92 vs. 0.84), faithfulness, and answer relevancy, while roughly matching it on BERTScore. CoT consistently made both open-source models more concise and less prone to burying the right answer in irrelevant context — though it didn't eliminate the “needle in a haystack” failure mode, where a correct fact gets lost inside a large retrieved context block.

A Second Finding: The Metrics Themselves Were Unreliable

Digging into individual cases, we found the automated metrics didn't always agree with what a human reader could plainly see. RAGAS scored one CoT answer's relevancy at 0.0 and the equivalent non-CoT answer at 0.4, despite the CoT answer being clearly better — traceable to known issues in the RAGAS library's implementation at the time. BLEU and ROUGE, being n-gram overlap metrics, didn't penalize small but critical factual errors. Most strikingly, BERTScore rated “Habib offers BBA” and “Habib does not offer BBA” as 95.6% similar — two sentences with opposite meanings. Because of this, every configuration was also manually, holistically reviewed rather than judged on automated scores alone.

Conclusion

A RAG chatbot built on an open-source, locally-hosted LLM (Mistral, with CoT prompting) can match or beat a much larger commercial model on this kind of narrow, grounded Q&A task — at a fraction of the ongoing cost. The bigger lesson for us was methodological: automated NLP evaluation metrics are a starting point, not a substitute for actually reading the model's answers.