Title: “Modular LLM Pipeline for Task-Oriented Customer Service Dialogue”

Abstract:
Task-oriented dialogue (TOD) systems are a cornerstone of modern customer-service
automation, promising to handle structured user goals such as searching, booking, and
information requests through natural conversation. The recent rise of instruction-tuned Large
Language Models (LLMs) has reopened a fundamental design question: should a single
general-purpose LLM handle the entire turn, or should the responsibilities of a dialogue agent
be split into a modular pipeline of smaller, role-specific components? Existing work explores
both ends of this spectrum, but rarely compares them head-to-head under a setup that uses the
same dataset, the same evaluation procedure, and the same model families across
architectures, leaving the practical trade-offs between architectural simplicity and architectural
decomposition never brought together under a single controlled comparison.
In this work, we introduce a controlled comparative study of three task-oriented dialogue
architectures for hotel and restaurant customer service, evaluated on the MultiWOZ 2.2
benchmark. The first architecture is a single-LLM baseline that performs dialogue state tracking,
database lookup, and response generation in one prompt. The second is a zero-shot modular
pipeline that separates the dialogue state tracker from the response generator, connected by a
deterministic database lookup and a policy gate, with a lightweight guardrail on the generated
response. The third applies role-specific LoRA fine-tuning to the two LLM-based modules of the
pipeline (the dialogue state tracker, DST, and the response generator, ResponseGen) by
training small open-source models for their assigned role via QLoRA. All three architectures are
evaluated with the standardized MultiWOZ evaluator, reporting both dialogue-state metrics
(Joint Goal Accuracy, Slot F1) and end-to-end metrics (Inform, Success, BLEU, Combined),
complemented by a custom set of operational metrics covering hallucination, policy violations,
system correctness, latency, and cost.
Our findings show that a) architectural decomposition is not universally beneficial: it
substantially helps weaker models that struggle to manage every responsibility in a single
prompt, but it can hurt the strongest open-source models that already coordinate all
responsibilities reliably, and b) role-specific LoRA fine-tuning lifts dialogue-state tracking of
small open-source models close to the level of much larger commercial APIs, making them
cost-efficient, fast, and competitive alternative for slot tracking. These results show that
architectural decomposition and role-specific fine-tuning are effective tools, but only under the
right conditions, for building reliable LLM-based customer-service dialogue systems.

Date-Time: Thursday 25/06/2026 – 10:00 AM
Examination Committee:
Dr. Kosmas Kritsis
Prof. Aggelos Pikrakis
Dr. Georgios Paraskevopoulos

Msc Thesis Presentation
Thursday , June 25 · 10:00am – 11:00pm
Time zone: Europe/Athens
Microsoft Teams meeting
Join:
https://teams.microsoft.com/meet/318082679942220?p=ne3PcOyBsgIcxTjyGu