Student: “Georgios Xydias”
Program: “Data Science and Information Technologies”
Title: “Modular LLM Pipeline for Task-Oriented Customer Service Dialogue”
Abstract:
Task-oriented dialogue (TOD) systems are a cornerstone of modern
customer-service
automation, promising to handle structured user goals such as searching,
booking, and
information requests through natural conversation. The recent rise of
instruction-tuned Large
Language Models (LLMs) has reopened a fundamental design question:
should a single
general-purpose LLM handle the entire turn, or should the
responsibilities of a dialogue agent
be split into a modular pipeline of smaller, role-specific components?
Existing work explores
both ends of this spectrum, but rarely compares them head-to-head under
a setup that uses the
same dataset, the same evaluation procedure, and the same model families
across
architectures, leaving the practical trade-offs between architectural
simplicity and architectural
decomposition never brought together under a single controlled comparison.
In this work, we introduce a controlled comparative study of three
task-oriented dialogue
architectures for hotel and restaurant customer service, evaluated on
the MultiWOZ 2.2
benchmark. The first architecture is a single-LLM baseline that performs
dialogue state tracking,
database lookup, and response generation in one prompt. The second is a
zero-shot modular
pipeline that separates the dialogue state tracker from the response
generator, connected by a
deterministic database lookup and a policy gate, with a lightweight
guardrail on the generated
response. The third applies role-specific LoRA fine-tuning to the two
LLM-based modules of the
pipeline (the dialogue state tracker, DST, and the response generator,
ResponseGen) by
training small open-source models for their assigned role via QLoRA. All
three architectures are
evaluated with the standardized MultiWOZ evaluator, reporting both
dialogue-state metrics
(Joint Goal Accuracy, Slot F1) and end-to-end metrics (Inform, Success,
BLEU, Combined),
complemented by a custom set of operational metrics covering
hallucination, policy violations,
system correctness, latency, and cost.
Our findings show that a) architectural decomposition is not universally
beneficial: it
substantially helps weaker models that struggle to manage every
responsibility in a single
prompt, but it can hurt the strongest open-source models that already
coordinate all
responsibilities reliably, and b) role-specific LoRA fine-tuning lifts
dialogue-state tracking of
small open-source models close to the level of much larger commercial
APIs, making them
cost-efficient, fast, and competitive alternative for slot tracking.
These results show that
architectural decomposition and role-specific fine-tuning are effective
tools, but only under the
right conditions, for building reliable LLM-based customer-service
dialogue systems.
Date-Time: Thursday 25/06/2026 – 10:00 AM
Examination Committee:
Dr. Kosmas Kritsis
Prof. Aggelos Pikrakis
Dr. Georgios Paraskevopoulos
Msc Thesis Presentation
Thursday , June 25 · 10:00am – 11:00pm
Time zone: Europe/Athens
Microsoft Teams meeting
Join:
https://teams.microsoft.com/meet/318082679942220?p=ne3PcOyBsgIcxTjyGu