China

Evaluating long-form answers in high-stakes domains such as law or medicine remains a fundamental challenge. Standard metrics like BLEU and ROUGE fail to capture semantic correctness, and current LLM-based evaluators often reduce nuanced aspects of answer quality into a single undifferentiated score. We introduce DeCE, a decomposed LLM evaluation framework that separates precision (factual accuracy and relevance) and recall (coverage of required concepts), using instance-specific criteria automatically extracted from gold answer requirements. DeCE is model-agnostic and domain-general, requiring no predefined taxonomies or handcrafted rubrics. We instantiate DeCE to evaluate different LLMs on a real-world legal QA task involving multi-jurisdictional reasoning and citation grounding. DeCE achieves substantially stronger correlation with expert judgments (r=0.78), compared to traditional metrics (r=0.12) and pointwise LLM scoring (r=0.35). It also reveals interpretable trade-offs: generalist models favor recall, while specialized models favor precision. Importantly, only 11.95% of LLM-generated criteria required expert revision, underscoring DeCE’s scalability. DeCE offers an interpretable and actionable LLM evaluation framework in expert domains.

EMNLP 2025

Beyond Pointwise Scores: Decomposed Criteria-Based Evaluation of LLM Responses

decomposed evaluation

llm-as-a-judge

llm evaluation

legal question answering

### Welcome!
The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. Our Virtual Poster Sessions will take place online Thursday, August 22, 2024.

You are required to register for this event. **Please register [here](https://2024.aclweb.org/registration). **

If you have already registered, please check your inbox for an email from Underline granting you access to ACL 2024 content.

Please register!

ACL 2024

The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. More information will be announced soon.

poster

## Welcome!
"I am excited to welcome you to this year’s edition of the Conference on Empirical Methods in Natural Language Processing! Importantly, it marks the 30th edition of EMNLP. With over 8,000 submissions, more than 3,000 accepted papers, and thousands of attendees, we have come a long way from that first
workshop, which had 14 accepted papers. As the field looks ahead, Suzhou is the fitting location for celebrating this milestone: rooted in a long literary tradition, yet modern and forward-looking, and home to a large share of the NLP community." 

*Message from the General Chair, Dirk Hovy*

Celebrate 30 Years of EMNLP! 
EMNLP 2025 will be held in Suzhou, China from November 5th to November 9th, 2025.

Telemarketing towards merchants is considerably more complex than traditional dialogue systems. 
Given a user utterance, the response must not only follow the context but also strategically and naturally guide the conversation toward marketing objectives. 
A common approach is to fine-tune LLMs using high-quality dialogue data from top sales. 
However, we find that even after careful data cleaning, these data cannot be used directly due to two issues:
(1) Poor strategy-following: Real-world conversations are highly random with much chit-chat topics, easily leading deviation from intended strategy.
(2) Insufficient expert knowledge learning: Expert knowledge appears infrequently or not at all in limited collected corpus.
To this end, we introduce a hybrid data synthesis framework with two main innovations. 
First, we unify the input schema with profile and strategy designed by top sales, and extract them via a Multi-task paradigm.
In addition, we propose Role-playing Simulation and Session Prefix Completion to complementarily improve the strategy-following and inject long-tail expert knowledge into our fine-tuned model -- TeleBot.
Comprehensive online and offline evaluations demonstrate its effectiveness.
In particular, in terms of the final marketing results -- High Intention Rate, TeleBot reaches the performance level of the top 25% of human sales.

Advancing E-commerce Merchants Telemarketing with Synthetic Data-Driven LLMs

Limited access to mental healthcare resources hinders timely depression diagnosis.
Social media platforms present a valuable data source for early detection, yet this task faces two significant challenges: 1) the need for medical knowledge to distinguish clinical depression from transient mood changes, and 2) the dual requirement for high accuracy and model explainability.
To address this, we propose DORIS, a framework that leverages Large Language Models (LLMs).
To integrate medical knowledge, DORIS utilizes LLMs to annotate user texts against established medical diagnostic criteria and to summarize historical posts into temporal \textit{mood courses.}
These medically-informed features are then used to train an accurate Gradient Boosting Tree (GBT) classifier.
Explainability is achieved by generating justifications for predictions based on the LLM-derived symptom annotations and mood course analyses.
Extensive experimental results validate the effectiveness and interpretability of our method, highlighting its potential as a supportive tool for mental health professionals.

Medical Knowledge-Guided Depression Detection on Social Media with Large Language Models

Recent investigations into effective context lengths of modern flagship large language models (LLMs) have revealed major limitations in effective question answering (QA) and reasoning over long and complex contexts for even the largest and most impressive cadre of models. While approaches like retrieval-augmented generation (RAG) and chunk-based re-ranking attempt to mitigate this issue, they are sensitive to chunking, embedding and retrieval strategies and models, and furthermore, rely on extensive pre-processing, knowledge acquisition and indexing steps. In this paper, we propose Tagging-Augmented Generation (TAG), a lightweight data augmentation strategy that boosts LLM performance in long-context scenarios, without degrading and altering the integrity and composition of retrieved documents. We validate our hypothesis by augmenting two challenging and directly relevant question-answering benchmarks --- NoLima and NovelQA --- and show that tagging the context or even just adding tag definitions into QA prompts leads to consistent performance gains over the baseline --- up to 17\% for 32K token contexts, and 2.9\% in complex reasoning question-answering for multi-hop queries requiring knowledge across a wide span of text.

Tagging-Augmented Generation: Assisting Language Models in Finding Intricate Knowledge In Long Contexts

Recent advancements in slow-thinking reasoning models have shown exceptional performance in complex reasoning tasks. However, their tendency for "overthinking" on simple problems leads to excessive computational resource usage and increased inference latency, which hinders their widespread industrial adoption. While current mitigation strategies uniformly reduce reasoning tokens, they risk degrading performance on challenging tasks that require extended reasoning. This paper introduces Difficulty-Adaptive Slow-Thinking (DAST), a novel framework that enables models to autonomously adjust Chain-of-Thought (CoT) length based on problem difficulty. We propose a Token Length Budget (TLB) metric and leverage budget-aware preference optimization to implement DAST, which penalizes inefficiency on simple problems while incentivizing deep reasoning for complex ones. Experiments demonstrate DAST's significant value for practical application: it effectively mitigates overthinking, substantially lowering costs and latency—while crucially preserving high accuracy on complex problems, paving the way for the efficient application of advanced reasoning models.

DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Models

Embedding retrieval systems learn a shared semantic representation space for queries and items, enabling efficient retrieval through an approximate nearest-neighbor search. However, current industrial implementations face a critical limitation: using a fixed retrieval cutoff for all queries inevitably compromises performance, yielding insufficient recall for high-frequency (head) queries and reduced precision for low-frequency (tail) queries. This persistent challenge stems fundamentally from the frequentist paradigms dominating existing loss function designs. In this work, we introduce a novel framework, probabilistic Embedding-Based Retrieval (pEBR): maximum likelihood estimation-based and contrastive estimation-based, which learns the underlying probability distribution of relevant items for each query, compute adaptive cosine similarity cutoffs via probabilistic cumulative distribution functions (CDF), and automatically adapts to the distinct characteristics of head vs. tail queries. Experiments and ablation studies demonstrate that pEBR simultaneously improves precision and recall while maintaining computational efficiency.

pEBR: A Probabilistic Approach to Embedding Based Retrieval

This paper introduces <SYNTACT>, a three-stage multi agent LLM framework designed to transform unstructured and ambiguous Standard Operating Procedure (SOP) into a structured plan and an executable code template. Unstructured SOPs—common across industries such as finance, retail, and logistics—frequently suffer from ambiguity, missing information, and inconsistency, all of which hinder automation. SYNTACT addresses this through: (1) a Clarifier module that disambiguate the SOP using large language models, internal knowledge base (RAG) and human-in-the-loop , (2) a Planner that converts refined natural language instructions into a structured plan of hierarchical task flows through function (API) tagging, conditional branches and human-in-the-loop check-points, and (3) an Implementor that generates executable code fragments or pseudocode templates. We evaluate SYNTACT on real-world SOPs and synthetic variants, demonstrating an 88.4% end-to-end accuracy and a significant reduction in inconsistency compared to leading LLM baselines. Ablation studies highlight the necessity of each component, with performance dropping notably when modules are removed.
Our findings show that structured multi-agent pipelines like SYNTACT can meaningfully improve consistency, reduce manual effort, and accelerate automation at scale.

<SYNTACT>: Structuring Your Natural Language SOPs into Tailored Ambiguity-Resolved Code Templates

The emergence of open-source large language models (LLMs) has expanded opportunities for enterprise applications; however, many organizations still lack the infrastructure to deploy and maintain large-scale models. As a result, small LLMs (sLLMs) have become a practical alternative, despite their inherent performance limitations. While Domain Adaptive Continual Pretraining (DACP) has been previously explored as a method for domain adaptation, its utility in commercial applications remains under-examined. In this study, we validate the effectiveness of applying a DACP-based recipe across diverse foundation models and service domains. Through extensive experiments and real-world evaluations, we demonstrate that DACP-applied sLLMs achieve substantial gains in target domain performance while preserving general capabilities, offering a cost-efficient and scalable solution for enterprise-level deployment.

ixi-GEN: Efficient Industrial sLLMs through Domain Adaptive Continual Pretraining

Phonetic Cloaking Replacement (PCR), defined as the deliberate use of homophonic or near-homophonic variants to hide toxic intent, has become a major obstacle to Chinese content moderation. While this problem is well-recognized, existing evaluations predominantly rely on rule-based, synthetic perturbations that ignore the creativity of real users. We organize PCR into a four-way surface-form taxonomy and compile PCR-ToxiCN, a dataset of 500 naturally occurring, phonetically cloaked offensive posts gathered from the RedNote platform. Benchmarking state-of-the-art LLMs on this dataset exposes a serious weakness: the best model reaches only an F1-score of 0.672, and zero-shot chain-of-thought prompting pushes performance even lower. Guided by error analysis, we revisit a Pinyin-based prompting strategy that earlier studies judged ineffective and show that it recovers much of the lost accuracy. This study offers the first comprehensive taxonomy of Chinese PCR, a realistic benchmark that reveals current detectors' limits, and a lightweight mitigation technique that advances research on robust toxicity detection.

Lost in Pronunciation: Detecting Chinese Offensive Language Disguised by Phonetic Cloaking Replacement

Traditional Retrieval-augmented Generation systems struggle with complex multi-hop questions, which often require reasoning over multiple passages. While GraphRAG approaches address these challenges, most of them rely on expensive LLM calls. In this paper, we propose textGRsmallIEVER, a lightweight, low-resource, multi-step graph-based retriever for multi-hop QA. Unlike prior work, textGRsmallIEVER does not rely on LLMs and can perform multi-step retrieval in a few hundred milliseconds. It efficiently indexes passages alongside an associated knowledge graph and employs a hybrid retriever combined with aggressive filtering to reduce retrieval latency. Experiments on multi-hop QA datasets demonstrate that textGRsmallIEVER outperforms conventional retrievers and shows strong potential as a base retriever within multi-step agentic frameworks.

Don’t Forget the Base Retriever! A Low-Resource Graph-based Retriever for Multi-hop Question Answering

Corporate history in corporate annual reports includes events related to organizational changes, which can provide useful cues for a comprehensive understanding of corporate actions.
However, extracting organizational changes requires identifying differences in companies before and after an event, raising concerns about whether existing information extraction systems can accurately capture the relations. 
This work introduces JaCorpTrack, a novel event extraction task designed to identify events related to organizational changes.
JaCorpTrack defines five event types related to organizational changes and is designed to identify the company names before and after each event, as well as the corresponding date.
Experimental results indicate that large language models (LLMs) exhibit notable disparities in performance across event types.
Our analysis reveals that these systems face challenges in identifying company names before and after events, and in interpreting event types expressed under ambiguous terminology.
We will publicly release our dataset and experimental code at \url{https://github.com/foo/bar}.

Downloads

Next from EMNLP 2025

Advancing E-commerce Merchants Telemarketing with Synthetic Data-Driven LLMs

Similar lecture

An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from EMNLP 2025

Advancing E-commerce Merchants Telemarketing with Synthetic Data-Driven LLMs

Similar lecture

An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing

Downloads