Цели обучения после базового предобучения. DiRL — пример этапов обучения с учителем и последующего обучения с подкреплением; проверяйте конкретную цель, не предполагая единого стандартного рецепта.
Как читать и сравнивать
Какие базовый чекпойнт, обучающие данные и цель оптимизации используются?
Сравниваются ли варианты одной базовой модели при неизменном протоколе оценки?
Одна поэтапная схема не доказывает, что обучение с учителем и последующее обучение с подкреплением необходимы или оптимальны для любой диффузионной модели.
Наличие кода не гарантирует воспроизводимость. Сверяйте набор данных, метрику, протокол и условия в оригинале.
Оригинальная аннотация: Recent advancements in large language models (LLMs) have significantly improved Natural Language to SQL (NL2SQL) tasks, yet most NL2SQL systems continue to rely on the autoregressive (AR) paradigm. The highly structured nature of SQL makes AR models susceptible to sequential error propagation due to their rigid left-to-right decoding process. Diffusion Language Models~(DLMs) have recently emerged as a promising alternative, replacing unidirectional decoding with iterative denoising to enable global sequence refinement. Nevertheless, the adoption of DLMs in NL2SQL is constrained by a fragmented
Полезно для: Не указано · Ограничение: Не указано
2026-05-24T16:14:54+00:00 · Маскированная / дискретная диффузия, Дообучение · Источник
Оригинальная аннотация: Joint Embedding Predictive Architectures (JEPAs) have reshaped self-supervised representation learning in vision. The recent LLM-JEPA ported JEPA to autoregressive language models but inherited two steep costs from the causal-attention substrate: it demands explicit multi-view data (e.g., text-code pairs), and it requires two gradient-carrying forward passes per step. We introduce DLLM-JEPA, which pairs JEPA with masked-diffusion language models to eliminate both costs at once. The bidirectional attention of diffusion models yields two semantically distinct views of the same input via differen
Полезно для: Не указано · Ограничение: Не указано
2026-05-22T19:46:08+00:00 · Маскированная / дискретная диффузия, Дообучение · Источник
Оригинальная аннотация: Memorization in large language models has been studied almost exclusively through prefix-conditioned extraction, a natural choice for autoregressive models. However, diffusion language models (DLMs) can denoise masked tokens at arbitrary positions. Thus, prefix-only probing reveals only one facet of memorization in DLMs and significantly underestimates the risk of training-data extraction. In order to realistically model extractability of training data in DLMs, we introduce \emph{infilling extraction}, a data-extraction protocol parameterized by an arbitrary binary mask that subsumes prefix-on
Полезно для: Не указано · Ограничение: Не указано
2026-05-21T18:16:17+00:00 · Рассуждения, Дообучение · Источник
Оригинальная аннотация: We aim to improve the reasoning capabilities of diffusion language models (DLMs). While SFT is a popular post-training recipe for autoregressive models, its use in DLMs faces challenges and can even hurt performance, though the underlying causes remain understudied. Our analysis reveals that vanilla SFT overlooks learnability, namely what and when tokens are learned. Specifically, rare tokens are difficult to learn when most of the input is masked, whereas it is straightforward and thus of little value to learn common tokens when most of the input is unmasked. Motivated by our analysis, we pro
Полезно для: Не указано · Ограничение: Не указано
2026-05-19T02:20:08+00:00 · Маскированная / дискретная диффузия, Дообучение · Источник
Оригинальная аннотация: Masked diffusion language models (MDLMs) are emerging as a compelling new paradigm for text generation, but their training-time security remains largely unexplored. Existing backdoor attacks on Gaussian diffusion models or autoregressive language models do not directly apply to MDLMs because MDLMs rely on discrete state corruption and iterative denoising rather than continuous noising or left-to-right prediction. In this work, we present the first systematic study of training-time backdoor attacks on MDLMs. We propose SHADOWMASK, a backdoor attack that modifies the MDLM forward corruption proc
Полезно для: Не указано · Ограничение: Не указано
2026-05-18T11:54:11+00:00 · Маскированная / дискретная диффузия, Дообучение · Источник
Оригинальная аннотация: Recent masked diffusion language models (MDLMs), such as LLaDA and Dream, have achieved performance comparable to autoregressive large language models. Unlike autoregressive models, which generate text sequentially, MDLMs generate text by iteratively denoising masked positions in parallel. During fine-tuning, MDLMs learn to recover responses from masked response states conditioned on a prompt, thereby shifting their predictions from a prompt-masked unconditional distribution toward a prompt-conditional distribution. Despite this distinct generative and fine-tuning mechanism, machine unlearning