RELISH: LLM Regression with Latent Iterative State Head

In the rapid advancement of enterprise-grade artificial intelligence, the ability to extract accurate numerical predictions from large-scale language models (LLMs) has become a key challenge. Until now, the predominant strategies were to decode numerical values as if they were text, averaging multiple generations or adding classic regression heads on frozen representations. However, the RELISH (REgression with a Latent Iterative State Head) architecture proposes a disruptive approach: a latent state that is iteratively refined by cross-attention on the internal representations of the LLM, and then projects that state to a scalar value with a linear regressor. This innovation, presented in arXiv:2604.01206v2, not only outperforms previous methods by six different datasets and four different backbones, but does so with extraordinary parameter efficiency: only about 3.4 to 3.7 million trainable parameters, representing only 0.01–0.04% additional overhead on the frozen LLM. For companies looking for AI for high-performing companies without sacrificing resources, these types of advances open up real possibilities for integrating language models into numerical decision flows, from time series predictions to risk assessments. At Q2BSTUDIO, as a software and technology development company, we closely monitor these innovations to offer our customers tailor-made applications that leverage the latest in artificial intelligence, combining efficiency and accuracy. For example, a RELISH-based demand forecasting system could be integrated into a business intelligence services platform with power bi to visualize estimates in real-time, or function as an autonomous AI agent within a broader ecosystem. In addition, the lightweight nature of RELISH allows it to be deployed even in environments with cybersecurity restrictions or in AWS and Azure cloud service infrastructures, without requiring costly tuning of the entire model. The key is in its latent iterative state head: instead of forcing the LLM to 'speak numbers', a refined representation is directly extracted that adapts to the numerical target. This dramatically reduces the need for labeled data and the computational cost. In practice, a company could, for example, develop custom software that, on top of a pre-trained and frozen LLM, adds this light head to predict the share price, the probability of fraud, or the lifespan of a component. Integration with our enterprise AI solutions allows these prototypes to be scaled to production while maintaining tight control over latency and consumption. Compared to alternatives such as LoRA, which grow in parameters with model size (0.26–0.42%), RELISH remains constant and minuscule, making it ideal for multi-model deployments or for environments where every millisecond counts. From a technical perspective, the iterative refinement process is reminiscent of memory mechanisms in neural networks, but applied to the semantic space of the LLM. Each cross-attention step adjusts the latent state based on the representations of all tokens, managing to converge to a stable estimate. Experiments consistently show that this method outperforms direct regression heads and decoding methods, even with models as diverse as Llama, Mistral, or Gemma. For Q2BSTUDIO, this represents an opportunity to offer multiplatform applications that incorporate this technology without the need to modify the base models, accelerating the time-to-market of predictive analytics solutions. In short, RELISH is not just an incremental improvement, but a paradigm shift in how LLMs can serve as regression engines. Companies that adopt these types of lightweight printheads will be able to build more specialized AI agents, more accurate recommendation systems, and more dynamic business intelligence tools, all with a minimal parameter footprint. In the end, the key is to understand which part of the model should be trainable and which should remain frozen, and RELISH proves that less is more when the attention architecture is intelligently designed.