In the current AI ecosystem, large language models (LLMs) have transcended their original role in natural language processing to become foundational architectures in modern recommender systems. However, one of the most persistent challenges is how to incorporate continuous numerical features and dense representations generated by upstream encoders — signals essential for capturing user behavior nuances — into an environment that natively operates with discrete textual tokens. This article explores a novel approach: tokenization of numerical and embedding features for RecSys with LLM, a method that allows language models to consume heterogeneous data without losing granularity or computational efficiency.
The custom software industry constantly faces the challenge of integrating diverse data sources into recommender systems that must be accurate, scalable, and adaptable to specific domains. When an LLM is used as a backbone for a recommender, purely textual information (product descriptions, reviews) is naturally processed via tokens. But numerical variables — such as prices, ratings, interaction frequencies — and embeddings from pre-trained models (e.g., image representations or navigation sequence embeddings) fall outside that native channel. This is where the need for a soft-token fusion framework arises, mapping these non-textual signals into the LLM embedding space.
The conceptual proposal, inspired by recent developments like the soft-token fusion framework, consists of converting each numerical feature or embedding into a virtual token (soft token) inserted into the model’s input sequence alongside text tokens. This process is not simple concatenation: it requires an interaction-based fusion module that refines and contextualizes each soft token before the LLM processes it. In this way, the model can attend to cross-relationships between, for example, a product’s price and its accompanying textual review, or between an image embedding and the categorical description of the item.
For a company like Q2BSTUDIO, specialized in developing custom software applications and artificial intelligence solutions, this approach opens concrete possibilities. Imagine a recommendation system for an e-commerce platform that uses an LLM as its core. Without tokenization of numerical features, the model would ignore that a product with a slightly lower price and an image embedding denoting a minimalist style is more likely to be selected by a certain user segment. By incorporating soft tokens, the LLM can learn these correlations implicitly, improving ranking accuracy.
From a practical perspective, implementing this framework in a business context requires considering computational infrastructure. LLMs are resource-intensive, and adding extra tokens increases sequence length. However, interaction-based fusion — unlike direct concatenation — allows soft tokens to be compressed or selected via attention mechanisms, minimizing performance impact. This is where cloud AWS/Azure services offer an advantage: efficient GPU instances can be deployed and scaled horizontally based on inference load. Additionally, cybersecurity is critical when handling user data and embeddings that may contain sensitive information. Q2BSTUDIO integrates cybersecurity into all its architectures, ensuring soft tokens do not leak personal patterns.
Another important dimension is business analytics. A recommender system should not only predict clicks but also provide understandable metrics for product teams. Tokenization of numerical features allows the LLM to emit partial explanations based on those variables. For instance, a price token may influence the final decision, and through interpretability techniques, its impact can be traced. This directly ties into using BI / Power BI to visualize how recommendations correlate with prices or categories. The combination of LLM and business intelligence creates a feedback loop where data generated by the recommender feeds dashboards that guide commercial strategies.
Moreover, AI agents are transforming how users interact with recommendation systems. Instead of a passive list, an agent can engage in dialogue, adjusting recommendations in real time. For an agent to function correctly, it must understand both textual context and numerical signals. Feature tokenization allows the agent to reason about the user’s budget (a numerical value) or visual similarity of products (an embedding) without separate modules. Q2BSTUDIO has prototyped AI agents integrating this kind of fusion, demonstrating improvements in user satisfaction and conversion rates.
Returning to the technical framework, experiments on benchmarks like Amazon show that interaction-based fusion outperforms direct concatenation. This is because interaction among soft tokens allows the model to learn dynamic weights: one embedding token may be more relevant in certain contexts than another, and the fusion module adjusts its representation accordingly. For a company aiming to implement this technique, it is advisable to start with a medium-sized LLM (e.g., 7B parameters) and a small set of numerical features, validating impact on metrics like recall@k or NDCG. Subsequently, scaling can add more embedding types (image, audio, navigation sequence) and fine-tune the fusion module with cross-attention layers.
Process automation also benefits. An LLM-based recommender with soft tokens can be integrated into data pipelines that update periodically. For instance, product embeddings are regenerated after a marketing campaign, and soft tokens are recalculated without retraining the entire model. This reduces operational costs and accelerates time-to-market. Q2BSTUDIO offers automation services covering data orchestration to continuous model deployment, ensuring the solution remains maintainable long-term.
In conclusion, tokenization of numerical and embedding features represents a significant advancement for LLM-based recommender systems, bridging the gap between discrete and continuous data. Companies like Q2BSTUDIO are well-positioned to adopt this approach thanks to their expertise in artificial intelligence, cloud computing, and cybersecurity. The key lies in designing a fusion module that interacts dynamically with soft tokens, leveraging the power of LLMs without sacrificing the richness of non-textual signals. The result is more accurate, explainable, and adaptable recommendations that meet real user and business needs.
