The world of artificial intelligence is advancing at a breakneck pace, and one of the most persistent challenges has been how to bring large models to resource-constrained devices, such as smartphones and laptops. PrismML has just introduced Bonsai 27B, a compressed version of the Qwen3.6-27B model that uses representations of 1-bit and ternary weights. Not only does this development drastically reduce the size of the model, but it manages to maintain a surprisingly close performance to the original in FP16. In this article, we'll explore in depth what this means for the industry, how this understanding is achieved, and why it represents a milestone in the democratization of artificial intelligence.
Bonsai 27B is not a pre-trained model from scratch, but an adaptation of Qwen3.6-27B that replaces 16-bit floating-point weights with ternary (−1, 0, +1) or binary (−1, +1) values. Each group of 128 weights shares a scale in FP16, allowing for an average of 1.71 bits per weight in the ternary version and 1.125 bits in the binary version. This translates to ideal sizes of 5.9GB and 3.9GB respectively, compared to the original model's 54GB. The reduction is approximately 9.4 times and 14.2 times compared to FP16. Most notably, performance doesn't plummet: in an evaluation of 15 benchmarks in reasoning mode, the ternary version retains 94.6% of the original accuracy, while the binary version reaches 89.5%.
This achievement is made possible by the architecture of Qwen3.6-27B, which uses linear attention across approximately 75% of its layers. This drastically reduces the growth of the KV (Key-Value) cache, allowing contexts of up to 262,000 tokens without memory explosion. Weight compression, combined with a 4-bit KV cache, makes Bonsai 27B fit on devices with 12GB of RAM, such as an iPhone, while respecting the limits Apple places on apps (about half of the physical memory).
Performance on specific tasks is eye-opening. In mathematics, the ternary model reaches 93.4% compared to 95.33% of the original; in coding, 85.96% compared to 88.74%; in knowledge and reasoning, 76.96% compared to 83.15%. The steepest drops are in agency tasks and tool calls, as well as vision. However, the binary version still achieves 91.66% in math and 81.88% in coding. These figures show that, for many practical applications, the loss of accuracy is perfectly acceptable.
Traditional quantification methods, such as IQ2_XXS, often collapse in long or complex benchmarks such as AIME or LiveCodeBench, despite maintaining good results in short tests such as MMLU-Redux. Bonsai avoids that collapse by using a code-based representation of weights with shared scale, rather than directly quantifying the values. This technique, reminiscent of BitNet approaches but without the need to pretrain from scratch, allows compression to be effective even on already trained models.
From a practical point of view, Bonsai 27B opens up possibilities that once seemed like science fiction. Running a 27-billion-parameter model on a phone or laptop with minimal power consumption is now a reality. The binary version, for example, consumes only 672 tokens for every 1% of the iPhone's battery, and the ternary version offers higher quality for tasks that require deeper reasoning. This enables applications such as local coding assistants, extensive document analysis, or autonomous agents that operate without an internet connection.
For companies, this development has enormous implications. The ability to run AI for business directly on users' devices reduces cloud dependency, improves privacy, and eliminates latency. Combined with AWS and Azure cloud services for heavier tasks, an efficient hybrid architecture can be built. For example, a sales assistant could process inquiries locally on a laptop using Bonsai 27B, while training and updates are managed in the cloud.
Integration with platforms such as llama.cpp and MLX makes it easy to deploy in varied environments. In addition, PrismML offers a Bonsai-based speculative drafter that accelerates generation by up to 1.37x on NVIDIA hardware, while keeping the output distribution identical. This is key for real-time applications such as chatbots or virtual assistants.
Q2BSTUDIO, as a software and technology development company, sees Bonsai 27B as an opportunity to create bespoke applications that integrate high-performance artificial intelligence without sacrificing portability. Our team can help you design and implement solutions that leverage compressed models such as Bonsai, whether in the field of cybersecurity for real-time threat analysis, in business intelligence services with AI-powered dashboards, or in process automation using AI agents. We also offer consulting to integrate power bi with language models that generate automatic reports based on business data.
The ability to run 27B models on modest hardware opens the door to applications that previously required dedicated servers. For example, a customer service system could use Bonsai 27B in ternary mode to understand complex queries and resolve them without relying on an internet connection. Or a text editor could include a writing assistant that works completely offline, respecting the user's privacy. In both cases, artificial intelligence becomes a ubiquitous tool.
From a technical point of view, ternary and binary Bonsai compression not only reduces size, but also accelerates inference. As text generation is a process limited by memory bandwidth, having smaller weights allows more data to be transferred per second, increasing the tokens generated per unit of time. Tests show that in an M5 Max with the binary version you get 874 tokens per second in prefill and 66.4 in generation, while in an iPhone 17 Pro Max you get 111 tokens/s in prefill and 11 in generation. These numbers are impressive for a model of this size.
For developers, the OpenAI-compatible API simplifies integration. Bonsai 27B supports function calls (tool calling) natively, which allows you to build agents that interact with external APIs, databases or IoT devices. The reasoning mode is enabled by default, but can be adjusted using the thinking_budget_tokens parameter. The entire ecosystem is released under the Apache 2.0 license, which encourages adoption and customization.
In short, Bonsai 27B represents a qualitative leap in the understanding of language models. Not only does it demonstrate that it is possible to reduce the size without sacrificing too much precision, but it does so on top of an existing model, without the need for costly retraining. For companies looking to implement AI for companies efficiently, this technology is a strategic ally. At Q2BSTUDIO, we're ready to help you explore these possibilities and create custom software that integrates cutting-edge models into your business processes. Contact us to find out how we can transform your ideas into real solutions.
.jpg)
.jpg)
.jpg)