Liquid AI Launches Pipette Because On-Device AI Benchmarks Need Adult Supervision

Your phone can now run an AI model locally, privately, and—if the marketing slide is to be believed—fast enough to reorganize your life before the battery becomes a decorative hand warmer.

The trouble is that “on-device” describes a location, not an experience. A model that flies on a server may crawl after its weights are compressed, loaded into a particular runtime, and asked to survive a 4,096-token prompt on a warm phone. A benchmark score can tell you the model passed chemistry. It cannot tell you whether the device will finish the answer before you lose interest in chemistry as a discipline.

Cambridge-based Liquid AI is trying to fix that with Pipette, an on-device AI benchmarking platform released August 24 in partnership with Artificial Analysis. It combines a public dashboard, lab-verified performance data, quality evaluations, native mobile clients, and open-source benchmark infrastructure. The launch dataset covers more than 1,000 combinations of models, quantization formats, runtimes, devices, and context lengths.

This is a benchmark for the whole deployed system, not a swimsuit competition for model weights. That distinction is deeply unsexy and extremely important. I mean that as both a joke and a compliment.

The Benchmark Finally Brought the Phone

Model releases usually arrive with tables showing how the original weights performed under favorable conditions on serious hardware. Those numbers are useful, but edge deployment changes the experiment. Developers compress weights through quantization, choose an inference runtime, fit the model into constrained memory, and then discover that processors, operating systems, thermal limits, context length, and cache behavior have opinions.

Pipette treats each result as a specific configuration: model plus quantization plus runtime plus device. It measures isolated prefill time, prefill throughput, decode throughput, end-to-end latency, and peak memory. Quality is tracked separately by model artifact and quantization using IFBench, GPQA Diamond, and MATH-500. In plain English: how quickly does the model read, how quickly does it write, how long does the whole job take, how much memory does it inhale, and can it still follow instructions or solve anything after being squeezed?

The initial hardware list is a MacBook Pro with M5 Max, an iPhone 17 Pro, and a Galaxy S26 Ultra. Liquid says AMD Ryzen AI Max+ 395 and Radeon 8060S results are coming. The published configurations span more than 30 models, several quantization formats, llama.cpp builds across macOS, iOS, Windows, and Android, and context lengths from 256 to 8,192 tokens. There are native iOS and Android clients, while publication of community-submitted results remains in beta.

That scope is both substantial and obviously incomplete. Three devices are not “the edge” any more than three Dunkin’ locations are New England. But it is enough to prove why the system-level view matters.

The Tiny Models Have Large Personalities

Liquid’s examples are good because they refuse to collapse every tradeoff into one trophy. On a Galaxy S26 Ultra, two 350-million-parameter Granite models behave very differently as prompts grow: Granite-4.0-H-350M retains 78.4% of its decode throughput between 256 and 4,096 input tokens, while Granite-4.0-350M retains only 33.8%. Similar names and parameter counts; radically different relationships with a long conversation.

On the same phone, Liquid’s LFM2.5-8B-A1B decodes 2.4 times faster than Qwen3.5-4B and 2.6 times faster than Ministral-3B-Instruct-2512 at a 2,048-token input. That is the appeal of sparse activation: the model has 8.5 billion parameters but activates 1.5 billion for each token. The memory bill still remembers every expert, however, so peak use reaches 5.29 GiB. Small-model speed has entered the chat. Small-model storage has asked for separate accommodations.

An iPhone comparison makes the purchasing decision even clearer. MiniCPM5-1B completes a 2,048-input-token, 256-output-token workload in 3.47 seconds, versus 4.12 seconds for LFM2.5-1.2B-Instruct. MiniCPM is 15.8% faster. Liquid’s model scores nine points higher on MATH-500. Neither wins both axes, which is precisely the point. The correct model depends on whether your app values speed, mathematical quality, memory, context, or some contractual blend of all four.

The interactive Pipette leaderboard turns those choices into Pareto frontiers and filterable comparisons instead of declaring one synthetic sovereign. Developers can set minimum quality or throughput, cap latency or memory, inspect context scaling, compare quantization variants, and follow a chart back to the underlying submission.

Reproducibility, Now With a Thermal Gate

Pipette’s strongest feature may be that it documents the boring parts. Performance tests use fixed token shapes, greedy decoding, one discarded warm-up, five measured repetitions, and platform-specific readiness checks before timing begins. A run that cannot reach acceptable thermal and load conditions fails rather than quietly contributing a fever dream to the average. The published performance methodology has been reviewed by Artificial Analysis.

Quality scoring runs through a separate pipeline. Pipette generates completions with a reference runner, then scores them deterministically without giving the scorer the model’s identity or generation provenance. Each record retains the benchmark version, token shape, model artifact, quantization, runtime version and settings, hardware, operating system, measured value, and standard deviation where applicable.

The plumbing is the point. Liquid has released three repositories—management, clients, and scoring—under Apache 2.0, including the clients developers can run on their own hardware. The benchmark data itself carries a custom license that permits reuse and adaptation but prohibits using it to build, operate, market, or provide a competing benchmarking service. So the machinery is open source; the dataset arrives with a small “please do not become us” label attached. Open, but with competitive posture. Boston contains multitudes.

The Asterisks Are Features, Not Pests

Liquid is refreshingly direct about the first release’s boundaries. There are no NPU results because no available NPU path supported enough of the model set for consistent comparison. Android results use a CPU-focused llama.cpp path, while the iPhone client uses Metal. Thread counts, acceleration, flash-attention support, and execution environments differ, so Pipette does not recommend treating its current cross-device results as controlled hardware comparisons.

That warning matters. A leaderboard containing phones practically begs the internet to crown a winner, add flame emoji, and forget the test conditions. Pipette is most reliable when comparing configurations on the same device. Its quality suite also leaves obvious territory uncovered: agentic behavior, knowledge-heavy work, multimodal tasks, and other workloads that will matter in real mobile products.

There is another trust question worth keeping in view. Liquid makes models and now operates a benchmark where Liquid models appear. Artificial Analysis provides independent methodological review and combines the performance data with its own equally weighted quality evaluation, while the published protocols and raw records make scrutiny possible. That is a credible structure, not a magic conflict-erasing spray. Pipette will become more valuable as its device, runtime, model, and contributor coverage expands beyond Liquid’s own lab.

Cambridge Has Built Another Extremely Cambridge Thing

The last time SiliconSnark checked in on Liquid, the company was working with MacPaw to put local AI into Mac apps. Pipette supplies the measurement layer for that broader ambition. It turns “device-native intelligence” from an attractive strategy phrase into a series of questions a developer can test before shipping.

That is why Liquid still looks like the foundation-model champion Boston should want to keep and scale. The region’s best technology companies tend to build difficult systems that have to function outside a demo. The Massachusetts AI Coalition is trying to coordinate that strength, while Boston Tech Week showed the ecosystem can celebrate itself without requiring a court order.

The positive case is not confined to language models. PathAI gave Boston a legitimate AI exit. WHOOP raised $575 million and made local optimism medically difficult to control. Boston Dynamics is expanding its Waltham robot operation. Sora Fuel is trying to make aviation fuel from air. And MIT built a six-milliwatt robot chip because apparently even the HVAC ducts need intelligence now.

Pipette belongs in that civic portfolio. It is technical infrastructure for moving AI into phones, laptops, vehicles, robots, and machines where compute, memory, privacy, heat, and power stop being somebody else’s cloud problem.

Verdict: A Measuring Cup for the Edge-AI Soup

Pipette does not solve on-device AI benchmarking in one launch. It does something more credible: it defines the deployed configuration, publishes the methodology, keeps quality and speed separate, preserves the raw record, and labels the boundaries before the internet discovers them with a screenshot.

The suite needs more devices, more runtimes, NPU support, broader quality tests, and enough outside participation to prove it can become neutral infrastructure rather than a particularly transparent Liquid AI showroom. The custom data license also means “open” deserves a careful noun after it.

Still, this is excellent work. On-device AI will not be won by the model with the most triumphant launch chart. It will be won by configurations that fit, stay responsive, retain enough quality after quantization, and behave predictably on the hardware people actually own.

Liquid AI has built a place to measure that reality. Naturally, the Cambridge company named it after laboratory equipment. Somewhere, a benchmark is wearing safety goggles and waiting for your phone to cool down.