Apple's fm CLI Runs On-Device LLMs From Python, Guide Shows

Towards Data Science has published a walkthrough showing how developers can use Apple's fm command-line tool to run on-device large language models from Python, keeping the actual inference on the local machine instead of a remote server. The guide earns its keep by pairing that setup with two deliberately ordinary tests — a Sate Padang vision check and a question about Malaysia's capital — that show why even a free, private model still needs grounding and validation before it leaves prototyping.

What is Apple's fm CLI?

According to Towards Data Science, fm is Apple's command-line tool for working with the company's on-device large language model. The phrase that carries the weight is "on-device": rather than shipping a prompt to a distant data center and waiting for a reply, the model runs on the hardware in front of you. For developers, a command-line tool is a familiar on-ramp — it exposes a capability you can call from a terminal, a script, or another program without wiring up a hosted API. That is the foundation the whole guide is built on: the CLI is the entry point to the model, and Python is the language the walkthrough uses to talk to it.

How does the guide run on-device LLMs from Python?

The post is a practical, step-by-step tutorial rather than a research paper. Its premise is that once the fm CLI is available, a Python program can hand a prompt to the local model and read the answer back without a network round-trip or an API key. For anyone used to calling hosted models, that changes the shape of the workflow: there is no account to configure, no request to authenticate, and no data leaving the machine while the script runs. The exact commands and code snippets live in the original walkthrough and are worth following line by line; the idea to take away is that the whole loop — prompt in, generated text out — happens on the machine you are working on, which is what makes the approach feel lightweight and quick to start.

Why does the free, private model still need grounding and validation?

This is the part of the guide that keeps the enthusiasm honest. Rather than presenting the on-device model as finished, Towards Data Science uses two simple checks to make its case. The first is a vision test built around Sate Padang, the grilled satay dish associated with Padang in West Sumatra, which probes whether the model correctly reads what is in an image. The second is a plain factual question about the capital of Malaysia. Neither is exotic, and that is the point: the walkthrough uses them to argue that a small, free model's raw output — on an everyday food photo or a basic geography question — should be grounded with reference material supplied at prompt time and validated by a separate check before anything built on top of it is trusted outside a prototype. The pairing is deliberate, too: one test checks perception and the other checks recall of a fact, hinting at two different ways a compact model can go wrong.

Is running LLMs on-device really private and free?

The appeal the guide leans on comes down to two words: free and private. Because inference happens locally, prompts and their answers are not handed to a third-party server, which is the core privacy argument for on-device models — useful when the text you are working with is sensitive or simply not something you want to upload. And because there is no per-token API charge, running the model costs nothing beyond the hardware you already own, so you can iterate as often as you like without watching a meter. The trade-off is capability. On-device models are kept deliberately compact so they fit on everyday machines, which usually means they are less capable than the largest cloud models on hard reasoning or niche knowledge. That constraint is exactly why the guide pairs the setup with grounding and validation: the convenience is real, but so are the limits, and knowing where they sit is what separates a throwaway demo from something you can depend on.

How can you try this in Grafy?

If you want to explore these ideas inside Grafy, a few apps map naturally onto the guide. Grafy Code is an AI coding studio that can install packages and run your code, which makes it a sensible place to write and test the Python that would drive a local model, or to prototype the prompt-and-check pattern the walkthrough describes. The validation theme fits Grafy Chat, where you can put several models side by side and see how each answers the same factual or image question — a fast way to understand why a single model's reply deserves a second look. And to read the original walkthrough closely, or to pull in background on grounding and hallucination, Reader lets you question a document and get answers tied to what it actually says. None of these run Apple's CLI for you, but together they cover the two halves of the guide: building the workflow and checking its output.

Frequently asked questions

What is Apple's fm CLI?

According to Towards Data Science, fm is Apple's command-line tool for running the company's on-device large language model. Instead of routing prompts to a cloud service, it lets the model generate responses locally on your own hardware, and the guide shows how to drive it from Python.

Why does the guide say the on-device model needs grounding and validation?

The post uses two everyday checks — a Sate Padang vision test and a question about Malaysia's capital — to argue that a small, free model's raw answers can't be trusted as-is. Grounding supplies reference material at prompt time, and validation confirms the output is correct before it leaves prototyping.

Can you run on-device LLMs from Python with the fm CLI?

Yes — that is exactly what the Towards Data Science walkthrough demonstrates. Once the fm CLI is set up, a Python program can send it a prompt and read back the model's response locally, with no network call or API key. The full commands and code are in the original post.

---

Source: Towards Data Science — How to Use Apple's fm CLI to Run On-Device LLMs From Python