Notes from running language models locally

A useful experiment

My experiments with language models on an Apple-Silicon Mac mini explored whether local inference could help with document work. The useful question was not simply whether a model would run, but whether the whole arrangement was suitable for a bounded task.

Record a test you can repeat

A useful test records the setup and uses a representative document task. Response time matters alongside output quality, memory use and the work needed to operate the workflow. Record the following:

• Model version, exact model file or checksum, and quantisation.
• Hardware, memory, software versions and inference backend.
• Test documents, prompt, context length and generation settings.
• Prompt-processing time, time to first token and generation speed, recorded separately over repeated runs.
• Memory use and, if measured, power consumption with the measurement method.
• Output checked against source documents, including errors and failed runs.

Keep the historical context

My original note, published on 17 May 2025, remains available as a historical account. The article provides no exact model files or measurement logs for its performance table. For new work, use a documented test and check the actual data flows, including any external services.

Historical Python configuration example from the original note; preserved for context, not a tested installation recipe. The hostname, HTTP endpoint, model identifier and environment-variable names are examples tied to that setup; compatibility, authentication and network exposure need checking before reuse. The API-key value is a published dummy placeholder, not a credential.

import os
os.environ["OPENAI_API_BASE"]   = "http://brainbox:1234/v1"
os.environ["OPENAI_MODEL_NAME"] = "openai/qwen2.5-coder-7b-instruct"
os.environ["OPENAI_API_KEY"]    = "lmstudio_placeholder"  # dummy key

Assess an AI workflow →

All articles →