Choosing a model

EDI is a harness. It supplies the spreadsheet, the prompts and the plumbing; the model is yours to pick, and the answers you get are a property of that choice rather than of this code. The default is ollama, on your own hardware. Everything below is what to run there, and what the hosted alternatives cost you in exchange for being easier.

You probably do not have to configure this

The first time you open a new install, EDI asks which model should answer your questions and shows what it found on the machine. Pick one and that is the end of it: the choice is written next to your workspaces, so it survives a restart, a cleared browser and a different browser. The dialog only appears while nothing has been chosen.

The dropdown at the bottom of the chat box lists what this machine can actually reach, asked fresh each time it opens: the models Ollama has pulled, Claude if the CLI is signed in, and any provider whose key is already in your environment, each under the provider it came from. Pick one and it takes effect on the next question. No restart, no .env.

The choice is written to .edi-data/model.json next to your workspaces and survives a restart. It also overrides the environment variables below, on the grounds that a person clicking a menu is being more deliberate than a file they edited last month. There is a Reset in the same dropdown that forgets it and goes back to the environment.

A provider with no key yet offers Add a key instead of a model list. What you type goes to a file on the machine running the backend and is never sent back to the browser. There is no endpoint that returns a key, only one that reports whether a key exists. That is fine while the disk is yours. Where it is not, set EDI_ALLOW_MODEL_SWITCHING=0, which switches the whole control surface off: otherwise a visitor can repoint your backend and a key typed into the picker lands on your disk rather than theirs.

Providers

openai-compatible is one entry for the long tail: OpenRouter, LM Studio, vLLM, Together, llama.cpp's server. They all speak the OpenAI wire format, so pointing at a different EDI_LLM_BASE_URL is the whole integration. A local server with no auth needs no key.

Two ways to Claude

anthropic and claude reach the same models and differ only in how they authenticate. anthropic wants an API key and bills per token. claude wants neither: it runs the Claude Code CLI already on your PATH, as you, on the subscription you already pay for. If claude auth status says you are signed in, it appears in the picker with no setup at all. EDI never reads those credentials, it runs a binary that is already holding them.

The picker lists the four aliases the CLI accepts: sonnet, opus, haiku, fable, each shown with the model it currently resolves to, so sonnet · claude-sonnet-5 rather than sonnet alone. The alias is what gets sent, which is what keeps you on the current model in that tier; the version is there so you can see which one that is.

There is nothing to read that mapping from. The ids live inside a 250MB binary, and claude models is not a command, it is a prompt. So EDI asks the CLI the only way that works: one one-word completion per alias, in the background, cached against the CLI version and never repeated. Ordinary answers report their own model id too, so the mapping mostly keeps itself current for free. An alias your account cannot reach (a tier off your plan, or one that has hit its spend limit) simply shows without a version.

On timing: about 3.6 seconds per model call once warm, and EDI makes two per question. The first call after five idle minutes is slower, nearer 25 seconds, because each call is a fresh process and pays for the prompt cache again. EDI runs it with the agent tooling disabled, which is what keeps the per-call prefix at ~4,400 tokens instead of ~22,000; the measurements are in backend/claude_code_llm.py.

Configuring

EDI_LLM_PROVIDER=ollama
EDI_LLM_MODEL=llama3.1:8b
EDI_LLM_BASE_URL=http://localhost:11434   # ollama / openai-compatible
EDI_LLM_API_KEY=...                       # or the provider's usual name
EDI_LLM_MAX_TOKENS=8192

Only EDI_LLM_PROVIDER and a key are usually needed; every provider carries a default model. What happens when you set none of it is worth knowing, because it is what a fresh clone does. EDI resolves in this order:

1. a model chosen in the picker, if there is one;
2. EDI_LLM_PROVIDER, which is how a deployment is pinned;
3. GOOGLE_API_KEY on its own, unchanged from before there was a registry, so an existing deployment that sets only that keeps resolving to Gemini;
4. whatever on this machine answers first.

Step 4 is the one that matters on a clean checkout: it is what makes a fresh clone answer questions without being configured first. "First" there is the order the provider table declares them, and whatever it lands on is a starting point; the picker changes it in a click. GET /api/health reports which step produced the answer, under llm_config.source.

Install only the provider you use. The imports in backend/llm_providers.py are lazy, so an uninstalled one costs nothing:

pip install langchain-ollama==0.3.3

What the harness asks of a model

Four things, in the order they are hit. A model that cannot do the earlier ones never reaches the later ones:

1. Follow a system message. Every call is a plain completion. There is no tool calling and no structured-output binding anywhere in this app.
2. Return strict JSON on request. Structured replies are produced by asking for JSON and parsing what comes back. Fences are stripped and a stray object is recovered from surrounding prose, but a model that answers in paragraphs will not get far.
3. Route the question. Before any SQL is written, EDI decides whether your message is about the data or is ordinary conversation. Get that wrong and even a model that writes flawless SQL never gets asked for any. It replies that it does not have the information.
4. Write SQL. The load-bearing one, and the one where a weak model hurts most: bad SQL does not raise an error, it returns a confident wrong number.

Reasoning models

Models that work out loud — DeepSeek-R1, QwQ, Qwen3 in thinking mode, OpenAI's o-series, Claude with extended thinking — work here. Their working is removed before anything reads the reply, because nothing downstream expects it: the chart path parses the reply as JSON, and the read-only SQL check tests that the query starts with SELECT rather than with a paragraph of deliberation.

Servers disagree about where the working goes, so all three shapes are handled:

• A separate field. Ollama returns it as message.thinking, and as reasoning on its OpenAI-compatible route; Anthropic returns a thinking content block. The reply text is already clean.
• Inline tags. llama.cpp's server, LM Studio and vLLM leave <think>…</think> in the message. The block is stripped.
• A closing tag with no opening one. The commonest shape and the least obvious: chat templates usually prefill <think> into the prompt, so the model emits only the </think>. Everything up to it is treated as working.

Unsupported parameters are dropped. OpenAI's o-series and gpt-5 reject a custom temperature, and want max_completion_tokens where other models want max_tokens — sending the wrong one fails the request outright rather than being ignored. Those models are recognised by name and sent what they accept.

They are slower, and rarely worth it here

The work this app asks of a model is short and well-specified: write one SQL query, return one JSON object, answer with one word. On qwen3:4b, a one-line GROUP BY took 713 characters of thinking to produce 54 characters of SQL, and a question that a non-reasoning model of the same size answers in seconds took around two minutes. Those tokens are billed and waited on without changing the answer. An instruction- or code-tuned model of the same size is usually the better choice; reach for reasoning when questions genuinely need several steps.

Budget for the wait, too. A reasoning model on modest hardware can take minutes on a single question, and anything sitting between the browser and the backend needs to allow for that. The bundled dev proxy is set to ten minutes; a reverse proxy of your own will have its own timeout, usually 30 or 60 seconds, and a request cut off there looks to the user exactly like the app being broken.

Running it on your own hardware

Ollama is the one provider where the machine is yours, so it is the one with anything to say about GPUs and memory. It decides all of this for itself and is usually right; these are for when it is not. All are optional, and unset means Ollama chooses.

GET /api/health reports whichever of these are set, under llm_config.runtime. An empty value there means every choice was left to Ollama. Anything your version of the client does not accept is dropped with a line in the log rather than failing to start.

There is no switch here for turning thinking off, on purpose. It is the first thing anyone reaches for and it makes the answer worse rather than faster, for the reason in the note above, and langchain-ollama did not accept the argument at all until 0.3.4, a release that wants a langchain-core this backend cannot import. If you want a model that does not think, use one.

Is it actually on the GPU?

Ollama will quietly fall back to the CPU: an unsupported card, a driver it does not like, a model too big for the memory available. Ask it:

curl http://localhost:11434/api/ps

Compare size_vram against size. Equal means all of it is on the GPU; zero means none of it is, and you are on the CPU whatever the card in the machine. Nothing is loaded at all until the first request, so ask a question first.

Testing yours

Rather than trusting a recommendation, measure the model you actually have:

python backend/check_model.py

It reads the same environment the app does, and reports on each of the four:

  provider   ollama
  model      qwen2.5-coder:7b
  endpoint   http://localhost:11434

  PASS  reachable      2.1s  OK
  PASS  strict JSON    0.5s  {"intent": "filter", "confidence": 0.9}
  PASS  SQL            0.5s  SELECT SUM(revenue) ... -> 1500.5
  PASS  routing        0.4s  3/3 routed correctly

The SQL check runs the query it gets back against a small fixture and compares the number, rather than eyeballing whether the SQL looks plausible.

Picking well

• Hosted models are the safe default. If you want it to just work, use a current model from any provider above.
• Local models vary enormously. Instruction- and code-tuned models do far better here than chat- or roleplay-tuned ones of the same size, because three of the four demands above are format-following rather than conversation.
• Bigger helps most at routing. That step is where small models most often fail in a way that looks like the app being broken.
• Everything is one call. There is no agent loop and no retry storm, so a slower local model costs you latency, not multiplied tokens.

How it works traces exactly where each of these calls happens.