Knowledge health and background operations

Your knowledge base is only useful if the assistant can find the right passage when a customer asks. This page explains three tools that keep it that way:

• the embedding model the knowledge base needs before anything can be added;
• background operations — long jobs such as refreshing a website, removing duplicates or learning from past conversations, with progress and a log;
• the weekly Knowledge health check.

You find all of them under Settings → Knowledge base (and the history import under Settings → Training). On the phone: Menü → AI training → Background operations.

The embedding model comes first

Every passage in the knowledge base is turned into an embedding — a list of numbers that captures its meaning — so that a question can be matched with the passages that answer it. That needs an embedding model.

If your workspace has none, you see "Choose an embedding model for the knowledge base" on the Knowledge base screen, in AI settings and in the phone's AI training hub, and:

• new sources cannot be added (the New button is disabled);
• "Use in training" and automatic learning do not store anything;
• the assistant does not search the knowledge base when it replies.

To fix it, open Settings → AI assistant → Model and either:

• use the platform key (the model marketplace) — embeddings then run on ConnectWiz's key and are paid from your AI credit; or
• add an OpenAI credential under Settings → Credentials and pick it as the embedding credential (an Anthropic, Gemini or other chat-only credential has no embedding model here).

The AI on/off switch does not affect the knowledge base: you can build it while the assistant is off. Only automatic learning waits for the assistant to be on (see below).

Background operations

Long knowledge-base jobs run in the background. Each one appears in the Background operations list with:

• its state — Queued, Running, Paused, Done, Failed or Cancelled;
• a progress bar ("245 of 820");
• the process log (the scroll icon): every step, in your language, updating while it runs;
• Pause, Resume and Cancel.

If the server restarts in the middle, the operation continues from where it stopped. Only one operation of a kind runs at a time for the same target — for example, one refresh per website.

Operation	What it does	Started from
Refresh source	Re-reads a web page or website and updates what changed	The refresh icon on a web source, or the source's automatic refresh schedule
Remove duplicates	Removes identical passages and stores site-wide blocks once	Check for duplicates → Remove duplicates
Re-embed knowledge	Embeds everything again with the current embedding model	Re-embed (after changing the embedding model)
Add missing titles	Adds the page title to older passages that lack it, then re-embeds them	Add missing titles
Learn from past conversations	See below	Settings → Training
Knowledge health check	See below	Weekly, automatically, or Check now

Refreshing a website safely

A refresh never empties a source. The new version of the site is built next to the one the assistant is using, and replaces it only when it is complete:

• If the refresh fails (the site is down, a page times out), the previous content stays in use and the source shows "The last refresh failed; the previous content is still in use."
• Unchanged pages cost nothing. A page the server reports as unchanged, or whose text is the same as last time, is kept as it is — no AI credit is spent on it.
• A page that disappears from the site is kept after one refresh and removed after the second refresh in a row, so a temporary glitch does not delete anything.
• Hand-picked sites stay hand-picked. If you chose the pages of a site yourself, a refresh re-reads only those pages; newly found links wait in the page list for you to select.

No duplicates

The same text is stored once:

• adding a passage that already exists in the same scope does not store it again — if it came from another source, the two sources share it, and deleting one of them does not remove it from the other;
• a block that appears on more than 30% of a website's pages (a footer, a contact box) is stored once for the site instead of once per page;
• adding the same website or page twice while the first one is still being read is refused.

For a knowledge base built before these rules, use Check for duplicates: it shows how many passages are duplicates without changing anything. Remove duplicates then cleans them up in the background.

Learning from past conversations

Automatic learning picks up your agents' answers from now on. Learn from past conversations (under Settings → Training) does the same for answers your team gave before:

1. Choose a date range and, if you like, some channels or departments.
2. Press Estimate. You see how many agent answers and conversations would be read, about how many AI calls and tokens it takes, and — if you use the platform key — the estimated cost from your AI credit. Nothing is changed yet.
3. Press Start learning. The run appears below with its progress and log.

Every answer goes through the same steps as automatic learning: it is rewritten as a generic question and an answer without personal details; a general answer is learned for everyone (and waits for your approval when Review learned answers before they are used is on); a personal answer stays with that contact, and is skipped if the contact is not saved with data capture on. An answer the knowledge base already holds is skipped, and so is one that could not be classified.

It runs only while the AI assistant and automatic learning are both on. If either is turned off during a run, the run pauses; resume it when they are on again. When the assistant is off, nothing is learned automatically at all.

Knowledge health

Once a week the Knowledge health check measures your knowledge base and the card at the top of the Knowledge base screen shows the result and its trend. Check now runs it at once.

Measure	What it means	Good value
Retrieval score	Once you have test questions: the share of them whose passage comes back among the top 6 results. Before that: for sample passages, the passage's own heading is asked as a question	High; a drop means the assistant finds the right passage less often
Duplicates	Share of passages that are exact copies	Near 0 — use Remove duplicates
Embedded	Share of passages embedded with the current model	100% — otherwise Re-embed
Failed sources	Sources whose last processing failed	0
Passages without title	Website passages missing their page title	0 — use Add missing titles
Stale pages	Website pages not re-read for 30 days	Low, if the site changes
Suspicious contact facts	Contact blocks with a "phone number" that looks like a year or a date	0 — refresh the source
Average passage length	Characters per passage	Informational

Retrieval evaluation

The Retrieval evaluation card (under Knowledge health) answers the question that matters most: when a customer asks something your knowledge base can answer, does the assistant find the right passage — in every language you serve? It works with test questions written from your own knowledge base.

Run an evaluation

1. Press Run evaluation. You first see an estimate: how many passages will get test questions, how many AI calls and tokens that takes, how many searches are made and — on the platform key — the estimated cost from your AI credit (on your own key, your provider bills it). Nothing is written to your knowledge base.
2. Press Start evaluation. It runs in the background; progress and the log appear under Background operations.

The first run writes the test questions; later runs ask the same questions again (no AI calls, only searches), so the numbers stay comparable. Tick Write new test questions to replace them, for example after a large change to your sources.

Test questions are written in each passage's own language. A Japanese page gets Japanese questions, an Arabic page Arabic ones — the way a customer of that page would ask. For every passage the AI writes one question the passage answers, in a customer's own words, and (for one passage in three) a realistic question your knowledge base does not answer. Passages that belong to a single contact (their personal learned answers) are never used.

"Not in knowledge base" questions are checked against your knowledge base. The AI writes such a question looking at one passage only, so it may ask something another page does answer. Right after writing, every new "not in knowledge base" question is searched once. If a passage comes back that both contains enough of the question's words and is close in meaning (the same two limits the assistant uses, applied together), the question is not kept as "not in knowledge base": a short AI check asks whether that passage answers it — if yes, it becomes an answerable question pointing at that passage; if not, or if the check cannot run, it is removed. Questions you wrote or edited are never changed. The estimate counts these checks, and the card shows how many questions were converted or removed.

Automatic test questions

With Write test questions automatically on (the default), a workspace that has no test questions yet gets them from the weekly Knowledge health check: it queues one evaluation that writes the questions (30 passages) and measures. It happens only when:

• the AI assistant is switched on (if it is switched off during the run, the run stops without spending more);
• an embedding model and an AI chat model are usable;
• on the platform key, your AI credit covers the estimate — otherwise it is skipped and your credit is never taken below zero.

It is tried at most once a week; if it is skipped or fails, the next weekly check decides again. Every decision — queued, or skipped and why — is written to the health check's log under Background operations. A check started with Check now never writes questions. Turn the switch off to run evaluations only by hand; on the phone the setting is shown read-only.

What the numbers mean

Measure	What it means	Good value
Found in top 6	Share of test questions whose passage is among the 6 passages the assistant is given	High
Found first	Share whose passage is the very first result	High
Average rank score (MRR)	1 when the right passage is always first, ½ when it is always second	Close to 1
"Not in knowledge base" answered anyway	Share of the questions your knowledge base cannot answer for which retrieval still returned passages — the assistant would then answer from something unrelated instead of saying it does not know or handing over	Low
Out-of-date questions	Their passage changed or was removed since the questions were written	0 — write new test questions

Below the numbers you see the change since the previous evaluation (What changed lists the questions that got better or worse), the trend, a table by language, and — when your questions suggest one — a distance limit that would give fewer "not in knowledge base" answers. The suggestion is shown only; nothing is changed.

Keyword evidence shows the other limit of the search: a passage that is far in meaning is still used when it contains enough of the question's words — today half of them. The table replays your questions with 34%, 50% and 67% of the words required (the distance limit unchanged) and shows how many passages would be found and how many "not in knowledge base" questions would be answered at each. When another level scores better on your questions, it is suggested — again only shown, never applied. This costs nothing extra: it re-reads the same search results.

Once you have at least five test questions, the weekly Knowledge health check asks them too, so the retrieval score on the health card follows your own questions. Apart from the one automatic first set, the weekly check never writes new questions.

Test questions

Test questions lists every question with its language, whether you edited it, its source and how the latest evaluation did (Found at #2, Not found, Correctly nothing found, 3 passages returned). You can:

• edit a question — an edited question is yours and is kept when the questions are written again;
• delete one that does not make sense (for example a "not in knowledge base" question that your knowledge base does answer elsewhere);
• regenerate the questions (an estimate is shown first).

On the phone, Menü → AI training → Background operations shows the latest numbers read-only; running an evaluation and editing questions are on the web panel.

How the assistant searches

For every question the assistant looks for passages in two independent ways and combines the two rankings:

• by meaning (the embedding), and
• by words, with each language's own rules: Turkish, German, Arabic and many other languages are matched by word stems ("adresiniz" finds "Adres"); Japanese, Chinese, Korean and Thai, which are written without spaces, are matched by pairs of characters; Arabic spelling variants (with or without hamza, ة or ه, diacritics) are treated as the same.

A passage that is far in meaning is used only when it matches at least half of the question's words — one shared word with an unrelated page is no longer enough. Every evaluation also measures 34% and 67% on your own questions (see Keyword evidence above).

For operators

Every command that changes stored knowledge is a dry run by default: it prints what it would do and changes nothing until you add --apply. Long jobs are queued as background operations, so their progress and log are visible in the panel.

php artisan knowledge:cleanup-failed-training [--tenant=acme] [--apply]
php artisan knowledge:dedupe [--tenant=acme] [--apply]
php artisan knowledge:reindex [--tenant=acme] [--source=<id>] [--apply]
php artisan knowledge:recontextualize [--tenant=acme] [--apply]
php artisan knowledge:index-lexical [--tenant=acme] [--apply] [--all]
php artisan knowledge:evaluate --tenant=acme [--questions=30] [--regenerate] [--apply]

• knowledge:evaluate runs the retrieval evaluation for one workspace. Without --apply it prints the estimate — whether questions are written, AI calls (writing and the "not in knowledge base" checks), searches, tokens, whose key pays and the platform-credit cost. --tenant is required.

• knowledge:index-lexical builds the word-search index of passages stored before it existed (until then they are still found, by exact words only). Resumable: stop it at any time and run it again; --all rebuilds every passage.

• knowledge:cleanup-failed-training removes learned answers that were captured automatically while AI was off and could never be embedded (failed, no passages). It lists the counts per workspace.

• knowledge:reindex --apply rebuilds sources with the current pipeline using the same safe refresh: nothing is deleted before the new version is complete, and passages whose text did not change keep their embedding.

The scheduler resumes interrupted operations every five minutes (knowledge:resume-operations), prunes process logs older than 30 days (knowledge:prune-operation-logs) and queues the weekly health check (knowledge:schedule-quality-checks).