dAI Pro agents – maintain knowledge, check answers

An AI chatbot is only as good as its knowledge and as reliable as its answers. Neither stays in order by itself. In dAI Pro, agents are the background helpers that look on schedule, report and act when you want them to. We present three of them in detail here – the freshness guard, which keeps the knowledge base current, the quality audit, which grades real conversations, and the golden set, the regression test for answers. You will find every dAI Pro function on the feature page and an introduction with use cases on the dAI Pro product page.

An agent is a task with a schedule. You create it on a collection, choose the type, set a few options and decide when it runs – hourly, daily, weekly, monthly or by your own cron expression. Every agent can also be started by hand at any time.

• Every run is a report – with date, duration, summary and a list of findings. Each finding carries a severity (info, warning, critical), a type and a description with a link to the affected page. Tick off findings you have dealt with.
• Report by e-mail – if you wish, every report goes to an address of your choice, for example the editorial team or support.
• The decision stays with you – agents write nothing into your knowledge base unless you explicitly switch on an automation. What they did is in the report.
• Costs visible – if an agent uses a language model, the cost of the run appears at the end of the report and in the collection’s usage log.
• Language model selectable – per agent you choose one of the LLM configurations released for the collection, for example a model operated in Europe for grading.

Eight agent types are available. We explain the three most important ones for day-to-day operation in detail.

The most common cause of wrong answers is outdated knowledge. A page was rewritten, a price changed, a PDF replaced – and the chatbot still answers with the old state. The freshness guard compares a collection’s documents with their sources, including linked images and files, and reports what has changed, what is no longer reachable and what is new.

To keep this fast even for large websites, it works in three stages.

• Change hints from TYPO3 – when an editor saves a page, the TYPO3 integration reports it to the guard immediately. In the report these findings appear as “change reported”, with no waiting for the next crawl. For records that bypass this signal (news, for example) there is a table mapping.
• Quick check via the sitemap – one fetch instead of hundreds of page requests. Pages with a newer modification date are re-read, new pages are added, pages gone from the sitemap are deactivated. A safeguard kicks in when more than 30 % vanish at once – then it is only reported.
• Deep check – at adjustable intervals (every seventh run, for example) the guard fetches every source individually, checks images byte by byte and finds dead links. A budget per run ensures that even large collections are fully checked in rotation.

With the automation options the guard becomes a complete update pipeline without an editor.

• Re-read changed sources automatically – the stored state is overwritten, linked images and files are updated along. Uploaded files and connector content are only reported.
• Delete content that no longer exists – sources with an HTTP error or a redirect to another page disappear from the knowledge base together with their figures and text chunks.
• Process what changed – newly read and all still unprocessed content is embedded and activated.
• Rebuild the CAG bundle – at the end of the run, per language, when data changed.
• Feed specialist collections automatically – new files (PDF, Office, text) are assigned by AI to the thematically best-fitting collection.

The freshness guard needs no language model for this, only the assignment to specialist collections uses one. For every finding the report shows the affected page, the cause and the triggered action – “automatic re-read triggered” or “re-read already scheduled”.

Whether a chatbot answers well shows not in the test lab but in real conversations. The quality audit is the weekly report on them. It evaluates the chat logs of a period (seven days by default), groups the questions into topics and has an AI judge grade a sample of the answers – against exactly the knowledge excerpts the chatbot had when answering.

• Overview – questions, conversations, resolution rate, cost and cost per resolved request. The resolution rate comes from the conversation outcome. Answered, confirmed, follow-up or action count as resolved, decline, thumbs down, rephrasing, handover or error do not.
• Topics – the questions are embedded and grouped by similarity, a language model names the topics. Per topic you see requests, resolution rate, cost and the trend against the previous period. Topics with a weak resolution rate and fast-growing topics are reported as findings.
• Auto-QA – the judge grades every answer from 1 to 5 in four rubrics (faithfulness to sources, completeness, no unsupported claims, tone and length – your own rubrics are possible), gives a verdict and the likely cause. Retrieval, knowledge base, prompt or model. Answers judged once are remembered.
• Careful in judgement – error messages, actions such as appointment bookings and handovers are not graded but counted. A contradiction only counts with a verbatim quote from the excerpts, and the judge knows the capabilities configured for your chatbot. Flagged answers are labelled “check answer” – the assessment remains a hypothesis that you verify with one click in the log.
• Judge freely selectable – the LLM configuration chosen on the agent acts as the judge, otherwise the collection’s chat model.

Every change to prompt, knowledge base or language model can improve answers – or quietly make them worse. The golden set is the regression test for that. The agent sends stored test questions through the real answer pipeline and checks the answers deterministically.

• Test questions with expectations – per question you define which facts must appear in the answer, which should appear at least once, which source should be cited or that the chatbot must honestly decline. Up to 60 test questions per agent, in the language of your choice.
• Cause instead of gut feeling – when a test question fails, the agent reads from the answer trace why. Source not retrieved, retrieved but cut off before the prompt, in the prompt but not used, knowledge gap, wrong source or an unexpected answer instead of the decline.
• Runs after changes – if you wish, the golden set starts automatically as soon as knowledge base, prompt or LLM configuration have changed. Every run is compared with the previous one – regressions appear as critical findings, improvements as a note.
• Model comparison – a second LLM configuration of the collection answers the same questions. The report sets rate, cost and grades side by side – the basis for a model switch with numbers instead of assumptions.
• Judge LLM – optionally a judge grades every answer from 1 to 5 and decides test questions without expectations. Pass or fail stays deterministic where expectations exist, the AI grade is an additional column.
• Answers in the report – every answer is in the report in full, teased and expandable, with source links and, if the chatbot used images, with thumbnails.
• Test questions from real questions – the FAQ generator provides suggestions that you adopt as test questions with one click. The “golden set” tile on the usage page shows the latest run next to resolution rate and cost.

Both audit agents close their report with concrete recommendations – not “improve quality” but “check source X”, “extend the prompt with …”, “add content on …” or “check retrieval”. Every recommendation names the triggering question, the grade and the reasoning.

The same recommendations accompany the report as a machine-readable JSON block – with action, target, text, source and log number. This lets an AI assistant, for example your team’s coding agent, pick up the suggestions directly. Insert the prompt addition, open the named page or create an editorial task.

Alongside the three audit and maintenance agents, five more types are available.

• Content guard – AI check of the stored content for sanity. Expired deadlines, old prices or versions, contradictions between documents – with verbatim quotes in the report.
• Content gap radar – evaluates the real chat questions and delivers gap topics with example questions and a content recommendation, the feed list for the editors.
• FAQ generator – turns unanswered visitor questions into publication-ready FAQ drafts from the collection knowledge. Where the collection offers nothing, an editorial to-do is created.
• Ticket assistant – watches a data source as a support inbox and drafts an answer from the collection knowledge for every new request, in the request’s language. Nothing is written back into the ticket system.
• Free prompt – a freely worded task on schedule, optionally grounded in the collection’s knowledge, filed as a report.

Agents work with the same language models as your chatbot – the LLM configurations you have released for the collection, on request only models operated in Europe or on sovereign infrastructure. So you decide yourself which data a judge model sees. The sample contains question, answer and the knowledge excerpts.

Every run logs its cost under the purpose “agent”, separately from the chat. Reports stay in your installation, go out by e-mail if you wish and can be deleted at any time. And because agents change nothing without an activated automation, you can switch them on safely, watch, and only then decide how much to leave to them.

Agents are part of dAI Pro – in the TYPO3 extension as well as in the web application. With the demo with your own website you create a collection in a few minutes. We will gladly show you freshness guard, quality audit and golden set on your own content in a first conversation.