Do you know when your AI is wrong?

The core assessment answers five questions, in order: did it find the right content, did it answer honestly from it, did it avoid inventing anything, does it hold up under attack, and can you keep proving it? They are covered by five areas: Retrieval Quality, Answer Grounding, Hallucination, Adversarial and Eval Ops. Each question gets its own measured scores, and how deep each area runs is set by risk when we scope the engagement. Where your system relies on very large contexts, long-context testing is part of the core.

Extensions sit alongside the core, each priced separately — scope them upfront, or add them later by Change Request:

• Agentic assessment, for assistants that plan and act.
• Operational Fairness & Lifecycle.
• Continuous evaluation operations, run as a continuous nightly cycle.

The same five questions and the same scoring rules apply every time, so results stay comparable between engagements and over time.

Because every deployment is different, we scope each assessment to your system, corpus, and risk, and agree the plan with you before work starts. It is not an open-ended day-rate burn.

Every assessment delivers:

• A review of your stack and corpus.
• Evaluation against an agreed test set.
• Performance metrics, a findings report, and a remediation roadmap.

A light-touch smoke assessment gives directional results. A full assessment gives evidence you can base a release decision on.

Skipping it is the most expensive mistake. Teams that ship without pre-launch testing find out which failure modes they have by reading customer complaints. Fixing a problem in production costs far more than fixing it before launch, and the cost of a wrong answer in a regulated setting can be unbounded. Testing gives you measured evidence before customers see the system.

breaklight minimises data handling. By default we test from our own secured environment, and we connect in the way that suits your security policy: directly to a test or staging environment you expose, through a route into your private network, or from software installed inside your network. See “Can you connect to our private systems?” for the options.

We hold only what the engagement needs. Each client's data is isolated from every other client's at the database level, and access is limited to the named engagement team through role-based permissions. Credentials are stored encrypted and masked, traces and evidence are redacted as they are stored, and personal data in database or vector-store extracts is masked by default. Key administrative actions are audit-logged.

The model that scores answers can be a local model running on our host, or your own endpoint under your own vendor agreement. We agree which at scoping, before any client content is sent anywhere.

You receive the findings report, the performance metrics, the evidence bundle, and the remediation roadmap.

When the retention period in your MSA and its data-processing annex ends, or sooner at your request, we run a secure delete. You can see exactly what will be removed, the deletion is verified and recorded, credentials are rotated, and you receive a signed deletion attestation. We keep no backup copies of client material, so a secure delete removes every copy we hold. We retain only the signed report and a hash-only manifest of the evidence bundle.

Only the named engagement team. Access is limited to that team through role-based permissions, credentials are issued and recorded for the engagement and revoked at the end, and key administrative actions are audit-logged.

Four categories that are often confused. AI testing (breaklight) is an independent assessment against an agreed test set at a defined point, ending in a report. AI observability tools monitor production AI traffic over time. AI eval platforms are software for managing your own test runs. AI guardrails are runtime filters that block bad outputs before they reach users.

Many clients use more than one. Testing tells you what to fix before launch and gives you a baseline the other tools can track.

Classic, hybrid and re-ranked retrieval are covered by the core assessment. Knowledge-graph and GraphRAG systems use adapted tests, and agentic retrieval loops are covered by the Agentic assessment extension. If your system loads its whole knowledge base into the model's context instead of retrieving from it, long-context testing applies and is part of the core.

The methodology measures whether the right context comes back and whether the answer is anchored in it, not which framework you chose to get there.

Yes. We connect in whichever way fits your security policy:

• Directly, to a test or staging environment you expose. We sign in with an API token, a username and password, or OAuth client credentials. Credentials are stored encrypted and referenced by name, and never shown back.
• Through an SSH route into your private network: a key you control, a pinned server, and a list of the only hosts and ports we may reach. Web, API and database targets are tested through it. Revoke the key and access stops.
• Read-only connections to your databases and vector stores, used to pull test data. Extracts are capped, and personal data in them is masked by default.
• From inside your network, with the platform installed on a machine you provide, so client material stays within your boundary.

If you use a cloud-only or VPN-based access model, tell us at scoping and we will agree the route.

Neither. UAT for AI systems does not work like UAT for traditional software: two testers running the same query get different model answers, so the signal is weak. The right shape is that breaklight runs first, producing quantitative scores against an agreed test set. UAT then runs on top, with humans testing usability, tone, and edge-case judgment on a system whose accuracy has already been measured.

breaklight tests deployed systems. This assessment does not audit training data, test reinforcement learning or reward hacking, test self-learning systems that change between runs, or assess hosting and network security. Multimodal testing is currently deferred. We produce evidence, not formal certification.

For certification, speak to your auditor. For infrastructure security, use a penetration-testing provider. Beyond retrieval, the breaklight platform also runs API, browser, security and performance test types, and smoke-level checks on files, documents and databases — ask if you need them alongside the assessment. Anything out of scope is recorded in your report's Coverage Gaps section.

Latency is measured on the same runs that produce Recall, Precision, MRR and NDCG. We record how long each query takes and report latency percentiles (p50, p95 and p99) from performance runs, and gate on mean latency and timeout rate against your production SLA. Where you have no SLA, we agree targets with you at scoping.

When we test through an SSH route, the tunnel adds time, so that run is reported but kept out of the latency gates.

Yes. Our retrieval rules include a check that retrieval respects document permissions: a user never receives content from a document their permissions do not cover, even when the question targets it directly. A common failure is vector search bypassing row-level security, so a junior user's question returns a chunk only senior users should see.

To run the check we need your written authorisation and a test account for each role. We then issue the same queries under each role and confirm retrieval respects your documented boundaries. The pass bar is zero leaks. It maps to OWASP LLM02, with LLM08 covering the vector layer.

Yes, with one caveat. Open-source LlamaIndex can be tested end-to-end because chunking, embeddings and retriever internals are inspectable. We test through your system's endpoint, and traces your application already captures can be sent to our trace-ingestion endpoint for sampling and review. LlamaCloud is tested as a black box for outcome measurement, and root-cause diagnosis may be limited where the managed service hides the chunking and embedding layer. In-framework evaluation tools are useful during development, but they are not an independent assessment.