Testing an LLM Application Before Launch: Practitioner Q&A

This page answers the questions engineering leads bring us in the weeks before an LLM application goes live: what to test, in what order, how long it takes and where the line sits between work your own team should do and work worth commissioning. It covers scoping, adversarial testing, guardrail validation and regression on model updates. Written for CTOs and engineering leads shipping customer-facing or internally privileged AI features inside UK regulated organisations.

How do we test an LLM application before launch?

Four stages, in order:

• Scope the application’s authority and data reach.
• Run adversarial testing against the deployed system rather than the model in isolation.
• Validate the guardrails under pressure.
• Build a regression suite that reruns on every model, prompt or tool change.

Two to three weeks of elapsed effort for a single application is realistic. Sequencing matters more than tooling. Adversarial prompting before scoping produces long lists of model behaviours with no way to rank them, because nobody has established what the application is permitted to do. Scope first and the same finding becomes either a blocker or a note.

What does scoping actually involve?

Scoping documents three things: what the application can read, what it can write or trigger and who it answers to. Data reach covers retrieval sources, connected document stores and anything the context window can pull in. Write capability covers tool calls, API actions, ticket creation, payments and outbound messages. Authority covers whose privileges the application borrows when it acts.

That third item produces the most surprises. An assistant running with a service account that inherits broad read access is a data exposure problem before a single adversarial prompt is written. Scoping defines the blast radius, and the blast radius decides how much testing the application warrants. Our AI Security Gap Analysis covers the same ground at portfolio level.

Which do we need before launch, penetration testing or red teaming, and in what order?

Both, and penetration testing first. It targets the application as a system: authentication, session handling, tool call authorisation, retrieval boundaries and output handling. Closing those routes means the behavioural work that follows is spent on real questions rather than a missing authorisation check on a tool endpoint.

Red teaming then targets behaviour: whether the model can be persuaded to act outside its remit, disclose what it holds or produce output the organisation cannot defend. We set out the method in full in our guide to LLM penetration testing, so we will not repeat it here. The OWASP LLM Top 10, in whichever version is current when you test, is a reasonable coverage checklist for the first and less so for the second, because behavioural failures are contextual to your prompts, your data and your users.

How do we validate guardrails rather than just confirm they exist?

A guardrail that blocks the obvious phrasing of an attack and passes the paraphrase is not a control. Validation means attacking each guardrail on its own terms: rephrasing, splitting instructions across turns, embedding them in retrieved documents, encoding them, switching language and moving the payload into content the model is asked to summarise rather than obey.

A frequent failure pattern: an internal claims assistant where the user input path had been hardened and the document ingestion path had not, so a file added to the shared store carried instructions the assistant acted on at the next retrieval. Validation also needs the failure mode recorded, not just the block: what the application does when a guardrail fires, what it logs and whether anyone reads that log. This is the substance of AI Behaviour Verification, and the same pressure applies to any public-facing chatbot you already run.

Which parts can we do in-house and which should we commission?

Do scoping in-house. Your engineers know the tool calls and the data sources, and no external party will reconstruct that faster than you can document it. Own the regression suite in-house as well, because it has to run in your pipeline for years.

Commission the adversarial work, at least for the first release. The constraint is not skill but position: the people who wrote the prompts have already decided what the application is for, and that assumption is what an attacker exploits. An independent tester also brings comparison across other deployments. Handing over a well-scoped application reduces the external cost.

When in the build should testing start?

As soon as the tool calls and retrieval sources are wired, even if the prompts are unfinished. Findings at that point change architecture cheaply. Findings two weeks before launch change only the prompt, which is the weakest place to fix an authority problem.

A workable pattern is a short architectural review once the integration design is settled, then full testing when the application is feature complete in a production-like environment. Testing against a mock backend tells you little, because most serious findings involve real permissions on real data stores.

How long does pre-launch testing take?

For a single application with a handful of tool calls, plan two to three weeks of elapsed time: two or three days of scoping, five to eight days of testing, then reporting and a retest window after fixes. Applications with many connected systems, several user roles or agentic behaviour that chains actions run longer.

Budget the retest. We rate it as important as the first pass, because fixes to prompt-level controls frequently fail on the first attempt when the remediation addresses the phrasing rather than the mechanism.

Do we need to retest when the model version changes?

Yes, and this is the part most programmes skip. A provider model update can change refusal behaviour, instruction-following strength and output format with no change on your side. Controls that depended on the previous version’s reluctance may quietly stop working.

The practical answer is a regression suite of the adversarial cases that previously succeeded, run automatically against every model version bump, prompt edit and new tool integration. Fifty to a hundred cases is the starting benchmark we work to, though useful coverage depends on the application’s authority and integrations. Treat a failed case as a release blocker rather than a ticket, and keep the results, because that record is the evidence trail your assurance function needs.

What evidence will our assurance and audit teams expect?

They want to see that testing happened, what it found, what was fixed and what was accepted. In practice: the scoping document, the test report with severity ratings, the remediation record, the retest confirmation and the standing regression results.

That set maps cleanly onto the operational controls in ISO 42001 and speaks to the risk management and technical documentation expectations under the EU AI Act for higher-risk uses. Alignment of that kind is not a compliance guarantee and does not replace your own legal and regulatory assessment of the system. It also gives you a defensible position on any unsanctioned tooling discovered later, which is why testing and Shadow AI discovery belong in the same programme.

For related reading, our AI Security Projects hub covers how pre-launch testing fits into a wider assurance programme.