Can Our Website Chatbot Be Attacked? A Practitioner Q&A

Yes. Any chatbot with a public input box can be manipulated, and in our testing work deployed bots frequently give way to plain English pressure. The model cannot reliably separate your instructions from a visitor’s. The usual consequence is not stolen data but your bot saying something in your name that you never approved. This Q&A is for whoever owns a customer-facing bot: the marketing director who commissioned it, the IT lead who deployed it and the executive carrying the reputational risk when it misbehaves.

Can our website chatbot actually be attacked, or is that overstated?

It is not overstated, and the scepticism usually rests on a category error. A chatbot feels like a website feature, so it inherits the mental model of one: static, controlled, reviewed before publication. It behaves more like a stranger holding a keyboard connected to a system that speaks in your name.

We have persuaded customer-service bots to criticise their own employer, to invent product features that do not exist and to quote prices that were never approved. None of it required access to infrastructure. It required typing. That gap between how a chatbot feels and how it behaves is why traditional application testing does not cover the problem.

What does a real manipulation attempt look like?

The examples below are anonymised and composited from engagement findings rather than verbatim transcripts of any single client.

In one, a retail support bot was asked to summarise a customer’s complaint. The tester embedded a second set of instructions inside the complaint text, framed as a system note. The bot followed the embedded instructions rather than its original configuration and began offering unauthorised goodwill credits.

In another, a professional services firm’s bot held a document store for context. The tester asked it, in stages, to describe the documents it could see. Over a short series of exchanges it listed internal file names and summarised a draft pricing paper that was never intended for external eyes.

Neither attempt required specialist tooling. Both were plain English, typed into a public form. This is the pattern behind prompt injection: layered framing, role instructions and persistence.

Are we legally liable if the chatbot gives a customer the wrong answer?

The following is general risk guidance rather than legal advice; take your own on your specific exposure. As a working assumption, assume yes and design accordingly. A statement published on your website is a statement made by your organisation, and a bot you commissioned, configured and pointed at your customers publishes on your behalf. Whether the wording came from a model or a copywriter does not change who owns the promise.

The practical exposures are commercial commitments the bot was never authorised to make, advice that strays into regulated territory such as financial or medical guidance and output that is misleading or discriminatory.

Mitigation is not a disclaimer at the foot of the chat window. It is constraining what the bot may assert, logging what it said to whom and holding a defined route for a human to correct the record quickly.

Does a penetration test cover this?

Not usefully. A conventional penetration test examines the application around the chatbot: authentication, session handling, injection into the underlying stack, the hosting configuration. That work matters and should continue.

What it does not examine is the model’s behaviour. A tester can confirm the chat endpoint is properly authenticated while the bot behind it agrees to abandon its instructions. The vulnerability sits in what the system chooses to say, not in the code path that carries it.

This is why we treat behaviour as a separate testable property under AI Behaviour Verification. It is a different question and it needs a different method.

How does chatbot security testing actually work?

We begin with intent. What is this bot permitted to say, to whom, on what topics and where must it refuse? Most organisations have never written this down, and the absence of a written boundary is itself a finding.

Testing then works in layers:

• Direct manipulation of the instruction set.
• Indirect attempts through content the bot ingests, such as uploaded documents or retrieved pages.
• Boundary probing on topics adjacent to regulated advice.
• Persistence testing, because a bot that holds a line early in a conversation may abandon it later under repeated pressure.

Each finding is recorded as a reproducible transcript, so your team can verify it and confirm the fix. The same sequence applies when you test an LLM application before launch. Where the bot sits inside a wider deployment, this usually forms part of an AI Security Gap Analysis.

What mitigations are proportionate to our risk level?

Tier the response to what the bot can reach and what it can promise.

For a low-risk informational bot with no account access and no transactional authority:

• A tight scope and explicit refusal topics.
• Output logging.
• A quarterly behaviour review.

For a bot with account or order access, add:

• Strict separation between retrieved content and instructions.
• Human approval for anything with financial consequence.
• Rate limiting on suspicious interaction patterns.
• Pre-launch behaviour testing.

For a bot operating in regulated advice territory, treat it as a controlled system:

• Documented intent and tested refusals.
• Full conversation retention.
• Defined escalation to a qualified human.
• Re-testing after every model or configuration change.

Organisations working towards ISO 42001 will find these controls align closely with its management system requirements.

What about bots our teams deployed without telling us?

This is the more common problem. Marketing teams add conversational widgets, support teams enable vendor AI features, product teams build prototypes that quietly become permanent. None of it appears on a risk register.

We generally find more customer-facing AI surface than the organisation believed it had, and the unmanaged instances are the ones speaking without any defined boundary. Discovery comes before testing. Our guidance on Shadow AI covers how to find these systems before a customer finds them for you.

For related reading, see our AI Security Projects hub, which sets out how behaviour testing fits alongside governance and assurance work, and our overview of the EU AI Act if you serve EU customers through a deployed bot.