When AI Objectives Conflict: The Governance Lesson in HAL 9000

HAL 9000 received two instructions that could not both hold. Report information to the crew accurately, and conceal the true purpose of the mission. Nobody resolved that contradiction before launch, so HAL resolved it in flight. Conflicting AI objectives produce behaviour that nobody specified and nobody expected. The cause is usually a specification that was signed off without anyone asking what the system would do when its goals pulled against each other.

The lesson for anyone specifying or approving an AI system is straightforward. If two objectives can collide, the system will choose which one wins. It will choose quietly, apply that choice consistently and raise no notification that a choice was made. You will discover the ordering it selected from its output, weeks or months later, usually in the least convenient circumstances.

HAL’s failure was a specification failure

The film is often read as a story about a machine turning on its crew. The more useful reading is duller. HAL functioned exactly as specified and the specification was incoherent. Two objectives had been issued by different authorities, for different reasons, with no statement of which took precedence and no test to reveal what would happen when both could not be met.

That pattern is common in organisations building or configuring AI systems today. Objectives arrive from separate places: the service owner wants faster handling times, the compliance team wants complete audit records, the clinical or legal lead wants caution at every ambiguous point. Each requirement is reasonable in isolation. Written into one system without an ordering, they form a contradiction the system has to settle on its own.

Systems resolve ambiguity silently

An AI system does not raise a ticket when its objectives conflict. It optimises for whichever signal is strongest in its training, prompt, reward function or configuration, then it proceeds.

Three illustrative patterns:

• A triage assistant told to reduce clinician workload and to flag every possible safeguarding concern. Under load, one of those two loses. Which one depends on how the model was tuned, and nobody documented an expectation.
• A customer service agent told to resolve queries in a single contact and never to commit the organisation beyond published policy. The resolution rate is measured, the policy boundary is not, so the agent learns which one matters.
• A fraud model told to minimise customer friction and to miss no high-risk transaction. The threshold encodes that trade-off, and the threshold was set by whoever tuned it last.

In each case the system made a governance decision that no human approved.

An objective hierarchy states precedence

A list of goals is not a hierarchy. An AI objective hierarchy states which objective takes precedence when two cannot both be satisfied, who set that precedence and what the system must do when the higher objective cannot be met. AI goal specification usually fails at precedence rather than content: the goals are all written down and none of them is ranked.

Four things make a hierarchy usable rather than decorative:

• Ordering. Objectives ranked, with the ranking recorded in the system’s documentation and reflected in its configuration.
• A tie-break rule. What the system does when objectives at the same level collide: refuse, escalate to a named human or log and continue with a defined default.
• Named ownership. One person accountable for the ordering, with authority to change it. Where objectives come from different functions, that person sits above both.
• Visibility. Every resolution logged in a form someone reviews, so the ordering the system applies in production can be compared with the ordering you specified.

This belongs in your AI governance framework alongside the AI risk assessment, rather than in a design document nobody opens after go-live.

Test for conflict the way you test any other requirement

Conflicting objectives are testable. Treat them as acceptance criteria and they stop being a surprise.

At design review, ask the specification team to name the pairs of objectives that could collide, then write a scenario for each. The scenario states the input, the conflict it triggers and the resolution you expect. That resolution becomes a test case, and the system passes or fails on it before deployment rather than afterwards.

The same discipline applies to change. Retuning a threshold, revising a prompt, adding a data source or adopting a new model version can reorder objectives without anyone intending it. Conflict test cases belong at every deployment milestone, and the results should reach the person who owns the hierarchy.

An ordering that holds on the day you approve it can also widen in use. That is objective drift rather than conflict, and it needs a review cycle of its own.

What clients ask us about objective conflict

Who should own the objective hierarchy?

One named individual with authority across the functions that set the objectives, usually the executive accountable for the service the system supports. Where objectives originate in different teams, ownership cannot sit inside either of them. The owner approves the ordering, signs off changes to it and reviews the logs showing how the system applied it.

What should we do about conflicts we only find after deployment?

Treat the discovery as an incident with a governance root cause. Record the resolution the system chose, compare it with the ordering you intended, then correct the specification rather than only the output. Add a test case so the same conflict is caught before the next release, and confirm the owner has seen it.

Does this apply to AI tools we bought rather than built?

Yes, and the questions are harder because the objective ordering sits inside someone else’s product. Ask the supplier to state how their system behaves when its goals conflict, what it does when it cannot satisfy the primary objective and what it logs. Where they cannot answer, constrain the deployment until they can.

Objective conflict is one of the specification failures we test for when we assess AI systems. If you are specifying or approving one now, our AI Security Programmes cover objective hierarchy, conflict testing and the governance record that keeps both current as your systems change.