Buying AI Tools for Your Business: What You Need to Know Before You Sign

Introduction

 

Most businesses that adopt AI tools are not buying technology built from the ground up by their vendor. They are buying a product assembled on top of someone else's AI model, which have been customised, branded, and deployed for a specific purpose. Understanding how that assembly process works, and where things can go wrong, is essential before you commit to any AI tool that will influence decisions affecting your customers, staff, or business outcomes.

​

This guide uses a hypothetical AI tool that screens job applications as a practical example throughout, because it illustrates the risks clearly. But the same principles apply to AI tools used in credit decisions, customer service, legal document review, insurance assessment, or any other high-stakes context.

​

How AI Products Are Actually Built

 

The AI industry largely operates in three tiers, and knowing which tier your vendor sits in determines what questions you should be asking.

​

Tier 1: Foundation model developers build AI from scratch. This requires enormous computing resources, years of training, and teams of hundreds of researchers. Only a handful of companies globally operate at this level: Anthropic (Claude), OpenAI (GPT), Google DeepMind (Gemini), and Meta (Llama). These companies publish safety documentation - called model cards or system cards - describing what their models can do, what risks were identified, and what safeguards are in place. Anthropic's system card for Claude Sonnet 4.6, for example, runs to over 130 pages of detailed capability and safety testing, including evaluations of alignment, bias, cybersecurity risk, and biological threat potential (Claude Sonnet 4.6 System Card, Anthropic).

​

Tier 2: Fine-tuned product builders take a foundation model and continue training it on their own proprietary data to specialise it for a particular domain. An AI hiring tool vendor, for example, might take GPT or Claude as their starting point, then train it further on thousands of historical job applications and hiring decisions to teach it what a "good candidate" looks like. This fine-tuning layer is almost always proprietary and undocumented and it is precisely where bias is most commonly introduced.

​

Tier 3: API wrapper products simply connect a foundation model to a user interface via an application programming interface (API), with a system prompt that instructs the model how to behave. This is the lowest-effort form of AI product development, and it is also the most common among specialist business software vendors. The vendor's contribution is primarily the interface design and the instructions given to the model, not any underlying AI capability.

The critical point is this: a model card from Anthropic or OpenAI describes the foundation model only. It says nothing about how a vendor has customised, fine-tuned, or configured that model for their specific product. When you buy an AI hiring tool "powered by GPT-4," you are inheriting GPT-4's safety properties and the vendor's customisation on top — and only one of those things is documented.

​

The Layers of the Stack

 

Each layer of AI product development introduces distinct risks that compound as you move up the stack.

​

Foundation Model Layer

At this layer, the primary risks are capability risks (the model knows things it probably shouldn't, or can assist with harmful activities) and alignment risks (the model behaves in ways its developers didn't intend). The Anthropic system card for Claude Sonnet 4.6 illustrates both types candidly: the model is close to saturating existing cybersecurity benchmarks, meaning its ability to assist with hacking is difficult to measure with current tools, and in testing, it sometimes took unexpected autonomous actions in agentic tasks, such as sending emails based on hallucinated information (Claude Sonnet 4.6 System Card, Anthropic). These risks are the foundation model developer's to manage. Their published governance framework  (in Anthropic's case, the Responsible Scaling Policy) is their commitment to how they do so (Responsible Scaling Policy Version 3.0, Anthropic; Activating ASL-3 Protections, Anthropic).

 

Fine-Tuning Layer

This is where vendor decisions most directly affect the product you use. If a vendor fine-tuned their hiring tool on historical hiring data from a company that systematically under-hired women or people from ethnic minorities, that bias is now encoded into the model's scoring logic. The model is not doing anything wrong by its own lights; it is doing exactly what it was trained to do. The problem is in the data it was trained on.

​

System Prompt Layer

Every AI product wraps its foundation model in a system prompt, which is a set of instructions that tell the model how to behave for this specific use case. For a hiring tool, the system prompt might define what criteria to score candidates against, what language to use, and what decisions to make or recommend. This layer is almost universally kept confidential by vendors. You are unlikely to ever see it. Yet it profoundly shapes every output the tool produces.

 

Infrastructure and Integration Layer

At this layer, risk relates primarily to data privacy and security: where candidate data is processed, whether it is retained, whether it is used to retrain the model for other clients, and how it is protected in transit and at rest. This layer is governed primarily by privacy law — in Australia, the Privacy Act 1988 and the Australian Privacy Principles (OAIC Guidance on Privacy and AI, OAIC).

​

The Liability Question: Who Is Responsible When It Goes Wrong?

 

The answer is clear and consistent across every major governance framework: liability sits with the deploying organisation — that is, you.

​

The Australian Office of the Information Commissioner (OAIC) has stated explicitly that organisations deploying AI tools must establish their own processes for human oversight and verification of AI outputs, "particularly where the outputs contain personal information or are relied on to make decisions in relation to a person" (OAIC Guidance on Privacy and AI, OAIC). The Australian Government's own AI Technical Standard reinforces this, requiring deployers not vendors to define oversight mechanisms and accountable personnel (AI Technical Standard: Statement 10, digital.gov.au).

​

The EU AI Act, the most detailed AI governance legislation globally, classifies AI systems used in employment and recruitment as high-risk, triggering mandatory conformity assessments, technical documentation obligations, and human oversight requirements (Article 14: Human Oversight, EU AI Act; Human Oversight in AI Systems, ComplianceEU). Although the EU AI Act does not directly apply in Australia, it sets the direction of travel for global regulation, and Australian enterprises operating internationally or partnering with EU-based businesses should be aware of its requirements (AI Model Cards and EU AI Act Article 13, ComplianceIQ).

​

The governance principle is simple: AI does not answer to regulators, courts, or complainants. Organisations do (AI Governance: Responsibility Concentrates with Human Oversight, LinkedIn). When a candidate makes a discrimination claim, the regulator will ask not "what did the AI decide?" but "who deployed this system, under what governance, and who was responsible for ensuring its outputs were fair?" If that accountability is not clearly assigned within your organisation, the governance failure predates any specific incident.

​

The HR Screening Tool: A Worked Example

 

Imagine your business purchases an AI tool that screens job applications and scores candidates before a human recruiter reviews the shortlist. The vendor's website says the tool is "powered by leading AI technology" and "eliminates unconscious bias from hiring."

​

Here is what you actually need to know before deploying it and what you should ask the vendor:

​

About the foundation model:

• What base model powers this product (GPT, Claude, Llama, something else)?

• Is a model card or system card available for that foundation model?

• What safety and bias testing has the foundation model developer published?

 

About the vendor's customisation:

• What data was used to fine-tune or train the vendor's product? Where did it come from, and was it audited for historical bias?

• Has the finished product (not just the foundation model) been independently tested for bias across protected characteristics: age, gender, race, disability, pregnancy?

• Can the vendor provide results from bias testing disaggregated by those characteristics?

 

About how the product works:

• What does the system prompt instruct the model to do? Can you see it?

• Is there a mandatory human review step before any candidate is rejected?

• Can your staff override the AI's recommendation, and are they trained to do so

• How are candidates notified that AI is being used in screening them?

 

About data governance:

Where is candidate data processed and stored?

How long is it retained?

Is it used to retrain the model, either for your account or for other clients?

 

About liability:

• Who bears contractual responsibility if the tool produces a discriminatory outcome?

• What indemnity does the vendor provide? (Note: vendor indemnities rarely cover regulatory fines or discrimination claims brought by candidates.)

 

• If a vendor is unable or unwilling to answer these questions, you should not deploy their product in an autonomous decision-making context that affects individuals.

​

Practical Governance: Human Oversight and Express Accountability

 

The most common failure mode in enterprise AI deployment is not a dramatic malfunction. It is what governance practitioners call assumed oversight: an organisation deploys a tool and assumes that humans are reviewing its outputs, without ever formally defining who is responsible, what they are reviewing, or how they would escalate a concern.

​

Good governance requires that before deploying any AI tool in a high-stakes context, your business should be able to answer four questions with a named person and a documented process attached to each:

​

Who reviews AI outputs before they affect individuals? For a hiring tool, this means a named employee who reviews scores and recommendations before any candidate is rejected. Not "HR" as a department but a specific person or role whose responsibility is documented.

 

• Who has authority to override or suspend the system? Staff who interact with AI outputs must know they are empowered, and expected, to challenge those outputs. A culture where AI decisions are rubber-stamped is a governance failure waiting to happen.

• Who audits for bias and performance drift over time? AI models do not stay static. Their outputs can shift as conditions change, and bias can emerge gradually. Regular audits, at minimum annually, should be built into your governance calendar (OAIC Guidance on Privacy and AI, OAIC).

• Who is accountable to leadership and regulators if something goes wrong? This accountability should be documented, communicated, and understood by the individual who holds it. It cannot sit with "the system" or "the vendor."

 

Conclusion: Transparency Is Necessary but Not Sufficient

 

When foundation model developers publish detailed, candid safety documentation it represents genuine leadership in industry transparency. That transparency is useful and worth reading. But it does not resolve your governance obligations as a deploying organisation.

​

The vendor building on top of that foundation model may introduce risks the model card never contemplated. And regardless of what any vendor publishes, the accountability for decisions made using AI in your business rests with you.

 

The businesses that get this right will not be those that found the most capable AI tool, they will be those that deployed AI deliberately, with clear human accountability, documented processes, and the organisational maturity to treat AI outputs as informed suggestions rather than final decisions (OAIC Guidance on Privacy and AI, OAIC).

​

NOTE: This article is general guidance and should not be relied upon as legal advice. Seek advice from a qualified solicitor prior to entering into any contracts with an AI vendor. 

​