Most vendor questionnaires are written by procurement and answered by sales. Here is how to assess an AI supplier so the answer actually protects you.
The standard AI vendor questionnaire has between sixty and one hundred and fifty questions, takes a sales engineer two days to complete, and provides almost no information that protects the buying organisation. It is the dominant form of AI supplier assessment in most enterprises. It is also, by a wide margin, the part of the governance program that delivers the least value relative to the effort invested.
The reason is structural. The questionnaire is written by procurement or legal, often by adapting a generic security questionnaire to mention AI. It is answered by sales, who optimise for the answer that gets the deal through. It is reviewed by a risk team that does not have the technical depth to challenge the answers. By the time the assessment is complete, it has produced a binder of yes responses that no one in the organisation actually believes are protective, but that everyone can point to if something goes wrong.
The good news is that vendor assessment, done differently, is among the highest-leverage controls in an AI governance program. The bad news is that doing it differently requires breaking habits that procurement, legal, and risk functions have invested years in building.
What standard questionnaires get wrong
Standard questionnaires get three things wrong, in roughly equal measure.
First, they focus on the vendor's general posture rather than your specific use case. Whether the vendor encrypts data at rest is a meaningful question for any data processor. It tells you almost nothing about whether their AI is safe to use in your specific workflow. The questions that would tell you, what does the model do when given inputs outside its training distribution, what is the latency on the human oversight loop, how often is the model retrained and on what data, are usually absent or so general as to be unanswerable.
Second, they confuse certification with capability. A SOC 2 Type II report tells you that the vendor has security controls. It does not tell you that those controls cover their AI development process, their model evaluation process, or their incident response for model failures. ISO 27001 certification has the same limitation. Asking whether the vendor is certified is a sensible first filter; treating the certification as a substitute for direct assessment is not.
Third, they are answered once, at the point of purchase, and never updated. AI vendors ship new model versions monthly. They change training data policies, adjust retention rules, sometimes pivot their entire architecture. The assessment that was true at the point of purchase may be wholly inaccurate twelve months later, and the assessment process has no mechanism to detect this.
Start with your use case, not their product sheet
A useful assessment starts with three questions about your own use case, before any vendor question is asked.
What decision does the model influence in our workflow? Be specific. Not improves customer experience but decides which support tier a customer is routed to, or summarises an internal document used by underwriters to make credit decisions. The level of specificity determines the entire downstream assessment.
What is the consequence of a wrong output? Again, be specific. Not customer dissatisfaction but mis-routed support tickets resulting in regulated response time breaches, or summarised information that omits a material risk factor and leads to a flawed credit decision. The consequence framing tells you how much protection you need.
Who in our organisation is accountable for that consequence? Not the AI team but the named individual whose performance review will reflect the failure, and the named function whose budget will absorb the remediation cost. Without a named accountability, the assessment is hypothetical.
Only after these three questions have specific answers should the vendor be brought into the conversation. The vendor assessment then becomes: given this use case, this consequence, and this accountability, what does the vendor need to provide to make the risk acceptable?
The questions that actually matter
Once the use case is defined, the questions that matter for an AI vendor narrow considerably. A short list:
How does the system behave when given inputs outside the training distribution? What is the failure mode, and how is the failure surfaced to the calling system? This is the single most important question for any non-trivial AI use. Vendors that cannot answer it with specifics, with examples, with measured behaviour, are not ready to be deployed in a workflow with real consequences.
What is the retraining cadence, and how is model behaviour validated against the previous version? Drift between versions is the most common source of governance incidents. A vendor that retrains continuously without version comparison is asking the buyer to absorb risk that the vendor has not bothered to measure.
What human oversight is designed into the system, and what is the latency of that oversight loop? Many AI products advertise human in the loop but mean that a human can theoretically review an output, with no measured latency and no enforcement that the review actually happens. Ask for the measured statistics, not the design intent.
What governance does the vendor itself operate? Do they have a named AI risk owner? Do they run their own model risk reviews? Do they conduct red team exercises against their own models, and do they publish the results? A vendor with no internal governance is asking the buyer to do the governance for them.
What is the disclosure policy for incidents? Specifically, how long after the vendor discovers a material model issue will the buyer be notified, and through what channel? Without this commitment, the buyer is on their own for incident detection.
The live failure mode walkthrough
The single most useful exercise in any AI vendor assessment is a live walkthrough of the model's failure modes. Not a demonstration of the model working well, which any vendor can stage. A demonstration of what happens when the model is wrong.
Ask the vendor to walk you through three specific examples: an input where the model produced a wrong output in their internal testing; the failure mode the vendor considers most likely in production; the failure mode that would be hardest to detect from the calling system's side. A vendor who can speak fluently about all three has a mature model risk practice. A vendor who deflects, generalises, or claims their model does not fail should not be deployed in any context with real consequences.
This exercise takes ninety minutes and produces more useful information than the entire standard questionnaire. It also reshapes the buyer-vendor relationship in a healthy way. The vendor learns that the buyer takes failure modes seriously. The buyer learns whether the vendor does.
Continuous, not one-time
The final structural fix is to treat vendor assessment as continuous rather than one-time. The point-of-purchase assessment establishes a baseline. A quarterly review, owned by a named individual on the buyer side, refreshes the baseline against what the vendor has changed in the intervening period. The review takes about ninety minutes per material vendor and addresses a short checklist: what has the vendor changed in their model, their training data, their retention policy, their incident history, their own governance posture.
Most organisations resist this on the basis that they do not have the capacity to review every vendor quarterly. The honest answer is that they should not have so many material AI vendors that quarterly review is infeasible. If the list is too long for review, the list is too long, full stop. Trim it. Consolidate. Move workloads onto fewer suppliers whose behaviour you can actually understand. A vendor relationship that cannot be reviewed quarterly is, by definition, a relationship in which the buyer has lost track of the risk.
What changes when you do this
Organisations that move to use-case-led assessment, failure mode walkthroughs, and quarterly review report two changes. First, the population of approved AI vendors shrinks, sometimes by half. The vendors that survive the new process are demonstrably more capable than the ones that drop out. Second, the relationship with surviving vendors becomes substantively different. The vendor knows the buyer is paying attention. The buyer knows what the vendor can and cannot do. When an incident happens, and it will, both sides have the context to respond well, rather than discovering each other's posture in the middle of the failure.
This is what vendor assessment is supposed to deliver. Most organisations are not getting it, because they are running a process designed for a different problem on a class of supplier it was never built for. The fix is not a bigger questionnaire. The fix is a different question.
