Vendor AI Evaluation Checklist

The organisation still carries the risk.

A supplier may describe a feature as AI-powered, but the organisation carries the risk if the tool affects tenants, staff, data, records, decisions or service quality. A good supplier should be able to explain what the AI does, what data it uses, how it is tested, how it fails, how it is monitored and how it can be switched off.

Ten areas to evaluate

1. Use Case and Housing Value

Board question: What housing problem does this supplier AI feature solve?

The supplier can describe the AI feature in plain English.

The feature is linked to a housing outcome, not only productivity.

The supplier can explain where AI sits in the workflow.

The feature has measurable success criteria.

The buyer has agreed what would count as failure.

Red flag: The supplier says "AI" often but cannot explain the task, workflow, data or decision point.

2. Disclosure and Transparency

The supplier declares whether AI is used in the product, service delivery, support or bid process.

The supplier identifies which parts use generative AI, machine learning, rules, automation or third-party models.

The supplier commits to notice before material AI changes.

3. Data Protection and Privacy

Data categories are listed, including personal data, special category data and sensitive housing records.

The supplier explains whether buyer data is used for training, fine-tuning or service improvement.

Data residency, retention and deletion are clear.

Subprocessors and model providers are listed.

The supplier supports DPIA completion.

4. Accuracy, Testing and Limits

The supplier has tested the AI feature against realistic use cases.

Accuracy, error types and known limits are stated.

The buyer can run local testing before rollout.

The supplier agrees to ongoing monitoring after rollout.

Red flag: The supplier relies on a polished demo and will not provide test evidence, limits or failure examples.

5. Fairness, Bias and Tenant Impact

The supplier has considered bias and fairness risks.

Accessibility, language, digital exclusion and vulnerability have been considered.

Residents can challenge or request human review where the use affects them.

6–10. Cyber, Supply Chain, Oversight, Contract, Cost

Cyber security risk has been assessed, including prompt injection, data leakage and unsafe output risks.

The supplier identifies foundation model providers, hosting providers and key subcontractors.

Human review points are built into the workflow and staff can override AI outputs.

Contract terms cover AI changes, data use, audit rights, incidents and termination.

AI functionality can be enabled, disabled or limited by the buyer (rollback capability).

Pricing is clear including usage-based charges, tokens, storage and future AI modules.

Ten questions to ask in a supplier demo

Show us exactly where AI is used in the workflow.

Show us what data the AI can see.

Show us how a staff member checks, edits or rejects the output.

Show us what happens when the AI is uncertain.

Show us a failure example.

Show us how the feature can be switched off.

Show us what the audit log records.

Show us what would be visible to a resident.

Show us how you notify us when the model or feature changes.

Show us the report we would receive after three months of use.

HAILIE position

Do not buy AI because it is in a roadmap. Buy or enable it when it solves a housing problem, protects residents, gives staff better information, and can be tested, governed and stopped. The best supplier conversations start with evidence.