Summary. Buying an AI tool is buying a probabilistic system whose behavior changes when the vendor updates a model. This checklist covers the diligence and contract terms that make that manageable: use-case classification, data rights and training exclusion, output ownership and IP indemnity, accuracy and evaluation, human oversight for consequential decisions, security and subprocessors, model change management, and the regulatory overlays that apply to deployers. A final phase addresses ongoing governance — inventory, shadow AI, incidents, and review.

What this checklist is for. Procuring an AI system and governing it after deployment. For the underlying analysis, see Artificial Intelligence Key Legal Issues.

Phase 1 — Classify the use case before anything else

  • Write a one-paragraph description of what the system does, what data goes in, what comes out, and who is affected.
  • Determine whether the output is advisory (a human decides) or consequential (the system decides or effectively decides).
  • Determine whether the use case touches a regulated decision: employment, credit, housing, insurance, education, healthcare, or essential government services.
  • Identify whether personal information, sensitive personal information, biometric data, PHI, or customer confidential information will be sent to the system.
  • Identify whether third-party confidential information subject to a contractual restriction will be sent.
  • Assign a risk tier and route accordingly: low-risk tools take the standard path; high-risk tools require an impact assessment and legal review.

Why this matters. Every meaningful control downstream — bias testing, notice, human review, documentation — is triggered by the classification. A tool that summarizes meeting notes and a tool that ranks job applicants should not go through the same procurement.

Phase 2 — Data rights: the most negotiated terms

  • No training on customer data without express opt-in, stated affirmatively and covering the vendor's affiliates and its model providers.
  • Confirm the default configuration matches the contract — many products train by default on the consumer tier and not on the enterprise tier.
  • No use of inputs or outputs to improve models, benchmarks, or products beyond providing the service.
  • Retention limits stated in days, with deletion on termination and a certificate of deletion.
  • Abuse-monitoring retention disclosed and bounded; ask whether human reviewers can see inputs and under what conditions.
  • Subprocessor list disclosed, including which foundation model providers sit behind the product, with notice and objection rights on change.
  • Data residency and cross-border transfer mechanism identified. See International Data Transfers After Schrems II.
  • DPA executed with the correct role characterization (controller/processor or business/service provider), and a service provider restriction on using personal information for the vendor's own purposes.
  • Confidentiality terms cover prompts and outputs, which are frequently omitted from a generic NDA's definition.

Phase 3 — Outputs, IP, and indemnity

  • Output ownership assigned to the customer to the maximum extent the vendor can convey, with an acknowledgment that outputs may not be copyrightable. See Copyright Infringement Claims Against Generative AI.
  • Non-exclusivity acknowledged: identical prompts by other customers may produce similar outputs.
  • IP infringement indemnity for outputs, with the exclusions read carefully — most are conditioned on using the vendor's safety filters, not providing infringing input, and not prompting for a specific third-party work.
  • Indemnity cap identified; an uncapped indemnity is the market for major vendors and worth asking for.
  • Human authorship workflow documented for any output intended for copyright registration, including a record of human contribution. See Copyright Registration: A Comprehensive Guide.
  • Patent inventorship: confirm a human conceived any invention arising from AI-assisted work. Thaler v. Vidal, 43 F.4th 1207 (Fed. Cir. 2022).
  • Open-source and license contamination addressed for code-generation tools, including license-filter settings and provenance reporting.

Phase 4 — Accuracy, evaluation, and oversight

  • Acceptance testing on your data, not the vendor's demo, with a documented pass threshold.
  • Documented error rates for the intended use, and a model card or system card.
  • Known limitations disclosed in writing.
  • Hallucination controls for any use touching facts, citations, or figures — retrieval grounding, citation to source, and mandatory verification.
  • Human review required for consequential decisions, with reviewers who have authority and information to override. A rubber stamp is not oversight.
  • Explainability sufficient to give the individual a meaningful reason for an adverse decision — required by the FCRA for credit and by state AI statutes for consequential decisions.
  • Bias testing performed and repeated, with results retained. Disparate impact analysis under Title VII, the ADA, the ADEA, ECOA, and the FHA where applicable. See Workplace Harassment and Hostile Work Environment Claims.
  • Accessibility of the AI interface confirmed, including for screen reader users and for candidates who need an accommodation from an automated assessment.

Phase 5 — Security, change management, and exit

  • Security review completed: SOC 2 Type II or ISO 27001, penetration test summary, and answers on prompt injection, data exfiltration through outputs, and tenant isolation.
  • Access controls confirmed — that the tool cannot reach data the requesting user could not otherwise access is a frequent failure in retrieval-augmented deployments.
  • Model change notice: advance notice of material model version changes, with the right to test and, for high-risk uses, to remain on a prior version for a stated period.
  • Performance floor that survives model updates, with a remedy if accuracy degrades.
  • Deprecation and end-of-life notice period.
  • Exit rights: export of prompts, outputs, fine-tuning data, and configurations in a usable format; deletion certificate; transition assistance. See Cloud and SaaS Agreements.
  • Incident notification for AI-specific incidents, not only security breaches — model failures, harmful outputs, and data leakage through outputs.
  • Insurance reviewed for AI exclusions in technology E&O and cyber policies. See Business Insurance and Coverage Disputes.

Phase 6 — Regulatory overlays

  • Notice to affected individuals where required — several states and cities require pre-use notice for automated employment decision tools and for consequential decisions generally.
  • Bias audit completed and published where required (New York City Local Law 144 requires an annual independent bias audit and publication of results for automated employment decision tools).
  • Colorado AI Act duties assessed for deployers of high-risk systems: reasonable care, impact assessments, notice, and adverse-decision explanation and appeal.
  • EU AI Act applicability assessed if the system or its output is used in the EU, with prohibited-practice screening, transparency obligations for synthetic content, and high-risk obligations where triggered.
  • FTC Act § 5 exposure reviewed for accuracy of AI claims — "AI washing" and unsupported performance claims are an active enforcement priority.
  • Sector rules identified: FCRA for background and credit tools, ECOA adverse action notices, HIPAA for clinical tools, state insurance regulations, and professional responsibility duties for legal and medical uses.
  • NIST AI Risk Management Framework mapped to the internal program as the documentation backbone.

Phase 7 — Ongoing governance

  • Maintain an AI inventory: system, owner, use case, risk tier, data, vendor, model version, last review date.
  • Run shadow AI discovery — expense reports, SSO logs, browser extension inventories, and network egress. Most organizations have far more AI in use than the inventory shows.
  • Publish an acceptable use policy naming approved tools, prohibited data, and disclosure duties.
  • Train employees on what may and may not be entered into a prompt.
  • Log prompts and outputs for high-risk uses, consistent with the retention schedule and any litigation hold.
  • Review each system at least annually, and after any material model change or incident.
  • Assign a named accountable owner for every system in the inventory.

Common mistakes

  • Reading the enterprise contract while employees use the consumer version, which trains on inputs by default.
  • Assuming no training because the sales engineer said so; the default is what governs.
  • No acceptance testing on real data.
  • Human review that is a rubber stamp — high override rates near zero are a warning sign.
  • No notice for automated employment decisions in jurisdictions that require it.
  • Ignoring model updates, so a validated system silently becomes an unvalidated one.
  • A generic NDA that does not cover prompts and outputs.
  • No exit plan for fine-tuning data.
  • Cataloguing only the tools procurement knows about.

Primary authority

Related

This checklist is educational and not legal advice. AI regulation is changing rapidly and varies by jurisdiction and sector. Consult qualified counsel before deploying an AI system in a regulated decision.