AI Governance Framework Icon

FINOS AI Governance Framework

A comprehensive collection of risks and mitigations that support on-boarding, development of, and running Generative AI solutions

AI, especially Generative AI, is reshaping financial services, enhancing products, client interactions, and productivity. However, challenges like hallucinations and model unpredictability make safe deployment complex. Rapid advancements require flexible governance.

Financial institutions are eager to adopt AI but face regulatory hurdles. Existing frameworks may not address AI’s unique risks, necessitating an adaptive governance model for safe and compliant integration.

The following framework has been developed by FINOS (Fintech Open Source Foundation) members, providing comprehensive catalogue or risks and associated mitigation. We suggest using our heuristic risk identification framework to determine which risks are most relevant for a given use case.

Risk Catalogue


Operational

AIR-OP-004

Hallucination and Inaccurate Outputs

SummaryLLM hallucinations occur when a model generates confident but incorrect ...

Read more
AIR-OP-005

Foundation Model Versioning

SummaryFoundation model instability refers to unpredictable changes in model behavior ...

Read more
AIR-OP-006

Non-Deterministic Behaviour

SummaryLLMs exhibit non-deterministic behaviour, meaning they can generate different outputs ...

Read more
AIR-OP-007

Availability of Foundational Model

SummaryFoundation models often rely on GPU-heavy infrastructure hosted by third-party ...

Read more
AIR-OP-014

Inadequate System Alignment

SummaryAI alignment risk arises when a system’s behaviour diverges from ...

Read more
AIR-OP-016

Bias and Discrimination

SummaryAI systems can systematically disadvantage protected groups through biased training ...

Read more
AIR-OP-017

Lack of Explainability

SummaryAI systems, particularly those using complex foundation models, often lack ...

Read more
AIR-OP-018

Model Overreach / Expanded Use

SummaryModel overreach occurs when AI systems are used beyond their ...

Read more
AIR-OP-019

Data Quality and Drift

SummaryGenerative AI systems rely heavily on the quality and freshness ...

Read more
AIR-OP-020

Reputational Risk

SummaryAI failures or misuse—especially in customer-facing systems—can quickly escalate into ...

Read more

Security

AIR-SEC-002

Information Leaked to Vector Store

SummaryLLM applications pose data leakage risks not only through vector ...

Read more
AIR-SEC-008

Tampering With the Foundational Model

SummaryFoundational models provided by third-party SaaS vendors are vulnerable to ...

Read more
AIR-SEC-009

Data Poisoning

SummaryData poisoning occurs when adversaries tamper with training or fine-tuning ...

Read more
AIR-SEC-010

Prompt Injection

SummaryPrompt injection occurs when attackers craft inputs that manipulate a ...

Read more

Regulatory and Compliance

AIR-RC-001

Information Leaked To Hosted Model

SummaryUsing third-party hosted LLMs creates a two-way trust boundary where ...

Read more
AIR-RC-022

Regulatory Compliance and Oversight

SummaryAI systems in financial services must comply with the same ...

Read more
AIR-RC-023

Intellectual Property (IP) and Copyright

SummaryGenerative AI models may be trained on copyrighted or proprietary ...

Read more

Mitigation Catalogue


Preventative

AIR-PREV-002

Data Filtering From External Knowledge Bases

PurposeThis control addresses the critical need to sanitize, filter, and ...

Read more
AIR-PREV-003

User/App/Model Firewalling/Filtering

Effective security for AI systems involves monitoring and filtering interactions ...

Read more
AIR-PREV-005

System Acceptance Testing

PurposeSystem Acceptance Testing (SAT) for AI systems is a crucial ...

Read more
AIR-PREV-006

Data Quality & Classification/Sensitivity

PurposeThe integrity, security, and effectiveness of any AI system deployed ...

Read more
AIR-PREV-007

Legal and Contractual Frameworks for AI Systems

PurposeRobust legal and contractual agreements are essential for governing the ...

Read more
AIR-PREV-008

Quality of Service (QoS) and DDoS Prevention for AI Systems

PurposeThe increasing integration of Artificial Intelligence (AI) into financial applications, ...

Read more
AIR-PREV-010

AI Model Version Pinning

PurposeModel Version Pinning is the deliberate practice of selecting and ...

Read more
AIR-PREV-012

Role-Based Access Control for AI Data

PurposeRole-Based Access Control (RBAC) is a fundamental security mechanism designed ...

Read more
AIR-PREV-014

Encryption of AI Data at Rest

PurposeEncryption of data at rest is a fundamental security control ...

Read more
AIR-PREV-017

AI Firewall Implementation and Management

PurposeAn AI Firewall is conceptualized as a specialized security system ...

Read more

Detective

AIR-DET-001

AI Data Leakage Prevention and Detection

PurposeData Leakage Prevention and Detection (DLP&D) for Artificial Intelligence (AI) ...

Read more
AIR-DET-004

AI System Observability

PurposeAI System Observability encompasses the comprehensive collection, analysis, and monitoring ...

Read more
AIR-DET-009

AI System Alerting and Denial of Wallet (DoW) / Spend Monitoring

PurposeThe consumption-based pricing models common in AI services (especially cloud-hosted ...

Read more
AIR-DET-011

Human Feedback Loop for AI Systems

PurposeA Human Feedback Loop is a critical detective and continuous ...

Read more
AIR-DET-013

Providing Citations and Source Traceability for AI-Generated Information

PurposeThis control outlines the practice of designing Artificial Intelligence (AI) ...

Read more
AIR-DET-015

Using Large Language Models for Automated Evaluation (LLM-as-a-Judge)

Purpose“LLM-as-a-Judge” (also referred to as LLM-based evaluation) is an emerging ...

Read more
AIR-DET-016

Preserving Source Data Access Controls in AI Systems

PurposeThis control addresses the critical requirement that when an Artificial ...

Read more