AI, especially Generative AI, is reshaping financial services, enhancing products, client interactions, and productivity. However, challenges like hallucinations and model unpredictability make safe deployment complex. Rapid advancements require flexible governance.

Financial institutions are eager to adopt AI but face regulatory hurdles. Existing frameworks may not address AI’s unique risks, necessitating an adaptive governance model for safe and compliant integration.

The following framework has been developed by FINOS (Fintech Open Source Foundation) members, providing comprehensive catalogue or risks and associated mitigation. We suggest using our heuristic risk identification framework to determine which risks are most relevant for a given use case.

Risk Catalogue

Operational

AIR-OP-004

Hallucination and Inaccurate Outputs

LLM hallucinations refer to instances when a large language model ...

AIR-OP-005

Instability in Foundation Model Behaviour

Instability in foundation model behaviour would manifest itself as deviations ...

AIR-OP-006

Non-Deterministic Behaviour

A fundamental property of LLMs is the non-determinism of their ...

AIR-OP-007

Availability of Foundational Model

RAG systems are proliferating due to the low barrier of ...

AIR-OP-011

Lack of Foundation Model Versioning

Inadequate or unpublished API versioning and/or model version control may ...

AIR-OP-014

Inadequate System Alignment

AlignmentThere is a specific goal you want to achieve when ...

AIR-OP-016

Bias and Discrimination

AI trained on historical/internet data may embed biases. Can lead ...

AIR-OP-017

Lack of Explainability

Black Box Nature of Generative Models Difficult to interpret and ...

AIR-OP-018

Model Overreach and Misuse

The impressive capabilities of GenAI can lead to overestimation of ...

AIR-OP-019

Data Quality and Drift

Generative AI’s outputs depend on the quality and recency of ...

AIR-OP-020

Reputational Risk

AI failures or misuse can quickly become public incidents, eroding ...

Security

AIR-SEC-002

Unauthorized Access & Data Leaks

TODO: Make this non-vector store specificVector stores are specialized databases ...

AIR-SEC-008

Tampering With the Foundational Model

The SaaS-based LLM provider is a 3rd party supplier and ...

AIR-SEC-009

Data Poisoning

Adversaries can tamper with AI training or fine-tuning data to ...

AIR-SEC-010

Prompt Injection

Users of the application or malitious internal agents can craft ...

Regulatory and Compliance

AIR-RC-001

Information Leaked To Hosted Model

In the provided system architecture, sensitive data is transmitted to ...

AIR-RC-022

Regulatory Compliance and Oversight

Financial services are heavily regulated, and AI use does not ...

AIR-RC-023

Intellectual Property (IP) and Copyright

Generative AI models often train on datasets that may include ...

Preventative

AIR-PREV-002

Data Filtering From Confluence Into The Samples

To mitigate the risk of sensitive data leakage and tampering ...

AIR-PREV-003

User/App/Model Firewalling/Filtering

As in any information system component, you can monitor and ...

AIR-PREV-005

System Acceptance Testing

System Acceptance Testing is the final phase of the software ...

AIR-PREV-006

Data Quality & Classification/Sensitivity

Data is classified within the Confluence data store, and filtered ...

AIR-PREV-007

Legal/Contractual Agreements

This control is about legal agreements between the SaaS inference ...

AIR-PREV-008

QoS/DDoS Prevention

Controls should be in place to ensure single or few ...

AIR-PREV-010

Model Version Pinning

Supplier Controls:Ensure the supplier is contractually obligated to provide enough ...

AIR-PREV-012

Role-Based Data Access

Ensure data provided by Confluence is aligned with the end-user ...

AIR-PREV-014

Encrypt Data at Rest

Encrypting data at rest involves converting stored information into a ...

AIR-PREV-017

Ai Firewall

DescriptionThe rapid and widespread integration of generative AI into application ...

Detective

AIR-DET-001

Data Leakage Prevention And Detection

Preventing Leakage of Session DataThe use of Third-Party Service Providers ...

AIR-DET-004

System Observability

What to log/monitorWhen talking about observability, these are the main ...

AIR-DET-009

Alerting / DoW Spend Alert

Level Scope Control Pros Cons / Residual Risk 0 Org-wide ...

AIR-DET-011

Human Feedback Loop

Human Feedback LoopImplementing a human feedback loop is crucial for ...

AIR-DET-013

Provide Citations

Provide citations / linkage to the source data in Confluence ...

AIR-DET-015

LLM-As-A-Judge

Testing (evaluating model responses against a set of test cases) ...

AIR-DET-016

Preserving Access Controls in the Ingested Data

When ingesting data to be queried using RAG architecture, the ...