Summary

Data poisoning occurs when adversaries tamper with training or fine-tuning data to manipulate an AI model’s behaviour, often by injecting misleading or malicious patterns. This can lead to biased decision-making, such as incorrectly approving fraudulent transactions or degrading model performance in subtle ways. The risk is heightened in systems that continuously learn from unvalidated or third-party data, with impacts that may remain hidden until a major failure occurs.

Description

Data poisoning involves adversaries deliberately tampering with training or fine-tuning data to corrupt the learning process and manipulate subsequent model behavior. In financial services, this presents several attack vectors:

Training Data Manipulation: Adversaries alter datasets by changing labels (marking fraudulent transactions as legitimate) or injecting crafted data points with hidden patterns exploitable later.

Continuous Learning Exploitation: Systems that continuously learn from new data are vulnerable if validation mechanisms are inadequate. Fraudsters can systematically feed misleading information to skew decision-making in credit scoring or trading models.

Third-Party Data Compromise: Financial institutions rely on external data feeds (market data, credit references, KYC/AML watchlists). If these sources are compromised, poisoned data can unknowingly introduce biases or vulnerabilities.

Bias Introduction: Data poisoning can amplify biases in credit scoring or loan approval models, leading to discriminatory outcomes and regulatory non-compliance.

The effects are often subtle and difficult to detect, potentially remaining hidden until major failures, financial losses, or regulatory interventions occur.