Supplier Controls:
Ensure the supplier is contractually obligated to provide enough of the below practices to allow for the development of an upgrade strategy:
- Version Numbers and Release Notes: Establish a clear versioning system (e.g., major/minor/patch updates) and detailed release notes to help users understand the scope and impact of updates. Proper versioning includes documenting key updates in model behavior, architecture, and training data. With each new version, document what’s changed (e.g., increased accuracy, better handling of certain tasks).
- User Notifications: Implement proactive notifications to inform users when new versions are available or when an older version will be deprecated.
- Documentation, API Flexibility, and Backward Compatibility Options: Provide extensive documentation and API options to allow users to choose which version they want to use, along with allowing users to continue using previous versions until they’re ready to migrate to a new version. The version of the model in use should not be impacted until migration to the new version is complete.
- Monitoring and Feedback Loops: Collect user feedback on new versions and monitoring any potential regressions or issues that arise in specific use cases.
- Rigorous Testing: Companies need to test new versions comprehensively before deploying them, especially in production environments where accuracy and reliability are crucial.
- Testing and Validation: Offering users tools to test new versions in sandbox environments before transitioning to them in production, minimizing disruptions in workflows.
Organization Controls:
Organizations using LLMs via APIs may also implement controls for effective versioning:
- Establish procedures to test new versions of models
- Establish procedures to deploy application to migrate a model version with rollback procedures
- Regular audits of deployed model versions
- Integration of version information in model outputs for traceability