Auditing Analytic Models
An analytic model is a mathematical equation that takes in data and produces a calculation such as a score, ranking, classification, or prediction. It is a very specific set of instructions for analyzing data to deliver a particular kind of result — behavior, decision, action, or cause — to support a business process.
The objective of analytics controls is to ensure that:
Analytics personnel have the appropriate skills and training.
Input data is appropriate, complete, authorized, and correct.
Model selection procedures are documented and justified.
Model validation and testing have been conducted in accordance with scientific principles.
Outputs are accurate, complete, and being used by the business as intended.
The model is refreshed and reevaluated periodically.
The organization maintains a record to track the processing of data from input, to processing, to the eventual output.
There are several types of analytics controls. Skills controls provide assurance that data analytics personnel are competent and sufficiently trained in relevant analytics methods. Business-use controls provide assurance that the model addresses the intended business objective. Data controls are used mainly to check the integrity of data entered into an analytic model. Model selection controls ensure model selection is appropriate and reasonable to provide decision support. Model validation controls address what is done to ensure the model output is reasonable and accurately reflects the underlying nature of the input data. Output controls provide assurance that the model output is presented and used in an appropriate and justified manner to ensure it remains consistent and correct. Maintenance controls address the need to reevaluate and refresh analytic models periodically to ensure they are still relevant in the current environment.
Although many auditors may be unfamiliar with analytic models, machine learning, and AI, the fundamentals of internal auditing remain the same. As with all new technologies and processes that organizations have embraced, internal auditors have a responsibility to learn how analytic models can be useful in their work and adapt their methods to serve their stakeholders.