Making AI for healthcare more trustable and accountable

One of the fears of enabling AI to make decisions in regulated industries is that the AI models will be black box and decision-making logic and process will be protected or hidden from end-users to protect intellectual property or fear of exploitation of the algorithms. This is contrary to the assistive nature of AI and there is an increasing awareness that “Transparency in AI” will enhance trust in AI implementations to improve operations. In this context transparency is a proxy for several characteristics of the AI model including but not limited to (a) modeling technique and parameters used in training, (b) reproducibility of the end-to-end process, and (c) explanability of the results produced by the model.

Explainable models in Artificial Intelligence are often employed to ensure transparency and accountability of AI systems. The fidelity of the explanations are dependent upon the algorithms used as well as on the fidelity of the data. Many real-world datasets have missing values that can greatly influence explanation fidelity. The standard way to deal with such scenarios is imputation. This can, however, lead to situations where the imputed values may correspond to a setting which refer to counterfactuals. Acting on explanations from AI models with imputed values may lead to unsafe outcomes. In this paper, we explore different settings where AI models with imputation can be problematic and describe ways to address such scenarios.

With the increasing ubiquity of machine learning in healthcare, there are increasing calls for machine learning and AI-based systems to be regulated and held accountable in healthcare. Interpretable machine learning models can be used for accountability in machine learning. Healthcare offers unique challenges for machine learning where the demands for explainability, model fidelity, and performance, in general, are much higher as compared to most other domains. The time is not to address the notion of interpretability within the context of healthcare, the various nuances associated with it, challenges related to interpretability which are unique to healthcare and the future of interpretability in healthcare.