Abstract

The healthcare industry is quickly assimilating generative AI to leverage its benefits. However, as the sector adopts AI, the patient community is skeptical about using the technology. They are worried about the safety of data shared with AI systems, the decision-making transparency of AI models and the potential bias of AI systems. This publication recognizes the legitimacy of these concerns and proposes various AI governance practices that can be leveraged to address the aforementioned issues.

Keywords: Gen AI, Healthcare, AI governance.

1. Introduction

Generative AI is taking over the healthcare industry. According to research by McKinsey & Company, 70 percent of institutions in the healthcare sector have adopted AI capabilities or are planning to deploy AI technology in the near future¹. Although the healthcare sector is actively adopting generative AI in its operations, the public does not fully approve the use of AI in healthcare. According to a Pew Research survey, 60 percent of Americans say they would be uncomfortable if their healthcare providers relied on AI for their medical care².

Figure 1: More Americans are uncomfortable about use of AI in healthcare

The study notes that negative public sentiment about AI is primarily influenced by ethical concerns surrounding AI use in healthcare settings. Most of the ethical issues that surround the use of AI can be addressed via proper governance of the technology.

1.1. Ethical issues related to the use of AI in healthcare

Although generative AI promises to transform healthcare delivery, its application in the sector is subject to various ethical implications. These issues must be addressed for AI to be seamlessly integrated into the healthcare sector. Some of the top ethical concerns surrounding the use of AI in healthcare include;

1.2. Data privacy concerns

The assimilation of AI in healthcare is intensifying discussions about the safety of patient data in healthcare systems. According to Pew Research, 37 percent of Americans believe that increased use of AI in healthcare will worsen the security of patients' health records. As gen AI is deployed, the need to gather more data increases. As institutions hold more patient data, the risk of exposure increases and the impact of exposure worsens. Besides, as gen AI is incorporated in healthcare systems, there is increased risk of these systems not meeting compliance requirements to regulations such as HIPAA. There is also the risk of AI systems collecting patients' data without consent and institutions not informing patients how their data is collected, stored and used.

1.3. AI bias

Artificial intelligence systems work based on how they were trained. If the training data constituted elements of bias toward certain gender, race or ethnic communities, these biases will show up when the models are deployed in working environments. In other industries, AI models have discriminated against people based on race. For example, in security, AI models have associated criminal risks to certain races. In healthcare, AI discrimination may involve models underestimating or overestimating risks in certain patient populations³. It may also involve misdiagnosing patients due to their age, gender or race. It is essential medical AI systems are designed to be objective in their functioning and strive to promote better and more equitable health outcomes.

1.4. Reduced transparency in decision-making

Transparency is one of the principal tenets of medical practice. Patients have a right to know how decisions regarding their treatment plans were reached. Healthcare practitioners are also obligated to have a precise understanding of patients' journeys and be able to explain how each decision was arrived at. AI algorithms are vulnerable to 'black-box' problem. ‘Black box’ issue is a term that describes a phenomenon where AI algorithms continuously fine-tune their parameters and evolve rules to the extent that users of the systems and even developers do not understand how the models arrive at decisions. This leads to a lack of transparency in decision-making, making it difficult for caregivers to explain diagnosis and treatment plans to patients⁴. This also raises concerns about who is responsible for mistakes committed by opaque decision-making AI models.

Concerns around the use of AI models in the medical field are monumental. Although the technology has been tipped to transform the healthcare sector, its use threatens to affect public trust, which is essential to effective healthcare. The good news is that these can be addressed via effective governance practices.

1.5. Governance practices for AI in healthcare

Some of the approaches that can exploited to mitigate ethical concerns that result from the use of gen AI in healthcare include;

1.6. Legal regulation of AI technology

EHR systems in the United States are regulated by The HHS Office for Civil Rights (OCR), which enforces HIPAA and HITECH regulations and the Office of the National Coordinator for Health Information Technology, which oversees EHR certification programs to ensure compliance with standards. The mandate of these bodies can be expanded to include medical AI systems. The roles of these bodies can be expanded to ensure AI systems used in healthcare adhere to data privacy regulations. This will impel healthcare organizations to instill strict data privacy standards in their AI systems.

1.7. Data governance panel

This is a team of public members, target group representatives, clinical experts, AI experts, ethics crusaders and legal professionals responsible for reviewing data sets for training medical AI systems. The panel's core responsibility is to ensure datasets used to train AI models are representative and do not result in the development of biased algorithms. Besides addressing bias, the panels can enforce data privacy in AI systems. Data governance panels can regulate how patient data is collected, stored and used. These panels can also enforce the use of consent in AI systems. They can set regulations that require healthcare facilities to seek permission from patients before collecting data and inform patients how their data is used.

1.8. Transparency

Transparency concerns can be addressed primarily through two strategies.

·Explainable AI (XAI): XAI is an emerging practice that constitutes a set of practices that enable explainability of how AI algorithms make decisions. The practices seek to explain the whole logic of the model (global level) or explain the reasoning for a specific decision (local level). AI governance should focus on encouraging AI developers to incorporate XAI in AI systems to increase transparency in decision-making.

·Self-identification of AI models: Although AI mimics human experts, the technology should not deceive patients into believing they are interacting with real humans. AI models should introduce themselves as artificial intelligence models. Policies should make it a requirement for AI models to introduce themselves appropriately.

1.9. Informed use of AI

In environments where gen AI is used, healthcare givers should inform patients that they rely on the technology. Patients should be informed about the benefits and drawbacks of the technology. In applications where patient data is shared with AI, patients should be informed. Patients must also have the autonomy to accept or reject the use of AI in their treatments. Policies should be ratified to formalize the informed use of AI in healthcare facilities.