Comparison of AI applications and anesthesiologist’s anesthesia method choices

Abstract

Background

In medicine, Artificial intelligence has begun to be utilized in nearly every domain, from medical devices to the interpretation of imaging studies. There is still a need for more experience and more studies related to the comprehensive use of AI in medicine. The aim of the present study is to evaluate the ability of AI to make decisions regarding anesthesia methods and to compare the most popular AI programs from this perspective.

Methods

The study included orthopedic patients over 18 years of age scheduled for limb surgery within a 1-month period. Patients classified as ASA I-III who were evaluated in the anesthesia clinic during the preoperative period were included in the study. The anesthesia method preferred by the anesthesiologist during the operation and the patient’s demographic data, comorbidities, medications, and surgical history were recorded. The obtained patient data were discussed as if presenting a patient scenario using the free versions of the ChatGPT, Copilot, and Gemini applications by a different anesthesiologist who did not perform the operation.

Results

Over the course of 1 month, a total of 72 patients were enrolled in the study. It was observed that both the anesthesia specialists and the Gemini application chose spinal anesthesia for the same patient in 68.5% of cases. This rate was higher compared to the other AI applications. For patients taking medication, it was observed that the Gemini application presented choices that were highly compatible (85.7%) with the anesthesiologists’ preferences.

Conclusion

AI cannot fully master the guidelines and exceptional and specific cases that arrive in the course of medical treatment. Thus, we believe that AI can serve as a valuable assistant rather than replacing doctors.

Introduction

In medicine, AI has begun to be utilized in nearly every domain, from medical devices to the interpretation of imaging studies. AI can analyze large amounts of data in a short time and provide better recommendations than humans analyzing the same data [1]. One of the early AI applications, ChatGPT, was launched in 2022. With a vocabulary of over 300 billion words, it possesses extensive knowledge on numerous subjects. In its first month of availability, ChatGPT reached 57 million users, and by January 2023, its user base exceeded 100 million [2]. The use of ChatGPT has facilitated rapid and practical access to information [3]. In the field of anesthesia, robots, software, and AI are currently being utilized for various purposes. AI is predominantly used for drug infusions, sedation applications, the monitoring of anesthesia depth, ultrasound guidance, and pain management [4]. Additionally, AI provides extra benefits such as increasing efficiency and reducing costs [5, 6]. Studies have also demonstrated that ChatGPT possesses sufficient knowledge to pass the United States Medical Licensing Examination (USMLE) [7, 8].

AI technology is now being frequently used in fields such as plastic surgery, oncology, pulmonology, cardiology, orthopedics, hepatology, and neurology [9]. Studies have shown that AI is particularly beneficial in imaging and diagnostic processes in these fields. AI-assisted systems approved by the US Food and Drug Administration have begun to be used to locate congenital heart pathologies [10]. In other areas of medicine, however, concerns about ethics and safety persist due to the insufficient quality and quantity of relevant academic studies [11].

The potential future use of AI in patient management and its adequacy in this field are subjects of great interest. There are also fears that AI might eventually replace physicians, though the prevailing view is that AI will complement, rather than replace, the human practice of medicine [12]. There is still a need for more experience and more studies related to the comprehensive use of AI in medicine. The aim of the present study is to evaluate the ability of AI to make decisions regarding anesthesia methods and to compare the most popular AI programs from this perspective.

Materials and methods

This study procedures were followed in accordance with the ethical standards of the responsible committee on human experimentation and with the Helsinki Declaration of 1975. The study was approved by The Ethics Committee of Mardin Artuklu University on February 13, 2024 (No. 2024/2–40). The researchers assessed the patients’ eligibility and secured their written consent before the procedure. The study included orthopedic patients over 18 years of age scheduled for extremity surgery within a 1-month period. Patients classified as ASA I-III who were evaluated in the anesthesia clinic during the preoperative period were included in the study. Patients directed by the surgeon regarding the choice of anesthesia method, as well as cases in which the patient’s preference conflicted with the anesthesiologist’s recommendation, were excluded from the study. Additionally, patients were excluded if the preferred anesthesia method was changed during the course of the operation (Fig. 1).

Fig.1

Flow charts of the patients

The anesthesia method preferred by the anesthesiologist during the operation and the patient’s demographic data, comorbidities, medications and surgical history were recorded. The obtained patient data were discussed as if presenting a patient scenario using the free versions of the ChatGPT, Copilot and Gemini applications by a different anesthesiologist who did not perform the operation. It was noted that the AI applications’ recommendations were not to be considered as medical consultations. Personal data were not shared with the AI applications; the shared data included the patient’s age, comorbidities, planned surgery, medications and surgical history. The preferences of the anesthesiologist were compared with the preferences suggested by the three AI applications in the context of the patient scenarios. Data for each patient were recorded.

The AI programs were instructed to recommend only one anesthesia method based on the patient information provided. The results were recorded and compared with the preferences of the anesthesiologist. Answers were recorded impartially, without any guidance. In cases where the AI applications suggested anesthesia methods using general terms, such as “brachial plexus block,” they were asked to provide more specific responses. Those detailed responses were then recorded.

Statistical analysis

Sample size calculation was performed using G-Power version 3.1.9.7 program (Universität Kiel, Germany). In calculation; For the chi-square test, effect size (w) was determined as 0.5, significance level (α) was 0.05, power (1 − β) was 0.95 and degree of freedom (Df) was 3. According to these parameters, the minimum required number of patients was calculated as 69 people [13].

The obtained data were analyzed using IBM SPSS Statistics 26. Categorical data were presented as numbers and percentages, while continuous data were presented as means and standard deviations. The chi-square test and Fisher’s exact test were used for statistical analysis. Values of p < 0.05 were considered statistically significant.

Results

Over the course of 1 month, a total of 72 patients were enrolled in the study. Of these patients, 35 (48.6%) were male and 37 (51.4%) were female. The average age of the patients was 47.86 ± 17.7 years. Twenty-two patients (30.6%) had comorbidities such as diabetes and hypertension. In 57 cases (79.2%), the surgical site was a lower extremity. Anesthesia specialists most commonly preferred the spinal anesthesia method, which was used in 75.0% of the cases.

The anesthesia specialists opted for general anesthesia for 4 patients, whereas ChatGPT selected general anesthesia for only 1 patient (Table 1). Like the anesthesia specialists, who preferred spinal anesthesia most frequently (61.1%), ChatGPT chose spinal anesthesia in 50% of the cases. The rate of selecting general anesthesia for the same patient was observed to be 100% with both Gemini and CoPilot.

Table 1 Artificial intelligence applications and anesthesiologist’s anesthesia method preferences

Among all patients, the highest preference for general anesthesia occurred with the CoPilot application, with this option being chosen for 42 patients (58.3%) (Table 1).

It was observed that both the anesthesia specialists and the Gemini application chose spinal anesthesia for the same patient in 68.5% of cases. This rate was higher compared to the other AI applications (Table 1).

The preference for anesthesia methods showed 70.2% agreement between the Gemini application and the anesthesiologists for patients undergoing lower extremity surgery, while this agreement decreased to 40% for upper extremity surgeries (p = 0.03) (Table 2). In contrast, the CoPilot application’s choice of anesthesia method for upper extremity surgeries was found to be 93.3% discordant with the anesthesiologists’ preferences (Table 2). Overall, when considering all patients, CoPilot had the highest level of discordance (70.8%) with the anesthesiologists’ choices (Table 3).

Table 2 Anesthesia method recommendations of AI for upper/lower patients and compliance with the anesthesiologist’s preference

Table 3 Anesthesia method recommendations for all patients and compliance with the anesthesiologist’s preference

For patients taking medication, it was observed that the Gemini application presented choices that were highly compatible (85.7%) with the anesthesiologists’ preferences (Table 4). Additionally, it was noted that the rate at which both the Gemini and CoPilot applications preferred general anesthesia for the same patient was 23.6% (17 patients) (Table 5).

Table 4 Anesthesia method recommendations of AI for patients taking drug and compliance with the anesthesiologist’s preference

Table 5 General anesthesia preferences and compatibility of Gemini and Copilot applications

Discussion

In this study evaluating the ability of AI applications to analyze patient information and determine the appropriate anesthesia method, it was observed that these applications often made choices close to those of anesthesiologists. Specifically, both the Gemini and CoPilot applications chose general anesthesia in 100% of cases in which anesthesiologists did the same. Furthermore, the Gemini application demonstrated a high concordance rate of 85.7% with anesthesiologists’ preferences for patients who were taking medication, indicating a robust capability to align with expert human decisions in specialized medical settings.

Clinicians’ confidence in AI is likely to increase as they understand its potential and limitations, especially when they observe its successful implementation in real-world scenarios [10]. In their review, Singh and Nath noted that allowing robots to perform routine typical procedures can save anesthesiologists time in critical situations, thereby enhancing their ability to think and make decisions. Additionally, studies have addressed instances in which robots have performed intubations in limited patient series or for simulated patients, or in which nerve recognition software-supported ultrasounds have been used to administer block procedures [14, 15].

Although there are publications related to drug infusion systems, pain management, and even intubation-performing robots in the literature, we did not encounter any studies specifically addressing AI technologies that determine anesthesia management in our literature review. Our study provides examples from current practice to increase confidence in AI. As noted in the reviews conducted by Singh and Nath and Bellini et al., the impact of AI in clinical practice generally remains confined to the level of digital display data. There is an ongoing need for more studies that directly explore real-life applications, as demonstrated by our research [14, 16].

In line with the objectives of our study, the review conducted by Lopes et al. emphasized that, despite the rapid AI advancements in the field of medicine, there is still a lack of clinical applications in anesthesia practice [17]. In many AI-related studies in the field of anesthesia, factors that an anesthesiologist could predict without the need for an extensive literature review have been considered, such as calculating the Cormack-Lehane score from human face photographs. In our planning of the present study, we aimed to analyze the results of AI analysis of patient data. For assessments of patients undergoing upper extremity surgery, we found that ChatGPT considered alternatives such as general anesthesia, axillary block, supraclavicular block, and interscalene block and made choices that were 80% consistent with the anesthesiologists’ preferences. This indicates significant potential for AI to support decision-making in anesthesia, with outcomes aligning closely with expert human judgments. AI can quickly select the most appropriate anesthesia method from among numerous alternatives, which is a substantial benefit. The fact that AI applications yield objective and emotion-free choices while considering anesthesia methods could also be a reason for their preference. Kambale and Jadhaw predicted that AI will contribute to the standardization of anesthesia management and the reduction of human errors. This capability highlights AI’s potential to enhance the efficiency and safety of medical procedures by leveraging consistent data-driven decision-making processes [18]. This leads us to suggest that, in the future, anesthesiologists could use AI applications to quickly review and validate their choices.

AI technology is currently being utilized in the field of anesthesia for critical tasks such as monitoring anesthesia depth or pain control and predicting adverse events [19]. The ability of AI applications to predict possible complications may put them a step ahead compared to physicians. The advantage of such AI applications lies in their ability to access and analyze far greater volumes of data in a short time compared to humans, and in a practical way. In the present study, AI was able to suggest anesthesia methods within seconds for patient scenarios. However, it did not respond according to current guideline practices regarding the use of regional anesthesia in patients using anticoagulants. It is clear that AI needs to be developed for specialized patient groups such as those with medication use and surgical history. Due to its incomplete mastery of guidelines and exceptional or specific situations, the prevailing view of AI at present is that it can be a good assistant for doctors [9, 10, 14, 16].

Singh and Nath noted in their review that AI is particularly beneficial in special cases, such as those of patients with rare diseases [14]. In surgeries where drug infusions are computer-assisted through closed-loop systems, the analysis of anesthesia depth and hemodynamic data has allowed the maintenance of vital parameters within narrower ranges. Joosten et al. demonstrated that patients using a computer-assisted closed-loop drug infusion system exhibited improved cognitive functions in the postoperative period [20]. In Hemmerling’s study, it was indicated that closed-loop anesthetic drug infusion systems under the control of an anesthesiologist could become routine in the future [21].

In our study, although AI was able to suggest anesthesia methods based on shared patient information, it was concluded that further development is needed regarding adherence to the current guidelines and the management of specific patient groups. Furthermore, while no statistical differences were found among the AI applications in terms of their approaches to patients, low-level differences were observed. The Copilot application chose general anesthesia more frequently than other applications. It was noted that for patients with pulmonary pathologies such as asthma or COPD, general anesthesia was sometimes recommended without querying vital parameters like the patient’s saturation value. Özsahin analyzed the chosen anesthetic agents and their alternatives while using various algorithms with the Fuzzy PROMETHEE application, emphasizing the necessity of expert support [22].

This study has several limitations due to its single-center design, the inclusion of patients within only a 1-month period, and the exclusion of patients who declined the anesthesiologist’s preference. The exclusion of patients who did not consent to the anesthesiologist’s choices and the lack of pediatric age groups prevent us from reflecting a broader population with our findings. Furthermore, as a potential limitation of the present study, only free versions of AI programs were used.

AI cannot fully master the guidelines and exceptional and specific cases that arrive in the course of medical treatment. Thus, we believe that AI can serve as a valuable assistant rather than replacing doctors.