As artificial intelligence becomes an increasingly common source of health information, new research from FIU Business suggests that public trust in AI-generated medical advice may be growing faster than the technology’s ability to safely deliver it—particularly in high-risk situations.
A recent study led by Pouyan Esmaeil Zadeh, associate professor of information systems and business analytics, found that users often perceive AI-generated medical guidance as credible, empathetic and authoritative, even when the advice may not be clinically appropriate. The findings raise concerns about overreliance on generative AI tools that were never designed to function as healthcare providers.
“People tend to trust AI responses because they are well written and confident,” Esmaeil Zadeh said. “But confidence does not always equal correctness, especially in healthcare.”
The study examined how individuals evaluated medical advice produced by large language models, or LLMs, commonly available to the public. Participants consistently rated AI responses as trustworthy and helpful, even in scenarios that would normally require professional medical judgment. According to the research, that perceived credibility may encourage users to delay or bypass appropriate care.
Those perception findings are reinforced by Esmaeil Zadeh’s broader body of work evaluating how AI systems actually perform across real-world medical scenarios. In a separate, large-scale study published in theInternational Journal for Quality in Health Care, Esmaeil Zadeh assessed AI-generated medical consultations across 200 simulated cases, ranging from routine illnesses to acute, life-threatening conditions.
The results revealed a clear pattern: AI performance declines as clinical risk and urgency increase. While the system performed strongly in routine care scenarios—such as common infections or general health questions—it showed significant limitations in acute care situations, including heart attacks and strokes. In those high-risk cases, accuracy dropped sharply, and evaluators identified mis-triage errors and inappropriate treatment recommendations.
“This is where the danger lies,” Esmaeil Zadeh said. “The same qualities that make AI appealing—speed, fluency, confidence—can be harmful when the situation demands immediate, expert intervention.”
Taken together, the two studies highlight a growing gap between how users perceive AI medical advice and how safely it performs. That gap is especially concerning as consumers increasingly turn to publicly available AI tools for health guidance, often without understanding their limitations.
One commonly cited example is Claude, an AI system developed by Anthropic. While Claude is frequently praised for its conversational tone and emphasis on safety, not all versions of the tool are appropriate for healthcare use. Standard consumer versions, including free and individual chat accounts, are not inherently compliant with the Health Insurance Portability and Accountability Act, or HIPAA, and should not be used to handle protected health information.
Anthropic does offer HIPAA-compliant configurations for enterprise and API users when the system is deployed within approved cloud environments and supported by a signed Business Associate Agreement. Without those safeguards, however, the use of AI tools for personal medical data poses significant privacy and regulatory risks.
“HIPAA compliance is not a label you can casually apply,” Esmaeil Zadeh said. “It requires specific technical controls, legal agreements and governance structures. Most consumer AI tools simply don’t meet that bar.”
Rather than replacing clinicians, the research suggests AI tools are best positioned as supportive technologies assisting with patient education, documentation and administrative tasks under human oversight. In low-risk settings, AI can help explain medical concepts, reinforce care instructions and reduce administrative burden. In high-risk or emergency situations, however, autonomous AI use poses unacceptable safety concerns.
The patient safety study proposes a risk-stratified framework for AI deployment, calling for strict limitations on AI use in acute care and greater oversight as clinical complexity increases. Such an approach aligns with emerging regulatory thinking around software used in healthcare and underscores the need for governance, transparency and accountability.
For healthcare leaders and policymakers, the takeaway is clear: trust in AI must be calibrated, not assumed. As AI tools continue to evolve and proliferate, understanding where they add value—and where they fall short—will be critical to protecting patients and maintaining quality of care.
“The future of AI in healthcare is not about replacing doctors,” Esmaeil Zadeh said. “It’s about using these tools responsibly, with clear boundaries, so that innovation enhances care rather than putting it at risk.”