When Kathy Callaghan called her local pharmacy for a prescription refill last Wednesday, she expected to hear the usual AI voice that had attempted to assist her for the last three months.

Instead, a pharmacist picked up.

“We were thrilled,” she said, describing her and her partner Dave Bellini’s reaction to the voice.

The call came as the pharmacy, Kinney Drugs, recently pulled back its AI assistant, John Marraffa, president of the Vermont and New York pharmacy chain, told WCAX on Friday.

In May, Kinney Drugs had implemented the AI tool, named Burt after the pharmacy’s founder, to improve patient communication and refill services. Customers, however, complained that it coincided with a range of issues, including medication delays, inaccurate refill information, and general confusion. Privacy experts raised concerns about the security of protected health information, a characterization Kinney Drugs denied.

“Burt was introduced to improve access, convenience, and responsiveness for patients. Burt is designed to supplement pharmacy teams by handling routine, non-clinical interactions so staff can focus more time on direct patient care, clinical questions, and in-pharmacy services,” a Kinney Drugs representative previously wrote in a statement to VtDigger.

But Marraffa acknowledged in his interview with WCAX that the new technology had been creating issues for customers, saying it did not get the customer experience “right,” while defending the AI tool’s security. Marraffa and a Kinney representative did not respond to requests for comment Monday from VtDigger.

The scaling back of the AI tool means that the pharmacy has returned to its previous prompt-based, touch-tone phone system for patient calls, Marraffa told WCAX. The AI tool will still be used for prescription refill texts. Now, however, patients must opt in to receiving these texts.

Previously, any customer who provided their telephone number to Kinney Drugs automatically consented to interacting with the AI tool, according to the pharmacy’s terms. That lack of transparent consent was one of the many issues that both Kinney Drugs customers and experts had with the AI.

Callaghan previously told VtDigger that she would frequently receive calls from the AI tool requesting refill orders on her prescriptions. Caught off guard, she would approve the requests, believing they were accurate. Often, however, they were not, and she received refills on prescriptions she did not need more of, she said.

“They got a winning team there, and they got great humans,” Bellini said. “They should have never gone to robots. Most of their customers are old, or they got an illness. They don’t want to deal with that.”

Bellini, who previously told VtDigger that the AI tool was eroding Kinney Drugs’ small-scale feel and personable customer service, said he also brought his issues to the company itself, calling their corporate office to voice his concerns.

A Kinney Drugs representative previously told VtDigger that the company was “aware that some patients have experienced frustration with Burt, especially during rollout,” and that it was reviewing each concern and attempting to reach each patient.

The implementation of the AI tool came as other healthcare systems nationwide are embracing the new technology. Legal experts said the implementation was outpacing regulation, leaving consumer protection thin. In Vermont, a new data privacy bill signed by Gov. Phil Scott in June does not take effect until 2028.

“The company finally admitted that it was a mistake,” Bellini said. “It’s just a lesson learned, and let’s go back to business as usual.”

This story was first published by VrDigger and is republished with permission as part of the Vermont Journalism Trust’s Community News Sharing Project.