For millions of people suffering from diabetes, new technology enabled by artificial intelligence promises to make management much easier. Medtronic’s Guardian Connect system promises to alert users 10 to 60 minutes before they hit high or low blood sugar level thresholds, thanks to IBM Watson , “the same supercomputer technology that can predict global weather patterns.”

SwissCognitiveStartup Beta Bionics goes even further: In May, it received Food and Drug Administration approval to start clinical trials on what it calls a “bionic pancreas system” powered by artificial intelligence, capable of “automatically and autonomously managing blood sugar levels 24/7.”

An artificial pancreas powered by artificial intelligence represents a huge step forward for the treatment of diabetes—but getting it right will be hard. Artificial intelligence (also known in various iterations as deep learning and machine learning) promises to automatically learn from patterns in medical data to help us do everything from managing diabetes to finding tumors in an MRI to predicting how long patients will live . But the artificial intelligence techniques involved are typically opaque. We often don’t know how the algorithm makes the eventual decision. And they may change and learn from new data—indeed, that’s a big part of the promise. But when the technology is complicated, opaque, changing, and absolutely vital to the health of a patient, how do we make sure it works as promised?

Diabetes devices provide an example in microcosm of broader issues with artificial intelligence in medicine. The potential here is enormous in terms of improved patient health, reduced costs, and increased access to high-quality care. Imagine if every primary care physician could diagnose certain eye problems as well as an ophthalmologist, or if pictures of skin lesions could be automatically evaluated for signs of cancer. These technologies are coming—the eye example is already FDA-approved. Soon, A.I. will make predictions, recommendations, and even decisions about patient care. But ensuring that medical A.I. consistently helps patients will demand careful study and continuing oversight.

To understand the challenge in diabetes, it helps to know the technological baseline. The traditional way to administer insulin to patients with Type 1 diabetes (where the pancreas doesn’t make insulin at all) or Type 2 diabetes (where the body becomes resistant to insulin) that can’t be managed with oral medication is through finger-sticks and injections. Patients check their blood sugar by pricking the tip of a finger and using a small test strip to measure the level of glucose in the blood. Patients also inject insulin manually, using a syringe, based on blood sugar readings and knowing when they are going to need extra insulin (typically around meals).  […]