The Curious Case of Doctor Ultron: How Well is English Law Currently Suited to Manage the Inherent Risks Associated with Black Box AI Medical Diagnostics?
| Jurisdiction | United Kingdom |
| Pages | 58-87 |
| Date | 01 October 2020 |
| Published date | 01 October 2020 |
| Author | Nikhil Dutt Sundaraj |
| Subject Matter | Civil Law |
The Curious Case of Doctor Ultron58
The Curious Case of Doctor Ultron: How Well
is English Law Currently Suited to Manage the
Inherent Risks Associated with Black Box AI
Medical Diagnostics?
N D S
I. I
Hollywood has given us plenty to cherish over the years, and interestingly,
a lot of modern technological inventions may very well have seen their creative
inception on the silver screen. Not least among these inventions is artificial
intelligence (AI) of which we have seen countless iterations, rarely benevolent. From
Terminator’s Skynet to Marvel’s Ultron, artificial intelligence has been portrayed
as a potentially uncontrollable force which could easily overrun its human creators.
So, how close are we to that? Well, at present we have AI in use in many
industries and to an extent, some have replaced several tasks which used to be
manned. However, those tasks tend to be relatively simple, like how a chess
software program can play against a human opponent according to the rules of the
game. However, what may seem a step closer to the ‘Ultron’ mould is a dynamic
AI. Dynamic AI programs solve a problem using past input-output data alone in
order to identify certain features common to each input that tend to influence the
occurrence of a certain output. They do this without a set of preprogramed rules on
how to arrive at that outcome.1 In the context of medical diagnostics, this means
that an AI which is meant to detect signs of skin cancer (for instance) would be
fed past data and images of various skin nevi and the corresponding diagnosis,
1 Anon, “Artificial Intelligence in Medicine: Machine Learning” (IBM) <https://www.ibm.com/
watson-health/learn/artificial-intelligence-medicine> (accessed 29 April 2020).
Cambridge Law Review (2020) Vol V, Issue ii, 58–87
The Curious Case of Doctor Ultron 59
and then form its own ‘process’ to determine how future input data would be
diagnosed.
These AI programmes function using predictive algorithms which enable it
to diagnose a condition based on past data alone. Their use is incredibly attractive
because AI’s ability to sift through vast quantities of data saves a lot of time and
money. This lightening of practitioners’ menial yet arduous workload would free
them up to attend to other cases or tasks with which AI is unable to help and allow
junior doctors to learn more about those.2 An AI program would eectively be a
supplementary member of the medical team. Furthermore, their large workload
capacity would make them useful allies in the fight against global pandemics like
Covid-19 where mass testing and diagnosis is essential.3
Another important feature of these AI is their ‘black box’ nature. They are
called as such because the exact mechanism and overarching ‘thought process’
the AI undergoes as the algorithms interact is unknown even to their software
programmers.4 As such, the doctors, hospitals and health authorities themselves
do not know exactly how the AI arrives at a diagnosis, merely that it does. This is a
distinctive feature of black box AI called ‘opacity’.5 Of course, with this opacity
comes a question of legal accountability. Who is to be held liable for harm done
by a black box diagnostic AI? That itself is a tough question to answer as there are
multiple parties involved in the production of the AI program and its use to arrive at
a diagnosis. 6 There is an inherent risk of har m for the patient that could arise from
a device malfunction, meaning its failure to meet its performance specifications or
otherwise perform as intended, especially if is leads to misdiagnosis by the AI.7
Those risks must be dealt with, not least by the Law. Such misdiagnoses could
themselves be caused by malfunctions such as contextual bias, software glitches,
or improper use of the machine.8 The parties in question that could contribute
2 Varun H Buch, Irfan Ahmed. & Mahiben Maruthappu, “Artificial intelligence in medicine: cur-
rent trends and future possibilities” (2018)British Journal of General Practice 68(668), 143–144.
3 Wim Naudé, “Artificial Intelligence against COVID-19: An Early Review” (Medium, 12 April
2020) <https://towardsdatascience.com/artificial-intelligence-against-covid-19-an-early-re-
view-92a8360edaba> (accessed 21 April 2020).
4 Yavra Bathaee, “The Artificial Intelligence Black Box and the Failure of Intent and Causation”
(2018) Harvard Journal of Law & Technology 31(2).
5 Dallas Card., “The ‘black box’ metaphor in machine learning” (Medium, 6 July 2017) <https://
towardsdatascience.com/the-black-box-metaphor-in-machine-learning-4e57a3a1d2b0> (accessed
21 April 2020).
6 US FDA Code of Federal Regulations (CFR) Title 21, s 803.3(k).
7 William Nicholson Price II, “Black-Box Medicine” (2015) 28 Harv JL & Tech 419.
8 Max Tegmark, “Benefits & Risks of Artificial Intelligence” (Future of Life Institute, 30 May 2016)
<https://futureoflife.org/background/benefits-risks-of-artificial-intelligence/?cn-reloaded=1>
(accessed 21 April 2020).
Get this document and AI-powered insights with a free trial of vLex and Vincent AI
Get Started for FreeStart Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
