Validating predictions of burial mounds with field data: the promise and reality of machine learning
| Date | 26 April 2024 |
| Pages | 1167-1189 |
| DOI | https://doi.org/10.1108/JD-05-2022-0096 |
| Published date | 26 April 2024 |
| Subject Matter | Library & information science,Records management & preservation,Document management,Classification & cataloguing,Information behaviour & retrieval,Collection building & management,Scholarly communications/publishing,Information & knowledge management,Information management & governance,Information management,Information & communications technology,Internet |
| Author | Adela Sobotkova,Ross Deans Kristensen-McLachlan,Orla Mallon,Shawn Adrian Ross |
Validating predictions of burial
mounds with field data:
the promise and reality
of machine learning
Adela Sobotkova
Department of History and Classical Studies, Aarhus Universitet, Aarhus, Denmark
Ross Deans Kristensen-McLachlan and Orla Mallon
Center for Humanities Computing, Aarhus Universitet, Aarhus, Denmark, and
Shawn Adrian Ross
Department of History and Archaeology, Macquarie University, Sydney, Australia
Abstract
Purpose –This paper provides practical advice for archaeologists and heritage specialists wishing to use ML
approaches to identify archaeological features in high-resolution satellite imagery (or other remotely sensed
data sources). We seek to balance the disproportionately optimisticliterature related to the application of ML to
archaeological prospection through a discussion of limitations, challenges and other difficulties. We further
seek to raise awareness among researchers of the time, effort, expertise and resources necessaryto implement
ML successfully, so that they can make an informed choice between ML and manual inspection approaches.
Design/methodology/approach –Automated object detection has been the holy grail of archaeological
remote sensing for the last two decades. Machine learning (ML) models have proven able to detect uniform
featuresacross a consistent background,but more variegatedimagery remains a challenge.We set out to detect
burial mounds in satellite imagery from a diverse landscape in Central Bulgaria using a pre-trained
Convolutional Neural Network (CNN) plus additional but low-touch training to improve performance. Training
was accomplished using MOUND/NOT MOUND cutouts, and the model assessed arbitrary tiles of the same
size from the image. Results were assessed using field data.
Findings –Validation of results against field data showed that self-reported success rates were misleadingly
high, and that the model was misidentifying most features. Setting an identification threshold at 60%
Validating ML
predictions of
burial mounds
1167
© Adela Sobotkova, Ross Deans Kristensen-McLachlan, Orla Mallon and Shawn Adrian Ross.
Published by Emerald Publishing Limited. This article is published under the Creative Commons
Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works
of this article (for both commercial and non-commercial purposes), subject to full attribution to the
original publication and authors. The full terms of this licence may be seen at http://creativecommons.
org/licences/by/4.0/legalcode
This research was funded by the Aarhus University Digital Literacy Initiative. We thank Cormac
Purcell for initial elaboration of manually trained CNN, which led us to our current approach. Data were
acquired by the participants of the Tundzha Regional Archaeological Project (2009–2011) with the
support of the Australian Research Council Linkage Projects Funding scheme LP0989901, University of
Michigan International Grant, GeoEye Foundation, Kazanlak Historical Museum, Sofia University of St.
Kliment Ohridski, University of New South Wales; Macquarie University; American Research Center,
Sofia; and the Institute for the Study of Aegean Prehistory. All of the machine-learning computation
done for this project was performed on the UCloud interactive HPC system, which is managed by the
eScience Center at the University of Southern Denmark.
Code: Data processing and analysis was performed using R and Python and the scripts are available
in public repositories: (1) Training data preparation and CNN prediction validation can be found in
https://github.com/adivea/cnn-testing (2) 2021 CNN classifier training and mound prediction is
implemented in https://github.com/c entre-for-humanities-computing/bu rial-mounds (3) 2022 CNN
classifier training and mound prediction is implemented in https://github.com/centre-for-humanities-
computing/MoundDetection
The current issue and full text archive of this journal is available on Emerald Insight at:
https://www.emerald.com/insight/0022-0418.htm
Received 10 August 2022
Revised 1 November 2023
Accepted 3 February 2024
Journal of Documentation
Vol. 80 No. 5, 2024
pp. 1167-1189
Emerald Publishing Limited
0022-0418
DOI 10.1108/JD-05-2022-0096
probability, and noting that we used an approach where the CNN assessed tiles of a fixed size, tile-based false
negative rates were 95–96%, false positive rates were 87–95% of tagged tiles, while true positives were only 5–
13%. Counterintuitively, the model provided with training data selected for highly visible mounds (rather than
all mounds) performed worse. Development of the model, meanwhile, required approximately 135 person-hours
of work.
Research limitations/implications –Our attempt to deploy a pre-trained CNN demonstrates the
limitations of this approach when it is used to detect varied features of different sizes within a heterogeneous
landscape that contains confounding natural and modern features, such as roads, forests and field boundaries.
The model has detected incidental features rather than the mounds themselves, making external validation
with field data an essential part of CNN workflows. Correcting the model would require refining the training
data as well as adopting different approaches to model choice and execution, raising the computational
requirements beyond the level of most cultural heritage practitioners.
Practical implications –Improving the pre-trained model’s performance would require considerable time
and resources, on top of the time already invested. The degree of manual intervention required –particularly
around the subsetting and annotation of training data –is so significant that it raises the question of whether it
would be more efficient to identify all of the mounds manually, either through brute-force inspectionby experts
or by crowdsourcing the analysis to trained –or even untrained –volunteers. Researchers and heritage
specialists seeking efficient methods for extracting features from remotely sensed data should weigh the costs
and benefits of ML versus manual approaches carefully.
Social implications –Our literature review indicates that use of artificial intelligence (AI) and ML
approaches to archaeological prospection have grown exponentially in the past decade, approaching adoption
levels associated with “crossing the chasm”from innovators and early adopters to the majority of researchers.
The literature itself, however, is overwhelmingly positive, reflecting some combination of publication bias and
a rhetoric of unconditional success. This paper presents the failure of a good-faith attempt to utilise these
approaches as a counterbalance and cautionary tale to potential adopters of the technology. Early-majority
adopters may find ML difficult to implement effectively in real-life scenarios.
Originality/value –Unlike many high-profile reports from well-funded projects, our paper represents a
serious but modestly resourced attempt to apply an ML approach to archaeological remote sensing, using
techniques like transfer learning that are promoted as solutions to time and cost problems associated with, e.g.
annotating and manipulating training data. While the majority of articles uncritically promote ML, or only
discuss how challenges were overcome, our paper investigates how –despite reasonable self-reported scores –
the model failed to locate the target features when compared to field data. We also present time, expertise and
resourcing requirements, a rarity in ML-for-archaeology publications.
Keywords Machine learning, Remote sensing, CNN, Satellite imagery, Archaeology
Paper type Article
Introduction
Machine learning (ML) has taken the world by a storm, and archaeologists too have tried to
harness the power of Convolutional Neural Networks (CNN) and other “deep learning”
approaches to extract archaeological features from remotely sensed data. A series of recent
articles have promoted the success of CNNs in cultural heritage applications (Can et al., 2021;
Caspari, 2020;Caspari and Crespo, 2019;Ekim et al., 2021;Ma et al., 2021). High success rates
reported for detecting and monitoring burial mounds in Siberia have highlighted CNNs as an
effective approach for large-scale prospection (Caspari and Crespo, 2019). Enthusiasm arising
from this study, and similar outcomes from Egypt (Woolf, 2018) must, however, be tempered
by the fact that the authors targeted uniform features situated in environments with little
variation in terrain or vegetation –indeed, with relatively little vegetation or other
confounding factors at all. Fewer studies explore the challenges presented by more difficult
environments where cultural heritage lies in diverse or thick vegetation, surrounded by
obtrusive natural and artificial features (Doyle et al., 2023;Fuentes-Carbajal et al., 2023;
Verschoof-van der Vaart et al., 2020).
Although few publications report the time, expertise, or costs associated with applying
ML to archaeological prospection, examples from projects trying to extract symbols and text
from historical maps indicate that it can be labour-intensive (Can et al., 2021;Ekim et al., 2021;
Ma et al., 2021). Correct classification of linear road features with a pre-trained model required
JD
80,5
1168
Get this document and AI-powered insights with a free trial of vLex and Vincent AI
Get Started for FreeStart Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting
Start Your Free Trial of vLex and Vincent AI, Your Precision-Engineered Legal Assistant
-
Access comprehensive legal content with no limitations across vLex's unparalleled global legal database
-
Build stronger arguments with verified citations and CERT citator that tracks case history and precedential strength
-
Transform your legal research from hours to minutes with Vincent AI's intelligent search and analysis capabilities
-
Elevate your practice by focusing your expertise where it matters most while Vincent handles the heavy lifting