AI and Archaeology — How can they cooperate?

By Bogdan K, Mael C, Mike M, Dhrithi G, Leonardo M — MehtA+ AI/Machine Learning Research Bootcamp students

In a project in partnership with CUNY professor, Prof. Elizabeth Macaulay, high school students in MehtA+ AI/Machine Learning Research Bootcamp were provided with a United Nations Gifts Dataset and tasked to use AI to understand why? In part 4 of a seven part series, students explore ways in which AI can help us understand archaeological gifts better.

If you would like to learn more about MehtA+ AI/Machine Learning Research Bootcamp, check out https://mehtaplustutoring.com/ai-ml-research-bootcamp/.

*******************

Research question: Why was a particular object given to the UN?

Problem Statement: It is valuable for archaeologists to know why a certain artifact was given as a gift for the UN. To fully answer this question, a researcher needs to deeply understand the country’s motives, the time period, relations between countries, etc. Unfortunately, to accomplish this research, one needs a huge amount of resources and a lot of time. Hence, our project investigates how it is possible to assist research in this topic by first providing relevant information from the UN website summarized by Google’s Large Language Model (LLM) Gemini and then giving an opportunity to investigate the question further through a Google search assisted by Gemini.

Code: https://github.com/MehtaPlusTutoring/studentprojects/blob/main/aimlresearchbootcamp/2024/midterm/MidprojectArchaeologyProject.ipynb

Description of the project: Our project is designed to help research on why a particular object was given to the UN using Google’s LLM Gemini. After entering the object’s name and the link of the appropriate UN website, the user is initially presented with relevant information relevant to the question and a particular object. From there, the user has a chance to choose to either stay with the answer or continue researching. In the latter case, Gemini generates possible Google queries based on its previous response that help the user expand on their research. The user then has an option to enter a query in the appropriate input box and receive a list of links. Finally, the user can ask Gemini to summarize a certain link to determine whether it’s useful or not.

It is important to note that the main goal of this project was to aid the user’s research process and provide research ideas and not to answer the question correctly by providing a series of particular chat messages with the Gemini because each object has different reasons for being granted to the UN.

Model: For our project, we used Gemini to filter information out of the articles provided from the United Nations Gifts catalog and then provide a reason and explanation as to why each item was gifted to the UN. We chose to use Gemini due to its interdisciplinary information about any topic, its ability to process and filter through large data sets quickly, concisely summarize articles, and ability to clearly present any needed information. Gemini is also seen to perform better than some AI models when pulling factual responses due to its access to more recent information. Other AI models we saw were better suited for creative writing, storytelling, and their ability to create responses in the form of images, diagrams, and such. Thus, the particular functions and advantages of Gemini were best suited to our project, which is why we chose Google’s LLM.

Steps to answer the question for a specific object:

1. Click Runtime and and press Run All
2. Enter the name of the desired object in cell 2
3. Enter the link of the appropriate UN webpage in cell 2
4. Read Gemini’s response in cell 7
5. Enter whether the response is sufficient or whether you want to go on searching in Google in cell 8
6. Read 5 generated Google search queries in Cell 10
7. Enter a query you want to research on in Cell 11
8. Look at the Generated links in cell 11
9. Enter the link you want to look into in Cell 12
10. Look at the summary of that link in cell 12 and enter whether or not you want to look at another link

Analysis (UN website pages):

Results: Evaluation of usefulness of the UN website

We measured the accuracy of Gemini when summarizing the UN page by using the metric bad/sufficient/excellent. These refer not to whether or not the question was answered correctly, but to what extent Gemini was able to produce a relevant and adequate response. We rated the accuracy based on our own judgement. The accuracy for the program was approximately 15% of the time wrong, 50% of the time it gave sufficient answers, and 35% of the time it gave excellent answers.

This data suggests that for most of the objects, Gemini is not able to provide a sufficient answer from the UN website and further research is required. However, in some cases, the AI worked really well as for “The replica of the Ngoc Lu Bronze Drum,” where the AI responded:

Here, it states a very specific reason for why the gift was given to the UN: it marked the 50th anniversary of both Vietnam and the United Nations. There were also cases for which the AI didn’t have an answer — as in the case of the Amphora, where Gemini did not give any concrete reason for why Cyprus may have given this gift. Despite this, the AI still provided inferences about the reasons for the gift, which aids further research. Finally, it is important to note that there were no cases where the AI outputted something that didn’t have to do anything with the topic.

Analysis (in-depth evaluation using Replica of “The Golden Man”):

In this section we evaluate to what extent Gemini was able to help to answer the question “Why was the Replica of “The Golden Man” given to the UN?

After entering the name of the object and the link of the appropriate UN website as shown above, the Gemini gave this response:

While this response indeed helps to answer the question and provides a multitude of reasons for why the replica was given to the UN, some points miss specific context. For example, in the sections “International Recognition” and “Diplomacy and Goodwill,” Gemini doesn’t give background information on the time period and the level of the relations between Kazakhstan and other nations at the time. Thus, we pick an option to go further into research.

Now, the Gemini generates 5 possible queries:

All of these queries seem to expand on the points from Gemini’s previous response. For instance, the first query appears to be useful to the research as it searches for specific articles about the time period when the object was given. This is indeed very helpful for the research process as we now understand better what we must specifically look for on the internet to answer the question as best as possible.

Now, let’s choose a query, say Kazakhstan’s foreign policy in the 1990s, and search up the relevant articles. Here are some of the links that were produced:

Finally, we input one of the links to be summarized by Gemini, and we get

As seen, this allows to quickly get the general idea of the article without actually reading through the article. This saves a lot of research time and allows you to get to the desired information faster.

Thus, our method was able to give excellent assistance in research by first providing a starting information, giving possibilities of further research in specific areas relevant to the question, and saving time looking at the article by summarizing the article’s contents.

Methods that didn’t work or were rejected:

1. At first, we thought about using Retrieval Augmented Generation for this project to extract appropriate information from the UN website. However, we rejected this method because we weren’t aiming to extract information solely based on a specific word or phrase, and the information we needed for our question didn’t necessarily have to stick to a specific word order.
2. To save time and to not copy and paste the link manually for every object, we tried to automatically generate the link for the UN web page containing the information about the object based on the name of the object. We thought this would be possible because all the UN links in the spreadsheet have the same starting portion (i.e. https://www.un.org/ungifts, after which follows the name of the object). However, that failed to work because the name of the object didn’t always match that in the link. For example, Replica of Ngoc Lu Bronze Drum was the replica-ngoc-lu-bronze-drum in the link. While we were able to make the name lowercase and add “-”s between spaces, the name missed the word “of”. Since in many links similar words were missing, we decided that it would be best to just copy and paste the link manually.

1. To save time on getting the link, we also attempted to extract links directly from the United Nations Gifts website, where we discovered that the images included both the link for the object and the object name. However, the image tags were inconsistent and challenging to sort through within our limited time frame. Ultimately, we opted to manually enter the website URL instead. Given more time, this method could have been feasible, allowing us to extract the relevant information more efficiently.

Limitations:

1. Majority of the passages mention what the gifted object is a symbol of and what it represents but do not explicitly mention why the object was gifted. Due to this, in most of the summaries, Gemini provides inferences and possible pathways to look into, but no direct reasoning to answer our question, “Why was a particular object given to the UN?” Additionally, the context of why each gift was given relies on multiple factors other than what is mentioned in the article and summaries, Gemini takes these factors into account but doesn’t go in-depth on them.
2. Since our research question is vague, we have no real way of evaluating the extent to which Gemini’s assistance is relevant to the question. Therefore, the information gathered from the UN websites is subjective and cannot be fully trusted.

Conclusion:

Our project proved to be effective for helping research the question “Why was a particular object given to the UN?” Our method allows the user to get helpful information from an official website and from there go in-depth into a guided and efficient research. However, our method does not provide a well-defined answer or allow us to accurately rate Gemini’s responses. Overall, despite the limitations, our project serves as a valuable tool for archaeological research with the help of AI.