Skoltech at TextGraphs-17 Shared Task: Finding GPT-4 Prompting Strategies for Multiple Choice Questions

2024

Full text Open access: Gold Metadata (CSL-JSON)
DOI 10.18653/v1/2024.textgraphs-1.14
Data confidence 70 medium
How this score is calculated

It is built from checkable properties of the record itself: whether the article has a DOI, how many independent sources confirmed its data, whether the required fields are filled in, and how recent the latest confirmation is. Disagreement between sources and a missing identifier lower it. A value of 80 or above means the record was published without further checking; 60 to 79 means it is published at reduced confidence, with a manual check advisable. A record verified by an editor by hand is never pushed below 90 by a recalculation. This scores our data about the article, not the article itself.

Abstract

In this paper, we present our solution to the TextGraphs-17 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering (KGQA).GPT-4 alone, with chain-of-thought reasoning and a given set of answers, achieves an F1 score of 0.78.By employing subgraph size as a feature, Wikidata answer description as an additional context, and a question rephrasing technique, we further strengthen this result.These tricks help to answer questions that were not initially answered and to eliminate irrelevant, identical answers.We have managed to achieve an F1 score of 0.83 and took 2nd place, improving the score by 0.05 over the baseline.An open implementation of our method is available on GitHub. 1