Evaluation of QA systems typically involve having a set of pre-defined questions, example queries, and answers to the questions. The Question Answering over Linked Data (QALD) is an evaluation campaign on multilingual question answering over linked data. QALD-4 provides a set of biomedical questions drawn from DrugBank, SIDER and Diseaseome. We wanted to extend this evaluation to a broader set of questions of increasing complexity, and to consider the data from DBpedia, Bio2RDF and BioGateway. Towards this goal, we
(1) established a set of descriptors to characterize questions [document][spreadsheet]
(2) created questions, queries, and answers over our KB.
At the moment, evaluation module uses NLP-1 dataset for the evaluation of the full module.
Input is a language tag with configuration containing the address of the module and the number of maximum answers.
"tgm" : ["http://184.108.40.206:1555/templategeneration/templator"],
"dm" : ["http://220.127.116.11:2357/agdistis/run"],
"kb" : ["http://dbpedia.org/sparql"],
Output includes accuracy of the full system applying it to the full NLP-1 dataset.