Tianci Xia,Yuan Sun,,Xiaobing Zhao,Wei Song and Yumiao Guo

Abstract:With the emergence of large-scale knowledge base,how to use triple information to generate natural questions is a key technology in question answering systems.The traditional way of generating questions require a lot of manual intervention and produce lots of noise.To solve these problems,we propose a joint model based on semi-automated model and End-to-End neural network to automatically generate questions.The semi-automated model can generate question templates and real questions combining the knowledge base and center graph.The End-to-End neural network directly sends the knowledge base and real questions to BiLSTM network.Meanwhile,the attention mechanism is utilized in the decoding layer,which makes the triples and generated questions more relevant.Finally,the experimental results on SimpleQuestions demonstrate the effectiveness of the proposed approach.

Keywords:Generating questions,semi-automated model,End-to-End neural network,question answering.

1 Introduction

Question answering (QA)over knowledge base (KB)has attracted more and more researchers’ attention in recent years.One of the main reasons is the rapid development of large-scale knowledge base,such as Freebase [Bollacker,Cook and Tufts (2007)],DBpedia[Hellmann,Morsey,Van Kleef et al.(2015)],and so on.However,QA systems need largescale question-answer pairs,so how to generate questions based on KB has become a difficult problem in the field of natural language processing (NLP).At present,there are two main ways to solve this problem.One is template-based method [Su,Sun,Sadler et al.(2016);Chen,Aist and Mostow (2009);Ali,Chali and Hasan (2010)],and the other is Endto-End neural network methods [Serban,Garc?adur??n,Gulcehre et al.(2016)].

The template-based method needs to build question templates manually and replace the entity position with special placeholder.Then using entities in KB to replace the placeholder of the original position,and generating the question after manual verification.These questions meet the requirement of the QA system,and there are almost no grammatical and semantic errors.But the process of constructing templates is time consuming,and the generalization of templates is very poor.The End-to-End neural network method directly sends triples of KB and questions to the neural network for training.After a certain of iterations,questions corresponding to triples are generated.This method is simple and does not require manual intervention.But the generated questions often have some grammatical and semantic errors.In our work,we avoid shortcomings of two ways.Firstly,we use the semi-automated model based on KB to generate question templates and real questions automatically.Then,we send the KB and real questions to the Bi-directional long short-term memory (BiLSTM)neural network to generate questions related to triples.Finally,we compare the questions generated by two models and choose the one with larger bilingual evaluation understudy (BLEU)[Papineni,Roukos,Ward et al.(2002)] value as the final output.

The contributions of this paper can be summarized as follows：

(1)We use center graph to automatically build question templates.Using templates and KB to make the question generation smoother.

(2)We use KB and End-to-End neural network to automatically generate questions.Also,the attention mechanism is utilized in the decoding layer to make triples and generated questions more relevant.

(3)We choose better questions by comparison automatically.

2 Related work

In recent years,the field of generating questions has attracted more and more attention.Some template-based methods have been proposed.Ali proposedwh-inversionandwhfrontingmethods which use the way of matching content entities like “who”,“where” to generate questions directly[Ali,Chali and Hasan (2010)].However,it does not make full use of the semantic information in the content.The semantic information can be obtained through named entity recognition [Mannem,Prasad and Joshi (2010);Yao (2010)] and semantic role labeling [Chen,Aist and Mostow (2009)].In Mendes et al.[Mendes,Piwek,and Boyer (2012)],the question is classified according to its syntactic structure,answer type and question prefix.Then the model learns patterns to generate questions.After identifying Keywords in the question,Graesser et al.[Graesser,Gordon and Brainerd(1992)] classified questions according to 18 predefined categories.On this basis,Chen et al.[Chen,Aist and Mostow (2009)] generated target questions by handcrafted-templates.Recently,the way of building templates integrating KB has become a research hotspot.There are some KB-based QA datasets constructed by manual annotation,such as QALD[Lopez,Unger,Cimiano et al.(2013)],FREE917 [Cai and Yates (2013)],SimpleQuestions [Petrochuk and Zettlemoyer (2018)],and so on.

Another effective way to construct QA dataset is based on End-to-End neural network model [Serban,Garc?adur?,Gulcehre et al.(2016)],which uses the recurrent neural network to automatically formulate questions based on KB.Olney et al.[Olney,Graesser and Person (2012)] took triples as inputs to Long Short-Term Memory (LSTM)network.

Subject entities of triples are replaced by placeholders in pre-defined position.Similarly,Duma et al.[Duma and Klein (2013)] fed the triple directly into the neural network and generated short text descriptions with placeholders,which can generate fact questions by filling slots.In addition,with the development of deep learning methods,Generative Adversarial Networks (GANs)are gradually used in NLP tasks,such as text generation[Yu,Zhang,Wang et al.(2017);Li,Monroe,Shi et al.(2017)],distant supervised relation extraction [Zeng,Dai,Li et al.(2018)],and so on.These works can be used for references in question generation.

Similar to Olney,our baseline model mainly defines some relational templates and uses placeholders to replace head entities.The main differences are that our model optimizes question templates by center graph,and uses the question selector to choose the best question as output.At the same time,in order to optimize the neural network model,we feed back the output to End-to-End model and get better parameters.

3 Model architecture

The overall framework of the joint model is shown in Fig.1.It is mainly divided into four parts：KB preprocessing,semi-automated model,End-to-End neural network and question selector.

The KB preprocessing is divided into two subtasks：center entity set for semi-automated model and triple embedding for End-to-End neural network.

The semi-automated model combines the center graph and ontology to generate question templates and real questions.The End-to-End neural network mainly encodes the triple of KB to BiLSTM,and uses the attention mechanism in the decoding layer to generate the related questions.Finally,we choose the question with larger BLEU value as the final output by the question selector.

3.1 Preprocessing for knowledge base

In this section,we treat the KB as two subtasks respectively：center entity set for semiautomated model and triple embedding for End-to-End neural network.Firstly,we define the triple in KB as ＜Subject,Relation,Object＞.It can be abbreviated asThe semi-automated model can find all triples in KB associated with the center entity like Yao Ming in Fig.1.It should be noted that if the number of selected triples is too small,we can find other entities of the same type as the center entity.For example,according the center entity Yao Ming,we get three triples related to it：＜Yao Ming,wife,Ye Li＞,＜Yao Ming,birthtime,1980＞,and ＜Yao Ming,birthplace,ShangHai＞.Then we retrieve all entities in the three triples and find the entity Ye Li has the same type as center entity,so we traverse with Ye Li and get a new triple ＜Ye Li,birthplace,ShangHai＞.We use the TransE [Bordes,Usunier,García-Durán et al.(2013)] algorithm to map the triple＜Yao Ming,wife,Ye Li＞ into the low-dimensional dense vector space,and get the triple embedding ＜Enc(Yao Ming),Enc(wife),Enc(Ye Li)＞.Enc(Yao Ming),Enc(wife),Enc(Ye Li)are embeddings of subject entity Yao Ming,relation wife and object entity Ye Li.And put the triple embedding ＜Enc(Yao Ming),Enc(wife),Enc(Ye Li)＞ to Endto-End neural network.

Figure1:The overall framework of the joint model

3.2 Semi-automated model

3.2.1 Center graph

Inspired by the structure diagram of KB,the center graph method is adopted.It can visualize the syntax information and the semantic information.For the syntax information,the graph queryqis built on a given KBK.It consists of three nodes：center node(double rounded rectangle),indirect node (rounded rectangle)and direct node (shaded rounded rectangle).For the semantic information,graph query is the proper subset ofλcalculus shown in the following.

The answer of the question can be easily obtained from the center node.For example,the center node “Yao Ming” is the answer to the question “Who was born in ShangHai and his wife born in the same place?”,shown in Fig.1.Different from [Yih,Chang,He et al.(2015)],graph query has good expression which is not restricted by tree structure.

3.2.2 Generating question templates and real questions

This section mainly describes how to generate question templates and real questions.The semi-automated model takes triples and ontology as input.Also,the operations are random to prevent the biases in generating questions.For example,we first select the birthperson node as the center node.Then traversing other nodes and edges connected to the center node in turn,we get question templates by combining these nodes.In addition,according to the ontology,the node is replaced with the corresponding entity to get the real question.

Functional words are very important components in the process of generating questions.Functional words generally refer to the interrogative words corresponding to the center entity.In Fig.1,the functional words corresponding to Yao Ming may be “who” or“whose”.These functional words are predefined and frequently used,such as “who”,“what”,“whose”,“when”,“where” and so on.

When the question template is generated,we use the question template to directly match the triples of the same relation in the KB to get more questions.And the answer to the question is the center entity of the current triple,described in Section 3.2.1.

3.3 End-to-End neural network model

We address this task with a model inspired by the recent successful neural machine translation model [Sutskever,Vinyals and Le (2014);Neco and Forcada (1997)].We can regard the task as “differential translation”,that is,the transformation from KB to natural language question,while ignoring the potential structured knowledge in the model.These models usually consist of two parts：encoding layer which mainly encodes the source triples into a fixed size intermediate vector and decoding layer which decodes the intermediate vector into the target question.

3.3.1 Encoding layer

Contrary to the neural machine translation network,our source language is not a natural language,but triples in KB.We use the TransE algorithm,which maps each element of the triple into embedding.Each element in fact {s,r,o}represents the subject entity,the relation and the object entity respectively.At the same time,the factF=(s,r,o)is mapped to a embedding of 1-of-kvector,which is obtained byef=Ein(s,r,o),whereEinis the embedding matrix of the whole input entities,andkis the vector size of the vocabulary.

The embedding matrixEinis a parameter to learn from the model.We have learned it in advance through the TransE algorithm.We adjust it in the pre-training process,and do not allow the model to be modified during the model training process.

3.3.2 Decoding layer

In the decoding layer,we use the BiLSTM with an attention-mechanism [Bahdanau,Cho and Bengio (2014)] to decode the output of the encoding layer into questionQrelated to the factF.Recently,BiLSTM is more effective and has achieved good results in many fields,which compared with other recurrent neural network architectures,such as LSTM[Greff,Srivastava,Koutník et al.(2017)] and Gated Recurrent Unit (GRU)[Chung,?aglar Gül?§ehre,Cho et al.(2014)].The time stepnof the decoding layer in each hidden state of BiLSTM is computed (forward is the same as backward),as shown in Eq.(1)-Eq.(4).

3.3.3 Generating questions

In order to solve the problem of sparse triples in KB,we draw on the idea of using placeholders in neural machine translation by Luong et al.[Luong,Sutskever,Le et al.(2014)].For each question and answer pair,we try to match the words in the question with the subject entity in the triple as much as possible.We estimate the sequence of words most likely to occur in the question from a heuristic perspective,and they correspond to the subject entity.

At the same time,we use the questions selected in question selector to optimize the parameters of the End-to-End neural network model.We use the selected questions as the standard answers in the training set,and the generated questions as the predictive answers.By optimizing the parameters,we can narrow the difference between the two results.

3.4 Question selector

We compare questions generated from the semi-automated model and End-to-End neural network.Using the BLEU value to choose better questions.The question selector has two functions.On the one hand,the selected question can be used as the final output of the whole model.On the other hand,the selected question can be sent into the neural network for parameter optimization.Also,human evaluation is important to help optimize the question structure and paraphrasing.Finally,we construct the QA pairs.

4 Experiments

4.1 Dataset

We use the SimpleQuestions dataset [Petrochuk and Zettlemoyer (2018)] to train our model.SimpleQuestions is mainly based on Freebase triples,using manual annotation to map the triples and questions.It contains over 100K Freebase English QA pairs.Topics and relationships are clearly presented in each question.

4.2 Training

All neural network models were implemented in Tensorflow [Abadi (2016)].In order to train our model effectively,we use gradient optimization algorithm Adam [Kingma and Ba (2015)] for maximum likelihood estimation.At the same time,in order to accelerate convergence,we use Practical Recommendations method [Bengio (2012)] to adjust parameters in the verification set.In this paper,the Simple Questions dataset is divided into training set,validation set,and testing set.We train the TransE algorithm for each subject entity,relation and object entity.We set the learning rate to 0.01 initially,which makes the model unable to converge.So,we try to reduce this value.When the value is 0.00001,the model can converge.We further set the embedding dimension of the word to 200 and the hidden state of the decoder to 600.And we initially set the num of units to 32,but the training speed is too slow.Therefore,we reduce it to 16 under the premise of guaranteeing the result.Hyper parameters for network training are shown in Tab.1.

Table1:Hyper parameters setting

4.3 Evaluation

In order to verify the validity of our model,we evaluate it using two methods：automated evaluation and human evaluation.

4.3.1 Automatic evaluation

BLEU and METEOR [Bahdanau,Cho and Bengio (2014)] are two commonly used evaluation indicators in machine translation and image-caption generation [Satanjeev(2005)].Similar to machine translation,the main evaluation indicators in this paper are the mapping of BLEU to natural questions from KB.It is especially effective for the short sentence.Furthermore,most of the effective outputs in both missions are mutually interpreted,BLEU and METEOR are mainly used to evaluate the correlation between input and output.Thus,BLEU and METEOR can be used to evaluate the correlation between triples and questions.

In this paper,we use BLEU as the evaluation because it performs well on statements and matches well at the corpus level.Firstly,we introduce the Brevity Penalty (BP)value which is computed in Eq.(6).

wherecis the length of the generating question,ris the length of the real question.And the BLEU value is computed shown in Eq.(7).

wherepnis the n-gram value,wnis the context sliding window.We set thewnis 1/4,that is,this evaluation indicator is BLEU-4.

The BLEU-4 value in each epoch is shown in Fig.2.At the 10thepoch,the BiLSTM +Attention model is clearly better than others.After the 10thepoch,each method is obviously improved,and the Semi-automated + BiLSTM + Attention model (our model)is better than others.By comparing the generated questions of BiLSTM with attention model and Semi-automated model,we select the questions with higher BLEU-4 value as the final output.The experimental results of different models are shown in Tab.2.Compared with Semi-automated,LSTM,BiLSTM and BiLSTM + Attention,our model increases BLEU-4 value by 84.2%,56.9%,32.6% and 4.9% respectively.And through Tab.3,we can see that the generated questions are no longer the simple questions,which have higher quality of our model than others in human evaluation.

Table2:The BLEU-4 value in various models

Figure2:Three models for each epoch

4.3.2 Human evaluation

Also,we consider the results for relevance and fluency separately.The most important part of the question depends on the relation in the triple,because the generating questions should be related to the relation and contain the semantic information of the relation.In order to validate this point,we try to evaluate the relevance of the question and relation to verify whether the question is good or bad.Tab.3 shows the results which using a fact to generate the most relevant question by different models.

Table3:Examples and corresponding questions

Cancer＞ of? was a singer? a singer?＜Venice,location,Italy＞＜Prince adventure,written work,subjects＞Name a location in Italy?What subjects is racing to the beginning of the road written about?Where is the Venice and is a beautiful city?What are the subjects?Which city is beautiful in Italy?What is about beginning of the road written?Where is the Venice and is a beautiful city?What is about beginning of the road written?

5 Conclusion

We propose a joint model based on semi-automated and End-to-End neural network for mapping KB facts into corresponding natural language questions.The End-to-End neural network with the attention mechanism is utilized to generate questions,which make the triples and generated questions more relevant.Generated questions are evaluated by BLEU-4 value and human evaluation.The experimental results show the effectiveness of the proposed method.

Acknowledgement:This work is supported by National Nature Science Foundation (No.61501529,No.61331013),National Language Committee Project of China (No.ZDI125-36)and Young Teachers' Scientific Research Project in Minzu University of China.