Learning to Select, Track, and Generate for Data-to-Text
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Learning to Select, Track, and Generate for Data-to-Text
∗
Hayate Iso† Yui Uehara‡ Tatsuya Ishigaki\‡ Hiroshi Noji‡
Eiji Aramaki†‡ Ichiro Kobayashi[‡ Yusuke Miyao]‡ Naoaki Okazaki\‡ Hiroya Takamura\‡
†
Nara Institute of Science and Technology ‡ Artificial Intelligence Research Center, AIST
\
Tokyo Institute of Technology [ Ochanomizu University ] The University of Tokyo
{iso.hayate.id3,aramaki}@is.naist.jp koba@is.ocha.ac.jp
{yui.uehara,ishigaki.t,hiroshi.noji,takamura.hiroya}@aist.go.jp
yusuke@is.s.u-tokyo.ac.jp okazaki@c.titech.ac.jp
Abstract In addition, the salient part moves as the sum-
mary explains the data. For example, when gen-
We propose a data-to-text generation model erating a summary of a basketball game (Table 1
with two modules, one for tracking and the
(b)) from the box score (Table 1 (a)), the input
other for text generation. Our tracking mod-
ule selects and keeps track of salient infor- contains numerous data records about the game:
mation and memorizes which record has been e.g., Jordan Clarkson scored 18 points. Existing
mentioned. Our generation module generates models often refer to the same data record mul-
a summary conditioned on the state of track- tiple times (Puduppully et al., 2019). The mod-
ing module. Our model is considered to simu- els may mention an incorrect data record, e.g.,
late the human-like writing process that gradu- Kawhi Leonard added 19 points: the summary
ally selects the information by determining the
should mention LaMarcus Aldridge, who scored
intermediate variables while writing the sum-
mary. In addition, we also explore the ef-
19 points. Thus, we need a model that finds salient
fectiveness of the writer information for gen- parts, tracks transitions of salient parts, and ex-
eration. Experimental results show that our presses information faithful to the input.
model outperforms existing models in all eval- In this paper, we propose a novel data-to-
uation metrics even without writer informa- text generation model with two modules, one for
tion. Incorporating writer information fur- saliency tracking and another for text generation.
ther improves the performance, contributing to
The tracking module keeps track of saliency in the
content planning and surface realization.
input data: when the module detects a saliency
1 Introduction transition, the tracking module selects a new data
record1 and updates the state of the tracking mod-
Advances in sensor and data storage technolo- ule. The text generation module generates a doc-
gies have rapidly increased the amount of data ument conditioned on the current tracking state.
produced in various fields such as weather, fi- Our model is considered to imitate the human-like
nance, and sports. In order to address the infor- writing process that gradually selects and tracks
mation overload caused by the massive data, data- the data while generating the summary. In ad-
to-text generation technology, which expresses the dition, we note some writer-specific patterns and
contents of data in natural language, becomes characteristics: how data records are selected to be
more important (Barzilay and Lapata, 2005). Re- mentioned; and how data records are expressed as
cently, neural methods can generate high-quality text, e.g., the order of data records and the word
short summaries especially from small pieces of usages. We also incorporate writer information
data (Liu et al., 2018). into our model.
Despite this success, it remains challenging The experimental results demonstrate that, even
to generate a high-quality long summary from without writer information, our model achieves
data (Wiseman et al., 2017). One reason for the the best performance among the previous models
difficulty is because the input data is too large for in all evaluation metrics: 94.38% precision of re-
a naive model to find its salient part, i.e., to deter- lation generation, 42.40% F1 score of content se-
mine which part of the data should be mentioned. lection, 19.38% normalized Damerau-Levenshtein
∗
Work was done during the internship at Artificial Intel-
1
ligence Research Center, AIST We use ‘data record’ and ‘relation’ interchangeably.
2102
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2102–2113
Florence, Italy, July 28 - August 2, 2019. c 2019 Association for Computational LinguisticsDistance (DLD) of content ordering, and 16.15% Graves et al., 2016). This module is often used
of BLEU score. We also confirm that adding in natural language understanding to keep track
writer information further improves the perfor- of the entity state (Kobayashi et al., 2016; Hoang
mance. et al., 2018; Bosselut et al., 2018).
Recently, entity tracking has been popular for
2 Related Work generating coherent text (Kiddon et al., 2016; Ji
2.1 Data-to-Text Generation et al., 2017; Yang et al., 2017; Clark et al., 2018).
Kiddon et al. (2016) proposed a neural checklist
Data-to-text generation is a task for generating de- model that updates predefined item states. Ji et al.
scriptions from structured or non-structured data (2017) proposed an entity representation for the
including sports commentary (Tanaka-Ishii et al., language model. Updating entity tracking states
1998; Chen and Mooney, 2008; Taniguchi et al., when the entity is introduced, their method selects
2019), weather forecast (Liang et al., 2009; Mei the salient entity state.
et al., 2016), biographical text from infobox in Our model extends this entity tracking module
Wikipedia (Lebret et al., 2016; Sha et al., 2018; for data-to-text generation tasks. The entity track-
Liu et al., 2018) and market comments from stock ing module selects the salient entity and appropri-
prices (Murakami et al., 2017; Aoki et al., 2018). ate attribute in each timestep, updates their states,
Neural generation methods have become the and generates coherent summaries from the se-
mainstream approach for data-to-text generation. lected data record.
The encoder-decoder framework (Cho et al., 2014;
Sutskever et al., 2014) with the attention (Bah- 3 Data
danau et al., 2015; Luong et al., 2015) and copy
mechanism (Gu et al., 2016; Gulcehre et al., 2016) Through careful examination, we found that in the
has successfully applied to data-to-text tasks. original dataset ROTOW IRE, some NBA games
However, neural generation methods sometimes have two documents, one of which is sometimes in
yield fluent but inadequate descriptions (Tu et al., the training data and the other is in the test or val-
2017). In data-to-text generation, descriptions in- idation data. Such documents are similar to each
consistent to the input data are problematic. other, though not identical. To make this dataset
Recently, Wiseman et al. (2017) introduced more reliable as an experimental dataset, we cre-
the ROTOW IRE dataset, which contains multi- ated a new version.
sentence summaries of basketball games with box- We ran the script provided by Wiseman et al.
score (Table 1). This dataset requires the selection (2017), which is for crawling the ROTOW IRE
of a salient subset of data records for generating website for NBA game summaries. The script col-
descriptions. They also proposed automatic evalu- lected approximately 78% of the documents in the
ation metrics for measuring the informativeness of original dataset; the remaining documents disap-
generated summaries. peared. We also collected the box-scores associ-
Puduppully et al. (2019) proposed a two-stage ated with the collected documents. We observed
method that first predicts the sequence of data that some of the box-scores were modified com-
records to be mentioned and then generates a pared with the original ROTOW IRE dataset.
summary conditioned on the predicted sequences. The collected dataset contains 3,752 instances
Their idea is similar to ours in that the both con- (i.e., pairs of a document and box-scores). How-
sider a sequence of data records as content plan- ever, the four shortest documents were not sum-
ning. However, our proposal differs from theirs maries; they were, for example, an announcement
in that ours uses a recurrent neural network for about the postponement of a match. We thus
saliency tracking, and that our decoder dynami- deleted these 4 instances and were left with 3,748
cally chooses a data record to be mentioned with- instances. We followed the dataset split by Wise-
out fixing a sequence of data records. man et al. (2017) to split our dataset into train-
ing, development, and test data. We found 14 in-
2.2 Memory modules stances that didn’t have corresponding instances in
The memory network can be used to maintain the original data. We randomly classified 9, 2, and
and update representations of the salient informa- 3 of those 14 instances respectively into training,
tion (Weston et al., 2015; Sukhbaatar et al., 2015; development, and test data. Finally, the sizes of
2103T EAM H/V W IN L OSS P TS R EB A ST F G P CT F G 3 P CT . . . The Milwaukee Bucks defeated the New York Knicks,
105-104, at Madison Square Garden on Wednesday. The
K NICKS H 16 19 104 46 26 45 46 ... Knicks (16-19) checked in to Wednesday’s contest looking
B UCKS V 18 16 105 42 20 47 32 ... to snap a five-game losing streak and heading into the fourth
quarter, they looked like they were well on their way to that
goal. . . . Antetokounmpo led the Bucks with 27 points, 13
P LAYER H/V P TS R EB A ST B LK S TL M IN C ITY ... rebounds, four assists, a steal and three blocks, his second
consecutive double-double. Greg Monroe actually checked
C ARMELO A NTHONY H 30 11 7 0 2 37 N EW YORK ...
in as the second-leading scorer and did so in his customary
D ERRICK ROSE H 15 3 4 0 1 33 N EW YORK ...
bench role, posting 18 points, along with nine boards, four
C OURTNEY L EE H 11 2 3 1 1 38 N EW YORK ...
assists, three steals and a block. Jabari Parker contributed
G IANNIS A NTETOKOUNMPO V 27 13 4 3 1 39 M ILWAUKEE ...
15 points, four rebounds, three assists and a steal. Malcolm
G REG M ONROE V 18 9 4 1 3 31 M ILWAUKEE ...
Brogdon went for 12 points, eight assists and six rebounds.
JABARI PARKER V 15 4 3 0 1 37 M ILWAUKEE ...
Mirza Teletovic was productive in a reserve role as well,
M ALCOLM B ROGDON V 12 6 8 0 0 38 M ILWAUKEE ...
generating 13 points and a rebound. . . . Courtney Lee
M IRZA T ELETOVIC V 13 1 0 0 0 21 M ILWAUKEE ...
checked in with 11 points, three assists, two rebounds, a
J OHN H ENSON V 2 2 0 0 0 14 M ILWAUKEE ...
steal and a block. . . . The Bucks and Knicks face off once
... ... ... ... ... ... ...
again in the second game of the home-and-home series, with
the meeting taking place Friday night in Milwaukee.
(a) Box score: Top contingency table shows number of wins and
losses and summary of each game. Bottom table shows statistics
of each player such as points scored (P LAYER’s P TS), and total (b) NBA basketball game summary: Each summary
rebounds (P LAYER’s R EB). consists of game victory or defeat of the game and
highlights of valuable players.
Table 1: Example of input and output data: task defines box score (1a) used for input and summary document of
game (1b) used as output. Extracted entities are shown in bold face. Extracted values are shown in green.
t 199 200 201 202 203 204 205 206 207 208 209
Yt Jabari Parker contributed 15 points , four rebounds , three assists
Zt 1 1 0 1 0 0 1 0 0 1 0
JABARI JABARI JABARI JABARI JABARI
Et - - - - - -
PARKER PARKER PARKER PARKER PARKER
At F IRST NAME L AST NAME - P LAYER P TS - - P LAYER R EB - - P LAYER A ST -
Nt - - - 0 - - 1 - - 1 -
Table 2: Running example of our model’s generation process. At every time step t, model predicts each random
variable. Model firstly determines whether to refer to data records (Zt = 1) or not (Zt = 0). If random variable
Zt = 1, model selects entity Et , its attribute At and binary variables Nt if needed. For example, at t = 202, model
predicts random variable Z202 = 1 and then selects the entity JABARI PARKER and its attribute P LAYER P TS.
Given these values, model outputs token 15 from selected data record.
our training, development, test dataset are respec- that have been referred to. hE NT is also used to up-
tively 2,714, 534, and 500. On average, each sum- date hLM , meaning that the referred data records
mary has 384 tokens and 644 data records. Each affect the text generation.
match has only one summary in our dataset, as Our model decides whether to refer to x, which
far as we checked. We also collected the writer data record r ∈ x to be mentioned, and how to ex-
of each document. Our dataset contains 32 differ- press a number. The selected data record is used to
ent writers. The most prolific writer in our dataset update hE NT . Formally, we use the four variables:
wrote 607 documents. There are also writers who
wrote less than ten documents. On average, each 1. Zt : binary variable that determines whether the
writer wrote 117 documents. We call our new model refers to input x at time step t (Zt = 1).
dataset ROTOW IRE -M ODIFIED.2 2. Et : At each time step t, this variable indi-
cates the salient entity (e.g., H AWKS, L E B RON
4 Saliency-Aware Text Generation JAMES).
3. At : At each time step t, this variable indicates
At the core of our model is a neural language the salient attribute to be mentioned (e.g., P TS).
model with a memory state hLM to generate a 4. Nt : If attribute At of the salient entity Et is
summary y1:T = (y1 , . . . , yT ) given a set of data a numeric attribute, this variable determines if
records x. Our model has another memory state a value in the data records should be output in
hE NT , which is used to remember the data records Arabic numerals (e.g., 50) or in English words
(e.g., five).
2
For information about the dataset, please follow
this link: https://github.com/aistairc/ To keep track of the salient entity, our model
rotowire-modified predicts these random variables at each time step
2104t through its summary generation process. Run- saliency is represented as the entity and its at-
ning example of our model is shown in Table 2 tribute being talked about. We therefore propose
and full algorithm is described in Appendix A. In a model that refers to a data record at each time-
the following subsections, we explain how to ini- point, and transitions to another as text goes.
tialize the model, predict these random variables, To determine whether to transition to another
and generate a summary. Due to space limitations, data record or not at time t, the model calculates
bias vectors are omitted. the following probability:
Before explaining our method, we describe our LM E NT LM E NT
p(Zt = 1 | ht−1 , ht−1 ) = σ(W z (ht−1 ⊕ ht−1 )),
notation. Let E and A denote the sets of en-
(3)
tities and attributes, respectively. Each record
r ∈ x consists of entity e ∈ E, attribute a ∈ A, where σ(·) is the sigmoid function. If p(Zt = 1 |
and its value x[e, a], and is therefore represented LM E NT
ht−1 , ht−1 ) is high, the model transitions to an-
as r = (e, a, x[e, a]). For example, the box- other data record.
score in Table 1 has a record r such that e = When the model decides to transition to another,
A NTHONY DAVIS, a = P TS, and x[e, a] = 20. the model then determines which entity and at-
tribute to refer to, and generates the next word
4.1 Initialization
(Section 4.3). On the other hand, if the model de-
Let r denote the embedding of data record r ∈ x. cides not transition to another, the model generates
Let ē denote the embedding of entity e. Note that the next word without updating the tracking states
ē depends on the set of data records, i.e., it de- htE NT = ht−1
E NT
(Section 4.4).
pends on the game. We also use e for static em-
bedding of entity e, which, on the other hand, does 4.3 Selection and tracking
not depend on the game. When the model refers to a new data record
Given the embedding of entity e, attribute a, (Zt = 1), it selects an entity and its attribute. It
and its value v, we use the concatenation layer also tracks the saliency by putting the informa-
to combine the information from these vectors tion about the selected entity and attribute into the
to produce the embedding of each data record memory vector hE NT . The model begins to select
(e, a, v), denoted as r e,a,v as follows: the subject entity and update the memory states if
the subject entity will change.
r e,a,v = tanh W R (e ⊕ a ⊕ v) ,
(1) Specifically, the model first calculates the prob-
ability of selecting an entity:
where ⊕ indicates the concatenation of vectors,
LM E NT
and W R denotes a weight matrix.3 p(Et = e | ht−1 , ht−1 )
We obtain ē in the set of data records x, by sum-
(
E NT LM
exp hs W O LD ht−1
if e ∈ Et−1
ming all the data-record embeddings transformed ∝ LM
, (4)
exp ēW N EW ht−1
otherwise
by a matrix:
! where Et−1 is the set of entities that have already
been referred to by time step t, and s is defined as
X
A
ē = tanh W a r e,a,x[e,a] , (2)
a∈A
s = max{s : s ≤ t − 1, e = es }, which indicates
the time step when this entity was last mentioned.
where W aA is a weight matrix for attribute a. The model selects the most probable entity as
Since ē depends on the game as above, ē is sup- the next salient entity and updates the set of enti-
posed to represent how entity e played in the game. ties that appeared (Et = Et−1 ∪ {et }).
To initialize the hidden state of each module, we If the salient entity changes (et 6= et−1 ), the
use embeddings of for hLM and averaged model updates the hidden state of the tracking
embeddings of ē for hENT . model hE NT with a recurrent neural network with
a gated recurrent unit (G RU; Chung et al., 2014):
4.2 Saliency transition
E NT
Generally, the saliency of text changes during text 0
ht−1
if et = et−1
E NT E E NT
generation. In our work, we suppose that the ht = G RU (ē, ht−1 ) else if et 6∈ Et−1
G RU E (W sS hsE NT , ht−1
E NT
) otherwise.
3
We also concatenate the embedding vectors that repre-
sents whether the entity is in home or away team. (5)
2105Note that if the selected entity at time step t, et , is Subsequently, the hidden state of language model
identical to the previously selected entity et−1 , the hLM is updated:
hidden state of the tracking model is not updated.
If the selected entity et is new (et 6∈ Et−1 ), the htLM = LSTM(y t ⊕ h0t , ht−1
LM
), (11)
hidden state of the tracking model is updated with where y t is the embedding of the word generated
the embedding ē of entity et as input. In contrast, at time step t.4
if entity et has already appeared in the past (et ∈
Et−1 ) but is not identical to the previous one (et 6= 4.5 Incorporating writer information
et−1 ), we use hsE NT (i.e., the memory state when We also incorporate the information about the
this entity last appeared) to fully exploit the local writer of the summaries into our model. Specif-
history of this entity. ically, instead of using Equation (9), we concate-
Given the updated hidden state of the tracking LM
nate the embedding w of a writer to ht−1 ⊕ htE NT
model htE NT , we next select the attribute of the 0
to construct context vector ht :
salient entity by the following probability:
h0t = tanh W 0H (ht−1LM
⊕ htE NT ⊕ w) , (12)
LM 0
p(At = a | et , ht−1 , htE NT ) (6)
0
where W 0H is a new weight matrix. Since this new
∝ exp r et ,a,x[et ,a] W ATTR (ht−1
LM
⊕ htE NT ) .
context vector h0t is used for calculating the proba-
After selecting at , i.e., the most probable attribute bility over words in Equation (10), the writer infor-
of the salient entity, the tracking model updates the mation will directly affect word generation, which
memory state htE NT with the embedding of the data is regarded as surface realization in terms of tra-
record r et ,at ,x[et ,at ] introduced in Section 4.1: ditional text generation. Simultaneously, context
vector h0t enhanced with the writer information is
0
htE NT = G RU A (r et ,at ,x[et ,at ] , htE NT ). (7) used to obtain htLM , which is the hidden state of
the language model and is further used to select the
4.4 Summary generation salient entity and attribute, as mentioned in Sec-
Given two hidden states, one for language model tions 4.2 and 4.3. Therefore, in our model, the
LM
ht−1 and the other for tracking model htE NT , the writer information affects both surface realization
model generates the next word yt . We also incor- and content planning.
porate a copy mechanism that copies the value of
the salient data record x[et , at ]. 4.6 Learning objective
If the model refers to a new data record (Zt = We apply fully supervised training that maximizes
1), it directly copies the value of the data record the following log-likelihood:
x[et , at ]. However, the values of numerical at-
tributes can be expressed in at least two different log p(Y1:T , Z1:T , E1:T , A1:T , N1:T | x)
manners: Arabic numerals (e.g., 14) and English T
X
LM E NT
words (e.g., fourteen). We decide which one to use = log p(Zt = zt | ht−1 , ht−1 )
t=1
by the following probability: X
LM E NT
LM
+ log p(Et = et | ht−1 , ht−1 )
p(Nt = 1 | ht−1 , htE NT ) = σ(W N LM
(ht−1 ⊕ htE NT )), t:Zt =1
(8) X
LM 0
+ log p(At = at | et , ht−1 , htE NT )
where W N is a weight matrix. The model then t:Zt =1
X
updates the hidden states of the language model: + LM
log p(Nt = nt | ht−1 , htE NT )
t:Zt =1,at is num attr
h0t = tanh W H (ht−1 LM
⊕ htE NT ) ,
(9) X
+ log p(Yt = yt | h0t )
where W H is a weight matrix. t:Zt =0
If the salient data record is the same as the pre-
vious one (Zt = 0), it predicts the next word yt via 4
In our initial experiment, we observed a word repetition
a probability over words conditioned on the con- problem when the tracking model is not updated during gen-
text vector h0t : erating each sentence. To avoid this problem, we also update
the tracking model with special trainable vectors v REFRESH
to refresh these states after our model generates a period:
p(Yt | h0t ) = softmax(W Y h0t ). (10) htE NT = G RUA (v R EFRESH , htE NT )
2106RG CS CO
Method B LEU
# P% P% R% F1% DLD%
G OLD 27.36 93.42 100. 100. 100. 100. 100.
T EMPLATES 54.63 100. 31.01 58.85 40.61 17.50 8.43
Wiseman et al. (2017) 22.93 60.14 24.24 31.20 27.29 14.70 14.73
Puduppully et al. (2019) 33.06 83.17 33.06 43.59 37.60 16.97 13.96
P ROPOSED 39.05 94.43 35.77 52.05 42.40 19.38 16.15
Table 3: Experimental result. Each metric evaluates whether important information (CS) is described accurately
(RG) and in correct order (CO).
5 Experiments T EMPLATES summaries. The G OLD summary is
exactly identical with the reference summary and
5.1 Experimental settings each T EMPLATES summary is generated in the
We used ROTOW IRE -M ODIFIED as the dataset for same manner as Wiseman et al. (2017).
our experiments, which we explained in Section 3. In the latter half of our experiments, we exam-
The training, development, and test data respec- ine the effect of adding information about writers.
tively contained 2,714, 534, and 500 games. In addition to our model enhanced with writer in-
Since we take a supervised training approach, formation, we also add writer information to the
we need the annotations of the random variables model by Puduppully et al. (2019). Their method
(i.e., Zt , Et , At , and Nt ) in the training data, as consists of two stages corresponding to content
shown in Table 2. Instead of simple lexical match- planning and surface realization. Therefore, by
ing with r ∈ x, which is prone to errors in the incorporating writer information to each of the
annotation, we use the information extraction sys- two stages, we can clearly see which part of the
tem provided by Wiseman et al. (2017). Although model to which the writer information contributes
this system is trained on noisy rule-based annota- to. For Puduppully et al. (2019) model, we attach
tions, we conjecture that it is more robust to errors the writer information in the following three ways:
because it is trained to minimize the marginalized
loss function for ambiguous relations. All training 1. concatenating writer embedding w with the in-
details are described in Appendix B. put vector for LSTM in the content planning
decoder (stage 1);
5.2 Models to be compared 2. concatenating writer embedding w with the in-
We compare our model5 against two baseline put vector for LSTM in the text generator (stage
models. One is the model used by Wiseman 2);
et al. (2017), which generates a summary with an 3. using both 1 and 2 above.
attention-based encoder-decoder model. The other For more details about each decoding stage, read-
baseline model is the one proposed by Puduppully ers can refer to Puduppully et al. (2019).
et al. (2019), which first predicts the sequence of
data records and then generates a summary condi- 5.3 Evaluation metrics
tioned on the predicted sequences. Wiseman et al.
As evaluation metrics, we use BLEU score (Pap-
(2017)’s model refers to all data records every
ineni et al., 2002) and the extractive metrics pro-
timestep, while Puduppully et al. (2019)’s model
posed by Wiseman et al. (2017), i.e., relation gen-
refers to a subset of all data records, which is pre-
eration (RG), content selection (CS), and content
dicted in the first stage. Unlike these models, our
ordering (CO) as evaluation metrics. The extrac-
model uses one memory vector htE NT that tracks
tive metrics measure how well the relations ex-
the history of the data records, during generation.
tracted from the generated summary match the
We retrained the baselines on our new dataset. We
correct relations6 :
also present the performance of the G OLD and
6
The model for extracting relation tuples was trained on
5
Our code is available from https://github.com/ tuples made from the entity (e.g., team name, city name,
aistairc/sports-reporter player name) and attribute value (e.g., “Lakers”, “92”) ex-
2107- RG: the ratio of the correct relations out of all Willie Green Ronny Turiaf
Phil Pressey
Chris Kaman Amir Johnson Shane Larkin Deyonta Davis
Furkan Aldemir
Nikola PekovicAlonzo Ryan
BrandanGee Shawn Hollins
WrightMarionTiago Splitter John JaVale
Salmons
the extracted relations, where correct relations Michael Carter-Williams Gary Harris
Josh Smith Ed Davis Sebastian
Carlos
Kendrick
Omer
Bernard
Quincy
Boozer
Tayshaun
Asik
Jerome
Lou
Solomon
Jarrett
Perkins
Joel
James
Ray
Spencer
Prince
Amundson
Telfair
Jodie Meeks
Elijah
Jordan
Pondexter
Jack Jose
Freeland
Hill
McCallum
Shavlik Randolph
Dinwiddie
Calderon
Henry
Millsap Sims JoeyRobert
Quincy Dorsey Dorell
McGeeDaye
MillerSacreAustin
Wright
Nikola
T.J.
Danilo Vucevic
McConnell
Landry
Gallinari FieldsJusuf Nurkic Alex DanteKirk Cunningham
Jannero Hollis
Pargo Jordan
PerryHamilton
Thompson Jones III Dunleavy
Mike
Joakim Noah Nick Ty Lawson
Chris
Young Bosh Andrea
Greg
Mason Danny
Stiemsma
Paul Kostas
Bargnani
Plumlee
Zipser Chase
Granger Papanikolaou
Pero Darrell
Antic Arthur
Budinger Nate Wolters
are relations found in the input data records x. Tyreke
Kyle Samuel
Dwyane
Wilson
Kenyon
Marcin
KorverEvans
WadeMo Chandler
Dalembert
John Henson
Kevin
Tobias
Andre
Jared
Martin
Williams
Gortat Gorgui
Ersan
Jerami
Martin
Dudley
Iman
Harris
Iguodala
Chris
Jordan
Dieng
Ilyasova
Chandler
Grant
Joe
Robbie
Jameer
Brandon
Parsons
Shumpert
Ramon
Courtney
Copeland
Danny Jonas
Ingles
Crawford
CotyHummel
Clarke
Brian
James
Sessions
Nelson
Green
Bass
Lee
Jerebko
Charlie
Roberts
Anderson
MauriceSteve Pierre
Anderson
Markel
Nemanja Joe KJ McDaniels
Cole
Villanueva Aldrich
Jackson
Varejao
Harris
Brown
Johnny
Novak Bjelica
Jimmer O'Bryant
Russ Smith
Fredette III
Brook Lopez
DeMarcusDirk Nowitzki
Cousins
Mike Lance
Conley
Langston
Brandon
Stephenson
Roy
Ingram
Trevor
Channing
Jae Crowder
Michael
Hibbert
Shawne
Galloway
Ariza
Luol
Jason
Frye Tyus Thompson
Cleanthony
Randy
Kidd-Gilchrist
Deng
Raymond
Williams
George Jorge
Avery Hill
Jones
Greivis
Ryan
Felton
Gutierrez
Bradley
Wayne
FoyeEarly
Vasquez
KellyJeff
CJ
Alexey
Ellington MilesJustin
Withey
Shved PJHarkless
Holiday Luc Mbah
Tucker FestusaBruno
JRTyrus
MouteCheick
Ezeli
Smith Caboclo Diallo
RajonGiannis
Rondo
Monta
Kevin Ellis
AlLove Chris
Horford Trey Allen
Andersen
Eric
Patrick Burke
Gordon Seth
Crabbe
PatrickAndre
Beverley Curry
Thabo Roberson
Patterson
Kentavious
Cory JosephAlexis
Sefolosha
Nate Ajinca
Arinze
Robinson
Caldwell-Pope
Alan Williams Robin
Anthony Onuaku
Lopez
Morrow
AJReggie
Denzel
DwightPrice
Valentine
Buycks
Evans Amar'e Stoudemire
Thomas
The average number of extracted relations is James
DeMar
Antetokounmpo
Gordon Tim
Kent
Nicolas
Harden
Derrick
Jimmy Otto
Rose
DeRozan
Carmelo
Duncan
Butler Draymond
Bazemore
Hayward
Kyle
Isaiah
Enes
Anthony
Derrick
Darius
Cody
Batum
Porter
JeremyTim
Julius
BrandonAndrew
Zeller
Dion
Jr.
Lin
Jeff
Randle
Kenneth
Lowry
Tyson
Whitehead
Manu
Dennis Kanter
Jonas
Marcus
Tony
Favors
Green
Hardaway Waiters
Elfrid
Green
Taj
Khris
Knight
Bogut
Caris
Valanciunas
Ginobili
Schroder
Miller ParkerLavoy
Derrick
Smart
Matt
Jr.
Brandon
Gibson
Faried
Chandler Tony Evan
Chris
Payton
Tyler
Middleton
Devin Gerald
Kelly
LeVert
Ben
Corey
Barnes Allen
Thomas
Alec
Timothe
Harris
Jared
Allen Turner
Dewayne
ShabazzJohnson
Hansbrough
Jennings
Henderson
Olynyk
Kirk Sullinger
Jeff Hinrich
Ayres
Lance
McLemore
Williams Brewer
Jon
Burks
Marcelo
Luis James
Luis
Gary
Ricky
Muhammad
Leuer
Nik
Vince
Robinson
Kyle
Thomas
Francisco
Damien
Luwawu-Cabarrot
Dedmon
Aron
Johnson
Stauskas
Scola
Norris
Montero
Neal
Jarell
Ledo Carter
Huertas Cole
Martin
Shaun
Singler
Drew Gooden
Baynes
Garcia
Georgios
Tyler
IanTim
Inglis
Jerian
Livingston
Jason
Zeller
Clark
Mindaugas
Alan
Grant
Terry
Shawn
Frazier
Rodney
JaKarr
Papagiannis
Chasson Long
Randle
Kuzminskas
Anderson
SampsonMcGruder
DeJuan
Thanasis Blair
Antetokounmpo
LaMarcus
Kyrie Paul Millsap
Aldridge
JohnIrving
LucasNerlens Alex
IIIRobert Abrines Marcus Thornton Willy Hernangomez
Ben GordonClint
CJRaul Capela
McCollum Danuel
Marc Gasol Zach
Russell
Damian
Randolph
Westbrook
Eric Bledsoe
Lillard Rudy
JoelKevin
Embiid
Hassan
John
Gobert
Myles
Markieff
Ricky
Michael
Greg Kyle
Isaiah
Aaron
Rodney
Noel
Wesley
DeAndre
Whiteside
Turner
Morris
Covington
Kyle
Monroe Arron
Rubio
David
Marvin
Jenkins O'Quinn
Beasley
Bojan
Terrence
Anderson
Thomas
Brooks
Stuckey
Donald
Omri Meyers
Matthews
Jordan
Reggie
Steven
Afflalo
Zaza
West
Williams
Ross
Will
Domantas
Bogdanovic
Casspi Steve
Ronnie
Quincy
Jason
Jackson
Adams
Barton
Leonard
Pachulia
Tony
Acy
Jordan
Kosta
Salah
Stanley
T.J.
Sloan Blake
Warren
Price
Smith
Boris
Kevon
Malcolm
Damjan
Sabonis
Snell
Henry
Koufos
Mejri
Johnson
Diaw
Clarkson
Tyler
Alex
Andre
Norman
Looney
Rudez
Walker
Ennis
Cory
Brogdon Neto
Stepheson
Darrun
Cristiano
Miller
Powell
Shelvin
Jefferson Semaj
Juancho
O.J. Mayo
Hilliard
Sasha
Felicio
Mack
Christon Jarell
JJElliot
Hernangomez
Manny Josh
Vujacic
HicksonEddie
HarrisHuestisHouse Jr.
also reported. LarryJrue
AlGarnett Drew
Jefferson Holiday Patty
IIMirza Cameron Mills
Timofey
Brandon
Teletovic
Isaiah
Matt Payne
Sean
Canaan Brandon
Kris
Ian
Mozgov
Davies
Bonner Humphries
Mahinmi
Davis
Justin
Toney
Kilpatrick Rush
Bertans
Hamilton
Douglas
Andrew Andrew
Bobby
Goudelock
Chris Nicholson
Portis
McCullough
Brice Jakob
Johnson Poeltl
Williams
AnthonyKevin Stephen
Jabari
DavisDurant
Bradley Kawhi
Beal Curry
Parker
ChrisPorzingis
Paul
Rudy Andre
Austin
Leonard
Pau
Gay Jahlil
Drummond
Nene
Gasol Okafor
Alex
Rivers Lou
Len
Joe Williams
Anthony
James
Johnson Bennett
Donatas
Ennis JaMychalC.J.Rashad
Motiejunas
IIIMiles
Shabazz
James Watson
Green
Napier
Jeremy
Young Vaughn Glenn
Evans Robinson III
LeBron Josh
Kristaps
JamesMcRoberts
Karl-Anthony Towns
Darren John Victor
Mike
Wall
Collison
J.J. Barea
Aaron Oladipo
Miller
Tyler
IshHarrisonMarcus
Tristan
Nikola
GordonUlis
Smith E'Twaun
Marreese Morris
Hedo
Thompson
Barnes
Mirotic MooreTurkoglu
Speights
Archie
Garrett
Dwight Plumlee
Goodwin
Temple
Powell Jonathon Simmons
Nemanja Nedovic
Deron JeffWilliams
Goran
Klay Buddy
Dragic
Thompson Hield
DeMarre Jamal Crawford
Carroll
Justise
Rondae
Al-Farouq Okaro
Winslow Yogi WhiteNick
Jaylen Troy
Ferrell
Jerryd
Hollis-Jefferson
Aminu Bryce Kelly
Calathes
Daniels
Jonathan
Brown
Sam
Bayless
Cotton Oubre
Mario
Skal
Gibson
Adreian
Zoran
Dekker
Montrezl Jr. Devyn
Hezonja
Joe
Labissiere
Payne
Dragic
Harrell YoungMarble
Rasual
Dahntay Butler
Jones
James Jones
Chris Douglas-Roberts Kobe Kemba
Paul George Teague
Walker Marco
Tony
Jeremy
AnthonyBelinelli
WrotenLamb Richaun
Brown Doug
Holmes
Tomas Troy
Leandro McDermott
Williams
Satoransky
Barbosa Fred VanVleet
- CS: precision and recall of the relations ex- Dwight
Reggie Bullock
Evan
Blake
D'Angelo
Bryant Howard Rodney
Trevor
Fournier
Griffin
Russell Hood
Booker
EmmanuelSerge
FrankGlen
Dario
Beno
Willie
Jabari Damian
DeAndre
Richard
Mario
Wesley
Ibaka
Davis
Kaminsky
Kevin
Ryan Terrence
Dante
Udrih
Brown Saric
Jefferson
Chalmers
Mike
MudiayMike
Seraphin
Cauley-Stein
JJJohnsonPaul
Jones
Redick David
Liggins
Scott
Anderson
Muscala
Nick
Exum
Dorian
Larry
Jones
Pierce
Gerald
Matthew
Jordan
ZachCollison
Lee Green
Dellavedova
Finney-Smith
Mitch McRae
LaVine
NanceTolliver
Anthony McGary
Jr. Carl Landry D.J. Augustin
Gigi Datome
Andrew Nikola Jordan
WigginsJokic TylerHill Ron Johnson
Baker Justin Anderson ZubacSheldon Mac
Andrew James Spencer
Harrison Michael McAdooIvica
Hawes Jordan Adams
tracted from the generated summary against Devin Jordan
Booker
Thaddeus Mickey
Jared
Kris DunnJason
Cunningham
Young
Bismack Biyombo Richardson Joffrey
Malcolm
Pascal Lauvergne
Delaney
Siakam
Tarik Black Jarrod UthoffRichardson
Malachi
those from the reference summary.
- CO: edit distance measured with normalized Figure 1: Illustrations of static entity embeddings e.
Damerau-Levenshtein Distance (DLD) between Players with colored letters are listed in the ranking top
100 players for the 2016-17 NBA season at https:
the sequences of relations extracted from the
//www.washingtonpost.com/graphics/
generated and reference summary. sports/nba-top-100-players-2016/.
Only LeBron James is in red and the other players in
6 Results and Discussions top 100 are in blue. Top-ranked players have similar
representations of e.
We first focus on the quality of tracking model and
entity representation in Sections 6.1 to 6.4, where
we use the model without writer information. We 1901). As shown in Figure 1, which is the vi-
examine the effect of writer information in Sec- sualization of static entity embedding e, the top-
tion 6.5. ranked players are closely located.
We also present the visualizations of dynamic
6.1 Saliency tracking-based model entity embeddings ē in Figure 2. Although we
As shown in Table 3, our model outperforms all did not carry out feature engineering specific to
baselines across all evaluation metrics.7 One of the NBA (e.g., whether a player scored double
the noticeable results is that our model achieves digits or not)8 for representing the dynamic en-
slightly higher RG precision than the gold sum- tity embedding ē, the embeddings of the players
mary. Owing to the extractive evaluation nature, who performed well for each game have similar
the generated summary of the precision of the rela- representations. In addition, the change in embed-
tion generation could beat the gold summary per- dings of the same player was observed depending
formance. In fact, the template model achieves on the box-scores for each game. For instance, Le-
100% precision of the relation generations. Bron James recorded a double-double in a game
The other is that only our model exceeds the on April 22, 2016. For this game, his embedding
template model regarding F1 score of the con- is located close to the embedding of Kevin Love,
tent selection and obtains the highest performance who also scored a double-double. However, he
of content ordering. This imply that the tracking did not participate in the game on December 26,
model encourages to select salient input records in 2016. His embedding for this game became closer
the correct order. to those of other players who also did not partici-
pate.
6.2 Qualitative analysis of entity embedding
Our model has the entity embedding ē, which de- 6.3 Duplicate ratios of extracted relations
pends on the box score for each game in addition As Puduppully et al. (2019) pointed out, a gen-
to static entity embedding e. Now we analyze the erated summary may mention the same relation
difference of these two types of embeddings. multiple times. Such duplicated relations are not
We present a two-dimensional visualizations of favorable in terms of the brevity of text.
both embeddings produced using PCA (Pearson, Figure 3 shows the ratios of the generated sum-
maries with duplicate mentions of relations in the
tracted from the summaries, and the corresponding attributes
(e.g., “T EAM NAME”, “P TS”) found in the box- or line-score. development data. While the models by Wiseman
The precision and the recall of this extraction model are re- et al. (2017) and Puduppully et al. (2019) respec-
spectively 93.4% and 75.0% in the test data.
7 8
The scores of Puduppully et al. (2019)’s model signifi- In the NBA, a player who accumulates a double-digit
cantly dropped from what they reported, especially on BLEU score in one of five categories (points, rebounds, assists,
metric. We speculate this is mainly due to the reduced amount steals, and blocked shots) in a game, is regarded as a good
of our training data (Section 3). That is, their model might be player. If a player had a double in two of those five cate-
more data-hungry than other models. gories, it is referred to as double-double.
2108Timofey Mozgov JR SmithLeBron James
4 James Jones
Jodie Meeks LeBron James
Mo Williams Beno Udrih
DahntaySpencer
Jones Dinwiddie
Kyrie Irving Kevin Love
2 Darrun Hilliard
JR Smith KyrieKevin
IrvingLove
Tristan Thompson
Joel Anthony Richard Jefferson Tristan Thompson
Andre Drummond
0 DeAndre Liggins Jon Leuer Kentavious Caldwell-Pope
Richard Jefferson Reggie Jackson Reggie Jackson
Tobias Harris
Kentavious Caldwell-Pope James Jones Marcus Morris
Channing
Steve Blake Frye Marcus
AndreMorris
Drummond Channing
Iman Frye
Tobias
Shumpert Harris
Iman Shumpert HenryDarrun Hilliard
Ellenson
2 Matthew Dellavedova Michael Gbinije Jordan McRae
Kay Felder
Ish Smith
Mike Dunleavy
Stanley Johnson Aron Baynes
Aron Baynes Stanley Johnson
Anthony Tolliver
4
4 2 0 2 4 4 2 0 2 4 6
April 22, 2016 December 26, 2016
Figure 2: Illustrations of dynamic entity embedding ē. Both left and right figures are for Cleveland Cavaliers vs.
Detroit Pistons, on different dates. LeBron James is in red letters. Entities with orange symbols appeared only
in the reference summary. Entities with blue symbols appeared only in the generated summary. Entities with
green symbols appeared in both the reference and the generated summary. The others are with red symbols. 2
represents player who scored in the double digits, and 3 represents player who recorded double-double. Players
with 4 did not participate in the game. ◦ represents other players.
1.00 15.8% 1 model, which mitigates redundant references and
12.7%
0.75 20.2% 2 therefore rarely contains erroneous relations.
>2
0.50 84.2% 95.8% However, when complicated expressions such
0.25 64.0% as parallel structures are used our model also gen-
0.00 erates erroneous relations as illustrated by the un-
Wiseman+'17 Pudupully+'19 Proposed derlined sentences describing the two players who
scored the same points. For example, “11-point
Figure 3: Ratios of generated summaries with dupli-
cate mention of relations. Each label represents number efforts” is correct for C OURTNEY L EE but not for
of duplicated relations within each document. While D ERRICK ROSE. As a future study, it is necessary
Wiseman et al. (2017)’s model exhibited 36.0% dupli- to develop a method that can handle such compli-
cation and Puduppully et al. (2019)’s model exhibited cated relations.
15.8%, our model exhibited only 4.2%.
6.5 Use of writer information
tively showed 36.0% and 15.8% as duplicate ra- We first look at the results of an extension of
tios, our model exhibited 4.2%. This suggests that Puduppully et al. (2019)’s model with writer in-
our model dramatically suppressed generation of formation w in Table 4. By adding w to con-
redundant relations. We speculate that the track- tent planning (stage 1), the method obtained im-
ing model successfully memorized which input provements in CS (37.60 to 47.25), CO (16.97
records have been selected in hsE NT . to 22.16), and BLEU score (13.96 to 18.18). By
adding w to the component for surface realization
6.4 Qualitative analysis of output examples (stage 2), the method obtained an improvement in
Figure 5 shows the generated examples from val- BLEU score (13.96 to 17.81), while the effects on
idation inputs with Puduppully et al. (2019)’s the other metrics were not very significant. By
model and our model. Whereas both generations adding w to both stages, the method scored the
seem to be fluent, the summary of Puduppully highest BLEU, while the other metrics were not
et al. (2019)’s model includes erroneous relations very different from those obtained by adding w
colored in orange. to stage 1. This result suggests that writer infor-
Specifically, the description about D ERRICK mation contributes to both content planning and
ROSE’s relations, “15 points, four assists, three surface realization when it is properly used, and
rounds and one steal in 33 minutes.”, is also used improvements of content planning lead to much
for other entities (e.g., J OHN H ENSON and W ILLY better performance in surface realization.
H ERNAGOMEZ). This is because Puduppully et al. Our model showed improvements in most met-
(2019)’s model has no tracking module unlike our rics and showed the best performance by incor-
2109RG CS CO
Method B LEU
# P% P% R% F1% DLD%
Puduppully et al. (2019) 33.06 83.17 33.06 43.59 37.60 16.97 13.96
+ w in stage 1 28.43 84.75 45.00 49.73 47.25 22.16 18.18
+ w in stage 2 35.06 80.51 31.10 45.28 36.87 16.38 17.81
+ w in stage 1 & 2 28.00 82.27 44.37 48.71 46.44 22.41 18.90
P ROPOSED 39.05 94.38 35.77 52.05 42.40 19.38 16.15
+w 30.25 92.00 50.75 59.03 54.58 25.75 20.84
Table 4: Effects of writer information. w indicates that W RITER embeddings are used. Numbers in bold are the
largest among the variants of each method.
The Milwaukee Bucks defeated the New York Knicks, 105-104, at
The Milwaukee Bucks defeated the New York Knicks, 105- Madison Square Garden on Saturday. The Bucks (18-16) checked in
104, at Madison Square Garden on Wednesday evening. The to Saturday’s contest with a well, outscoring the Knicks (16-19) by
Bucks (18-16) have been one of the hottest teams in the league, a margin of 39-19 in the first quarter. However, New York by just a
having won five of their last six games, and they have now won 25-foot lead at the end of the first quarter, the Bucks were able to pull
six of their last eight games. The Knicks (16-19) have now away, as they outscored the Knicks by a 59-46 margin into the second.
won six of their last six games, as they continue to battle for the 45 points in the third quarter to seal the win for New York with the
eighth and final playoff spot in the Eastern Conference. Giannis rest of the starters to seal the win. The Knicks were led by Giannis
Antetokounmpo led the way for Milwaukee, as he tallied 27 Antetokounmpo, who tallied a game-high 27 points, to go along with
points, 13 rebounds, four assists, three blocked shots and one 13 rebounds, four assists, three blocks and a steal. The game was
steal, in 39 minutes . Jabari Parker added 15 points, four re- a crucial night for the Bucks’ starting five, as the duo was the most
bounds, three assists, one steal and one block, and 6-of-8 from effective shooters, as they posted Milwaukee to go on a pair of low
long range. John Henson added two points, two rebounds, one low-wise (Carmelo Anthony) and Malcolm Brogdon. Anthony added
assist, three steals and one block. John Henson was the only 11 rebounds, seven assists and two steals to his team-high scoring total.
other player to score in double digits for the Knicks, with 15 Jabari Parker was right behind him with 15 points, four rebounds,
points, four assists, three rebounds and one steal, in 33 min- three assists and a block. Greg Monroe was next with a bench-leading
utes. The Bucks were led by Derrick Rose, who tallied 15 18 points, along with nine rebounds, four assists and three steals.
points, four assists, three rebounds and one steal in 33 minutes. Brogdon posted 12 points, eight assists, six rebounds and a steal.
Willy Hernangomez started in place of Porzingis and finished Derrick Rose and Courtney Lee were next with a pair of {11 / 11}
with 15 points, four assists, three rebounds and one steal in 33 -point efforts. Rose also supplied four assists and three rebounds, while
minutes. Willy Hernangomez started in place of Jose Calderon Lee complemented his scoring with three assists, a rebound and a steal.
( knee ) and responded with one rebound and one block. The John Henson and Mirza Teletovic were next with a pair of {two / two}
Knicks were led by their starting backcourt of Carmelo An-
thony and Carmelo Anthony, but combined for just 13 points -point efforts. Teletovic also registered 13 points, and he added a re-
on 5-of-16 shooting. The Bucks next head to Philadelphia to bound and an assist. Jason Terry supplied eight points, three rebounds
take on the Sixers on Friday night, while the Knicks remain and a pair of steals. The Cavs remain in last place in the Eastern
home to face the Los Angeles Clippers on Wednesday. Conference’s Atlantic Division. They now head home to face the
Toronto Raptors on Saturday night.
(a) Puduppully et al. (2019) (b) Our model
Table 5: Example summaries generated with Puduppully et al. (2019)’s model (left) and our model (right). Names
in bold face are salient entities. Blue numbers are correct relations derived from input data records but are not
observed in reference summary. Orange numbers are incorrect relations. Green numbers are correct relations
mentioned in reference summary.
porating writer information w. As discussed in generated highest quality summaries that scored
Section 4.5, w is supposed to affect both content 20.84 points of BLEU score.
planning and surface realization. Our experimen-
tal result is consistent with the discussion.
Acknowledgments
7 Conclusion
We would like to thank the anonymous reviewers
In this research, we proposed a new data-to-text for their helpful suggestions. This paper is based
model that produces a summary text while track- on results obtained from a project commissioned
ing the salient information that imitates a human- by the New Energy and Industrial Technology De-
writing process. As a result, our model outper- velopment Organization (NEDO), JST PRESTO
formed the existing models in all evaluation mea- (Grant Number JPMJPR1655), and AIST-Tokyo
sures. We also explored the effects of incorpo- Tech Real World Big-Data Computation Open In-
rating writer information to data-to-text models. novation Laboratory (RWBC-OIL).
With writer information, our model successfully
2110References Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK
Li. 2016. Incorporating Copying Mechanism in
Tatsuya Aoki, Akira Miyazawa, Tatsuya Ishigaki, Kei- Sequence-to-Sequence Learning. In Proceedings of
ichi Goshima, Kasumi Aoki, Ichiro Kobayashi, Hi- the 54th Annual Meeting of the Association for Com-
roya Takamura, and Yusuke Miyao. 2018. Gener- putational Linguistics, volume 1, pages 1631–1640.
ating Market Comments Referring to External Re-
sources. In Proceedings of the 11th International
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati,
Conference on Natural Language Generation, pages
Bowen Zhou, and Yoshua Bengio. 2016. Pointing
135–139.
the Unknown Words. In Proceedings of the 54th An-
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- nual Meeting of the Association for Computational
gio. 2015. Neural machine translation by jointly Linguistics, pages 140–149.
learning to align and translate. In Proceedings of the
Third International Conference on Learning Repre- Luong Hoang, Sam Wiseman, and Alexander Rush.
sentations. 2018. Entity Tracking Improves Cloze-style Read-
ing Comprehension. In Proceedings of the 2018
Regina Barzilay and Mirella Lapata. 2005. Collective Conference on Empirical Methods in Natural Lan-
content selection for concept-to-text generation. In guage Processing, pages 1049–1055.
Proceedings of the conference on Human Language
Technology and Empirical Methods in Natural Lan- Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin
guage Processing, pages 331–338. Choi, and Noah A Smith. 2017. Dynamic Entity
Antoine Bosselut, Omer Levy, Ari Holtzman, Corin Representations in Neural Language Models. In
Ennis, Dieter Fox, and Yejin Choi. 2018. Simulating Proceedings of the 2017 Conference on Empirical
Action Dynamics with Neural Process Networks. In Methods in Natural Language Processing, pages
Proceedings of the Sixth International Conference 1830–1839.
on Learning Representations.
Chloé Kiddon, Luke Zettlemoyer, and Yejin Choi.
David L Chen and Raymond J Mooney. 2008. Learn- 2016. Globally coherent text generation with neural
ing to sportscast: a test of grounded language ac- checklist models. In Proceedings of the 2016 Con-
quisition. In Proceedings of the 25th international ference on Empirical Methods in Natural Language
conference on Machine learning, pages 128–135. Processing, pages 329–339.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gul- Sosuke Kobayashi, Ran Tian, Naoaki Okazaki, and
cehre, Dzmitry Bahdanau, Fethi Bougares, Hol- Kentaro Inui. 2016. Dynamic entity representation
ger Schwenk, and Yoshua Bengio. 2014. Learn- with max-pooling improves machine reading. In
ing Phrase Representations using RNN Encoder– Proceedings of the 15th Conference of the North
Decoder for Statistical Machine Translation. In Pro- American Chapter of the Association for Computa-
ceedings of the 2014 Conference on Empirical Meth- tional Linguistics: Human Language Technologies,
ods in Natural Language Processing, pages 1724– pages 850–855.
1734.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, Rémi Lebret, David Grangier, and Michael Auli. 2016.
and Yoshua Bengio. 2014. Empirical evaluation of Neural Text Generation from Structured Data with
gated recurrent neural networks on sequence model- Application to the Biography Domain. In Proceed-
ing. arXiv preprint arXiv:1412.3555. ings of the 2016 Conference on Empirical Methods
in Natural Language Processing, pages 1203–1213.
Elizabeth Clark, Yangfeng Ji, and Noah A Smith. 2018.
Neural Text Generation in Stories Using Entity Rep- Percy Liang, Michael I Jordan, and Dan Klein. 2009.
resentations as Context. In Proceedings of the 16th Learning semantic correspondences with less super-
Conference of the North American Chapter of the vision. In Proceedings of the Joint Conference of
Association for Computational Linguistics: Human the 47th Annual Meeting of the ACL and the 4th In-
Language Technologies, pages 2250–2260. ternational Joint Conference on Natural Language
Processing of the AFNLP, pages 91–99.
Xavier Glorot and Yoshua Bengio. 2010. Understand-
ing the difficulty of training deep feedforward neu-
ral networks. In Proceedings of the thirteenth in- Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang,
ternational conference on artificial intelligence and and Zhifang Sui. 2018. Table-to-text Generation by
statistics, pages 249–256. Structure-aware Seq2seq Learning. In Proceedings
of the Thirty-Second AAAI Conference on Artificial
Alex Graves, Greg Wayne, Malcolm Reynolds, Intelligence.
Tim Harley, Ivo Danihelka, Agnieszka Grabska-
Barwińska, Sergio Gómez Colmenarejo, Edward Thang Luong, Hieu Pham, and Christopher D Man-
Grefenstette, Tiago Ramalho, John Agapiou, et al. ning. 2015. Effective Approaches to Attention-
2016. Hybrid computing using a neural net- based Neural Machine Translation. In Proceedings
work with dynamic external memory. Nature, of the 2015 Conference on Empirical Methods in
538(7626):471. Natural Language Processing, pages 1412–1421.
2111Hongyuan Mei, Mohit Bansal, and Matthew R Walter. Yasufumi Taniguchi, Yukun Feng, Hiroya Takamura,
2016. What to talk about and how? Selective Gener- and Manabu Okumura. 2019. Generating Live
ation using LSTMs with Coarse-to-Fine Alignment. Soccer-Match Commentary from Play Data. In Pro-
In Proceedings of the 15th Conference of the North ceedings of the Thirty-Third AAAI Conference on
American Chapter of the Association for Computa- Artificial Intelligence.
tional Linguistics: Human Language Technologies,
pages 720–730. Zhaopeng Tu, Yang Liu, Zhengdong Lu, Xiaohua Liu,
and Hang Li. 2017. Context gates for neural ma-
Soichiro Murakami, Akihiko Watanabe, Akira chine translation. Transactions of the Association
Miyazawa, Keiichi Goshima, Toshihiko Yanase, Hi- for Computational Linguistics, 5:87–99.
roya Takamura, and Yusuke Miyao. 2017. Learning
to generate market comments from stock prices. Jason Weston, Sumit Chopra, and Antoine Bordes.
In Proceedings of the 55th Annual Meeting of the 2015. Memory Networks. In Proceedings of the
Association for Computational Linguistics, pages Third International Conference on Learning Repre-
1374–1384. sentations.
Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Sam Wiseman, Stuart Shieber, and Alexander Rush.
Matthews, Waleed Ammar, Antonios Anastasopou- 2017. Challenges in Data-to-Document Generation.
los, Miguel Ballesteros, David Chiang, Daniel In Proceedings of the 2017 Conference on Empiri-
Clothiaux, Trevor Cohn, et al. 2017. Dynet: The cal Methods in Natural Language Processing, pages
dynamic neural network toolkit. arXiv preprint 2253–2263.
arXiv:1701.03980.
Zichao Yang, Phil Blunsom, Chris Dyer, and Wang
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Ling. 2017. Reference-Aware Language Models.
Jing Zhu. 2002. BLEU: a method for automatic In Proceedings of the 2017 Conference on Empiri-
evaluation of machine translation. In Proceedings cal Methods in Natural Language Processing, pages
of the 40th annual meeting on association for com- 1850–1859.
putational linguistics, pages 311–318.
Karl Pearson. 1901. On lines and planes of closest fit to A Algorithm
systems of points in space. The London, Edinburgh,
and Dublin Philosophical Magazine and Journal of The generation process of our model is shown
Science, 2(11):559–572. in Algorithm 1. For a concise description, we
omit the condition for each probability notation.
Ratish Puduppully, Li Dong, and Mirella Lapata. 2019.
Data-to-Text Generation with Content Selection and and represent “start of the doc-
Planning. In Proceedings of the Thirty-Third AAAI ument” and “end of the document”, respectively.
Conference on Artificial Intelligence.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar.
2018. On the convergence of adam and beyond. In B Experimental settings
Proceedings of the Sixth International Conference
We set the dimensions of the embeddings to 128,
on Learning Representations.
and those of the hidden state of RNN to 512 and all
Lei Sha, Lili Mou, Tianyu Liu, Pascal Poupart, Sujian of parameters are initialized with the Xavier ini-
Li, Baobao Chang, and Zhifang Sui. 2018. Order-
planning neural text generation from structured data.
tialization (Glorot and Bengio, 2010). We set the
In Proceedings of the Thirty-Second AAAI Confer- maximum number of epochs to 30, and choose the
ence on Artificial Intelligence. model with the highest B LEU score on the devel-
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al.
opment data. The initial learning rate is 2e-3 and
2015. End-to-end memory networks. In Advances AMSGrad is also used for automatically adjusting
in neural information processing systems, pages the learning rate (Reddi et al., 2018). Our imple-
2440–2448. mentation uses DyNet (Neubig et al., 2017).
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.
Sequence to sequence learning with neural net-
works. In Advances in neural information process-
ing systems, pages 3104–3112.
Kumiko Tanaka-Ishii, Kôiti Hasida, and Itsuki Noda.
1998. Reactive content selection in the generation
of real-time soccer commentary. In Proceedings of
the 36th Annual Meeting of the Association for Com-
putational Linguistics and 17th International Con-
ference on Computational Linguistics, pages 1282–
1288.
2112Algorithm 1: Generation process
Input: Data records s,
Annotations Z1:T , E1:T , A1:T , N1:T
LM E NT
1 Initialize {r e,a,v }r∈x , {ē}e∈E , h0 , h0
2 t ← 0
3 et , yt ← N ONE , < S O D >
4 while yt 6=< E O D > do
5 t←t+1
6 if p(Zt = 1) ≥ 0.5 then
/* Select the entity */
7 et ← arg max p(Et = e0t )
8 if et 6∈ Et−1 then
/* If et is a new entity */
E NT0 E E NT
9 ht ← G RU (ēt , ht−1 )
10 Et ← Et−1 ∪ {et }
11 else if et 6= et−1 then
/* If et has been observed before, but is different from
the previous one. */
E NT ’ E S E NT E NT
12 ht ← G RU (W hs , ht−1 ),
13 where s = max{s : s ≤ t − 1, e = es }
14 else
15 htE NT ’ ← ht−1
E NT
/* Select an attribute for the entity, et . */
16 at ← arg max p(At = at ) 0
0
17 htE NT ← G RU A (r et ,at ,x[et ,at ] , htE NT )
18 if at is a number attribute then
19 if p(Nt = 1) ≥ 0.5 then
20 yt ← numeral of x[et , at ]
21 else
22 yt ← x[et , at ]
23 end
24 else
25 yt ← x[et , at ]
0
ht ← tanh W H (ht−1 LM
⊕ htE NT )
26
27 htLM ← L STM(y t ⊕ h0t , ht−1 LM
)
28 else
29 et , at , htE NT ← et−1 , at−1 , ht−1
E NT
0 H
ht ← tanh W (ht−1 ⊕ htE NT )
LM
30
31 yt ← arg max p(Yt )
32 htLM ← L STM(y t ⊕ h0t , ht−1 LM
)
33 end
34 if yt is “.” then
35 htE NT ← G RU A (v R EFRESH , htE NT )
36 end
37 return y1:t−1 ;
2113You can also read
