Improving Zero-Shot Entity Retrieval through Effective Dense Representations
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Improving Zero-Shot Entity Retrieval through Effective Dense
Representations
Eleni Partalidou Despina Christou
Department of Informatics Department of Informatics
Aristotle University of Thessaloniki Aristotle University of Thessaloniki
54124 Thessaloniki – Greece 54124 Thessaloniki – Greece
epartalid@csd.auth.gr christoud@csd.auth.gr
Grigorios Tsoumakas
Department of Informatics
Aristotle University of Thessaloniki
54124 Thessaloniki – Greece
greg@csd.auth.gr
Abstract writing, digital humanities, and biomedical data
arXiv:2103.04156v1 [cs.CL] 6 Mar 2021
Entity Linking (EL) seeks to align entity men- analysis.
tions in text to entries in a knowledge-base While EL systems typically rely on external KBs
and is usually comprised of two phases: candi- and assume entities at inference time known, real-
date generation and candidate ranking. While world applications are usually accompanied by min-
most methods focus on the latter, it is the imal to zero labeled data, highlighting the impor-
candidate generation phase that sets an upper tance of approaches that can generalize to unseen
bound to both time and accuracy performance
entities. Logeswaran et al. (2019) introduced Zero-
of the overall EL system. This work’s contri-
bution is a significant improvement in candi- shot EL, where mentions must be linked to unseen
date generation which thus raises the perfor- entities, without in-domain labeled data, given only
mance threshold for EL, by generating candi- the entities’ text description. By comparing two
dates that include the gold entity in the least texts - a mention in context and a candidate en-
candidate set (top-K). We propose a simple ap- tity description - the task goes closer to reading
proach that efficiently embeds mention-entity comprehension.
pairs in dense space through a BERT-based bi-
Most EL systems consist of two subsystems:
encoder. Specifically, we extend (Wu et al.,
2020) by introducing a new pooling function Candidate Generation (CG), where for each en-
and incorporating entity type side-information. tity mention the system detects entities related to
We achieve a new state-of-the-art 84.28% ac- the mention and document, and Candidate Ranking
curacy on top-50 candidates on the Zeshel (CR) where the system chooses the most proba-
dataset, compared to the previous 82.06% on ble entity link among the found candidates. Most
the top-64 of (Wu et al., 2020). We report the state-of-the-art models (Logeswaran et al., 2019;
results from extensive experimentation using
Li et al., 2020; Yao et al., 2020) rely on traditional
our proposed model on both seen and unseen
entity datasets. Our results suggest that our frequency-based CG and focus on building robust
method could be a useful complement to ex- candidate rankers using cross-encoders to jointly
isting EL approaches. encode mention and entity candidate descriptions.
However, the memory-intensive CR phase depends
1 Introduction on the set of candidates provided by CG. Therefore,
Entity Linking (EL) aims to disambiguate entity the accuracy of CR (and, by extension, the overall
mentions in a document against entries in a knowl- EL system), is capped by that of the CG phase.
edge base (KB) or a dictionary of entities. Ac- The goal of this work is to advance the state of
curately linking these entities plays a key role in the art in the CG phase (Wu et al., 2020), in order
various natural language processing (NLP) tasks, to set a higher performance ceiling for CR and for
including information extraction (Lin et al., 2012; EL overall. We start from (Wu et al., 2020), which
Hasibi et al., 2016), KB population (Dredze et al., encodes mention and entity descriptions in dense
2010), content analysis (Huang et al., 2018) and space through a BERT-based bi-encoder and then
question answering (Li et al., 2020), with poten- retrieves the closest K entities to each mention us-
tial applications in many fields, including technical ing the dot product method. Although this methodincreased accuracy at top-64 retrieved candidates ize to unknown named entities, either via a re-
from 69.13% (Logeswaran et al., 2019) to 82.06%, duced candidate set or through a more robust can-
an approach that allows higher CG accuracy with didate ranker. Our work falls into the first category,
fewer candidates is yet to be developed. namely the candidate generation phase and is re-
We introduce a new state-of-the-art CG model lated to BERT-based sequence modeling, pooling
that achieves 84.28% accuracy on the top-50 candi- techniques, entity type side-information, and can-
dates; namely, we obtain a 2.22 higher unit thresh- didate retrieval methods.
old in CR and EL overall using 21.88% fewer can-
didates, compared to (Wu et al., 2020). Our pro- Candidate Generation (CG) Traditional CG
posed model uses a BERT-based bi-encoder and has been based on string comparison (Phan et al.,
incorporates a special pooling function that effi- 2017) and alias tables. These methods lack rich rep-
ciently encodes mention and entity descriptions in resentation and restrict to a small entity set, respec-
dense space by aggregating information from spe- tively, thus proving inefficient for the challenging
cial tokens. We experimented on various pooling task of entity candidates retrieval, especially in the
function alternatives, retrieval methods, and the ef- zero-shot set, where generalization to new entities
fect of entity type side-information. At the same is on the spot. Over the last years, many works
time, we report our results on both seen and unseen (Sil et al., 2012; Murty et al., 2018; Logeswaran
entity sets to evaluate our model’s efficiency on et al., 2019) focused on frequency-based methods,
both typical and zero-shot sets of EL. Our results with most of them following TF-IDF and BM25 ap-
revealed that concatenating special tokens of the proaches. Gillick et al. (2019) introduces a simple
proposed BERT-based input representation and us- neural bi-encoder and shows that encoding mention
ing dot product to retrieve candidates yield the best and entities in dense space works well. Inspired
outcomes on both test and heldout train seen sets. by this idea, (Wu et al., 2020) proposed the, to
Meanwhile, state-of-the-art results were preserved our knowledge, current state-of-the-art CG model,
without the entity type information, warranting our using a more robust transformer-based bi-encoder
model’s robustness regardless of additional infor- (Humeau et al., 2019). Precisely, they achieve an
mation. 82.06% acc. on top-64 retrieved candidates in the
The contributions of this paper are summarized Zeshel dataset. Their model uses a BERT-based
as follows: bi-encoder to encode mention and entities’ descrip-
tions in dense space, where the top-K entities are
• We introduce a new CG approach, based on
then retrieved based on the two vectors’ maximum
BERT-based bi-encoder and a new pooling
dot product. Our work extends the work of (Wu
function that manages to encode mention and
et al., 2020) by investigating other than the default
entities in the same dense space very effec-
BERT (Devlin et al., 2019) pooling functions, ad-
tively, achieving state-of-the-art results.
ditional entity type side-information, and alternate
• A 84.28% accuracy on top-50 candidates, retrieval methods.
compared to 82.06% at top-64 of (Wu et al.,
2020), measured on the Zeshel test dataset. Entity Type side-information Apart from
We thus increase the CR accuracy by 2.22 model architecture, several methods (Raiman and
units while requiring 21.88% fewer candi- Raiman, 2018; Khalife and Vazirgiannis, 2018)
dates, allowing more accurate and faster infer- show that fine-grained entity type side-information
ence. helps to further disambiguate entities in EL. First
attempts to integrate type information in the task
• State-of-the-art results on both seen and un- consisted of end-to-end systems (Durrett and Klein,
seen entity sets, with and without entity type 2014; Luo et al., 2015; Nguyen et al., 2016) that
information, reveal the robustness of our jointly modeled entity linking and named entity
model in both the typical and the zero-shot recognition to capture the mutual dependency
EL settings. between them, typically using structured CRF and
hand-crafted features. Automatically extracted
2 Related Work
entity features were captured by (Yamada et al.,
Former work has pointed out the importance of 2016; Ganea and Hofmann, 2017) using typical
building entity linking systems that can general- embedding methods proposed by (Mikolovet al., 2013), while (Martins et al., 2019) learned Input Representation The mention context and
entity features using Stack-LSTMs (Dyer et al., the entity description are encoded into the follow-
2015) in multi-task learning. Latest works in EL ing vectors:
(Hou et al., 2020; Chen et al., 2020) suggest to
build entity type embeddings by modeling the ym = red(T1 (tm ))
mentions’ near context and its semantics using
ye = red(T2 (te ))
Word2Vec (Mikolov et al., 2013) and BERT
(Devlin et al., 2019) embeddings, respectively. In where tm is the representation of mention, te is
our work, we choose to model entities as special the representation of entity, T1 and T2 are two trans-
tokens in BERT representation and let the model formers. The function red(.) reduces the sequence
automatically capture relevant information across of vectors into one. By default, the last layer of the
all sequence, following similar BERT-based entity output of the [CLS] token is returned.
representations (Shi and Lin, 2019; Poerner et al., The representation tm is composed of the men-
2020; Christou and Tsoumakas, 2021) applied in tion text surrounded by two special tokens that de-
other NLP applications. note the existence of the mention, the context from
the left and the right side of the mention text plus
Candidate Retrieval methods As mentioned the special tokens that BERT inserts. Specifically,
above, most approaches (Logeswaran et al., 2019; the construction of the mention is:
Sil et al., 2012) retrieve entity candidates using
frequency-based methods, such as BM25 (Robert- [CLS] ctxtl [M s] mention [M e] ctxtr [SEP ]
son et al., 2009). Meanwhile, recent bi-encoder
approaches in EL (Gillick et al., 2019; Wu et al., and with the addition of the entity type it is:
2020) score mention-candidate pairs using dot
[CLS] [ent type] mention[H SEP ] ctxtl...
product during training while retrieving top-K can-
didates using the same vector similarity measure. where [Ms] and [Me] are the special tokens that
In our work, we experiment on alternate candidate tag the mention, ctxtl and ctxtr are the word-pieces
retrieval methods, i.e., euclidean, cosine, and dot tokens of the context before and after the mention
methods, to find out cosine (Manning, 2008) and and [CLS] and [SEP] are BERT’s special tokens.
dot product to outperform the euclidean metric sig- The representation te is composed of the entity
nificantly. title, the special token [ENT] separating the entity’s
title and description plus the special tokens that
3 Methodology BERT inserts.
Our proposed method for candidate generation uses
[CLS] title [EN T ] description [SEP ]
BERT (Devlin et al., 2019) in a bi-encoder set. The
bi-encoder uses two independent BERT transform- and with the addition of the entity type it is:
ers to encode the mention’s context and all entity
descriptions into two dense vectors. Then, the best [CLS] [ent type] title...
candidates are retrieved by scoring the mention-
entity pair vectors with a vector similarity measure. For simplicity in both input representations a
This section presents specific sequence representa- maximum length of the document is retrieved.
tion methods, and other than the typical dot-product The score of the entity candidate ei given a men-
metric used to retrieve the best candidates. tion is computed by the dot-product:
3.1 Bi-encoder s(m, ei ) = ym ·yei
We use a BERT-based bi-encoder, following (Wu The network is trained to maximize the score
et al., 2020) to model the mention-entity pairs, be- of the correct entity with respect to the entities of
cause it allows for fast, real-time inference, as the the same batch. For each training pair (mi , ei ) in a
candidate representations can be cached. Below, batch of B pairs, the loss is computed as:
we present our structured input to the bi-encoder,
our proposed pooling functions for sequence repre- B
sentation and brief notes on entity type incorpora- L(mi , ei ) = −s(mi , ei )+log(
X
exp(s(mi , ej )))
tion in the structured input. j=1Input Encoding The dataset needed to be pro- • Concatenation of special tokens: Concate-
cessed by the models to understand information nates only the special token representations
and to extract representative vectors from it. For from BERT’s last hidden layer.
that reason, a Tokenizer was used to split each doc-
ument into a list of tokens and to give each token conc special = [hL [j1 ], ..., hL [jM ]],
a respective id. The Bert Tokenizer used a spe-
cial technique called Byte Pair Encoding (Sennrich where j refers to the index of special tokens
et al., 2016). BPE makes use of a simple data with M being the total number of special to-
compression technique that iteratively replaces the kens.
most frequent pair of bytes in a sequence with a
We note that hL corresponds to the last hid-
single, unused byte. Instead of merging frequent
den layer of BERT, n = 128, namely the max
pairs of bytes, characters or character sequences are
length, and special tokens declare specific parts
merged. The reason this technique is used by the
of the sequence (i.e. [CLS], [SEP ], [Ms ], [Me ],
algorithm is to compute subwords from unknown
[ent type]) that are not split by the tokenizer.
tokens, so words that the algorithm may known can
be associated with unknown words that tend to be Entity Type Representation In the extent that
similar to them and consequently are given proper entity types put some constraint on the possible
token ids. entity to link, we also experimented on incorporat-
ing the entity type in the model’s structured input.
Pooling Functions The different sequence repre-
Precisely, we incorporate 18 generic entity types,
sentations that were tested fall into the following
captured from recognizing mention’s and entity ti-
categories:
tles’ entity types with the spaCy model1 . SpaCy
• Usage of [CLS]: The default representation of is an open-source software library for advanced
the sequence. natural language processing, written in the pro-
gramming languages Python and Cython (Honni-
CLS = hL [0] bal et al., 2020). We note that entity types, include
the < unk > type for unclassified mentions/entity
• Average of tokens: Averages all token repre- titles. Also, all entity types are encoded as special
sentations from BERT’s last hidden layer. tokens; consequently, sequence representations that
make use of special tokens also consider the entity
Pn
i=0 (hL [i])
type embeddings.
avg =
n
3.2 Retrieval Methods
• Sum of tokens: Sums all token representations In the candidate generation step different methods
from BERT’s last hidden layer. were performed, in order to find a proper candi-
date set. The first method was to initialize a k-
n
sum =
X
(hL [i]) nearest neighbors classifier and fit the model with
i=0 the representations of the entity descriptions from
the bi-encoder and their respective id. After that,
• Average special: Averages only the special to- for each context mention the k closest neighbors
ken representations from BERT’s last hidden were returned, where k the number of candidates
layer. the candidate set should have.
Pn Another approach was to compute the similarity
i=0 (hL [i]) between each context mention with each entity and
avg special =
n construct the candidate set with the h more com-
mon ones. To find the similarity between two vec-
• Sum of special tokens: Sums only the spe- tors A = [a1 , a2 , ..., an ] and B = [b1 , b2 , ..., bn ] ,
cial token representations from BERT’s last some main similarity measures were used to choose
hidden layer. from.
1
n
X https://github.com/explosion/
sum special = (hL [i]) spacy-models/releases/tag/en_core_web_
i=0
lg-2.3.0The cosine similarity method measures the co- Set World Entities
sine of angle θ between two vectors. The cosine Train American Football 31929
increases as the two vectors tend to look alike. Doctor Who 40281
Fallout 16992
AT B Final Fantasy 14044
cos(A, B) = Military 104520
|A|·|B|
Pro Wrestling 10133
The dot product method measures the cosine StarWars 87056
multiplied by lengths of both vectors. For very World of Warcraft 27677
similar vectors the value is increased and also for Val Coronation Street 17809
vectors with big length. Muppets 21344
Ice Hockey 28684
Elder Scrolls 21712
A · B = a1 b1 + a2 b2 + ... + an bn = |A||B|cos(θ) Test Forgotten Realms 15603
Lego 10076
Star Trek 34430
The euclidean distance method measures the dis-
YuGiOh 10031
tance between ends of vectors. For vectors that are
similar and consequently very close to the vector Table 1: Number of Entities per world and train, val,
space the value is small. test sets in the Zeshel dataset.
q
d(A, B) = (a1 − b1 )2 + ... + (aN − bN )2 4.2 Hyper-parameter Settings
4 Experimental Setup In our experiments we utilize bert-base-uncased
model with hidden layer dimension Dh = 768,
4.1 Dataset while we fine-tune the model with max seq length
For our experiments, we use the Zeshel2 dataset, Dt = 128. Regarding model’s hyper-parameters,
which to our knowledge, is the prevailing bench- we manually tune them on the training set. For fine-
mark for zero-shot EL. The dataset was constructed tuning, we assigned batch size = 8, epochs =
using documents from Wikias, namely community- 5, all the BERT’s layers are updated during
written encyclopedias, each specialized in a par- back-propagation, learning rate lr = 3e−5 and
ticular subject. Documents in Wikias exhibit rich weight decay = 0.01. Moreover, we fine-tune
document context, thus are an excellent source for our model using the Adam optimization scheme
natural language comprehension models. Mean- (Loshchilov and Hutter, 2017) with β1 = 0.9, β2 =
while, each encyclopedia contains domain-specific 0.999 and a linear learning rate decay schedule. We
entity dictionaries, making it an ideal dataset for minimize loss using cross entropy criterion.
evaluating the EL system’s generalization on new Experiments were conducted on a PC with 15
domains. GB RAM, an AMD FX-8350 Eight-Core Processor
Table 1 presents the number of unique entities with a base frequency of 4.00 GHz and a NVIDIA
per domain (world) and train/val/test sets. Regard- GeForce GTX TITAN X graphics card with 12 GB
ing mentions, the training set includes 49,275 la- memory.
beled mentions, while the validation and test sets
consist of 10,000 and 10,000 unseen mentions, re- Training time took about 150min for the whole
spectively. Moreover, an extra 5,000 seen and process. For the CG task and to process all test set
5,000 unseen mentions from the training set are pro- (10K mentions, 70.140 entities), the model takes
vided. The goal of this discrimination is to explore 90min using cosine method, 60min with the eu-
the models’ performance to known domains. We clidean distance method and 50min with the dot
present our model’s performance on both known product method. For the heldout train seen set, the
and unknown worlds in section 5.3. model takes 30min with the cosine method, 25min
with the euclidean distance method and 20min with
2
https://github.com/lajanugen/zeshel the dot product method.4.3 Evaluation Method Test Top-K
To evaluate the performance of our model against (Logeswaran et al., 2019) 69.13 64
the rest state-of-the-art models, we report the ac- (Wu et al., 2020) 82.06 64
curacy at top-k. Namely, we assess the perfor- Ours (sum)† 80.27 50
mance on the subset of test instances for which the Ours (sum special)† 83.23 50
gold entity is among the top-k retrieved candidates. Ours (CLS)† 83.83 50
We highlight that since EL systems’ overall per- Ours (avg special)† 83.88 50
formance is upper-bounded by the CG accuracy Ours (avg)† 84.19 50
and their speed is associated with the number of Ours (conc special) 84.28 50
retrieved candidates, a model that will increase CG Table 2: Normalized accuracy on the test set of Zero-
performance in fewer candidate entities is crucial. shot EL dataset. Baseline models are compared against
our model’s various sequence representation alterna-
4.4 State-of-the-art CG Models tives, with selected pooling function noted in paren-
For evaluating our method, we compare against the theses. † indicates methods including entity type side-
following state-of-the-art models in Zero-Shot EL: information.
BM25 (Logeswaran et al., 2019): Use a traditional,
frequency-based method (BM25) for the candidate accuracy than (Wu et al., 2020) and (Logeswaran
generation. To re-rank the top K candidates pro- et al., 2019) respectively, using 21.88% fewer can-
pose the use of a cross-encoder to jointly encode didates and without incorporating any additional
mention context and entity description. side-information.
Bi-encoder (Wu et al., 2020): First to perform the
candidate retrieval on a dense space. Mention con-
text and candidate entities are encoded into vectors
using a bi-encoder and the retrieval phase is then
reduced to finding the maximum dot product be-
tween mention and entity candidate representations.
Final entity selection is performed using the top K
candidates from the retrieval phase and re-ranked
through a cross-encoder, similar to (Logeswaran
et al., 2019). As an encoder BERT model is se- Figure 1: Top-K entity retrieval accuracy of Ours (conc
lected. special) model.
While state-of-the-art models in Zero-shot EL (Lo-
geswaran et al., 2019; Yao et al., 2020) focus on the Meanwhile, in Figure 1, which shows our best
CR phase, Wu et al. (2020) are the only to propose model’s accuracy across various top-K retrieved
a different to traditional IR approach for CG. Our candidates, we can see that choosing more can-
focus is to further push the boundaries of the CG didates could further increase accuracy. Still, we
phase and set a higher performance threshold to decided on fifty (50) candidates due to its small
CR and EL overall. enough size and performance improvement over
the rest baselines.
5 Results
5.2 Ablation Studies
5.1 Comparison with Baselines
In this section, we review the effect of pooling
Here, we compare our CG model against the pre- functions, entity type side-information and retrieval
vious state-of-the-art works and our model’s var- methods.
ious alternatives that use different pooling func-
tions for sequence representation and incorporate 5.2.1 Effect of Entity Type across all Pooling
or not entity type side-information. Table 2 shows Functions
our model to significantly outperform the previ- We experimented with various pooling functions
ous state-of-the-art works with fewer candidates re- to find the most efficient sequence representation,
gardless of chosen pooling function and additional whereas for each representation, we explore the ad-
side-information. Precisely, the best version of our dition of the mention’s entity type at the sequence
model achieves 2.22 and 15.15 higher normalized encoding (see section 3.1).Test Euclidean distance presents the worst performance
Pooling Functions w/o Ent.Type Ent.Type for all methods, with dot product consistently
Sum tokens 79.64% 80.27% slightly outperform the cosine. Figure 2, which
CLS token 83.52% 83.83% shows our best model’s performance for all three
Avg special tokens 83.83% 83.88% distance metrics across top-{1, 10, 25, 50} re-
Avg tokens 84.06% 84.19% trieved candidates, exhibits similar observations.
Conc special tokens 84.28% 83.30% Namely, the dot product achieves higher accuracy
across all top candidates, closely followed by co-
Table 3: Comparison of results with and without entity
sine and then by the euclidean distance. Conse-
type side-information for each pooling function on the
Test set of Zero-shot EL dataset. We report results on
quently, our findings are in sync with (Manning,
the top-50 candidates, using the dot product as retrieval 2008). Comparing the three distance metrics, co-
method. sine and dot product metrics are advantageous to
euclidean as they measure the angle between docu-
Table 3 presents the normalized accuracy on top- ments; namely, they measure documents’ similarity
50 candidates for all pooling functions presented regardless of their size. Moreover, the dot product
in section 3.1 incorporating or not the entity type considers both the angle and the magnitude of two
side-information. As can be seen, state-of-the-art vectors, which makes it competitive to the cosine
accuracy is achieved by representing the sequence similarity metric.
as the concatenation of its special tokens without
incorporating entity-type information. Meanwhile,
the addition of entity type in the rest representation
techniques seems to improve performance, but only
slightly (up to 0.63%). We suspect that the reason
that entity type does not add significant value is
that about 60% of mentions are detected with UN-
KNOWN entity type; thus, the model was not able
to capture robust patterns. From manual inspection,
we notice that most UNKNOWN entity types are
associated with pronouns or co-references. Figure 2: Performance of (conc special) model for all
Euclidean, Cosine, and Dot product distance metrics
5.2.2 Effect of the Retrieval Methods across the top retrieved candidates.
Regarding the retrieval phase, we wanted to explore
the effect of other common distance metrics, apart
from the typical one used, i.e., the dot product. 5.3 Case Study: Performance on the typical
EL set
Method/ Test So far, we have tested our methods’ efficiency in
Distance Metrics Euclid. Cosine Dot the most challenging set of EL, the zero-shot set,
Ours (CLS) 79.51 82.74 83.52 where we must prove the model’s generalization in
Ours (avg) 81.27 83.73 84.06 out-of-domain, unseen entities. In this section, our
Ours (sum) 76.05 78.02 79.64 scope is to examine the in-domain generalization
Ours (avg special) 82.08 83.55 83.83 performance and prove our method as a valuable
Ours (sum special) 81.56 82.36 82.90 complement to all existing EL approaches.
Ours (conc special) 79.96 83.63 84.28 To test the performance on in-domain, seen enti-
ties, we exploit the “Heldout train seen” set, pro-
Table 4: Comparison of all methods using Euclidean,
vided by (Logeswaran et al., 2019), that includes
Cosine and Dot product distance metrics on the top-
50 retrieved candidates on the test set of the Zero-shot 5,000 mentions linked to seen entities during train-
EL dataset. Reported results refer to Normalized Acc., ing. In Table 5, we present the normalized accuracy
while all methods do not incorporate entity type. on top-50 candidates for all pooling functions in-
corporating or not the entity type side-information,
Table 4 compares all methods’ performance us- whereas in Table 6 we show the performance of our
ing the Euclidean, Cosine, and Dot product dis- best model using different distance metrics in the
tance metrics to retrieve the best candidates. The retrieval phase.Heldout Train Seen the one hand, that proves our model’s robustness
Pooling functions w/o Ent.Type Ent.Type regardless of side-information, but from the other
[CLS] 98.42% 98.56% suggest specific enhancements for the future. Fu-
Avg tokens 98.56% 98.05% ture enhancements could incorporate entity graph
Sum tokens 97.60% 97.62% information instead of entity types to capture co-
Avg special tokens 98.76% 98.54% references better and integrate a generic document
Sum special tokens 98.26% 98.30% entity type to locate the main concept of the doc-
Conc special tokens 98.86% 98.36% ument. At last, we plan to extend this work with
respective experimentation in the candidate ranking
Table 5: Comparison of results with and without entity phase.
type for each pooling function on the Heldout Train
Seen set. We report results on the top-50 candidates,
using the dot product as retrieval method.
References
Shuang Chen, Jinpeng Wang, Feng Jiang, and Chin-
Both tables reveal same observations to our Yew Lin. 2020. Improving entity linking by model-
methods’ performance in the Zero-shot set. ing latent entity type information. In Proceedings of
Namely, using the concatenation of special tokens the AAAI Conference on Artificial Intelligence, vol-
ume 34, pages 7529–7537.
to represent sequences and the dot product to re-
trieve the best candidates leads to greater accuracy. Despina Christou and Grigorios Tsoumakas. 2021.
Moreover, entity type only slightly boosts few se- Improving distantly-supervised relation extraction
quence representation methods, and euclidean dis- through bert-based label & instance embeddings.
arXiv preprint arXiv:2102.01156.
tance performs the poorest. At last, we provide a
great upper threshold (98.86%) for EL under its Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
typical, supervised set. Kristina Toutanova. 2019. Bert: Pre-training of
deep bidirectional transformers for language under-
Distance metrics Results standing. In Proceedings of the 2019 Conference of
the North American Chapter of the Association for
Euclidean 97.3% Computational Linguistics: Human Language Tech-
Cosine 98.64% nologies, Volume 1 (Long and Short Papers), pages
Dot product 98.86% 4171–4186.
Table 6: Comparison of the similarity methods for con- Mark Dredze, Paul McNamee, Delip Rao, Adam Ger-
catenation of special tokens and 50 candidates. Results ber, and Tim Finin. 2010. Entity disambiguation
for the heldout train seen set. for knowledge base population. In Proceedings
of the 23rd International Conference on Computa-
tional Linguistics (Coling 2010), pages 277–285.
6 Conclusion
Greg Durrett and Dan Klein. 2014. A joint model
We proposed a simple yet very effective candidate for entity analysis: Coreference, typing, and link-
generation model that sets a higher performance ing. Transactions of the association for computa-
tional linguistics, 2:477–490.
bar for candidate ranking (CR) and EL overall.
Precisely, our model achieves at least 2.22 higher Chris Dyer, Miguel Ballesteros, Wang Ling, Austin
threshold for CR and EL using 22% fewer candi- Matthews, and Noah A Smith. 2015. Transition-
dates, compared to previous state-of-the-art works based dependency parsing with stack long short-
term memory. In Proceedings of the 53rd Annual
in the zero-shot set. Meanwhile, we tested our Meeting of the Association for Computational Lin-
model’s efficiency with in-domain corpus, seen guistics and the 7th International Joint Conference
entities to also accomplish great results with the on Natural Language Processing (Volume 1: Long
same parameters as in zero-shot EL, proving that Papers), pages 334–343.
the suggested method can be a valuable comple- Octavian-Eugen Ganea and Thomas Hofmann. 2017.
ment to all existing EL approaches. We also exper- Deep joint entity disambiguation with local neural
imented with adding entity types to boost perfor- attention. In Proceedings of the 2017 Conference on
mance, without success, though. After inspecting Empirical Methods in Natural Language Processing,
pages 2619–2629.
the data, we found out that 60% of the mentions
were not classified with an entity type, with most of Dan Gillick, Sayali Kulkarni, Larry Lansing, Alessan-
them constituting pronouns, not named entities. On dro Presta, Jason Baldridge, Eugene Ie, and DiegoGarcia-Olano. 2019. Learning dense representa- Christopher D Manning. 2008. Prabhakar raghavan,
tions for entity retrieval. In Proceedings of the and hinrich schutze. Introduction to information re-
23rd Conference on Computational Natural Lan- trieval.
guage Learning (CoNLL), pages 528–537.
Pedro Henrique Martins, Zita Marinho, and André FT
Faegheh Hasibi, Krisztian Balog, and Svein Erik Brats- Martins. 2019. Joint learning of named entity recog-
berg. 2016. Exploiting entity linking in queries for nition and entity linking. In Proceedings of the
entity retrieval. In Proceedings of the 2016 ACM In- 57th Annual Meeting of the Association for Com-
ternational Conference on the Theory of Information putational Linguistics: Student Research Workshop,
Retrieval, pages 209–218. pages 190–196.
Matthew Honnibal, Ines Montani, Sofie Van Lan- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Cor-
deghem, and Adriane Boyd. 2020. spaCy: rado, and Jeffrey Dean. 2013. Distributed represen-
Industrial-strength Natural Language Processing in tations of words and phrases and their composition-
Python. ality. In Proceedings of the 26th International Con-
Feng Hou, Ruili Wang, Jun He, and Yi Zhou. 2020. Im- ference on Neural Information Processing Systems-
proving entity linking through semantic reinforced Volume 2, pages 3111–3119.
entity embeddings. In Proceedings of the 58th An-
Shikhar Murty, Patrick Verga, Luke Vilnis, Irena
nual Meeting of the Association for Computational
Radovanovic, and Andrew McCallum. 2018. Hier-
Linguistics, pages 6843–6848.
archical losses and new resources for fine-grained
Zhipeng Huang, Bogdan Cautis, Reynold Cheng, Yu- entity typing and linking. In Proceedings of the
dian Zheng, Nikos Mamoulis, and Jing Yan. 2018. 56th Annual Meeting of the Association for Compu-
Entity-based query recommendation for long-tail tational Linguistics (Volume 1: Long Papers), pages
queries. ACM Transactions on Knowledge Discov- 97–109.
ery from Data (TKDD), 12(6):1–24.
Dat Ba Nguyen, Martin Theobald, and Gerhard
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, Weikum. 2016. J-nerd: joint named entity recogni-
and Jason Weston. 2019. Poly-encoders: Architec- tion and disambiguation with rich linguistic features.
tures and pre-training strategies for fast and accurate Transactions of the Association for Computational
multi-sentence scoring. In International Conference Linguistics, 4:215–229.
on Learning Representations.
Minh C Phan, Aixin Sun, Yi Tay, Jialong Han, and
Sammy Khalife and Michalis Vazirgiannis. 2018. Scal- Chenliang Li. 2017. Neupl: Attention-based seman-
able graph-based individual named entity identifica- tic matching and pair-linking for entity disambigua-
tion. arXiv preprint arXiv:1811.10547. tion. In Proceedings of the 2017 ACM on Confer-
ence on Information and Knowledge Management,
Belinda Z Li, Sewon Min, Srinivasan Iyer, Yashar pages 1667–1676.
Mehdad, and Wen-tau Yih. 2020. Efficient one-pass
end-to-end entity linking for questions. In Proceed- Nina Poerner, Ulli Waltinger, and Hinrich Schütze.
ings of the 2020 Conference on Empirical Methods 2020. E-bert: Efficient-yet-effective entity embed-
in Natural Language Processing (EMNLP), pages dings for bert. In Proceedings of the 2020 Con-
6433–6441. ference on Empirical Methods in Natural Language
Thomas Lin, Oren Etzioni, et al. 2012. Entity linking Processing: Findings, pages 803–818.
at web scale. In Proceedings of the Joint Workshop
Jonathan Raiman and Olivier Raiman. 2018. Deep-
on Automatic Knowledge Base Construction and
type: multilingual entity linking by neural type sys-
Web-scale Knowledge Extraction (AKBC-WEKEX),
tem evolution. In Proceedings of the AAAI Confer-
pages 84–88.
ence on Artificial Intelligence, volume 32.
Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee,
Kristina Toutanova, Jacob Devlin, and Honglak Lee. Stephen Robertson, Hugo Zaragoza, et al. 2009. The
2019. Zero-shot entity linking by reading entity de- probabilistic relevance framework: Bm25 and be-
scriptions. In Proceedings of the 57th Annual Meet- yond. Foundations and Trends® in Information Re-
ing of the Association for Computational Linguistics, trieval, 3(4):333–389.
pages 3449–3460.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
Ilya Loshchilov and Frank Hutter. 2017. Decou- 2016. Neural machine translation of rare words
pled weight decay regularization. arXiv preprint with subword units. In Proceedings of the 54th An-
arXiv:1711.05101. nual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1715–
Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Za- 1725.
iqing Nie. 2015. Joint entity recognition and disam-
biguation. In Proceedings of the 2015 Conference Peng Shi and Jimmy Lin. 2019. Simple bert models for
on Empirical Methods in Natural Language Process- relation extraction and semantic role labeling. arXiv
ing, pages 879–888. preprint arXiv:1904.05255.Avirup Sil, Ernest Cronin, Penghai Nie, Yinfei Yang,
Ana-Maria Popescu, and Alexander Yates. 2012.
Linking named entities to any database. In Pro-
ceedings of the 2012 Joint Conference on Empirical
Methods in Natural Language Processing and Com-
putational Natural Language Learning, pages 116–
127.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian
Riedel, and Luke Zettlemoyer. 2020. Scalable zero-
shot entity linking with dense entity retrieval. In
Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing (EMNLP),
pages 6397–6407.
Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and
Yoshiyasu Takefuji. 2016. Joint learning of the em-
bedding of words and entities for named entity dis-
ambiguation. In Proceedings of The 20th SIGNLL
Conference on Computational Natural Language
Learning, pages 250–259.
Zonghai Yao, Liangliang Cao, and Huapu Pan. 2020.
Zero-shot entity linking with efficient long range se-
quence modeling. In Proceedings of the 2020 Con-
ference on Empirical Methods in Natural Language
Processing: Findings, pages 2517–2522.
A Examining model errors and
predictions
In tables 7,8, 9 we show some example mentions
and model predictions.... When he took it off , a boy working for Insector Haga swiped it and inserted the card
Mention P̈arasite Paracide ı̈nto his Deck as he ran away with it . Jonouchi managed to catch the
boy and met up with Anzu , Bakura and Sugoroku as he did so . ....
Haga ’ s helper ( manga ) Haga ’ s helper was a boy
employed by Insector Haga to help him cheat in a Duel
against Katsuya Jonouchi . Biography . Haga promised
Haga ’ s helper ( manga )
the boy a rare card if he could sneak the card ” Parasite
Paracide ” into Jonouchi ’ s Deck . ...
Queue cutter The ” queue cutter ” is an elementary
or junior high school kid , who appeared once in the
manga . Biography . The boy cut past Yugi in a queue
Queue cutter
to a Capsule Monster Chess coin machine , outside Old
Man Dentures store . ...
Daichi Daichi is one of the children in Crow ’ s care in ”
Daichi Yu - Gi - Oh ! 5D ’ s ” . He has black hair in a bowl cut
and dark blue eyes . ...
Table 7: First example with the mention and the entity candidates. The correct entity candidate is the first from
the candidate set.
... He stole the Type 7 shuttlecraft ” ” , intending to join a freighter on Beltane IX and
Mention
asked Captain Picard to tell his father he ’ s sorry but had to do this . ...
Jack Crusher Lieutenant Commander Jack R . Crusher
was a Starfleet officer . Considered by Jean - Luc Picard
to have been his best friend , he served under Picard ’ s
Jack Crusher
command on the . He was husband to Beverly Crusher
and father to Wesley Crusher . ...
Kurland Kurland was a Human male aboard the in 2364
. He had one son , Jake Kurland . When Jake tried to
leave the ” Enterprise ” - D with a shuttlecraft that year ,
he asked Captain Picard to tell his father he was sorry
Kurland
but he could not stay aboard the ship . Picard told him
he should bring back the shuttle and tell this his father
himself . ( ) ...
Gabriel Lorca Captain Gabriel Lorca was a male Human
Starfleet officer who lived during the mid - 23rd century
Gabriel Lorca
. He served as the commanding officer on board at least
one Federation starship , ...
Table 8: Second example with the mention and the entity candidates. The correct entity candidate is the second
from the candidate set.... Riker argues with him and is generally uncooperative . Remmick asks La Forge in
Mention engineering about the incident with Kosinski and the Traveler , and La Forge is forced
to acknowledge that the captain lost control of the ship . ...
USS Enterprise bridge holoprogram The USS ” Enter-
prise ” bridge holoprogram was a holodeck recreation
of the bridge of the original . The program was accessed
USS Enterprise bridge
by Captain Montgomery Scott when he was on board
holoprogram
the in 2369 after having been rescued from transporter
stasis on the . ...
Captain ’ s log , USS Enterprise ( NCC - 1701 ) The
captain ’ s log on the was the method used by the
Captain ’ s log , USS En- commanding officer to record the ship ’ s events . These
terprise ( NCC - 1701 ) logs included Captain James T . Kirk ’ s famous five -
year mission . ( ) ...
Traffic accident A traffic accident was an incident involv-
ing vehicle s and their occupants in which the vehicle
in question malfunctioned or crashed . Some accidents
Traffic accident
resulted in death . In 1930 , Edith Keeler died in a traf-
fic accident after she crossed the street to find out why
James T . Kirk had left her abruptly . ...
Table 9: Third example with the mention and the entity candidates. The correct entity candidate is not part of the
candidate set.You can also read
