TAC KBP 2015 Slot Descriptions - Text Analysis Conference
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
TAC KBP 2015 Slot Descriptions
Version 1.0 – May 15, 2015
Linguistic Data Consortium
Created by: Joe Ellis (joellis@ldc.upenn.edu)
With contributions from: Jeremy Getman, Heather Simpson, Kira Griffitt, Hoa Trang Dang,
Ralph Grishman, Heng Ji, Catherine DePrince, Thomas Riese, & Neil Kuster
http://projects.ldc.upenn.edu/kbp/
© 2015 Trustees of the University of Pennsylvania
***********This document is unpublished and intended solely for the use of the individual or
entity to whom it was delivered. Redistribution is strictly prohibited without the express
authorization of the Linguistic Data Consortium.**********
1Changes from 2014:
1. Added note to definition of per:employee_or_member_of regarding media publications
2Table of Contents
1 Introduction ......................................................................................................................................................4
2 Slot Categorizations .........................................................................................................................................4
2.1 Slot Content.............................................................................................................................................4
2.1.1 Name slots ..........................................................................................................................................4
2.1.2 Value Slots ..........................................................................................................................................5
2.1.3 String Slots ..........................................................................................................................................5
2.2 Slot Quantity ............................................................................................................................................5
2.2.1 Single-value Slots ...............................................................................................................................5
2.2.2 List-value Slots ....................................................................................................................................5
2.3 Slots Table ..............................................................................................................................................5
2.4 Inverse Slots ............................................................................................................................................6
2.5 Inverse Slots Table..................................................................................................................................7
3 Person Slots .....................................................................................................................................................7
3.1 per:alternate_names ...............................................................................................................................7
3.2 per:date_of_birth .....................................................................................................................................8
3.3 per:age ....................................................................................................................................................9
3.4 per:country_of_birth ................................................................................................................................9
3.5 per:stateorprovince_of_birth .................................................................................................................10
3.6 per:city_of_birth .....................................................................................................................................11
3.7 per:origin ...............................................................................................................................................11
3.8 per:date_of_death .................................................................................................................................12
3.9 per:country_of_death ............................................................................................................................12
3.10 per:stateorprovince_of_death ...............................................................................................................13
3.11 per:city_of_death ...................................................................................................................................13
3.12 per:cause_of_death...............................................................................................................................14
3.13 per:countries_of_residence ...................................................................................................................14
3.14 per:statesorprovinces_of_residence .....................................................................................................15
3.15 per:cities_of_residence .........................................................................................................................16
3.16 per:schools_attended ............................................................................................................................17
3.17 per:title ...................................................................................................................................................17
3.18 per:employee_or_member_of ...............................................................................................................19
3.19 per:religion .............................................................................................................................................20
3.20 per:spouse .............................................................................................................................................21
3.21 per:children ............................................................................................................................................21
3.22 per:parents ............................................................................................................................................21
3.23 per:siblings ............................................................................................................................................22
3.24 per:other_family.....................................................................................................................................22
3.25 per:charges ...........................................................................................................................................23
4 Organization Slots ..........................................................................................................................................23
4.1 org:alternate_names .............................................................................................................................23
4.2 org:political_religious_affiliation ............................................................................................................24
4.3 org:top_members_employees ...............................................................................................................25
4.4 org:number_of_employees_members ..................................................................................................26
4.5 org:members .........................................................................................................................................27
4.6 org:member_of ......................................................................................................................................27
4.7 org:subsidiaries .....................................................................................................................................28
4.8 org:parents ............................................................................................................................................29
4.9 org:founded_by .....................................................................................................................................30
4.10 org:date_founded ..................................................................................................................................31
4.11 org:date_dissolved ................................................................................................................................31
4.12 org:country_of_headquarters ................................................................................................................31
4.13 org:stateorprovince_of_headquarters ...................................................................................................32
4.14 org:city_of_headquarters.......................................................................................................................33
4.15 org:shareholders ...................................................................................................................................34
4.16 org:website ............................................................................................................................................34
31 Introduction
Text Analysis Conference (TAC) is a series of workshops organized by the National Institute
of Standards and Technology (NIST). TAC was developed to encourage research in natural
language processing (NLP) and related applications by providing a large test collection,
common evaluation procedures, and a forum for researchers to share their results. The
Knowledge Base Population (KBP) track of TAC aims to develop systems that can determine
whether or not specific entities already appear in existing knowledge bases, extract
information about those entities from web and newswire texts, and use the extracted
information to populate a new or existing knowledge base.
KBP uses a snapshot of BaseKB as the existing knowledge base (KB) that systems and
annotators scan for existing information and update with new information they obtain from
source corpora.
This document contains descriptions of the slots and fillers that apply to the Cold Start
annotation task for TAC KBP. For more information on task-based specifics, see the separate
guidelines for each task.
2 Slot Categorizations
The TAC KBP slots are primarily categorized as being either Person (PER) or Organization
(ORG) slots based on the type of entities about which they seek to extract information.
However, slots are also categorized by the content and quantity of their fillers.
2.1 Slot Content
Slots are labeled as Name, Value, or String based on the content of their fillers.
2.1.1 Name slots
Name slots are required to be filled by the name of a person, organization, or geo-
political entity (GPE). Be sure to review the individual slot descriptions in sections 3
and 4 for slot-specific considerations for person, organization, or GPE fillers. However,
general considerations are listed below
Person Entities (PER) – PERs are limited to individual humans. Groups of
people (including families) are not valid person entities.
Organization Entities (ORG) – ORGs are corporations, agencies, and other
groups of people defined by an established organizational structure. Note that
musical groups are considered to be organizations but individual artists (e.g.
Brittany Spears) are considered persons. Programs or projects should not be
considered organizations and different iterations of the same organization (e.g.,
the 111th U.S. Congress and the 112th U.S. Congress) should not be considered
as distinct entities. Media publications and productions (newspapers, magazines,
TV shows, films, etc.) are not themselves considered organizations, though the
entities that produce such works are often organizations.
4 Geo-political Entities (GPE) – Generally speaking, GPEs are composite entities
comprised of a government, a physical location, and a population, with common
types including countries, states, provinces, counties, cities, and towns. Note,
however, that for the purposes of TAC KBP, all top-level governments of GPEs
should also be categorized as GPEs, not as ORGs.
Regions like “the southeast US” are not GPEs because, while they have the
physical location and population qualities, they do not have their own government.
Given the text “southeast Texas”, only “Texas” could be annotated as GPE, as
southeast Texas has neither its own government nor a defined location.
2.1.2 Value Slots
Value slots are required to be filled by either a numerical value or a date. The numbers
and dates in these fillers can be spelled out (forty-two; December 7, 1941) or written as
numbers (42; 12/7/1941).
2.1.3 String Slots
String slots are basically a “catch all”, meaning that their fillers cannot be neatly
classified as names or values. The text excerpts (or “strings”) that make up these fillers
can sometimes be just a name, but are often expected to be more than a name.
2.2 Slot Quantity
Slots are labeled as Single-value or List-value based on the number of fillers they can take.
2.2.1 Single-value Slots
Single-value slots can have only a single filler. While most single-value slots are
obvious (e.g., a person can only have one date of birth), some may be less apparent
(see per:age).
2.2.2 List-value Slots
List-value slots can take multiple fillers as they are likely to have more than one correct
answer in the source data. For example, people may have multiple children,
employers, or alternate names.
2.3 Slots Table
The following table of all 41 slots and their categorizations is provided for reference:
Type Slot Name Content Quantity
PER per:alternate_names Name List
PER per:children Name List
PER per:cities_of_residence Name List
PER per:city_of_birth Name Single
PER per:city_of_death Name Single
PER per:countries_of_residence Name List
PER per:country_of_birth Name Single
5PER per:country_of_death Name Single
PER per:employee_or_member_of Name List
PER per:origin Name List
PER per:other_family Name List
PER per:parents Name List
PER per:schools_attended Name List
PER per:siblings Name List
PER per:spouse Name List
PER per:stateorprovince_of_birth Name Single
PER per:stateorprovince_of_death Name Single
PER per:statesorprovinces_of_residence Name List
PER per:age Value Single
PER per:date_of_birth Value Single
PER per:date_of_death Value Single
PER per:cause_of_death String Single
PER per:charges String List
PER per:religion String Single
PER per:title String List
ORG org:alternate_names Name List
ORG org:city_of_headquarters Name Single
ORG org:country_of_headquarters Name Single
ORG org:founded_by Name List
ORG org:member_of Name List
ORG org:members Name List
ORG org:parents Name List
ORG org:political_religious_affiliation Name List
ORG org:shareholders Name List
ORG org:stateorprovince_of_headquarters Name Single
ORG org:subsidiaries Name List
ORG org:top_members_employees Name List
ORG org:date_dissolved Value Single
ORG org:date_founded Value Single
ORG org:number_of_employees_members Value Single
ORG org:website String Single
2.4 Inverse Slots
In the Cold Start task, there are 41 regular slots. Of these, 26 are entity-value slots: slots that
require fillers that are themselves entities (ORG, PER, or GPE). Each entity-value slot has an
inverse. Some slots, such as per:siblings, are symmetric. Others, such as per:parents, have
inverses that are already slots (in this case, per:children). The remaining slots (e.g.
org:founded_by) have no pre-existing corresponding inverse slot. Cold Start specifies new
slot names for these inverses. All such newly-introduced slots are list-value slots.
62.5 Inverse Slots Table
The following table lists all entity-value slots and their corresponding inverse slots, for use in
Cold Start. Slots with asterisks represent relations that are defined specifically for Cold Start.
Slot Inverse(s)
per:children per:parents
per:other_family per:other_family
per:parents per:children
per:siblings per:siblings
per:spouse per:spouse
per:employee_or_member_of {org,gpe}:employees_or_members*
per:schools_attended org:students*
per:city_of_birth gpe:births_in_city*
per:stateorprovince_of_birth gpe:births_in_stateorprovince*
per:country_of_birth gpe:births_in_country*
per:cities_of_residence gpe:residents_of_city*
per:statesorprovinces_of_residence gpe:residents_of_stateorprovince*
per:countries_of_residence gpe:residents_of_country*
per:city_of_death gpe:deaths_in_city*
per:stateorprovince_of_death gpe:deaths_in_stateorprovince*
per:country_of_death gpe:deaths_in_country*
org:shareholders {per,org,gpe}:holds_shares_in*
org:founded_by {per,org,gpe}:organizations_founded*
org:top_members_employees per:top_member_employee_of*
org:member_of org:members
org:members {org,gpe}:member_of
org:parents {org,gpe}:subsidiaries
org:subsidiaries org:parents
org:city_of_headquarters gpe:headquarters_in_city*
org:stateorprovince_of_headquarters gpe:headquarters_in_stateorprovince*
org:country_of_headquarters gpe:headquarters_in_country*
3 Person Slots
Note that future, hypothetical, fictional, and metaphorical fillers are incorrect for ALL of the
following person slots. Former fillers are valid responses for most slots but you should check
the description of each to be sure.
3.1 per:alternate_names
Content: Name
Quantity: List
Description: Names used to refer to the assigned person that are distinct from the "official"
name (the namestring for your entity appearing at the top of the annotation or assessment
7tool). Alternate names may include aliases, stage names, alternate transliterations,
abbreviations, alternate spellings, nicknames, or birth names.
Inverse Slot: n/a
Alternate names must be used separately in the document in order to be correct (“Barack
Hussein” from document text “Barack Hussein Obama” would be wrong unless “Barack
Hussein” was also used by itself in the document).
Compared with other slots, more inference can be used when selecting fillers for
per:alternate_names because entities' canonical names will often be absent from source
documents.
Generally, a given (first) name alone is not a correct alternate name unless the person is
unambiguously known that way (e.g., ‘the Donald’ for Donald Trump, or ‘Oprah’ for Oprah
Winfrey, but not ‘John’ for ‘John Smith’).
A last name alone is not a valid alternate name.
Titles or other extraneous information added to a name do not justify an alternate name
(e.g. “Sen. Barbara Boxer” is not a valid alternate name for “Barbara Boxer).
Differing punctuation does not make a name distinct (e.g. “John Q Adams” in a list that
already contains “John Q. Adams” is not valid). Also, for personal names, initials are
considered equivalent to the fully spelled out names (Jonathan David Smith and Jonathan
D. Smith would be equivalent).
Combinations of first and middle names are considered as different from first and last
name combinations (“Barack Hussein” is different from “Barack Obama”).
Former alternate names are also correct as long as the source document indicates that
the name was used at some time in the past.
Entity Justification Correct Filler
Abdurrahman Wahid, also known as Gus Dur Gus Dur
Wahid
Rudolf Rudolf Walter Wanderone, Jr. was an American professional pocket Minnesota Fats
Wanderone billiards player, best known as "Minnesota Fats"
Bob Wilson Bob Wilson's middle name really is Primrose. But his parents Bob Primrose
weren't really being that cruel - apparently it's a Scottish tradition to Wilson
pass on a mother's maiden name as a middle name, and he was
unlucky enough to be the son of a Miss Primrose.
Hillary Clinton Secretary of State Hilary Clinton Hilary Clinton
Roy Scheider Roy Scheider, the actor best known for his role in 'Jaws,' has died. N/A
Scheider died Sunday...
Balbina del Balbina Herrera será la candidata del Partido Revolucionario Balbina Herrera
Carmen Democrático (PRD), que fundó el general Omar Torrijos, para
Herrera Araúz aspirar a la presidencia de Panamá en las elecciones de mayo
洪一峰 出生台南縣鹽水鎮的洪一峰,本名洪文路,出道時藝名為洪文昌 洪文昌
3.2 per:date_of_birth
Content: Value
Quantity: Single
Description: The date on which the assigned person was born.
Inverse Slot: n/a
8 Note the distinction between birthdays and birthdates for an individual. A full date (i.e.
day, month, and year) listed as someone’s birthday might only provide information on the
day and month on which they were born, not the correct year (see Augustus II the Strong
example below).
Entity Justification Correct Filler
Hank Williams the Jan. 1 anniversary of Williams' death, and on his Sept. 17
Sept. 17 birthday. Norm: XXXX-09-17
Augustus II the It was written to celebrate the birthday of King Augustus May 12
Strong II the Strong and was performed for him on his birthday, Norm: XXXX-05-12
May 12, 1727, on the Marktplatz of Leipzig
Haydée Santamaría Haydée Santamaría se suicidó en julio de 1980. Nació el el 21 de enero de 1923
21 de enero de 1923 en Villa Clara. Norm: 1923-01-21
卓琳 卓琳是邓小平的夫人。1916 年 4 月 6 日出生于云南省宣 1916 年 4 月 6 日
威县，1937 年参加革命工作 Norm:1916-04-06
3.3 per:age
Content: Value
Quantity: Single
Description: A reported age of the assigned person.
Inverse Slot: n/a
Age of the person at death is an acceptable answer.
Previous ages are valid responses.
Approximate ages are valid responses. For example, if a source document states that the
assigned person was “about 50”, then “50” would be a valid filler.
Responses for per:age must be explicitly mentioned in the document; annotators cannot
calculate ages.
Although this is a single-value slot, annotators and systems are not required to find the
most recent age mentioned in the source documents.
Entity Justification Correct Filler
Megawati “Today is a day of tragedy for democracy,'' Megawati, 49, told 49
Sukarnoputri her supporters before the demonstration in Jakarta.
Michael Jackson Michael Jackson died June 25 2009 at the age of 50. 50
Tom Lehman Nor Tom Lehman, who turned fifty last month fifty
Norm: 50
Christian Brando Christian Brando, problemático hijo del actor Marlon Brando, 49
murió de neumonía en un hospital de Los Angeles este
sábado, a los 49
王建泰 八十一歲的王建泰民國一九四八年跑船到祕魯,因與船員失散滯 八十一
留祕魯, Norm: 81
3.4 per:country_of_birth
Content: Name
Quantity: Single
Description: The country in which the assigned person was born.
Inverse Slot: gpe:births_in_country*
9 During the Assessment task, adjectival forms (e.g. Canadian) are acceptable responses
for this slot. However, in the slot-filling task, annotators should edit adjectival text strings
to match the appropriate countries.
Note that a person’s stated nationality cannot be used to infer country of birth (for
appropriate placement of information about a person’s nationality, see per:origin)
Geographical areas of contention are valid fillers (e.g. Palestine, Taiwan, Western
Sahara).
City-states (e.g. Vatican City, Singapore) should be classified at the country level.
Entity Justification Correct Filler
Abu Musab Al- Al-Zarqawi, the Jordanian-born militant… Jordanian
Zarqawi Norm: Jordan
Martin Burnham ..for the kidnapping last year of Americans Martin and Gracia N/A
Burnham and Guillermo Sobero.
Rafael Esquivel Esquivel, nacido en Tenerife (España)… España
Melo
梅志坚 梅志坚出生于中国广东,幼年随全家移民到美国纽约,大学毕业后 中国
移居加州并开始从政。
3.5 per:stateorprovince_of_birth
Content: Name
Quantity: Single
Description: The geopolitical entity at state or province level in which the assigned person
was born. This slot must be filled with the name of a state or province.
Inverse Slot: gpe:births_in_stateorprovince*
Capitol districts (e.g. Washington D.C.), emirates (e.g. Dubai), British counties, and
several Chinese “municipalities” and “special administrative regions” (e.g., Chongqing,
Macau) should be classified at the state or province level (you should use an Internet
search to clarify any uncertainties about foreign government systems).
Geographical areas of contention are valid fillers.
Entity Justification Correct Filler
Stephen Harper Harper, born in April of 1959 in Toronto, Ontario Ontario
Stephen Harper Harper was born in Toronto in April 1959. Toronto, Ontario is a Ontario
beautiful city.
Stephen Harper Harper was born in Toronto in April 1959. N/A
Stephen Harper Harper, a Toronto, Ontario native N/A
Yipsi Moreno Moreno, mejoró su propia marca del evento, los 74,25 Camaguey
conseguidos hace cuatro años en los Juegos de Santo
Domingo, donde se llevó también la medalla de oro. La atleta
de 26 años, nacida en la provincia de Camaguey…
黄琪梓 黄琪梓出生于 1991 年，目前还不满 20 周岁，生于广西来宾县 广西
一个小山村
103.6 per:city_of_birth
Content: Name
Quantity: Single
Description: The geopolitical entity at the municipality level (city, town, or village) in which
the assigned person was born. This slot must be filled with the name of a city, town, or village.
Inverse Slot: gpe:births_in_city*
Gaza and Jewish settlements should be classified as cities.
Capitol districts (e.g. Washington D.C.) and China’s Municipalities of Shanghai, Tianjin,
and Chongqing and Special Administrative Regions of Hong Kong and Macau should not
be classified at the city level, rather they should be classified at the state or province level.
GPEs below the city level (e.g. 5 boroughs of New York City) are not valid fillers.
City-states (e.g. Vatican City, Singapore) should be classified at the country level.
Geographical areas of contention are valid fillers.
Entity Justification Correct Filler
Hank Williams Williams lived in Georgiana in the mid 1930's with his mother, Mount Olive West
Lillie, and his sister, Irene, after his birth in Mount Olive West
Tom Lehman Lehman was born in Austin, Minnesota but … Austin
Samuel Moreno Samuel Moreno, elegido este domingo alcalde de Bogotá, es Miami
un economista graduado en Harvard, nacido hace 47 años en
Miami.
楼柏安 楼柏安 1942 年出生于浙江杭州,1980 年毕业于杭州工艺美术研 杭州
究所。
3.7 per:origin
Content: Name
Quantity: List
Description: The nationality and/or ethnicity of the assigned person.
Inverse Slot: n/a
Former nationalities are acceptable responses.
When both nationality and ethnicity are mentioned in source documents, both are valid
answers. For example, if a document states that the assigned person “is a Chinese
American”, both “Chinese” and “American” would be correct fillers.
If country of birth is stated, nationality can be inferred unless specifically stated otherwise
(such as "Egyptian-born Canadian").
For Assessment purposes, names of countries or ethnicities are acceptable responses.
However, during annotation, annotators should edit text to the adjectival form (except
Chinese language slot fillers, which should take the name’s noun form, e.g., 法国 for 法国
人; 彝 for 彝族人).
Continental ethnic categories are acceptable responses (e.g. “African”).
Ethno-religious groups (e.g. “Jewish”) are not acceptable fillers. Use per:religion instead.
Entity Justification Correct Filler
Carla Del Ponte Del Ponte, a Swiss citizen Swiss
11Zacarias Moussaoui Frenchman Zacarias Moussaoui Frenchman
Norm: French
Pascal Yoadimnadji African leader Pascal Yoadimnadji died Friday. African
Marcelo Martins El ariete boliviano Marcelo Martins Moreno boliviano
Moreno
考瑞・哈里斯 新加盟吉林男篮的加拿大籍球员考瑞・哈里斯是美国 加拿大
前 NBA 选手,今年 26 岁,身高 2 米 06,司职中锋,
3.8 per:date_of_death
Content: Value
Quantity: Single
Description: The date of the assigned person’s death.
Inverse Slot: n/a
Entity Justification Correct Filler
Hank Williams But Williams never played the Opry again. At age 29, while New Year’s Day
on the way to a concert in Canton, Ohio, he was found dead 1953
in the back seat of his Cadillac on New Year's Day 1953. Norm: 1953-01-01
Ahmed Shah Massoud is a brother of Ahmed Shah Massoud, the September 2001
Massoud legendary resistance fighter in northern Afghanistan who was Norm: 2001-09-XX
assassinated by operatives of al-Qaida in September 2001.
Haydée Santamaría Haydée Santamaría Cuadrado luchó en la clandestinidad y julio de 1980
la Sierra Maestra y se suicidó en julio de 1980 Norm: 1980-07-XX
马季 2006 年 12 月 20 日，相声大师马季在家中卫生间里的马桶上 2006 年 12 月 20 日
坐着时突发心脏病，终因抢救无效去世。 Norm: 2006-12-20
3.9 per:country_of_death
Content: Name
Quantity: Single
Description: The country in which the assigned person died. This slot must be filled with the
name of a country.
Inverse Slot: gpe:deaths_in_country*
Geographical areas of contention are valid fillers (e.g. Palestine, Taiwan, Western
Sahara).
City-states (e.g. Vatican City, Singapore) should be classified at the country level.
Entity Justification Correct Filler
Ahmed Said Khadr Ahmed Said Khadr, an Egyptian-born Canadian, who was Pakistan
killed last October in the South Waziristan region of
Pakistan.
Mohammed Khan Khan was killed in the North West Frontier Province N/A
Mohammed Khan Khan was killed in North West Frontier Province… NWFP is Pakistan
located in Pakistan
Isidra Sarmiento Tres personas muertas dejó hoy el inicio de la temporada de Honduras
lluvias en el norte de Honduras, incluso la señora Isidra
Sarmiento
雷远高 正军职离休干部、原南京高级陆军学校研究部部长雷远高同志, 南京
因病于 2009 年 9 月 16 日在南京逝世,享年 88 岁。
123.10 per:stateorprovince_of_death
Content: Name
Quantity: Single
Description: The geopolitical entity at state or province level in which the assigned person
died. This slot must be filled with a state or province name.
Inverse Slot: gpe:deaths_in_stateorprovince*
Capitol districts (e.g. Washington D.C.), emirates (e.g. Dubai), British counties, and
several Chinese “municipalities” and “special administrative regions” (e.g., Chongqing,
Macau) should be classified at the state or province level (you should use an Internet
search to clarify any uncertainties about foreign government systems).
Geographical areas of contention are valid fillers.
Entity Justification Correct Filler
Ahmed Said Khadr Ahmed Said Khadr, an Egyptian-born Canadian, who was N/A
killed last October in the South Waziristan region of
Pakistan.
Mohammed Khan Khan was killed in the North West Frontier Province North West Frontier
Province
Jay Smith Smith died in Austin, Texas Texas
Isidra Sarmiento La señora Isidra Sarmiento y su hija Karla Romero perecieron Cortés
al ser arrastradas por la corriente de agua en Puerto Cortés
del departamento de Cortés.
师占荣 离休干部、空军连云港场站原政治委员师占荣同志(正师职待遇) 江苏
,因病于 2009 年 9 月 21 日在江苏南京逝世,享年 85 岁。
3.11 per:city_of_death
Content: Name
Quantity: Single
Description: The geopolitical entity at the level of city, town, village in which the assigned
person died. This slot must be filled with a city, town, or village name.
Inverse Slot: gpe:deaths_in_city*
Gaza and Jewish settlements should be classified as cities.
Capitol districts (e.g. Washington D.C.) and China’s Municipalities of Shanghai, Tianjin,
and Chongqing and Special Administrative Regions of Hong Kong and Macau should not
be classified at the city level, rather they should be classified at the state or province level.
GPEs below the city level (e.g. 5 boroughs of New York City) are not valid fillers.
City-states (e.g. Vatican City, Singapore) should be classified at the country level.
Geographical areas of contention are valid fillers.
Entity Justification Correct Filler
Ahmed Said Khadr Ahmed Said Khadr, an Egyptian-born Canadian, who was N/A
killed last October in the South Waziristan region of
Pakistan.
Mohammed Khan Khan was killed in the North West Frontier Province N/A
Jay Smith Smith died in Wilkes-Barre General Hospital Wilkes-Barre
13Isidra Sarmiento La señora Isidra Sarmiento y su hija Karla Romero perecieron Puerto Cortés
al ser arrastradas por la corriente de agua en Puerto Cortés
del departamento de Cortés.
拉姆塞 拉姆塞曾效力的伊普斯维奇俱乐部宣布,拉姆塞是 28 日在伊普 伊普斯维奇
斯维奇一家护理所因病去世的。
3.12 per:cause_of_death
Content: String
Quantity: Single
Description: The explicit cause of death for the assigned person.
Inverse Slot: n/a
A mention of a serious illness is not an acceptable filler unless it is explicitly linked to the
death of the assigned person in the document.
Assessors should be lenient in their judgment of the fullness of selected strings for
per:cause_of_death (in the Roy Scheider example below, “complications” or “staph
infection” would be acceptable answers, though not preferred).
Entity Justification Correct Filler
Hank Williams The coroner's report said Williams died of a ‘heart condition heart condition and
and hemorrhaging,' hemorrhaging
Marilyn French Author Marilyn French, 79, passed away of heart failure on heart failure
May 2, 2009, in New York City
Roy Scheider Scheider had suffered from multiple myeloma for several complications from a
years, and died of complications from a staph infection staph infection
Christian Brando Christian Brando, el hijo mayor del afamado actor Marlon neumonía
Brando, murió de neumonía
赵丽蓉 2000 年 7 月 17 日早晨，赵丽蓉因肺癌在家中安然辞世，享年 肺癌
73 岁。
3.13 per:countries_of_residence
Content: Name
Quantity: List
Description: All countries in which the assigned person has lived.
Inverse Slot: gpe:residents_of_country*
Use caution when selecting per:origin fillers for use as countries of residence. While a
person’s national background can often justify residence, one’s ethnic background cannot.
Former countries of residence are correct responses.
Residence must be lexically supported in source documents (e.g., “home”, “house”,
“resides”, “grew up”, etc.) but there is no duration standard to define residence.
Prison stays and similar legal holdings can support residence fillers.
Vacation homes can support residence fillers as long as it is clear the residence is owned
by the entity and is not just a rental.
Although annotators cannot generally infer a person’s country of residence from the
location of his or her employer, national-level government employees or valid candidates
for such positions are an exception to this rule (e.g., "U.S. Senator Jesse Helms" and
14"Former Indian Prime Minister V.P. Singh" could be said to have resided in, respectively,
the U.S. and India).
Geographical areas of contention are valid fillers (e.g. Palestine, Taiwan, Western
Sahara).
City-states (e.g. Vatican City, Singapore) should be classified at the country level.
Entity Justification Correct Filler
Paavo Nurmi Finland's Nurmi won his nine golds in the 1920's. Finland
Jon Wong Surveillance cameras show a Chinese man, later identified N/A
as Jon Wong, leaving the White House
Abdurrahman Wahid Abdurrahman returned to his house in Cilandak, Indonesia Indonesia
Al Gore The Gore family resides in Nashville, Tennessee N/A
Remy Smith A judge in New York City said Remy Ma, whose real name is United States
Remy Smith, could not leave the United States for a five-
country European concert tour.
Mohammad Baqer "... said Qalibaf, who succeeded President Mahmoud Iran
Qalibaf Ahmadinejad as mayor after losing in Iran's 2005 presidential
race.
Ehud Barak Israeli Defense Minister and Labour party chairman Ehud Israeli
Barak on Wednesday demanded… Norm: Israel
Belisario Betancourt El ex presidente colombiano Belisario Betancourt colombiano
Norm: Colombia
李善喜 李善喜 1970 年出生于汉城，8 岁的时候她移居到了美国。 美国
3.14 per:statesorprovinces_of_residence
Content: Name
Quantity: List
Description: Geopolitical entities at the state or province level in which the assigned person
has lived. This slot must be filled with state or province names.
Inverse Slot: gpe:residents_of_stateorprovince*
Former states or provinces of residence are correct responses.
Residence must be lexically supported in source documents (e.g., “home”, “house”,
“resides”, “grew up”, etc.) but there is no duration standard to define residence.
Capitol districts (e.g. Washington D.C.), emirates (e.g. Dubai), British counties, and
several Chinese “municipalities” and “special administrative regions” (e.g., Chongqing,
Macau) should be classified at the state or province level (you should use an Internet
search to clarify any uncertainties about foreign government systems).
Prison stays and similar legal holdings can support residence fillers.
Vacation homes can support residence fillers as long as it is clear the residence is owned
by the entity and is not just a rental.
Although annotators cannot generally infer a person’s residence from his or her place of
employment, U.S. senators and representatives are exceptions to this rule because they
are constitutionally required to have lived in the states they represent.
Without other supporting language, birthplace is not sufficient to justify residence. (see
Roy Scheider example below for an exception based on other supporting language).
Geographical areas of contention are valid fillers.
15Entity Justification Correct Filler
Abdurrahman Wahid Abdurrahman returned to his house in Cilandak, Indonesia N/A
Al Gore The Gore family resides in Nashville, Tennessee Tennessee
George W. Bush US President George W. Bush will meet his French Maine
counterpart Nicolas Sarkozy on Saturday at the Bush
family's summer home in Kennebunkport, Maine,
Roy Scheider Born into a working class family in Orange, New Jersey, Roy New Jersey
Scheider...
Yipsi Moreno Moreno, mejoró su propia marca del evento, los 74,25 Camaguey
conseguidos hace cuatro años en los Juegos de Santo
Domingo, donde se llevó también la medalla de oro. La
atleta de 26 años, nacida y crecida en la provincia de
Camaguey...
梅志坚 梅志坚大学毕业后移居加州并开始从政。2003 年在加州地方 加州
选举中当选圣盖博市首位华裔市议员。
3.15 per:cities_of_residence
Content: Name
Quantity: List
Description: Geopolitical entities at the level of city, town, or village in which the assigned
person has lived. This slot must be filled with the name of a city, town, or village.
Inverse Slot: gpe:residents_of_city*
Former cities of residence are correct responses.
Residence must be lexically supported in source documents (e.g., “home”, “house”,
“resides”, “grew up”, etc.) but there is no duration standard to define residence.
Prison stays and similar legal holdings can support residence fillers.
Vacation homes can support residence fillers as long as it is clear the residence is owned
by the entity and is not just a rental.
Gaza and Jewish settlements should be classified as cities.
Capitol districts (e.g. Washington D.C.) and China’s Municipalities of Shanghai, Tianjin,
and Chongqing and Special Administrative Regions of Hong Kong and Macau should not
be classified at the city level, rather they should be classified at the state or province level.
Note that proof of employment in a city does not justify a residence filler.
Without other supporting language, birthplace is not sufficient to justify residence. (see
Roy Scheider example below for an exception based on other supporting language).
Unlike countries and states of residence, top-level government employees of cities cannot
be inferred to reside in their respective GPEs.
GPEs below the city level (e.g. 5 boroughs of New York City) are not acceptable answers.
City-states (e.g. Vatican City, Singapore) should be classified at the country level.
Geographical areas of contention are valid fillers.
Entity Justification Correct Filler
Abdurrahman Wahid Abdurrahman returned to his house in Cilandak, Indonesia N/A
16Al Gore The Gore family resides in Nashville, Tennessee Nashville
George W. Bush US President George W. Bush will meet his French Kennebunkport
counterpart Nicolas Sarkozy on Saturday at the Bush
family's summer home in Kennebunkport, Maine,
Rudy Giuliani Former New York City mayor Rudy Giuliani said... N/A
Roy Scheider Born into a working class family in Orange, New Jersey, Roy Orange
Scheider...
Roy Scheider Scheider lived in Sag Harbor, NY... Sag Harbor
Nixon Moreno Nixon Moreno, quien está refugiado en la sede de la Caracas
Nunciatura Apostólica de Caracas, ….
平多夫 现年 89 岁的平多夫 1955 年移居加拿大多伦多, 他是两家大型 多伦多
音乐器材公司的总裁。
3.16 per:schools_attended
Content: Name
Quantity: List
Description: Any school (college, high school, university, etc.) that the assigned person has
attended.
Inverse Slot: org:students*
Enrollment at a school does not justify a filler for this slot; there must be evidence in the
document that the person actually attended the school.
Entity Justification Correct Filler
Roy Lubove Lubove was born in Queens, N.Y., and graduated from Columbia College;
Columbia College in 1956. He earned his doctorate at Cornell University
Cornell University in 1960, and joined the faculty at Harvard
that year.
Tom Lehman He attended the University of Minnesota, graduating with a University of
degree in Business/Accounting and turned professional in Minnesota
1982.
Nixon Moreno Moreno aprovechó el período de refugio en la Nunciatura Universidad de los
Apostólica para concluir sus estudios de Ciencias Políticas Andes
de la Universidad de los Andes
黄佐临 我国一代艺术大师黄佐临先生 6 月 1 日因病去世。青年时期, 伯明翰大学
他曾在英国伯明翰大学学习
3.17 per:title
Content: String
Quantity: List
Description: Official or unofficial name(s) of the employment or membership positions that
have been held by the assigned person.
Inverse Slot: n/a
Generic professions such as ‘doctor’ or ‘lawyer’, are acceptable answers, but if more
specific titles within those professions such as ‘Chief Neurosurgeon’ or ‘Associate Partner’
17are mentioned in the document, those more specific versions are preferred (and 'doctor' or
'lawyer' would be considered inexact).
Extremely generic/vague position names within an organization such as “senior leader” or
“member” are not acceptable answers. However, absent any more descriptive
information, vague profession-like titles are acceptable (see examples below for Frankie
Delgado and Bryan Fuller).
Unofficial or metaphorical titles that do not refer to an employment or membership
position, (such as “hero” or “socialite”) are not acceptable fillers.
Although "founder" does not justify a per:employee_or_member_of filler, it is a valid title.
The name of the person’s employer should not be included as part of the title, so ‘reporter
with the LA Times’ and ‘reporter for the Sacramento Union’ would be inexact -- the answer
in these cases should be simply 'reporter'.
In selecting modifiers to include in title fillers, a general rule is to include them if they
describe positions within organizations (e.g., “record producer”, “defense chief”,
“advertising account manager”, and “executive producer”) and exclude them if they do not
(e.g., text excerpts “talent show judge”, “rookie driver”, “meteorology professor”, and
“LGBT activist” would produce the fillers “judge”, “driver”, “professor”, and “activist”).
Some titles will be very specific to the organizations that bestow them and, therefore,
seem strange out of context. However, these titles are correct fillers (see the examples
below for Kendra Wilkinson, Simon Cowell, and Jake Pavelka).
Positions on a sports team (short stop, quarterback, center) are considered titles for
professional athletes.
Aristocratic titles (e.g. “Duchess”, “King”) are acceptable fillers, as are infixed titles (e.g.
“Lord” in “Alfred Lord Tennyson”).
Entity Justification Correct Filler
Kofi Annan United Nations Secretary General Kofi Annan Secretary General
Carla Del Ponte Del Ponte, a Swiss citizen, was elected as its prosecutor in prosecutor
September 1999 for a four-year period which is up for
renewal.
Tom Lehman Thomas Edward Lehman is an American professional golfer. golfer
Mia Kirshner Mia Kirshner, star of “the L Word” Star
Kendra Wilkinson Wearing a white veil and surrounded by her girlfriends, the bunny
former Playboy bunny looked to...
Simon Cowell The American Idol judge is donating a day's pay to Idol Gives judge
Back, PEOPLE has confirmed.
Jake Pavelka ABC announces an All Star cast on Dancing with the Stars Bachelor
including Bachelor Jake Pavelka.
George Boyd it is likely that Boyd will be a Nottingham Forest player at the winger
end of the season as the two clubs are believed to have
agreed on a fee for the winger.
Frankie Delgado Frankie Delgado is an American musician and television musician,
personality known for his appearances on the reality series. television
personality
Bryan Fuller The episode of was written by series creator Bryan Fuller. series creator
Alberto Gonzales Gonzales, an architect of contentious US 'war on terror' legal N/A
tactics...
Fredrik Reinfeldt Persson served as Sweden's premier for 10 years, but premier
stepped down in October 2006 after his party was ousted by a
center-right coalition led by Fredrik Reinfeldt.
18Balbina Herrera Balbina Herrera, una veterana dirigente popular y ex ministra ministra de
de Vivienda Vivienda
余耀文 余耀文今年 47 岁,2004 年 8 月至今兼任县工商联合会会长, 会长
2007 年 3 月当选安阳县人大常委会副主任。
3.18 per:employee_or_member_of
Content: Name
Quantity: List
Description: The organizations or geopolitical entities (governments) of which the assigned
person has been an employee or member.
Inverse Slot: {org,gpe}:employees_or_members*
Programs or projects should not be considered organizations and different iterations of the
same organization (e.g., the 111th U.S. Congress and the 112th U.S. Congress) should not
be considered as distinct entities.
Government officials at the top level of government (Supreme Court, Department/Ministry
heads) and military personnel should be considered employees/members of their
countries as well as employees/members of their respective government or military
bodies. Government employees in lower departments (e.g. CIA, Centers for Disease
Control) should be employees/members of those organizations only. Note that, with
respect to this rule, the European Union and other “countries of countries” should be
categorized as a country.
The title of “Fellow” can be used as justification for membership.
People can sometimes become members of an organization posthumously (such as halls
of fame).
Actors, directors, screenwriters, etc. should not be considered employees/members of TV
shows or movies in which they appeared or helped to produce. However, they should be
considered as employees of networks/companies that produced such works.
Writers, photographers, etc. should not be considered employees of newspapers,
magazines, etc. They should be considered employees of the organizations that produce
such publications.
Organizations of which the assigned person was formerly, but no longer, an employee or
member are correct slot fillers.
Although employment generally means that individuals have received payment from
organizations, unpaid internships and volunteer positions would also fall into this category.
Organizations that have hired individuals as independent contractors are valid fillers for
per:employee_or_member_of.
Organizations of which the assigned entity is a founder or owner with no other
position/relationship are not acceptable answers.
Employees/members of subsidiary organizations are not considered employees/members
of the respective parent organizations (e.g. if the assigned entity were an annotator at
LDC, “UPenn” would not be a valid filler for per:employee_or_member_of). See
org:subsidiaries and org:parents for more details about these types of organizations.
Entity Justification Correct Filler
Hillary Clinton Newly-appointed US Secretary of State Hillary Clinton has State Department
been greeted warmly by State Department employees
19Muhammad Mohammed Najjar, a senior leader of the PLO's Black Black September
Assessorsssef al- September unit that carried out a number of terrorist attacks.
Najjar
Tom Lehman He played on the PGA Tour with little success from 1983 to PGA Tour
1985, and was then obliged to play elsewhere for the following
six seasons. [...] He regained his PGA Tour […] and has
enjoyed unbroken membership of the PGA Tour since 1992.
David Petraeus Gen. David Petraeus, U.S. Army general, said... Army
Romano Prodi Prodi was president of the EU’s European Commission from European
September 1999 to October 2005. Commission
Ronnie Lott Lott was known for his toughness and was inducted into the Pro Football Hall of
Pro Football Hall of Fame in 2000. Fame
Samuel Alito Supreme Court justice Samuel Alito Supreme Court
Donald Wildmon American Family Association Founder and Chairman: The American Family
Rev. Donald Wildmon Association
Alberto Gonzales Sampson told the committee that Gonzales, who leads the Department of
Department of Justice, was involved. Justice
Mostafa Iranian Defense Minister Mostafa Mohammad Najjar Iran
Mohammad-Najjar
Ashraf Al-Sayed He works as an owner and manager of Betra Balad firm for Betra Balad
Aassessorsb building and contracting investments
Robert Menendez U.S. Senator for New Jersey Robert Menendez… U.S., New Jersey
Brian Fuller Brian Fuller, creator of the critically-acclaimed new ABC series ABC
"Pushing Daisies"…
David Petraeus Gen. David Petraeus, U.S. Army general, said... U.S.
Romano Prodi Prodi was president of the EU’s European Commission from EU
September 1999 to October 2005.
Justin Theroux Directed by Ben Stiller; written by Justin Theroux, Stiller and DreamWorks
Ethan Cohen, based on a story by Stiller and Theroux; Pictures
released by DreamWorks Pictures...
Jacob Zuma Jacob Zuma, new leader of South Africa's governing party, South Africa
says business and labor will together 'find workable solutions.'
Abdul Rahim Noor Others on the list are Daim, Malaysia's police chief Abdul Malaysia
Rahim Noor…
Miguel Sanz Miguel Sanz, tomará posesión de su cargo este jueves tras Congreso de los
proponer que su formación, socia del conservador Partido Diputados
Popular, forme un grupo independiente de Éste en el
Congreso de los Diputados.
考瑞・哈里斯 新加盟吉林男篮的加拿大籍球员考瑞・哈里斯是美国 吉林男篮
3.19 per:religion
Content: String
Quantity: Single
Description: The religion to which the assigned person has belonged.
Inverse Slot: n/a
Former religions are acceptable answers.
Ethno-religious groups (Judaism) should always be annotated as per:religion (as opposed
to per:origin).
Entity Justification Correct Filler
20Abubakar Abdurajak Afghan-trained Muslim firebrand Abubakar Abdurajak Muslim
Janjalani Janjalani
Tom Lehman the entire Ryder Cup leadership – Tom Lehman, Corey Pavin Christians
and Loren Roberts – are self-professed born-again Christians. Norm: Christian
Julio Duarte Julio Duarte, un cristiano, es un miembro del Templo cristiano
Evangélico Alianza
奥巴马 奥巴马在自传中谈到他如何在长大成人后改信基督教。希拉里 基督教
也公开谈论她作为美以美教派基督徒的信仰。
3.20 per:spouse
Content: Name
Quantity: List
Description: The spouse(s) of the assigned person.
Inverse Slot: per:spouse
Former spouses are acceptable answers.
Marriages do not have to be legally recognized in order for resulting spouses to be correct
fillers. However, the word “partner”, by itself, does not justify a per:spouse filler (it must be
coupled with a word like “marriage”, “husband”, “wife”, etc.)
Entity Justification Correct Filler
Hank Williams Williams’ wife at the time of his death, Billie Jean Jones Billie Jean Jones
Hyman Rickover And the latest lady to take up the cudgels for her husband via Ruth Rickover
a newspaper interview was Mrs. Ruth Rickover. She is the
wife of Rear Adm. Hyman G. Rickover
Juan Manuel Santos Juan Manuel Santos está casado con María Clemencia María Clemencia
Rodríguez Rodríguez
戴美玉 吴伯雄夫人戴美玉访沪 观“大飞机”感动落泪 吴伯雄
3.21 per:children
Content: Name
Quantity: List
Description: The children of the assigned person, including adopted and step-children.
Inverse Slot: per:parents
Entity Justification Correct Filler
Hank Williams Williams’ son, Hank Williams Jr. Hank Williams Jr.
Hank Williams Jr. Jr.’s youngest child, Katie Williams Katie Williams
Isidra Sarmiento La señora Isidra Sarmiento y su hija Karla Romero Karla Romano
侯耀文 侯耀文长女侯瓒状告伯父侯耀华侵占父亲遗产 侯瓒
3.22 per:parents
Content: Name
Quantity: List
Description: The parents of the assigned person. In addition to biological parents, step-
parents and adoptive parents are also acceptable answers.
21Inverse Slot: per:children
Entity Justification Correct Filler
Hank Williams Hank lived in Georgiana in the mid 1930’s with his mother, Lillie Williams Lillie Williams
Megawati Roping in Guruh would have helped the government party counter to Sukarno
Sukarnoputri some extent Megawati’s popularity, which largely stems from the
reverence that millions of Indonesians have for her father, Indonesia’s
first post-independence president, Sukarno
Samuel Para esos comicios Moreno figuraba como favorito. “Esto es un montaje Juan Antonio
Moreno político que le metieron a él porque como tenía las elecciones ganadas Moreno
de la universidad de Mérida,” asintió el padre del dirigente, Juan Antonio Zambrano
Moreno Zambrano
洪荣宏 洪荣宏表示,父亲洪一峰生前致力于闽南语歌谣的保存,因此他们决定将所 洪一峰
有奠仪收入以及
3.23 per:siblings
Content: Name
Quantity: List
Description: The brothers and sisters of the assigned person.
Inverse Slot: per:siblings
In addition to full siblings, step-siblings and half-siblings are acceptable answers.
Brothers- or sisters-in-law are not acceptable responses for per:siblings (they are fillers for
per:other_family).
Entity Justification Correct Filler
Ahmed Wali Massoud is a brother of Ahmed Shah Massoud, the legendary Ahmed Shah
Massoud resistance fighter in northern Afghanistan who was assassinated Massoud
by operatives of al-Qaida in September 2001. On Wednesday
another brother of the late commander, Ahmed Wali Massoud,
who had not previously endorsed any candidate, said he
supported Karzai's election
Elizabeth Blackwell Unable to get a post in American hospitals, Blackwell, her sister Emily
Emily and friend Dr Marie Zakrzewaska started their own hospital,
the New York Infirmary for Indigent Women and Children.
Alcides Moreno Alcides Moreno, de 37 años, cayó accidentalmente junto a su Edgard
andamio el 7 de diciembre desde el piso 47 de una torre de la
calle 66 junto a su hermano Edgard, que murió instantáneamente
李远鹏 前“中研院”院长李远哲的胞弟、交通大学理学院院长李远鹏当选新 李远哲
科院士,再加上大哥李远川,创下一门三院士的纪录。
3.24 per:other_family
Content: Name
Quantity: List
Description: Family other than siblings, parents, children, and spouse (or former spouse).
Correct fillers for this slot include brothers-in-law, sisters-in-law, grandparents, grandchildren,
cousins, aunts, uncles, etc.
Inverse Slot: per:other_family
Entity Justification Correct Filler
22You can also read
