graviti logoProductOpen DatasetsAbout
Request DemoSign in
214
0
0
Children Book Test
General
Discussion
Code
Activities
c77e026b-8cd1-11eb-88ae-0e1f58d5e9a9
724bcf2·
Jun 20, 2021 9:56 AM
·1Commits

Overview

Children’s Book Test (CBT), designed to measure directly how well language models can exploit wider linguistic context. The CBT is built from books that are freely available thanks to Project Gutenberg. Details and baseline results on this dataset can be found in the paper:

Felix Hill, Antoine Bordes, Sumit Chopra and Jason Weston. The Goldilocks Principle: Reading Children’s Books with Explicit Memory Representations, arXiv:1511.02301.

After allocating books to either training, validation or test sets, we formed example ‘questions’ from chapters in the book by enumerating 21 consecutive sentences. In each question, the first 20 sentences form the context, and a word is removed from the 21st sentence, which becomes the query. Models must identify the answer word among a selection of 10 candidate answers appearing in the context sentences and the query. For finer-grained analyses, we evaluated four classes of question by removing distinct types of word: Named Entities, (Common) Nouns, Verbs and Prepositions

Here is an example of question (context + query) from Alice in Wonderland by Lewis Carroll:

Context:
1 So they had to fall a long way .
2 So they got their tails fast in their mouths .
3 So they could n't get them out again .
4 That 's all .
5 `` Thank you , " said Alice , `` it 's very interesting .
6 I never knew so much about a whiting before . "
7 `` I can tell you more than that , if you like , " said the Gryphon .
8 `` Do you know why it 's called a whiting ? "
9 `` I never thought about it , " said Alice .
10 `` Why ? "
11 `` IT DOES THE BOOTS AND SHOES . '
12 the Gryphon replied very solemnly .
13 Alice was thoroughly puzzled .
14 `` Does the boots and shoes ! "
15 she repeated in a wondering tone .
16 `` Why , what are YOUR shoes done with ? "
17 said the Gryphon .
18 `` I mean , what makes them so shiny ? "
19 Alice looked down at them , and considered a little before she gave her answer .
20 `` They 're done with blacking , I believe . "

Query: `` Boots and shoes under the sea , " the XXXXX went on in a
deep voice , `` are done with a whiting ".

Candidates: Alice|BOOTS|Gryphon|SHOES|answer|fall|mouths|tone|way|whiting

Answer: gryphon

Citation

Please use the following citation when referencing the dataset:

@article{hill2015goldilocks,
  title={The goldilocks principle: Reading children's books with explicit memory representations},
  author={Hill, Felix and Bordes, Antoine and Chopra, Sumit and Weston, Jason},
  journal={arXiv preprint arXiv:1511.02301},
  year={2015}
}
🎉Many thanks to Graviti Open Datasets for contributing the dataset
Basic Information
Application ScenariosNot Available
AnnotationsNot Available
TasksNot Available
LicenseUnknown
Updated on2021-01-20 03:42:36
Metadata
Data TypeNot Available
Data Volume0
Annotation Amount0
File Size0B
Copyright Owner
Facebook Research
Annotator
Unknown
More Support Options
Start building your AI now
Get StartedContact