List of Links
Chapter 7 Conclusion
This paper makes the following contributions:
• In Chapter 3, we improved the self-supervised learning objective for large-scale pre-trained language models. In Section 3.1, we replaced the next sentence prediction loss in language model pre-training with a new sentence ordering prediction loss, and showed that this change resulted in a state-of-the-art pre-trained encoder. In Section 3.2, we showed that in contrast to previous work that fine-tuned pre-trained decoders on human-annotated datasets, a well-designed self-supervised task can have a similar effect in in-context few-shot learning settings, promoting the cross-task generalization ability of the model.
• In Chapter 4, we transformed various naturally occurring data structures on Wikipedia into training data for various NLP tasks. In Section 4.1, we leveraged hyperlinks as training data to pre-train entity representations and created a model that can encode any entity. In Section 4.2, we used article structures such as sections and document titles to train sentence representations. Evaluation results on discourse-related tasks showed that such training improved model performance. In Section 4.3, we extracted training data from article category graphs and demonstrated that the extracted data improved model performance on textual entailment tasks. These results clearly demonstrate the benefits of pre-training a structure-aware model.
• In Chapter 5, we defined a new task of separating semantics and syntax and tackled it by designing a training objective and neural architecture. In Section 5.1, we built the first neural model to separate semantics and syntax in sentence representations. The model exploits the fact that paraphrasing pairs share meaning but differ in syntax. In addition to the semantic evaluation metric, we proposed an evaluation metric for the syntactic representation and found that the best performance on both metrics is obtained when the separation between the two latent representations is maximized. In Section 5.2, we adapt this framework to controlled paraphrasing, attempting to control the output text with syntactic and sentence examples. To formally define this controlled generation task, we annotated an evaluation set and proposed an evaluation metric. In subsequent work, we extended this framework and task setting to machine translation (Chen et al., 2020b) and showed that this idea may be generalizable to any data with a pair data structure.
• In Chapter 6, we constructed a challenging dataset from fan-contributed websites. We also proposed evaluation criteria and possible solutions, and conducted extensive experiments to characterize the new challenge. In Section 6.1, we cast the task as long-form data-to-text generation and generate arbitrary Wikipedia section text from a variety of tabular data by creating a large-scale dataset. This task is challenging because the model must generate coherent sentences that connect all entities in the tabular data, and the story must also fit the background knowledge of the tabular data. In Section 6.2, we summarize long TV show transcripts. This task has several challenges, such as plot information is not explicitly stated, but only implied in the dialogue, and the need to draw information from a wide range of the input transcripts. Since characters are fundamental to the plot of a TV show, we also proposed two character-centric evaluation criteria. In Section 6.3, we generate long-form stories from character descriptions and summaries. This task poses several challenges for story generation models, such as long inputs and outputs, and consistency in character modeling.
Below we discuss some future directions.
• Identify underlying contributing factors. In Chapter 5, we introduced a neural model that uses implicit yet natural supervision from paraphrases and bilingual texts to improve interpretability and controllability. Future work can generalize this idea to any resource formed by a data pair, such as dialogue and summaries. It can be used to separate shared and non-shared elements between pairs of inputs, such as intent and personalized style in dialogues, sentence-level fluency and document-level discourse in sentence modeling, and important events and irrelevant details in summaries.
Another possibility is to separate task supervision when a task can be decomposed into two subtasks. For example, cross-language summarization can be thought of as a combined task of summarization and translation. Separating task supervision can improve a model's ability to generalize across tasks and allow us to extract valuable intermediate supervision that is usually not available.
In general, uncovering latent factors is an attractive research direction. Large pre-trained models are producing superhuman performance, but researchers have yet to understand the behavior of these models. Outside of this paper, research has also been completed that benefits from interpretable latent variables, leading to efficient neural models (Chen and Gimpel, 2018) and effective semi-supervised learning (Chen et al., 2018a). Increased interpretability can improve robustness and worst-case behavior to better suit user-facing applications.
• Natural supervision for text generation. In Chapter 4, we introduced an approach that leverages various natural supervision for representation learning. It will be interesting in future research to see if the same can be applied to text generation. In particular, future research could explore using hyperlinks to improve the entity tracking performance of text generation systems and using article structures to increase the discourse coherence of generated text.
• A unified model for supervision of different languages. In Chapter 4, we discussed modeling options for taking into account different linguistic knowledge (e.g., entities and discourse), but it remains to be seen how best to design an integrated model that can incorporate all these learning signals. Future research may find that integrated models perform better as humans rely on multiple linguistic features simultaneously to solve tasks. Additionally, future research may also explore combining discourse, linking, and paraphrasing goals with models like BERT, as well as other kinds of natural supervision, such as bold/italic/underline annotations that occur naturally in web texts, or long-distance discourse cues such as there being two paragraphs in two chapters.
• Learn common sense knowledge from natural supervision. Future research could also explore learning common sense knowledge from naturally occurring data, for example learning domain-specific common sense from conversations on specific subreddits.[1] Extract common sense knowledge (e.g., technical or social) from existing pre-trained models. Common sense understanding is ubiquitous in language. It is rarely explicitly described because humans often assume that knowledge is familiar to all they interact with. This assumption also leads humans to believe that any intelligent system should be able to understand such knowledge. Two properties of common sense knowledge make this a challenging and necessary capability: In practice, when deploying a model in a real application, this knowledge improves language understanding and therefore the model's reliability.
• Richly explanatory text generation. Future research could explore text generation that includes rich, detailed descriptions of the world in which the task is situated. This direction is related to the work in Chapter 6, as the descriptions could be either tabular data about specific background knowledge (Section 6.1) or long documents about fictional characters (Section 6.3). These detailed descriptions explicitly describe the knowledge that the generated text should follow, so they are like “controlled environments” that simulate the real world, providing an opportunity to improve the evaluation of text generation and increase the fidelity of neural models.
[1] Subreddits are found on the social media website Reddit and are dedicated to specific topics on which people write.
