Visually Analyzing Contextualized Embeddings



Matthew Berger

 External link (DOI) 

 View presentation:2020-10-30T14:50:00ZGMT-0600Change your timezone on the schedule page
2020-10-30T14:50:00Z

Exemplar figure — Our interface allows the exploration of contextualized embeddings produced by language models. Our design shows (A) co-occurrences of phrases via their assigned clusters, (B) per-cluster span lengths and (C) how much context a given cluster captures. One may also inspect example sentences in detail (D), here highlighting terms that describe building structures.

Fast forward

Direct link to video on YouTube: https://youtu.be/H9E7IeivX-Y

Keywords

Machine Learning Visual Analytics Natural Language Processing

Abstract

In this paper we introduce a method for visually analyzing contextualized embeddings produced by deep neural network-based language models. Our approach is inspired by linguistic probes for natural language processing, where tasks are designed to probe language models for linguistic structure, such as parts-of-speech and named entities. These approaches are largely confirmatory, however, only enabling a user to test for information known a priori. In this work, we eschew supervised probing tasks, and advocate for unsupervised probes, coupled with visual exploration techniques, to assess what is learned by language models. Specifically, we cluster contextualized embeddings produced from a large text corpus, and introduce a visualization design based on this clustering and textual structure - cluster co-occurrences, cluster spans, and cluster-word membership - to help elicit the functionality of, and relationship between, individual clusters. User feedback highlights the benefits of our design in discovering different types of linguistic structures.