Visualizing textual distributions of repeated LLM responses to characterize LLM knowledge

Richard Brath, Adam James Bradley, David Jonker

Room: 110

2023-10-22T03:00:00ZGMT-0600Change your timezone on the schedule page
Exemplar figure, described by caption below
Issuing the same prompt to an LLM many times generates similar responses that approximate distributions. These distributions can be visualized with proportionally encoded correct words to depict exact matches; or processed into mind-maps to show LLM knowledge breadth.
Fast forward

The breadth and depth of knowledge learned by Large Language Models (LLMs) can be assessed through repetitive prompting and visual analysis of commonality across the responses. We show levels of LLM verbatim completions of prompt text through aligned responses, mind-maps of knowledge across several areas in general topics, and an association graph of topics generated directly from recursive prompting of the LLM.