Semantic scaffolding: Augmenting textual structures with domain-specific groupings for accessible data exploration (Preprint)
Authors: J. Zong; I. Pedraza Pineros; M. K. Chen; D. Hajas; A. Satyanarayan
Published: arXiv (preprint) (2025)
DOI · Publisher’s page · PDF · Accessible HTML
Abstract
Drawing connections between interesting groupings of data and their real-world meaning is an important, yet difficult, part of encountering a new dataset. A lay reader might see an interesting visual pattern in a chart but lack the domain expertise to explain its meaning. Or, a reader might be familiar with a real-world concept but struggle to express it in terms of a dataset’s fields. In response, we developed semantic scaffolding, a technique for using domain-specific information from large language models (LLMs) to identify, explain, and formalize semantically meaningful data groupings. We present groupings in two ways: as semantic bins, which segment a field into domain-specific intervals and categories; and data highlights, which annotate subsets of data records with their real-world meaning. We demonstrate and evaluate this technique in Olli, an accessible visualization tool that exemplifies tensions around explicitly defining groupings while respecting the agency of readers to conduct independent data exploration. We conducted a study with 15 blind and low-vision (BLV) users and found that readers used semantic scaffolds to quickly understand the meaning of the data, but were often also critically aware of its influence on their interpretation.
Media
-
📝
Helping Blind Readers of charts Make Sense of Data with a Semantic Scaffold of Domain Knowledge
When you open a dataset for the first time, the numbers rarely speak for themselves. You might see patterns in a chart but not know what they mean — or you might have an idea in mind (“sports cars”) but struggle to translate it into the dataset’s fields. For blind and low-vision screen reader users, this challenge is even sharper, since interfaces rarely provide the kind of overview that sighted readers get at a glance.
-
🎧
Dial a domain expert: Scaffolding your data exploration
Tune in to discover semantic scaffolding, an innovative technique that uses AI to make complex data more understandable by connecting interesting data patterns to their real-world meaning. We'll explore how this approach helps lay readers and those with visual impairments to quickly grasp the context of new datasets, making data exploration more accessible and insightful.