Wed 15 Jun 2022 14:30 - 14:50 at Kon-Tiki - Synthesis I Chair(s): Ruben Martins
Thu 16 Jun 2022 02:30 - 02:50 at Kon-Tiki - Synthesis I

While data visualization plays a crucial role in gaining insights from data, generating answers over complex visualizations from natural language questions is far from an easy task. Mainstream approaches reduce data visualization queries to a semantic parsing problem, which either relies on expensive-to-annotate supervised training data that pairs natural language questions with logical forms, or weakly supervised models that incorporate a larger corpus but fail on long-tailed queries without explanations. This paper aims to answer data visualization queries by automatically synthesizing the corresponding program from natural language. At the core of our technique is an abstract synthesis engine that is bootstrapped by an off-the-shelf weakly supervised model and an optimal synthesis algorithm guided by \emph{triangle alignment constraints}, which represent \emph{consistency} among natural language, visualization, and the synthesized program.

Starting with a few tentative answers obtained from an off-the-shelf statistical model, our approach first involves an \emph{abstract synthesizer} that generates a set of sketches that are consistent with the answers. Then we design an instance of optimal synthesis to complete one of the candidate sketches by satisfying common type constraints and maximizing the consistency among three parties, i.e., natural language, the visualization, and the candidate program.

We implement the proposed idea in a system called POE that can answer visualization queries from natural language. Our method is fully automated and does not require users to know the underlying schema of the visualizations. We evaluate POE on 629 visualization queries and our experiment shows that POE outperforms state-of-the-arts by improving the accuracy from 44% to 59%.

Wed 15 Jun

Displayed time zone: Pacific Time (US & Canada) change

13:30 - 14:50
Synthesis I PLDI at Kon-Tiki +12h
Chair(s): Ruben Martins Carnegie Mellon University
13:30
20m
Talk
WebRobot: Web Robotic Process Automation using Interactive Programming-by-Demonstration
PLDI
Rui Dong University of Michigan, Zhicheng Huang University of Michigan, Ian Iong Lam University of Michigan, Yan Chen University of Michigan, Xinyu Wang University of Michigan
DOI Pre-print
13:50
20m
Talk
Synthesizing Analytical SQL Queries from Computation Demonstration
PLDI
Xiangyu Zhou University of Washington, Rastislav Bodik University of Washington, Alvin Cheung University of California, Berkeley, Chenglong Wang University of Washington, USA
DOI
14:10
20m
Talk
Type-Directed Program Synthesis for RESTful APIs
PLDI
Zheng Guo University of California, San Diego, David Cao University of California, San Diego, Davin Tjong University of California, San Diego, Jean Yang Akita Software, Cole Schlesinger Akita Software, Nadia Polikarpova University of California at San Diego
DOI
14:30
20m
Talk
Visualization Question Answering Using Introspective Program Synthesis
PLDI
Yanju Chen University of California, Santa Barbara, Xifeng Yan University of California, Santa Barbara, Yu Feng University of California, Santa Barbara
DOI

Thu 16 Jun

Displayed time zone: Pacific Time (US & Canada) change

01:30 - 02:50
Synthesis I PLDI at Kon-Tiki
01:30
20m
Talk
WebRobot: Web Robotic Process Automation using Interactive Programming-by-Demonstration
PLDI
Rui Dong University of Michigan, Zhicheng Huang University of Michigan, Ian Iong Lam University of Michigan, Yan Chen University of Michigan, Xinyu Wang University of Michigan
DOI Pre-print
01:50
20m
Talk
Synthesizing Analytical SQL Queries from Computation Demonstration
PLDI
Xiangyu Zhou University of Washington, Rastislav Bodik University of Washington, Alvin Cheung University of California, Berkeley, Chenglong Wang University of Washington, USA
DOI
02:10
20m
Talk
Type-Directed Program Synthesis for RESTful APIs
PLDI
Zheng Guo University of California, San Diego, David Cao University of California, San Diego, Davin Tjong University of California, San Diego, Jean Yang Akita Software, Cole Schlesinger Akita Software, Nadia Polikarpova University of California at San Diego
DOI
02:30
20m
Talk
Visualization Question Answering Using Introspective Program Synthesis
PLDI
Yanju Chen University of California, Santa Barbara, Xifeng Yan University of California, Santa Barbara, Yu Feng University of California, Santa Barbara
DOI