Improving Unsupervised Visual Program Inference with Code Rewriting Families

Programs offer compactness and structure that makes them an attractive representation for visual data. We explore how code rewriting can be used to improve systems for inferring programs from visual data. We first propose Sparse Intermittent Rewrite Injection (SIRI), a framework for unsupervised bootstrapped learning. SIRI sparsely applies code rewrite operations over a dataset of training programs, injecting the improved programs back into the training set. We design a family of rewriters for visual programming domains: Parameter Optimization (PO), Code Pruning (CP), and Code Grafting (CG). For three shape programming languages in 2D and 3D, we show that using SIRI with our family of rewriters improves performance: better reconstructions and faster convergence rates, compared with bootstrapped learning methods that do not use rewriters or use them naively. Finally, we demonstrate that our family of rewriters can be effectively used at test time to improve the output of SIRI predictions. For 2D and 3D CSG, we outperform or match the reconstruction performance of recent domain-specific neural architectures, while producing more parsimonious programs that use significantly fewer primitives.

Visual data is often highly structured: manufactured shapes are produced by assembling parts; vector graphics images are built from layers of primitives; detailed textures can be created via intricate compositions of noise functions. Visual programs, i.e. programs that produce visual outputs when executed, are a natural approach to capturing this complexity in a structure-aware fashion. Access to well-written visual programs supports downstream applications across visual computing domains, including editing, generative modeling, and structural analysis.

However, crafting these visual programs for a given visual datum can be laborious, even for skilled visual programmers. Therefore, its desirable create systems which perform Visual Program Inference (VPI), i.e. the task of automatically inferring programs that represent the input visual datum. Here, we delve into unsupervised neural VPI, i.e., we are given examples of shapes from a target distribution (such as 3D chairs), without their associated programs and our objective is to train a neural network that aids inferring programs for new shapes from the same target distribution.

In this work, we demonstrate that code rewriting techniques can be effectively integrated into neural VPI technqiues, for training the neural network as well as while performing neurally guided inference. By integrating a family of code rewriters into boostrapped learning methods, we achieve state-of-the-art performance on a variety of visual programming domains while also inferring highly parsimonious programs. We propose: 1) Sparse Intermittent Rewrite Injection (SIRI) a framework for unsupervised visual program inference that leverages a family of code rewriters, and 2) A family of code rewriters applicable to multiple DSLs that benefit VPI learning methods and can be used in a test-time rewriting scheme.

We propose Sparse Intermittent Rewrite Injection (SIRI), a framework for unsupervised bootstrapped learning. SIRI sparsely applies code rewrite operations over a dataset of training programs, and injects the improved programs back into the training set in a controlled manner to ensuring diversity of training programs. Please refer to the paper for more details.

We identify three rewrite operators that generalize across multiple shape-program domains, namely Parameter Optimization (PO), Code Pruning (CP), and Code Grafting (CG). Please refer to the paper for more details.