Search Space Synthesis for Parametric Functions

We present a general framework for synthesizing search spaces of parametric functions, along with strategies for traversing these spaces to find optima. We formalize an algebraic theory for the categorical model of parametric functions in finite combinatory logic with predicates (FCLP). Based on a component-oriented synthesis framework for FCLP we automate composition from given components and search for parametric functions. Components are language-agnostic and may be instantiated as any implementation of parametric functions, e.g., as PyTorch modules. A proof-of-concept implementation demonstrates how to represent more specific concepts, such as neural architecture search and hyperparameter optimization, within the framework.

Modern AI systems often rely on experts to hand‑design the “blueprints” that specify how different components of the system are connected. This is time‑consuming and can prevent us from discovering unusual but powerful designs. In our work, we introduce a method for the computer to generate large collections of possible model blueprints from simple building blocks, ensuring every design is valid. Behind the scenes, we use a mathematical language for describing how pieces of a model can be plugged together, and then let standard search methods look for designs that work well on a given task. When we tested this approach, it not only worked with existing optimization tools but also uncovered a highly effective model with a structure rarely used in today’s practice. By automating this design process, our method reduces expert effort and opens the door to discovering novel AI architectures that people might never think to try on their own.