Introducing SymTorch: A PyTorch library that transforms deep learning models into human-readable formulas

Is symbolic regression the key to converting opaque deep learning models into interpretable closed-form mathematical equations? Or, suppose you have trained a deep learning model. it works. But do you know what we actually learned? A team of researchers from the University of Cambridge proposes ‘SymTorch’, a library designed to integrate Symbolic regression (SR) Introduce it into your deep learning workflow. This allows researchers to approximate neural network components with closed-form mathematical expressions, potentially making it easier to interpret features and speeding up inference.

Core Mechanism: Wrap, Distill, Switch Workflow

SymTorch simplifies the engineering required to extract symbolic equations from trained models by automating data movement and hook management.

• Wrap: The user is SymbolicModel any wrapper nn.Module or a callable function.
• distillation: The library registers forward hooks to record input and output activations during the forward pass. These are cached and transferred from GPU to CPU for symbolic regression via PySR.
• switch: Once distilled, the original neural weights can be replaced in the forward pass by the equations found using: switch_to_symbolic.

The library interfaces with PySRwe use a multi-population genetic algorithm to find an equation that balances accuracy and complexity. Pareto front. The “best” equation is selected by maximizing the fractional reduction in log-mean absolute error with increasing complexity.

Case study: Accelerating LLM inference

The main applications considered in this study are: Multilayer Perceptron (MLP) Use symbolic surrogates in layers in your Transformer model to increase throughput.

Implementation details

Due to the high level of LLM activation, the research team Principal component analysis (PCA) Compress input and output before running SR. for Quen 2.5-1.5B For the model, we selected 32 principal components for the input and 8 for the output across the three layers of interest.

Performance tradeoffs

As a result of the intervention, Token throughput increased by 8.3%^{. However, this advantage was accompanied by a significant increase in complexity, mainly caused by the dimensionality reduction of the PCA rather than the symbolic approximation itself..}

metric	Baseline (Qwen2.5-1.5B)	symbolic surrogate
Confused (Wikitext-2)	10.62	13.76
Throughput (tokens/sec)	4878.82	5281.42
Average latency (ms)	209.89	193.89

GNNs and PINNs

SymTorch has been validated for its ability to recover known physical laws from latent representations of scientific models^.

• Graph Neural Network (GNN): The research team used SymTorch to recover empirical force laws such as gravity (1/r) by training a GNN on particle mechanics.²) and spring force directly from the edge message.
• Physically informed neural network (PINN): This library successfully extracts analytical solutions for one-dimensional heat equations from trained PINNs. With the induced bias of PINN, we were able to achieve a mean squared error (MSE) of 7.40 x 10 .^-6.
• LLM Arithmetic Analysis: Symbolic distillation was used to examine how models like Llama-3.2-1B perform three-digit additions and multiplications. The extracted equations reveal that the model, while correct in many cases, relies on internal heuristics that contain systematic numerical errors.

Important points

• Automated symbolic distillation: SymTorch is a library that automates the process of replacing complex neural network components with interpretable closed-form mathematical equations by wrapping the components and collecting their input and output behavior.
• Removal of engineering barriers: This library addresses key engineering challenges that have previously hindered the adoption of symbolic regression, including GPU-CPU data transfer, input/output caching, and seamless switching between neural and symbolic forward passes.
• Accelerating LLM inference: A proof of concept demonstrates that replacing the MLP layer in a transformer model with a symbolic surrogate increases throughput by 8.3% at the expense of performance due to complexity.
• discovery of scientific laws: Using SymTorch, we successfully recovered physical laws from graph neural networks (GNNs) and analytical solutions of one-dimensional heat equations from physically informed neural networks (PINNs).
• Functional Interpretability of LLM: By extracting the end-to-end behavior of an LLM, researchers can examine the explicit mathematical heuristics used for tasks such as arithmetic, and reveal where the internal logic deviates from precise operation.

---

Please check paper, report and Project page. Please feel free to follow us too Twitter Don’t forget to join us 120,000+ ML subreddits and subscribe our newsletter. hang on! Are you on telegram? You can now also participate by telegram.