How to win the “mainly AI” synthetic data challenge

I was primarily an AI award and won both flat and sequential data challenges. The competition was an amazing learning experience. In this post I would like to provide insight into winning solutions.

competition

The goal of the competition was to generate synthetic datasets with the same statistical properties as the source dataset, without copying the data.

The competition was divided into two independent challenges.

1. Flat Data Challenge: Generates 100,000 records with 80 columns.
2. Sequential Data Challenge: Generates 20,000 sequences (groups) of records.

To measure the quality of synthetic data, competition is Overall accuracy metric. This score measures the similarity of the composite and source distributions of single columns (univariate), column pairs (bivariate), and column triples (trivariate) using L1 distance. Plus, like privacy metrics DCR (distance to the nearest record) and NNDR (Nearest Neighbor Distance Ratio) It was used to ensure that submissions do not merely overfit or copy training data.

Solution Design

Initially, my goal was to create an ensemble of several different cutting-edge models and combine the generated data. I have done a lot of experiments with different models, but the results didn't improve as much as I had hoped.

I swiveled my approach and focused Post-processing. First, instead of training a single generative model mostly from the AI SDK and generating the samples needed for submission, Oversampling Create a large pool of candidate samples. From this pool, the final output was selected in a way that more closely matches the statistical characteristics of the source dataset.

This approach has resulted in a significant jump in leaderboard scores. For the flat data challenge, the raw synthetic data of the model got about 0.96, but after post-processing, the score jumped 0.992. A modified version of this approach was used for the sequential data challenge, resulting in similar improvements.

My final pipeline for the flat challenge consisted of three main steps.

1. Repeated proportional fitting (IPF) Choose a subset of oversized high quality.
2. Greedy trimming To reduce the subset to the target size by removing the worst samples.
3. Repeated refinement Replace samples to improve the sample and polish the final dataset.

Step 1: Repeated proportional fitting (IPF)

The first step in my post-processing pipeline was to get a strong initial subset from an oversampled pool (2.5 million generated columns). For this I used Repeated proportional fitting (IPF).

IPF is a classic statistical algorithm used to adjust sample distribution to a known set of limits. In this case, we wanted the bivariate (two column) distribution of the synthetic data to match the distribution of the original data. We also tested single and trivariate distributions, and found that focusing on bivariate relationships gives us computationally faster and best performance.

Here's how this worked:

1. I've identified it 5,000 most correlated column pairs With training data using mutual information. These are the most important relationships to preserve.
2. IPF then calculated fractional weights for each of the 2.5 million composite columns. Weights were repeatedly adjusted so that the weighted sum of the bivariate distribution of the synthetic pool coincided with the target distribution from the training data.
3. Finally, we used a predicted round approach to convert these partial weights into integer numbers for the number of times each row was selected. This gave us an oversized subset of 125,000 rows (1.25 times the required size) with already very strong bivariate accuracy.

The IPF step provided a high quality starting point for the next phase.

Step 2: Trimming

Generating an oversized subset of 125,000 rows from IPF was an intentional choice that allowed this additional trimming step to remove samples that do not suit properly.

We used a greedy approach that iteratively calculates the “error contribution” for each row of the current subset. The rows that contribute most to the statistical distance from the target distribution are identified and deleted. This process repeats until only 100,000 rows remain, causing the worst 25,000 rows to be discarded.

Step 3: Refine (swapping)

The final step was an iterative refinement process to exchange rows from the subset with better rows from a much larger, unused data pool (the remaining 2.4 million rows).

For each iteration, the algorithm:

1. Identifies the worst row (the row that contributes most to the L1 error) within the current 100K subset.
2. Search for the best replacement candidates from the external pool that reduces L1 errors when replaced.
3. If the swap gives a better overall score, perform the swap.

The accuracy of the synthetic samples is already very high, so the additional gain from this process is considerably less.

Suitable for sequential challenges

A similar approach was required for consecutive challenges, but two changes were required. First, the sample consists of several lines connected by group ID. Second, the conflict metric adds measurements Consistency. This means that not only must the statistical distribution be matched, but the sequence of events must also be similar to the source dataset.

My post-processing pipeline was fitted to process groups and optimize for consistency.

1. Coherence-based preselection: Specialized refinement steps were performed before optimizing for statistical accuracy. This algorithm repeatedly exchanged the entire group (sequence) to specifically match the coherence metrics of the original data, such as the distribution of “unique categories per sequence” and “sequences per category”; This allowed us to continue post-processing with a healthy sequential structure.
2. Sophistication (swapping): The 20,000 groups selected for coherence went through the same statistical improvement process as the flat data. The algorithm swapped the entire group for better ones in the pool to minimize L1 errors in the distribution of Uni-, Bi, and Trivariate. The secret ingredient was to include “sequence length” as a function, so group length is also considered in swapping.

This two-stage approach has made the final dataset more robust to both statistical accuracy and continuous coherence. Interestingly, the IPF-based approach, which worked very well for flat data, was not very effective in sequential challenges. So, we removed it and focused on coherence and swapping in computing time, which gave us better results.

Fast: Major Optimization

The post-processing strategy itself was computationally expensive, and implementing it within competitive time limits was a challenge in itself. To be successful, I relied on some important optimizations.

First, Reduced data types Wherever you can handle large sample data pools without running out of memory. Changing the numeric type of a large matrix from 64-bit to 32- or 16-bit will significantly reduce the memory footprint.

Secondly, if changing the data type isn't enough, I used it Sparse Matrix From Scipy. This technique allowed us to store the statistical contributions of each sample in a very memory-efficient way.

Finally, the core improvement loop involved a lot of specialized calculations, some of which were very slow numpy. To overcome this, I used it numba. The bottleneck in my code @numba.njit The decorator, Numba, was automatically converted to highly optimized machine code that runs at speeds comparable to C.

This is an example required to speed up the sum of rows in a sparse matrix. This was the major bottleneck of the original Numpy version.

import numpy as np
import numba

# This can make the logic run hundreds of times faster.
@numba.njit
def _rows_sum_csr_int32(data, indices, indptr, rows, K):
    """
    Sum CSR rows into a dense 1-D vector without creating
    intermediate scipy / numpy objects.
    """
    out = np.zeros(K, dtype=np.int32)
    for r in rows:
        start = indptr[r]
        end = indptr[r + 1]
        for p in range(start, end):
            out[indices[p]] += data[p]
    return out

However, Numba is not a silver bullet. It's useful for numerical and loopy code, but for most calculations it's easier to stick to faster and vectorized numpy operations. I recommend you try the Numpy approach only if you haven't reached the speed you need.

Final Thoughts

Although ML models are getting stronger and stronger, I think that in most of the problems data scientists are trying to solve, the secret ingredient is often not included in the model. Of course, powerful models are an integral part of the solution, but pre-processing and post-processing are equally important. Because of these challenges, post-processing pipelines specifically targeted for evaluation metrics were led to victory solutions without additional ML.

I learned a lot from this task. I would like to thank AI and ju umpires mostly for the great work they did in holding this amazing competition.

My code and solution for both issues is open source and can be found here: