Deep-learning model for embryo selection using time-lapse imaging of matched high-quality embryos

Study design and population

This research was a retrospective observational study carried out of IVF cycles peformed in the reproductive medicine center at Angers hospital (France) between January 2020 and February 2023. We included women from 18 to 43 years old (French standard age requirements for IVF) who obtained embryos cultured in an EmbryoScope+^® time-lapse incubator and for whom the stimulation cycle resulted in several embryo transfers (fresh and/or frozen).

Namely clinical pregnancy (positive known implantation data, KIDp) or implantation failure (negative known implantation data, KIDn).

Ovarian stimulation and laboratory procedures

Pituitary suppression was achieved using either an antagonist (Ganirelix; 0.25 mg daily), or a gonadotrophin-releasing hormone agonist (Triptorelin; 0.1 mg subcutaneously daily). Ovarian stimulation was performed using recombinant or urine-derived follicle stimulation hormone (FSH). Gonadotropin dose selection was based on patient characteristics, ovarian reserve biomarkers, and response to any previous ovarian stimulation cycle. Triggering criteria included a minimum of three follicles ≥ 17 mm and serum estradiol (E2) levels. Ultrasound-guided transvaginal oocyte retrieval was scheduled 36 h after ovulation triggering, which was performed by injection of 6500 IU choriogonadotropin alfa and/or 0.2 mg triptorelin.

Collected oocyte cumulus complexes were washed with G-MOPS™ PLUS (Vitrolife, Sweden), and placed for 1–3 h in FertiCult^® IVF medium (FertiPro, Belgium) at 5% O2, 6% CO2, and 37 °C. Sperm preparation was performed using a standard gradient separation at 300 g for 20 min, followed by washing with FertiCult^® IVF medium (FertiPro, Belgium) at 600 g for 10 min. Oocytes were inseminated by IVF or ICSI according to semen quality parameters and patient’s history (failed or poor fertilization on a previous cycle). For conventional IVF, the cumulus-oocyte complexes were incubated with 100,000 motile spermatozoa in 1 mL of FertiCult^® IVF medium and denuded the following day (19–20 h after insemination). For ICSI cycles, metaphase II (MII) oocytes were injected using a RI Integra 3™ Micromanipulator (Cooper Surgical Company, Trumbull, Conn., USA) after denudation with hyaluronidase (FertiPro, Belgium). Oocytes were then cultured in an EmbryoScope+^® time-lapse incubator (Vitrolife, Sweden), in pre-equilibrated EmbryoSlides™ with a global culture medium (G-TL™, Vitrolife, Sweden) under a controlled atmosphere (5% O2, 6% CO2). Images were acquired automatically every 10 min in 11 focal planes with illumination from a single red LED (635 nm) until use. Embryo development was assessed with the EmbryoViewer software (Vitrolife, Sweden). Fertilization was checked ~ 19 h post insemination or injection. Abnormally fertilized oocytes (1 or 3 or more pronuclei) were excluded from further consideration. Number of cells, fragmentation level, symmetry among blastomeres and compaction degree were evaluated on days 2 and 3 of development, using the BLEFCO classification¹⁴. According to this classification, embryos that were ≥ 4.1.2. or 4.2.1. at day 2 and ≥ 8.1.2. or 8.2.1. at day 3 were deemed good grade, other types were deemed poor grade. Blastocysts were assessed according to the Gardner and Schoolcraft classification¹⁵. Based on these criteria, we defined good-quality blastocysts as follows: expansion grade ≥ 3, inner cell mass (ICM) grade ≥ B, and trophectoderm grade ≥ B on day 5. Any combination of ICM or trophectoderm quality grading of “C”, or embryos with developmental stage graded as early blastocyst or below were classified as poor quality. Morphokinetic parameters were manually annotated according to published guidelines¹⁶ and included: time to syngamy (tPNf), time to two (t2), three (t3), four (t4), five (t5), and eight (t8) cells, as well as time to blastocyst (tB). Embryos were then scored from 1 to 5 by the KIDScore D3 v1.2 algorithm⁴ and from 1 to 9.9 by the KIDScore D5 v3.1 algorithm¹⁷.

Embryos were selected for transfer or freezing according to the result of the KIDScore™ Day 3 or Day 5, and the conventional grading criteria. Other parameters such as multinucleation, direct or reverse cleavage, and blastocyst collapse were also monitored to further classify embryos. Embryos with abnormal (direct or reverse) cleavage were discarded. According to age, cycle number and quality of embryos, one or two embryos were transferred under transabdominal ultrasound guidance at cleaved or blastocyst stage. Vitrification was performed at cleaved or blastocyst stage using closed CBS High Security Vitrification (HSV) straws (Cryo Bio System, France) in combination with ethylene glycol, DMSO and sucrose as the cryoprotectants (Vit Kit-Freeze and Vit Kit-Thaw, Irvine Scientific, USA), as described previously¹⁸.

Luteal phase support of fresh embryo transfers was provided with oral dydrogesterone (30 mg/day) and personalized in case of abnormal endometrial receptivity analysis or previous implantation failure. In case of frozen embryo transfers, endometrial preparation was conducted via artificial cycles with oral or transdermal estrogen, and the addition of intravaginal progesterone when endometrial thickness was ≥ 8mm¹⁹. Biological pregnancy was confirmed 14 days after oocyte retrieval with a serum β-HCG level above 100 IU/L. Clinical pregnancy was defined as the presence of at least one fetal heartbeat on ultrasound 5 weeks after embryo transfer. Double embryo transfers (DET) that resulted in a single gestational sac were excluded. During the inclusion period, there was no significant difference in pregnancy outcomes between fresh and frozen embryo transfers. We matched embryos obtained from a same cycle in pairs according to their clinical outcome.

Data collection and image preprocessing

Raw videos were exported using the EmbryoViewer software (Vitrolife, Sweden) and then processed in Python. Data preprocessing was applied to convert the videos into usable images, given that raw videos could not be exploited directly. First, the initial resolution (400 × 400) was too large to be directly processed as sequences of images with the available computational resources. Thus, we cropped all images by restricting them to the view around the embryo. We also discarded all frames of poor quality, containing artefactual visual defects, such as bubbles or low luminosity and hindering the visibility of the embryo, and all frames without embryo (due to an asynchronous time of transfer and freezing for the same embryo cohort).

To carry out this step, we trained a YOLO v6 object detection deep learning algorithm to detect and locate the embryos in raw images²⁰. We first curated a random subset of 2000 images of embryos at various stages. Using color binarization and thresholding, we semi-automatically located the embryos on these images, before manually reviewing these annotations. Finally, after image augmentation (random rotation, flipping, cropping, affine and elastic transformations, noise, blur), YOLO was trained to detect and locate embryos. After training, YOLO was used to preprocess all videos into embryo images: images in which an embryo was detected and located were kept and cropped/padded with black to 362 × 362, centered on the embryo. We manually reviewed all images to ensure the quality of our dataset. Images in which an embryo could be visually identified but was undetected by YOLO due to visual artefacts were discarded.

Pre-training: extracting embryo morphology features

We used self-supervised contrastive learning, namely the SimCLR architecture, to pre-train a deep learning model (encoder) to learn to map (encode) embryo images into feature vectors summarizing static morphological features²¹. This step was performed on videos of embryos that fulfilled the inclusion criteria summarized in Table 1. In contrast to the original SimCLR architecture, we used a more conservative image augmentation pipeline, limited to random rotation and flipping followed by centered cropping, minor contrast, and saturation jittering. However, instead of using two augmentations of the same original image, each positive pair was composed of two different images of the same embryo taken 10 min to < 1 h apart. This was performed to ensure that the model could learn to map both the stage of the embryo but also its intrinsic morphological characteristics into the final feature vector. Two architectures were trained and compared, namely VGG16 ²² and ResNet18.

Table 1 Inclusion and exclusion criteria of the study.

During this step, the encoders were trained on all images of embryos selected for transfer or freezing, regardless of the transfer outcome. To validate the model during and after pre-training, we used a publicly available external dataset^23,24. The dataset comprised 302,134 frames of videos accompanied by the annotations of 16 cellular events, from the extrusion of the second polar body to the hatched blastocyst stage. Annotation of these timings was performed and manually checked by experienced embryologists using the definition of key events proposed by Ciray et al.¹⁶: tPB2 (extrusion of the second polar body), tPNa (pronuclei appearance), tPNf (pronuclei fading), t2, t3, t4, t5, t6, t7, t8, t9+, tM (end of compaction), tSB (start of blastulation), tB (full blastocyst), tEB (expanded blastocyst), and finally tHB (hatched blastocyst). Since the number of images per event in the dataset was imbalanced, we randomly sampled 99 images per stage of development to ensure a balanced external validation dataset, thus including a total of 1,584 images.

After each epoch and during the final validation step, we used the deep learning pre-trained model to encode the images of embryos into feature vectors, and then trained an XGBoost model to predict the stage of the embryo based on these feature vectors. We used mean one-versus-one Area Under the Receiver Operating Characteristic Curve (ROC-AUC) and F1-score after 5-fold cross-validation as the metric to test how much information from the embryos images were kept and encoded by the pre-trained model into the feature vector.

Fine-tuning: extracting morphokinetic features

After pre-training, we used supervised one-shot learning (Siamese network) to fine-tune the encoders to predict the pregnancy outcome of the embryo transfer, namely clinical pregnancy (positive known implantation data, KIDp) or implantation failure (negative known implantation data, KIDn). For fine-tuning, the training dataset was limited to cycles with at least three transfers, including at least one positive and one negative. This was done to ensure that, in the same cohort, each embryo could be matched both to an embryo with a similar outcome and to an embryo with a different one, in order to prevent overfitting. Random stratified partitioning per patient was used to separate videos of embryos into a training set (80%) and a validation set (20%). No embryo could be found both in the training and in the validation set. All videos unsuitable for fine-tuning (i.e. less than three usable embryos or lack of a positive or a negative outcome) were assigned to the validation set.

Each training step was performed as follows (Fig. 1): (1) for a same cycle, two embryo videos were randomly selected, and a sequence of N random images of each of these videos was randomly selected, ensuring that each pair of images was from the same cycle and at the same time point; (2) each image was encoded using the previously pre-trained model into a “static” feature vector; (3) each of the two sequences was input to a long short-term memory (LSTM) model, which output one “dynamic” feature vector for each embryo of the same patient, summarizing their morphokinetic features; (4) the morphokinetic feature vectors were compared and the model was penalized into predicting similar morphokinetic feature vectors(low Euclidean distance) for videos of embryos with the same clinical outcome (KIDp/KIDp or KIDn/KIDn), and dissimilar (high Euclidean distance) for embryos with a different clinical outcome (KIDn/KIDp or KIDp/KIDn).

Several hyperparameters were compared during fine-tuning, namely: input size of images (64 × 64 or 128 × 128), training or freezing the weights of the pre-trained static encoder model, number of frames per 48 h (6, 12, 24, or 48), learning rate, batch size and number of epochs.

After fine-tuning, the models’ predictions were tested using two validation tasks. The first validation task consisted of predicting the clinical pregnancy outcome by knowing the outcome of a previous transfer performed with an embryo from the same cohort. For this task, we included cycles with at least two paired embryos (with identical or different outcomes). After encoding both paired embryo videos by the fine-tuned model, the Euclidean distance between the two encodings was used as a metric to determine whether the two embryos of the pair should be associated with the same outcome. We performed a 1000-iteration bootstrap analysis; for each bootstrap, we used the samples resampled with replacement to determine the ROC-AUC and optimal Euclidean distance threshold using the Youden index. Then, we used the samples left out of the bootstrap to determine F1-score, sensitivity and specificity at the determined threshold. Finally, results from all iterations were aggregated to compute 95% confidence intervals.

The second validation task was to predict the clinical pregnancy outcome of a transfer without knowing the outcome of a previous transfer. For this task, we included one embryo for each cycle. Based on the encodings output by the fine-tuned model, we trained a simpler machine learning model, namely XGBoost with 5-fold cross-validation repeated over 200 iterations, to predict the clinical outcome based on these morphokinetic feature vectors. For each model, we computed F1-score, ROC-AUC, sensitivity and specificity, and merged the means of all 200 iterations to compute 95% confidence intervals. In parallel, we also ran a permutation test with 200 random permutations for each 5-fold cross-validated model to assess the significance of the model’s performance compared to random.

Statistical analyses and model performance

The performance of the deep learning model was assessed using AUC of ROC curve generated by plotting the true positive rate (sensitivity) against the false positive rate (1-specificity) across all possible thresholding values. AUC score ranges from 0 to 1, where 0.5 indicates random classifier, and 1 indicates perfect predictive performance. AUC values were interpreted according to the classification as described elsewhere²⁵and 95% confidence intervals were compared to an AUC of 0.5 to assess performance against a random model. We also used the weighted F1-score, sensitivity and specificity, using a threshold of 0.5 (50%). 95% confidence intervals were computed over 1000-iteration bootstrapping, or using 5-fold cross-validation repeated over 200 iterations, by selecting the 2.5th and 97.5th percentiles over repeats.