Real-world pharmacovigilance study of cardiovascular adverse events of tisagenlecleucel using a machine learning approach

Information source

This was an observational, retrospective, pharmacovigilance study using AEs reported in WHO's VigiBase. VigiBase contains over 30 million safety reports on AEs collected from over 150 countries. These reports originate from a variety of sources, including healthcare professionals, patients, and pharmaceutical companies. Each report contains information on the reporter's qualifications, patient information, drug, and suspected AE. Drugs mentioned in the safety report are recorded as “suspected” or “interacting” if they may cause the AE, as determined by the reporter, or “concomitant” if they are not causative. AEs were coded according to the Dictionary for Medical Regulatory Activities (MedDRA) terminology. Study data were obtained through the “VigiBase Extract Case Level” service provided by the WHO-Uppsala Monitoring Centre. This service provides raw data as fixed-length text files that can be configured as a relational database with unique report identification numbers that join across provided datasets. The extracted data used in this study included individual case safety reports up to February 2024. All analyses were conducted using medDRA version 26.1.

Data Processing

From all reports in VigiBase from January 1976 to February 2024, AE reports that listed tisagenlecleucel as a “suspected” or “interacting” drug were included in the study analysis, and reports recorded as other “concomitant use” or reports with missing suspected drug information were excluded. We then constructed label and feature data for training and fitting the machine learning method.

Label data contains information on known and unknown AEs of the target drug. Known AEs are used to provide direction for machine learning training and testing, while unknown AEs are used to mine new safety signals. In this study, we constructed the label data using AEs recorded for tisagenlecleucel in the study data. We classified them into three groups according to whether they were described in the EMA product label or in the scientific literature: (1) drug-related AEs (e.g., positive controls), if the relationship between the drug and the AE has been described in a randomized controlled trial (RCT), observational cohort study, or described in the product label; (2) non-drug-related AEs (e.g., negative controls), if there is no documentation describing the relationship; and (3) unknown AEs (unknown), if the relationship has only been described in case reports, case series, or other relevant literature but not in the product label information or has not been described in an RCT or observational cohort study. All AEs were identified using MedDRA Preferred Term (PT) codes.

Feature data represent variables recorded in AE reports that are used to improve the predictive performance of machine learning. Specifically, the distribution of variables such as reporter characteristics, demographics, and frequency of reported cases for each AE are used to fit the model. Among the variables available from the AE reports of tisagenlecleucel in this study, we selected the following variables based on the European Medicines Agency (EMA) Good Vigilance Practice (GVP) guidelines, module IX-signal management:^{twenty four}Number of cases, response after drug discontinuation, response after drug rechallenge, number of reports by healthcare professionals, severity of adverse events, and outcome of adverse events.

end point

The primary endpoint was serious cardiovascular AEs classified as “unknown.” We used the EMA Significant Medical Events Terminology List to classify 11 serious cardiac AEs (pericardial effusion, cardiotoxicity, bradycardia, cardiopulmonary arrest, cardiorenal syndrome, pulseless electrical activity, arrhythmia, cardiomyopathy, cardiopulmonary failure, acute myocardial infarction, and aortic valve insufficiency). From safety reports that described any of these AEs, we collected information on reporter type, designation of “serious AE,” suspected drug indication, age group (pediatric, adult, geriatric), and time to onset of the AE. The secondary endpoint was cardiovascular AEs reported concomitantly with CRS. For this endpoint, we analyzed safety reports that described both tisagenlecleucel CRS and cardiovascular AEs.

Modeling Strategy

In the process of model building, safety reports designated as positive or negative controls were included in the construction of an input dataset, 75% of the dataset was used for model training and 25% was used for model testing, and then the established input dataset was applied to the GBM algorithm-based model.^{twenty five}GBM is a boosting structure algorithm that learns by reducing the error between predicted data and actual data. Based on the error of randomly generated decision trees, new decision trees are created in the direction that reduces the gradient of the model, and the decision trees are combined to create one optimal model. Specifically, we used extreme gradient boosting (XGBoost) as a method for fitting the GBM algorithm. XGBoost has been shown to reduce overfitting of algorithms by allowing models to learn boosting algorithms in parallel.²⁶Using the training dataset, we generated and validated an XGBoost-based model that calculates the probability of whether an AE (cardiovascular AE) is related to the drug of interest (tisagenlecleucel). Finally, we implemented a model to calculate the predicted probability of a cardiovascular AE identified by a PT code included in the EMA's Significant Medical Events List. We determined that a given AE was related to tisagenlecleucel if the predicted probability was greater than 0.5, the default decision threshold for the binary classifier. This was also a generous prediction threshold that allowed more safety signals to be detected and further evaluated for possible association with the suspected drug.

Statistical analysis

To deal with the imbalance in the label data, the synthetic minority oversampling technique (SMOTE) was used. SMOTE is one of the oversampling techniques for data processing based on the k-NN algorithm.²⁷To evaluate the performance of the developed signal prediction model, the area under the receiver operating characteristic curve (AUROC), accuracy, and F1 score were measured. \((=\frac{{sensitivity + specificity}}{{2}})\) It represents the average accuracy, and its value is usually between 0.5 and 1, with the closer it is to 1, the better the model's predictive accuracy.\((=\frac{{2}}{{\frac{{1}}{{Precision}}+\frac{{1}}{{Recall}}}})\) is the harmonic mean of precision and recall, and the closer the value is to 1, the better the performance of the binary classification model. Precision is the percentage of data correctly predicted by the classification model in the entire dataset. These performance metrics were selected and measured based on the guidelines of GVP – Module IX – Signal Management.^{twenty four}.

Furthermore, we compared the statistical performance (accuracy, sensitivity, specificity, PPV, negative predictive value, AUROC) of our prediction machine with traditional signal detection methods such as information component (IC), proportional reporting ratio (PRR), reporting odds ratio (ROR), and empirical Bayesian geometric mean (EBGM).

All analyses were conducted using Python version 3.9.13 and SAS version 9.4. All methods used in this study were performed in accordance with the relevant guidelines and regulations.

Ethics approval

The Institutional Review Board of Sungkyunkwan University approved this study (IRB number SKKU 2023-02-024). Because this study uses de-identified administrative data, the board waived the requirement to obtain informed consent.