NEW APPROACH TO ADDRESSING CLASS IMBALANCE IN MEDICAL DATASETS CONSIDERING SPECIFICS

Scientific Journal of Astana IT University, 2025

Full text Open access: Gold Metadata (CSL-JSON)
DOI 10.37943/21vwqh9068
Data confidence 70 medium
How this score is calculated

It is built from checkable properties of the record itself: whether the article has a DOI, how many independent sources confirmed its data, whether the required fields are filled in, and how recent the latest confirmation is. Disagreement between sources and a missing identifier lower it. A value of 80 or above means the record was published without further checking; 60 to 79 means it is published at reduced confidence, with a manual check advisable. A record verified by an editor by hand is never pushed below 90 by a recalculation. This scores our data about the article, not the article itself.

Abstract

Currently, the popularization of the integration of machine learning into the field of medicine for data processing and analysis is being traced, but at the same time difficulties such as class imbalance and noisy datasets arise. Due to the prevalence of the problem, there are already existing solutions, but in all of them there is an abstraction from the field of medicine, namely, gender, racial and other differences are not taken into account. It is this side of the problem that is solved in our resampling algorithm. A feature of our algorithm is the use of splitting the dataset by an important feature through the p-value of Spearman correlation, which helps to consider subgroups of observations without losing their unique characteristics and removing noise data using LOF and Z-score separately for minority and majority classes, respectively. Synthetic data is generated in a flexible way, adapting to the data set using algorithm parameters. Work is provided with both quantitative and nominative features. The algorithm was tested on datasets for heart attack, chronic kidney disease, and liver disease, and the Random Forest ensemble method was used to train the model. After applying this class balancing method, improvements were recorded on average in Accuracy by 36%, in AUC by 15-25%, in Precision by 39-42%, and in Recall by 21-37% compared with SMOTE, ADASYN algorithms and the data set before balancing. Applying the algorithm on medical data can improve the accuracy of the algorithm and reduce the loss of reliability compared to other resampling methods.