NEW APPROACH TO ADDRESSING CLASS IMBALANCE IN MEDICAL DATASETS CONSIDERING SPECIFICS

Scientific Journal of Astana IT University, 2025

Полный текст Открытый доступ: Gold Метаданные (CSL-JSON)
DOI 10.37943/21vwqh9068
Достоверность данных 70 средняя
Как считается эта оценка

Складывается из проверяемых признаков самой записи: есть ли у статьи DOI, сколько независимых источников подтвердили её данные, заполнены ли обязательные поля и насколько свежо последнее подтверждение. Снижают оценку расхождения между источниками и отсутствие идентификатора. Значение от 80 означает, что запись опубликована без дополнительной проверки; от 60 до 79 — что она опубликована с пониженной достоверностью и ручная проверка желательна. Запись, выверенную редактором вручную, пересчёт не опускает ниже 90. Это оценка наших данных о статье, а не оценка самой статьи.

Аннотация

Currently, the popularization of the integration of machine learning into the field of medicine for data processing and analysis is being traced, but at the same time difficulties such as class imbalance and noisy datasets arise. Due to the prevalence of the problem, there are already existing solutions, but in all of them there is an abstraction from the field of medicine, namely, gender, racial and other differences are not taken into account. It is this side of the problem that is solved in our resampling algorithm. A feature of our algorithm is the use of splitting the dataset by an important feature through the p-value of Spearman correlation, which helps to consider subgroups of observations without losing their unique characteristics and removing noise data using LOF and Z-score separately for minority and majority classes, respectively. Synthetic data is generated in a flexible way, adapting to the data set using algorithm parameters. Work is provided with both quantitative and nominative features. The algorithm was tested on datasets for heart attack, chronic kidney disease, and liver disease, and the Random Forest ensemble method was used to train the model. After applying this class balancing method, improvements were recorded on average in Accuracy by 36%, in AUC by 15-25%, in Precision by 39-42%, and in Recall by 21-37% compared with SMOTE, ADASYN algorithms and the data set before balancing. Applying the algorithm on medical data can improve the accuracy of the algorithm and reduce the loss of reliability compared to other resampling methods.