Reproducible Performance Evaluation of a Hybrid Hadoop–Spark Big Data Architecture Using Urban Mobility and Global Event Datasets

Serik Aliaskarov

2026

Full text Open access: Hybrid Metadata (CSL-JSON)
DOI 10.59543/jidmis.v3.1068
Data confidence 70 medium
How this score is calculated

It is built from checkable properties of the record itself: whether the article has a DOI, how many independent sources confirmed its data, whether the required fields are filled in, and how recent the latest confirmation is. Disagreement between sources and a missing identifier lower it. A value of 80 or above means the record was published without further checking; 60 to 79 means it is published at reduced confidence, with a manual check advisable. A record verified by an editor by hand is never pushed below 90 by a recalculation. This scores our data about the article, not the article itself.

Abstract

This study presents a reproducible framework for evaluating the performance of Big Data workflows inspired by Hadoop–Spark processing logic using two open datasets with distinct workload characteristics. The first dataset, the NYC TLC Yellow Taxi Trip Record Data for January 2023, represents an urban mobility scenario and contains temporal and spatial trip attributes. The second dataset, the GDELT Event Daily Export for 1 January 2023, represents global event data with temporal, geographic, categorical, and quantitative characteristics. Three reproducible processing configurations are compared: a disk-based batch baseline, a Spark-only workflow, and a Prepared-storage + Spark workflow, which serves as a storage-oriented approximation of a hybrid Hadoop–Spark processing pipeline. The experimental design includes batch aggregation, spatial–temporal analysis, stream-like processing, scalability testing, and task restart-based fault recovery. Performance is evaluated using execution time, throughput, latency, scalability behavior, CPU utilization, RAM delta, I/O-related behavior, and recovery time metrics. To ensure reproducibility, a unified preprocessing procedure was applied across all configurations. The TLC dataset was reduced from 3,066,766 to 2,883,546 valid records, while the GDELT dataset was reduced from 43,575 to 42,053 valid records. The study does not claim universal superiority of the storage-oriented workflow. Instead, its primary contribution lies in providing a transparent and reproducible methodology for comparing storage-oriented and Spark-based data processing workflows using heterogeneous open datasets under identical preprocessing rules and comparable workload execution conditions.