Abstract
Intelligent systems operate in constantly changing environments with increasing workloads and complex structures, making performance and reliability analysis challenging. This paper details the design and evaluation of an intelligent system with an observability-based self-monitoring mechanism. To ensure complete and comprehensive continuous workload monitoring and analysis, the monitoring system utilized Prometheus for metric collection, Grafana for metric and log visualization, and Loki for workload aggregation. Controlled experiments utilized synthetic workloads to evaluate request throughput, response latency, and high percentile performance metrics. The results demonstrate that an observability-based monitoring system provides a deeper understanding of the performance characteristics of complex intelligent systems and helps identify performance bottlenecks, thereby providing users with a clear understanding of performance under various workloads. The proposed methodology, as a viable approach to achieving self-monitoring of intelligent systems, offers a simple and practical way to replicate the self-monitoring capabilities of intelligent systems.