Abstract
The present study addresses the three-body problem (point masses) undergoing non-isotropic mass variation while incorporating the influence of reactive forces within different coordinate systems. The primary contribution of this research is the formulation, first obtained, of the governing differential equations of motion for three-body problem with variable mass in the barycentric coordinate system. In addition, center-of-mass invariants has been established for this formulation. These invariants constitute the only analytical relationships that remain valid for arbitrary laws of mass variation and provide a direct connection between the coordinates and velocities of the interacting bodies. The proposed mathematical model extends the theoretical framework of celestial mechanics. Consequently, the derived equations and invariants provide a theoretical basis for qualitative investigations, analytical studies, and high-precision numerical simulations of variable-mass three-body systems evolution. Furthermore, the developed approach offers significant potential for investigation of dynamical evolution of many body systems. That is, it makes it possible to derive differential equations describing systems of many bodies with a non-isotropic mass variation and accounting for reactive forces in the barycentric coordinate system. In particular, it can be used to study the long-term dynamic evolution, orbital stability of exoplanet systems subject to arbitrary mass-variation laws and reactive-force effects. Overall, the obtained results provide a comprehensive mathematical foundation for advancing the study of variable-mass dynamical systems and broaden the range of opportunities of modern celestial mechanics.