Efficient Sim-to-Real Transfer in Reinforcement Learning Through Domain Randomization and Domain Adaptation

Aidar Shakerimov, Tohid Alizadeh, Hüseyin Atakan Varol

IEEE Access, 2023

Full text Open access: Gold Metadata (CSL-JSON)
DOI 10.1109/access.2023.3339568
Data confidence 85 high
How this score is calculated

It is built from checkable properties of the record itself: whether the article has a DOI, how many independent sources confirmed its data, whether the required fields are filled in, and how recent the latest confirmation is. Disagreement between sources and a missing identifier lower it. A value of 80 or above means the record was published without further checking; 60 to 79 means it is published at reduced confidence, with a manual check advisable. A record verified by an editor by hand is never pushed below 90 by a recalculation. This scores our data about the article, not the article itself.

Abstract

Reinforcement learning has gained significant interest in modern industries for its advancements in tackling challenging control tasks compared to rule-based programs. However, the robustness aspect of this technique is still under development, limiting its widespread adoption. This problem has become more pronounced as users switch to training simulations to reduce costs, resulting in a reality gap that negatively affects real-world performance. One popular method employed to mitigate this problem is randomizing uncertain parameters of the environment during training. Nevertheless, this approach requires expert knowledge to determine the appropriate range of randomization. On the other hand, there is a technique that introduces fine-tuning of agents by adapting their policies to new environments. However, it is challenging to adapt policies when the new environment requires shifting them off their initial distribution. These obstacles limit the practical utilization and popularization of both techniques. Our study proposes a hybrid approach that handles these issues by fine-tuning agents trained with domain randomization through additional real-world training. To assess the efficacy of our approach, we conducted experiments involving a rotary inverted pendulum, augmented with an extra weight not represented in the simulation. Additionally, we employed simulated environments including a cart pole, a simple pendulum, a quadruped, and an ant robot scenario. These environments were given in two distinct versions with mismatching parameters to imitate a gap between training and testing conditions. The results demonstrate that adding as few as twenty to fifty additional real-world training episodes can significantly enhance the performance of agents trained with domain randomization. Moreover, including fifty to two hundred additional episodes can elevate it to a level comparable to those fully trained in the real world. Our study concludes that achieving efficient simulation-to-reality transfer is feasible with domain randomization and relatively small amounts of real-world training.