Highlights This study identifies the restricted exploration problem encountered when training a signal control model which causes a model to obtain a partially-trained policy. The goal of this research is to obtain a transferable policy that enhances the model’s applicability on traffic states of unexperienced demand pattern using already-experienced information. The key idea consists of two parts; 1) to represent the traffic state as graph-structured data by embedding it into a graph, 2) to process the information-condensed graph by employing message-passing graph neural network (GNN). The experiment conducted on five unexperienced test demand scenarios which have different levels of traffic volumes and traveling patterns shows that the proposed GNN-based model obtains a transferable policy so that it adapts better to the unexperienced traffic states, while the conventional RL model fails.

    Abstract Reinforcement learning (RL) has emerged as an alternative approach for optimizing the traffic signal control system. However, there is a restricted exploration problem encountered when a signal control model is trained with a predefined demand scenario in the traffic simulation. With the restricted exploration, the model learns a policy based only on partial experiences in the search space, which yields a partially-trained policy. Partially-trained policy fails to adapt to some unexperienced (‘unexplored’, ‘never-before-seen’) dataset that have different distributions from the training dataset. Although this issue has critical effects on training a signal control model, it has not been considered in the literature. Therefore, this research aims to obtain a transferable policy to enhance the model’s applicability on unexperienced traffic states. The key idea is to represent the state as graph-structured data, and train it using a graph neural network (GNN). Since this approach enables to learn the relationship between the features resulting from the spatial structure of the intersection, it is able to transfer the already-learned knowledge of the relationship to the unexperienced data. In order to investigate the transferability, an experiment is conducted on five unexperienced test demand scenarios. For the evaluation, the performance of the proposed GNN model is compared with the conventional DQN model that is based on vector-valued state. At first, the models are trained with only a single dataset (training demand scenario). Then, they are tested with different unexperienced dataset (test demand scenarios) without additional trainings. The results show that the proposed GNN model obtains a transferable policy so that it adapts better to the unexperienced traffic states, while the conventional DQN model fails.


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    Transferable traffic signal control: Reinforcement learning with graph centric state representation


    Beteiligte:
    Yoon, Jinwon (Autor:in) / Ahn, Kyuree (Autor:in) / Park, Jinkyoo (Autor:in) / Yeo, Hwasoo (Autor:in)


    Erscheinungsdatum :

    2021-07-18




    Medientyp :

    Aufsatz (Zeitschrift)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    HumanLight: Incentivizing ridesharing via human-centric deep reinforcement learning in traffic signal control

    Vlachogiannis, Dimitris M. / Wei, Hua / Moura, Scott et al. | Elsevier | 2024


    Traffic Signal Control Based on Reinforcement Learning with Graph Convolutional Neural Nets

    Nishi, Tomoki / Otaki, Keisuke / Hayakawa, Keiichiro et al. | IEEE | 2018


    IG-RL: Inductive Graph Reinforcement Learning for Massive-Scale Traffic Signal Control

    Devailly, Francois-Xavier / Larocque, Denis / Charlin, Laurent | IEEE | 2022


    Self-adaptive traffic signal control method based on graph deep reinforcement learning

    ZHAO ZHONGNAN / WANG KUN / SONG XIN | Europäisches Patentamt | 2022

    Freier Zugriff

    Deep Reinforcement Learning-based Traffic Signal Control

    Ruan, Junyun / Tang, Jinzhuo / Gao, Ge et al. | IEEE | 2023