Highlights A Multi-layer Graph Mask Q-Learning algorithm for traffic signals is proposed. This study constructs a new upper and lower graph structure to extract geometric and spatial features of intersections. This article introduces an action masking mechanism, enabling the model to be applicable to intersections of arbitrary structures when utilizing any phase as the action. Model tests were conducted on synthetic road networks with different traffic volumes and two real urban road networks, and the test results showed that the algorithm has strong generalization ability and scalability.

    Abstract Due to its capability in handling complex urban intersection environments, deep reinforcement learning (DRL) has been widely applied in Adaptive Traffic Signal Control (ATSC). However, most existing algorithms are designed for specific road networks or traffic conditions, making it difficult to transfer them to new environments. Moreover, current graph-based algorithms do not fully capture the geometric and spatial features of intersections, leading to incomplete embedding of the agent's environment depiction. Additionally, the actions adopted by these algorithms are inherently based on fixed-cycle phases, limiting the flexibility of traffic signal control. To address the aforementioned issues, this paper proposes a Multi-layer Graph Mask Q-Learning (MGMQ) algorithm for multi-intersection ATSC to optimize traffic and reduce delay. Unlike previous graph-based algorithms, this paper introduces a method for computing multi-layer graphs, dividing the traffic environment into upper-level traffic network-layer graphs and lower-level intersection-layer graphs, and employs the graph attention algorithm and an improved GraphSAGE algorithm for computation. This method not only enables the generation of embedded state for intersections that include geometric and spatial features, but also allows the algorithm to adapt to different traffic conditions and road networks. Additionally, we introduce an action masking mechanism, allowing this algorithm can be adapted to different action spaces. As a result, the algorithm uses arbitrary signal phases as actions to achieve flexible traffic flow control, and can be directly applied to intersections with arbitrary geometry. The final test results demonstrate that a model trained solely on synthetic road networks can be directly transferred to other synthetic network configurations or real-world urban road networks, outperforming current state-of-the-art algorithms.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    A large-scale traffic signal control algorithm based on multi-layer graph deep reinforcement learning


    Contributors:
    Wang, Tao (author) / Zhu, Zhipeng (author) / Zhang, Jing (author) / Tian, Junfang (author) / Zhang, Wenyi (author)


    Publication date :

    2024-03-19




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English




    Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control

    Chu, Tianshu / Wang, Jie / Codeca, Lara et al. | IEEE | 2020


    Self-adaptive traffic signal control method based on graph deep reinforcement learning

    ZHAO ZHONGNAN / WANG KUN / SONG XIN | European Patent Office | 2022

    Free access

    Multi-agent road traffic signal control method based on deep reinforcement learning algorithm

    LIU LIJUAN / SI HUA | European Patent Office | 2023

    Free access

    Deep Reinforcement Learning-based Traffic Signal Control

    Ruan, Junyun / Tang, Jinzhuo / Gao, Ge et al. | IEEE | 2023