密集异构网络中基于强化学习的流量卸载算法
|
摘要
近年来互联网用户规模和网络流量呈现爆炸式的增长,不断逼近蜂窝移动通信网络的容量极限.流量卸载技术可充分利用现有网络,将蜂窝网络的部分流量卸载到空闲网络中,进行跨网协作实现对蜂窝网络容量的极大提升,可有效解决有限的无线带宽资源与海量高速业务需求的矛盾.将强化学习的思想引入流量卸载算法中,提出了一种异构网络中基于强化学习的流量卸载算法.该算法把流量卸载问题映射为一个强化学习问题.基于前一状态完成的动作,以WiFi网络吞吐量作为回报函数,准确地预测需卸载的流量,并计算当前网络的最大卸载量,寻找最佳的WiFi网络接入点(access point,AP),并推导出最优的流量卸载判决规则,达到异构网络整体吞吐量最大化.仿真结果表明:基于Q学习的流量卸载算法可有效地实现自适应流量卸载控制规则,有效地避免过度卸载引起的碰撞冲突和系统性能急剧恶化,达到跨网协作的负载均衡点,在保证WiFi用户服务质量的条件下,最大限度地提高LTE系统吞吐量,保证密集异构网络的整体性能.
With the explosive growth of numbers of Internet users and network traffic,the capacity of cellular mobile communication is already limited.In order to solve the contradiction between the increasing demand for high capacity and the limited resource,traffic offloading technology makes full use of the existing network,which offloads part of traffic from the cellular network into the other network and carries on the cooperation between networks,to improve the capacity of the cellular network greatly.Traffic offloading becomes one of the hot topics in the future research of wireless communication technology.In this paper,based on reinforcement learning,we propose a novel reinforcement learning algorithm for traffic offloading in dense heterogeneous network.Based on the previous experience and performance gain of the user offloading,this algorithm considers the system throughput of each state,and finds the optimal WiFi network access point(AP)by calculating the reward value. We also derive the optimal policy of traffic offloading decision to maximize the throughput of the system.Simulation results show that the reinforcement learning for traffic offloading can effectively avoid the collision caused by over offloading and rapid deterioration of system performance.Our scheme can effectively implement the adaptive traffic offloading control policy and achieve the cooperation between LTE and WiFi network guaranteeing the quality of service for users.The overall throughput of the dense heterogeneous network also reaches the maximum.
With the explosive growth of numbers of Internet users and network traffic,the capacity of cellular mobile communication is already limited.In order to solve the contradiction between the increasing demand for high capacity and the limited resource,traffic offloading technology makes full use of the existing network,which offloads part of traffic from the cellular network into the other network and carries on the cooperation between networks,to improve the capacity of the cellular network greatly.Traffic offloading becomes one of the hot topics in the future research of wireless communication technology.In this paper,based on reinforcement learning,we propose a novel reinforcement learning algorithm for traffic offloading in dense heterogeneous network.Based on the previous experience and performance gain of the user offloading,this algorithm considers the system throughput of each state,and finds the optimal WiFi network access point(AP)by calculating the reward value. We also derive the optimal policy of traffic offloading decision to maximize the throughput of the system.Simulation results show that the reinforcement learning for traffic offloading can effectively avoid the collision caused by over offloading and rapid deterioration of system performance.Our scheme can effectively implement the adaptive traffic offloading control policy and achieve the cooperation between LTE and WiFi network guaranteeing the quality of service for users.The overall throughput of the dense heterogeneous network also reaches the maximum.
引文
[1]Ding Zhiguo,Lei Xianfu,Karagiannidis G K,et al.A survey on non-orthogonal multiple access for 5Gnetworks:Research challenges and future trends[J].IEEE Journal on Selected Areas in Communications,2017,35(10):2181-2195
[2]Shafi M,Molisch A F,Smith P J,et al.5G:A tutorial overview of standards,trials,challenges,deployment,and practice[J].IEEE Journal on Selected Areas in Communications,2017,35(6):1201-1221
[3]Rebecchi F,De Amorim M D,Conan V,et al.Data offloading techniques in cellular networks:A survey[J].IEEE Communications Surveys&Tutorials,2015,17(2):580-603
[4]Wang Xijun,Quek T Q S,Sheng Min,et al.Throughput and fairness analysis of Wi-Fi and LTE-U in unlicensed band[J].IEEE Journal on Selected Areas in Communications,2017,35(1):63-78
[5]Zhang Ning,Zhang Shan,Wu Shaohua,et al.Beyond coexistence:Traffic steering in LTE networks with unlicensed bands[J].IEEE Wireless Communications,2016,23(6):40-46
[6]Babaei A,Andreoli-Fang J,Pang Y,et al.On the impact of LTE-U on Wi-Fi performance[J].International Journal of Wireless Information Networks,2015,22(4):336-344
[7]Chen Qimei,Yu Guanding,Shan Hangguan,et al.Cellular meets WiFi:Traffic offloading or resource sharing/[J].IEEE Trans on Wireless Communications,2016,15(5):3354-3367
[8]He Yejun,Chen Man,Ge Baohong,et al.On WiFi offloading in heterogeneous networks:Various incentives and trade-off strategies[J].IEEE Communications Surveys&Tutorials,2016,18(4):2345-2385
[9]Cheng Nan,Lu Ning,Zhang Ning,et al.Opportunistic WiFi offloading in vehicular environment:A game-theory approach[J].IEEE Trans on Intelligent Transportation Systems,2016,17(7):1944-1955
[10]Ho D,Park G S,Song H.Game-theoretic scalable offloading for video streaming services over LTE and WiFi networks[J].IEEE Trans on Mobile Computing,2018,17(5):1090-1104
[11]Suh D,Ko H,Pack S.Efficiency analysis of WiFi offloading techniques[J].IEEE Trans on Vehicular Technology,2016,65(5):3813-3817
[12]Ko H,Lee J,Pack S.Performance optimization of delayed WiFi offloading in heterogeneous networks[J].IEEE Trans on Vehicular Technology,2017,66(10):9436-9447
[13]Jung B H,Song N O,Sung D K.A network-assisted usercentric WiFi-offloading model for maximizing per-user throughput in a heterogeneous network[J].IEEE Trans on Vehicular Technology,2014,63(4):1940-1945
[14]Jiang Chunxiao,Zhang Haijun,Ren Yong,et al.Machine learning paradigms for next-generation wireless networks[J].IEEE Wireless Communications,2017,24(2):98-105
[15]Lee K,Lam M,Pedarsani R,et al.Speeding up distributed machine learning using codes[J].IEEE Trans on Information Theory,2018,64(3):1514-1529
[16]Hong Mingyi,Razaviyayn M,Luo Zhiquan,et al.A unified algorithmic framework for block-structured optimization involving big data:With applications in machine learning and signal processing[J].IEEE Signal Processing Magazine,2016,33(1):57-77
[17]Kato N,Fadlullah Z M,Mao Bomin,et al.The deep learning vision for heterogeneous network traffic control:Proposal,challenges,and future perspective[J].IEEE Wireless Communications,2017,24(3):146-153
[18]Klaine P V,Imran M A,Onireti O,et al.A survey of machine learning techniques applied to self-organizing cellular networks[J].IEEE Communications Surveys&Tutorials,2017,19(4):2392-2431
[19]Kaelbling L P,Littman M L,Moore A W.Reinforcement learning:A survey[J].Journal of Artificial Intelligence Research,1996,4:237-285
[20]Zhu Yuke,Mottaghi R,Kolve E,et al.Target-driven visual navigation in indoor scenes using deep reinforcement learning[C]//Proc of 2017 IEEE Int Conf on Robotics and Automation(ICRA).Piscataway,NJ:IEEE,2017:3357-3364
[21]Hwangbo J,Sa I,Siegwart R,et al.Control of a quadrotor with reinforcement learning[J].IEEE Robotics and Automation Letters,2017,2(4):2096-2103
[22]Liu Yanjun,Tang Li,Tong Shaocheng,et al.Reinforcement learning design-based adaptive tracking control with less learning parameters for nonlinear discrete-time MIMO systems[J].IEEE Trans on Neural Networks and Learning Systems,2015,26(1):165-176
[23]Bianchi G.Performance analysis of the IEEE 802.11distributed coordination function[J].IEEE Journal on Selected Areas in Communications,2000,18(3):535-547
[24]NihtilT,Tykhomyrov V,Alanen O,et al.System performance of LTE and IEEE 802.11 coexisting on a shared frequency band[C]//Proc of 2013IEEE Wireless Communications and Networking Conf(WCNC).Piscataway,NJ:IEEE,2013:1038-1043
[25]Watkins C J C H,Dayan P.Q-learning[J].Machine Learning,1992,8(3/4):279-292
[26]Jiang Chunxiao,Zhang Hai,Ren Yong,et al.Machine learning paradigms for next-generation wireless networks[J].IEEE Wireless Communications,2017,24(2):98-105
[27]Peng Haixia,Li Dazhou,Abboud K,et al.Performance analysis of IEEE 802.11 p DCF for multiplatooning communications with autonomous vehicles[J].IEEE Trans on Vehicular Technology,2017,66(3):2485-2498
[2]Shafi M,Molisch A F,Smith P J,et al.5G:A tutorial overview of standards,trials,challenges,deployment,and practice[J].IEEE Journal on Selected Areas in Communications,2017,35(6):1201-1221
[3]Rebecchi F,De Amorim M D,Conan V,et al.Data offloading techniques in cellular networks:A survey[J].IEEE Communications Surveys&Tutorials,2015,17(2):580-603
[4]Wang Xijun,Quek T Q S,Sheng Min,et al.Throughput and fairness analysis of Wi-Fi and LTE-U in unlicensed band[J].IEEE Journal on Selected Areas in Communications,2017,35(1):63-78
[5]Zhang Ning,Zhang Shan,Wu Shaohua,et al.Beyond coexistence:Traffic steering in LTE networks with unlicensed bands[J].IEEE Wireless Communications,2016,23(6):40-46
[6]Babaei A,Andreoli-Fang J,Pang Y,et al.On the impact of LTE-U on Wi-Fi performance[J].International Journal of Wireless Information Networks,2015,22(4):336-344
[7]Chen Qimei,Yu Guanding,Shan Hangguan,et al.Cellular meets WiFi:Traffic offloading or resource sharing/[J].IEEE Trans on Wireless Communications,2016,15(5):3354-3367
[8]He Yejun,Chen Man,Ge Baohong,et al.On WiFi offloading in heterogeneous networks:Various incentives and trade-off strategies[J].IEEE Communications Surveys&Tutorials,2016,18(4):2345-2385
[9]Cheng Nan,Lu Ning,Zhang Ning,et al.Opportunistic WiFi offloading in vehicular environment:A game-theory approach[J].IEEE Trans on Intelligent Transportation Systems,2016,17(7):1944-1955
[10]Ho D,Park G S,Song H.Game-theoretic scalable offloading for video streaming services over LTE and WiFi networks[J].IEEE Trans on Mobile Computing,2018,17(5):1090-1104
[11]Suh D,Ko H,Pack S.Efficiency analysis of WiFi offloading techniques[J].IEEE Trans on Vehicular Technology,2016,65(5):3813-3817
[12]Ko H,Lee J,Pack S.Performance optimization of delayed WiFi offloading in heterogeneous networks[J].IEEE Trans on Vehicular Technology,2017,66(10):9436-9447
[13]Jung B H,Song N O,Sung D K.A network-assisted usercentric WiFi-offloading model for maximizing per-user throughput in a heterogeneous network[J].IEEE Trans on Vehicular Technology,2014,63(4):1940-1945
[14]Jiang Chunxiao,Zhang Haijun,Ren Yong,et al.Machine learning paradigms for next-generation wireless networks[J].IEEE Wireless Communications,2017,24(2):98-105
[15]Lee K,Lam M,Pedarsani R,et al.Speeding up distributed machine learning using codes[J].IEEE Trans on Information Theory,2018,64(3):1514-1529
[16]Hong Mingyi,Razaviyayn M,Luo Zhiquan,et al.A unified algorithmic framework for block-structured optimization involving big data:With applications in machine learning and signal processing[J].IEEE Signal Processing Magazine,2016,33(1):57-77
[17]Kato N,Fadlullah Z M,Mao Bomin,et al.The deep learning vision for heterogeneous network traffic control:Proposal,challenges,and future perspective[J].IEEE Wireless Communications,2017,24(3):146-153
[18]Klaine P V,Imran M A,Onireti O,et al.A survey of machine learning techniques applied to self-organizing cellular networks[J].IEEE Communications Surveys&Tutorials,2017,19(4):2392-2431
[19]Kaelbling L P,Littman M L,Moore A W.Reinforcement learning:A survey[J].Journal of Artificial Intelligence Research,1996,4:237-285
[20]Zhu Yuke,Mottaghi R,Kolve E,et al.Target-driven visual navigation in indoor scenes using deep reinforcement learning[C]//Proc of 2017 IEEE Int Conf on Robotics and Automation(ICRA).Piscataway,NJ:IEEE,2017:3357-3364
[21]Hwangbo J,Sa I,Siegwart R,et al.Control of a quadrotor with reinforcement learning[J].IEEE Robotics and Automation Letters,2017,2(4):2096-2103
[22]Liu Yanjun,Tang Li,Tong Shaocheng,et al.Reinforcement learning design-based adaptive tracking control with less learning parameters for nonlinear discrete-time MIMO systems[J].IEEE Trans on Neural Networks and Learning Systems,2015,26(1):165-176
[23]Bianchi G.Performance analysis of the IEEE 802.11distributed coordination function[J].IEEE Journal on Selected Areas in Communications,2000,18(3):535-547
[24]NihtilT,Tykhomyrov V,Alanen O,et al.System performance of LTE and IEEE 802.11 coexisting on a shared frequency band[C]//Proc of 2013IEEE Wireless Communications and Networking Conf(WCNC).Piscataway,NJ:IEEE,2013:1038-1043
[25]Watkins C J C H,Dayan P.Q-learning[J].Machine Learning,1992,8(3/4):279-292
[26]Jiang Chunxiao,Zhang Hai,Ren Yong,et al.Machine learning paradigms for next-generation wireless networks[J].IEEE Wireless Communications,2017,24(2):98-105
[27]Peng Haixia,Li Dazhou,Abboud K,et al.Performance analysis of IEEE 802.11 p DCF for multiplatooning communications with autonomous vehicles[J].IEEE Trans on Vehicular Technology,2017,66(3):2485-2498