基于容器技术的天文应用软件自动部署方法
|
摘要
平方千米阵列即将开始建设,各子工作包也进入关键设计评估阶段。基于云与容器技术是平方千米阵列科学数据处理器未来可能采用的平台技术。针对超大规模海量数据处理面临的天文应用软件快速部署、运行与实测要求,充分考虑天文应用软件运行环境复杂、云计算环境下超大规模计算集群部署困难等问题,系统研究并给出了一种使用容器技术的天文应用软件通用自动部署方法。以目前较为常用的可见度函数校准软件SAGECaL为例,首先分析了SAGECaL的相关特性和分布式部署方面存在的困难,进而给出了基于容器技术的SAGECaL分布式集群的自动部署方法。实验结果表明,自动部署方法极大地提高了SAGECaL分布式集群的部署效率,满足项目组承担平方千米阵列科学数据处理器相关测试工作所需要的基础平台部署与切换等需求,同时也为其它天文软件在云端的快速部署与执行提供了有益的思路。
The Square Kilometer Array(SKA) is under construction. And each sub-work package will also enter the critical design evaluation phase. The cloud-based and container-based technologies are the platform technologies that SKA Scientific Data Processor(SDP) may adopt in the future. This paper is aimed at the rapid deployment, operation and measurement requirements of astronomical application software faced by SDP ultra-large-scale massive data processing. It fully considers the complex operation environment of astronomical application software and the difficulty in deploying ultra-large-scale computing clusters in cloud computing environment. For this reason, a general automatic deployment method for astronomical application using Docker technology is systematically studied and presented. The paper takes SAGECaL, a calibration software commonly used for visibility function as an example. Firstly, the characteristics of SAGECaL and the difficulties in distributed deployment are analyzed. The automatic deployment method of SAGECaL distributed cluster based on Docker container technology is given. The experimental results show that the automatic deployment method proposed in this paper greatly improves the deployment efficiency of SAGECaL distributed cluster, and meets the needs of the project team to undertake the deployment and switching of the basic platform required for SKA-SDP related testing. At the same time, this work also provides useful ideas for the rapid deployment and implementation of other astronomical software in the cloud.
The Square Kilometer Array(SKA) is under construction. And each sub-work package will also enter the critical design evaluation phase. The cloud-based and container-based technologies are the platform technologies that SKA Scientific Data Processor(SDP) may adopt in the future. This paper is aimed at the rapid deployment, operation and measurement requirements of astronomical application software faced by SDP ultra-large-scale massive data processing. It fully considers the complex operation environment of astronomical application software and the difficulty in deploying ultra-large-scale computing clusters in cloud computing environment. For this reason, a general automatic deployment method for astronomical application using Docker technology is systematically studied and presented. The paper takes SAGECaL, a calibration software commonly used for visibility function as an example. Firstly, the characteristics of SAGECaL and the difficulties in distributed deployment are analyzed. The automatic deployment method of SAGECaL distributed cluster based on Docker container technology is given. The experimental results show that the automatic deployment method proposed in this paper greatly improves the deployment efficiency of SAGECaL distributed cluster, and meets the needs of the project team to undertake the deployment and switching of the basic platform required for SKA-SDP related testing. At the same time, this work also provides useful ideas for the rapid deployment and implementation of other astronomical software in the cloud.
引文
[1]陈泰燃,王威,王锋,等.基于MPI的高性能UVFITS数据合成研究与应用[J].天文研究与技术,2016,13(2):184-189.
[2]石聪明,张晓丽,王锋,等.MUSER的负数据库接口设计与实现[J].天文研究与技术,2018,15(2):169-175.
[3]MORRIS D,VOUTSINAS S,HAMBLY N C,et al.Use of Docker for deployment and testing of astronomy software[J].Astronomy&Computing,2017,20:105-119.
[4]YOUNG M D,HAYASHI S,GOPU A.StarDock:shipping customized computing environments to the data[C]//Proceedings of the SPIE.2016.
[5]MERKEL D.Docker:lightweight Linux containers for consistent development and deployment[J].Linux Journal,2014,2014(239):Article No.2.
[6]ROSADO T,BERNARDINO J.An overview of openstack architecture[C]//Proceedings of the18th International Database Engineering&Applications Symposium.2014:366-367.
[7]WEI S L,WANG F,DENG H,et al.OpenCluster:a flexible distributed computing framework for astronomical data processing[J].Publications of the Astronomical Society of the Pacific,2016,129(972):024001(16pp).
[8]KAZEMI S,YATAWATTA S,ZAROUBI S,et al.Radio interferometric calibration using the SAGE algorithm[J].Monthly Notices of the Royal Astronomical Society,2011,414(2):1656-1666.
[9]KUMAZAKI K,YATAWATTA S,ZAROUBI S.Performance of SAGECal calibration[C]//Proceedings of the General Assembly and Scientific Symposium.2014.
[10]NAIK N.Building a virtual system of systems using docker swarm in multiple clouds[C]//IEEE International Symposium on Systems Engineering.2016.
[11]NGUYEN N,BEIN D.Distributed MPI cluster with Docker Swarm mode[C]//Computing and Communication Workshop and Conference.2017.
[2]石聪明,张晓丽,王锋,等.MUSER的负数据库接口设计与实现[J].天文研究与技术,2018,15(2):169-175.
[3]MORRIS D,VOUTSINAS S,HAMBLY N C,et al.Use of Docker for deployment and testing of astronomy software[J].Astronomy&Computing,2017,20:105-119.
[4]YOUNG M D,HAYASHI S,GOPU A.StarDock:shipping customized computing environments to the data[C]//Proceedings of the SPIE.2016.
[5]MERKEL D.Docker:lightweight Linux containers for consistent development and deployment[J].Linux Journal,2014,2014(239):Article No.2.
[6]ROSADO T,BERNARDINO J.An overview of openstack architecture[C]//Proceedings of the18th International Database Engineering&Applications Symposium.2014:366-367.
[7]WEI S L,WANG F,DENG H,et al.OpenCluster:a flexible distributed computing framework for astronomical data processing[J].Publications of the Astronomical Society of the Pacific,2016,129(972):024001(16pp).
[8]KAZEMI S,YATAWATTA S,ZAROUBI S,et al.Radio interferometric calibration using the SAGE algorithm[J].Monthly Notices of the Royal Astronomical Society,2011,414(2):1656-1666.
[9]KUMAZAKI K,YATAWATTA S,ZAROUBI S.Performance of SAGECal calibration[C]//Proceedings of the General Assembly and Scientific Symposium.2014.
[10]NAIK N.Building a virtual system of systems using docker swarm in multiple clouds[C]//IEEE International Symposium on Systems Engineering.2016.
[11]NGUYEN N,BEIN D.Distributed MPI cluster with Docker Swarm mode[C]//Computing and Communication Workshop and Conference.2017.
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |