학술논문
The IceProd framework: Distributed data processing for the IceCube neutrino observatory
Document Type
article
Author
Aartsen, MG; Abbasi, R; Ackermann, M; Adams, J; Aguilar, JA; Ahlers, M; Altmann, D; Arguelles, C; Auffenberg, J; Bai, X; Baker, M; Barwick, SW; Baum, V; Bay, R; Beatty, JJ; Tjus, J Becker; Becker, K-H; BenZvi, S; Berghaus, P; Berley, D; Bernardini, E; Bernhard, A; Besson, DZ; Binder, G; Bindig, D; Bissok, M; Blaufuss, E; Blumenthal, J; Boersma, DJ; Bohm, C; Bose, D; Böser, S; Botner, O; Brayeur, L; Bretz, H-P; Brown, AM; Bruijn, R; Casey, J; Casier, M; Chirkin, D; Christov, A; Christy, B; Clark, K; Classen, L; Clevermann, F; Coenders, S; Cohen, S; Cowen, DF; Silva, AH Cruz; Danninger, M; Daughhetee, J; Davis, JC; Day, M; De Clercq, C; De Ridder, S; Desiati, P; de Vries, KD; de With, M; DeYoung, T; Díaz-Vélez, JC; Dunkman, M; Eagan, R; Eberhardt, B; Eichmann, B; Eisch, J; Euler, S; Evenson, PA; Fadiran, O; Fazely, AR; Fedynitch, A; Feintzeig, J; Feusels, T; Filimonov, K; Finley, C; Fischer-Wasels, T; Flis, S; Franckowiak, A; Frantzen, K; Fuchs, T; Gaisser, TK; Gallagher, J; Gerhardt, L; Gladstone, L; Glüsenkamp, T; Goldschmidt, A; Golup, G; Gonzalez, JG; Goodman, JA; Góra, D; Grandmont, DT; Grant, D; Gretskov, P; Groh, JC; Groß, A; Ha, C; Ismail, A Haj; Hallen, P; Hallgren, A; Halzen, F; Hanson, K
Source
Subject
Language
Abstract
IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework.