jagomart
digital resources
picture1_Aws Ppt Templates 82615 | Google Dfio


 202x       Filetype PPTX       File size 0.13 MB       Source: indico.cern.ch


File: Aws Ppt Templates 82615 | Google Dfio
hep data format activity there is literally a flurry for some time focus on columnar formats storage or conversion dianna hep parquet awkward array femtocode oamap etc iris hep idds ...

icon picture PPTX Filetype Power Point PPTX | Posted on 10 Sep 2022 | 3 years ago
Partial capture of text on file.
       HEP Data Format Activity
        There is literally a flurry for some time
          Focus on columnar formats (storage or conversion)
            Dianna-HEP 
                 Parquet, Awkward Array, Femtocode, OAMap, etc
            iris-HEP
                 iDDS, Service-X, DOMA R&D
            ROOT Project
                 RDataFrame
            Others
                 COFFEA
       HEP-Google TIM             March 24-26, 2020            2
       HEP Public Cloud Activity
        Many projects leverage public clouds
         HEPCloud (AWS)
         HTCondor (AWS)
         ICCEP GCPM Project (GCP)
         Atlas Data Ocean Project (GCP)
         Many other independent projects
      HEP-Google TIM             March 24-26, 2020            3
       The Synthesis
        Why not combine all these ideas
         Analysis using a public cloud
            E.G. Google Cloud Platform (GCP)
         With a cloud storage friendly data format
            E.G. Parquet (https://parquet.apache.org/)
         Suitable for efficient memory representation
            E.G. PANDAS (https://pandas.pydata.org/)
         That Python oriented physicists find useful
        We should learn quite a lot
      HEP-Google TIM             March 24-26, 2020            4
        How We Got Here
        August 2019
             Informal discussion started (Andrew Hanushevsky & Ross Thomson)
        September 2019
             Project conceptualized
        October 2019
             Project formalized
             On-boarded 20% Google engineer (Guilhem Tesseyre) 
        November 2019 onwards
             Various approaches investigated and tried
        February 2020
             On-boarded physics analyst (Shawfeng Dong SLAC ACF)
       HEP-Google TIM                March 24-26, 2020               5
        Project Goals I
        Demonstrate efficient use of GCP for 
        physics analysis
          We are only addressing analysis here
             Using Python as the language
          The demonstration has two aspects
             Workflow for needed data flow setup
                   This usually requires data conversion
             Workflow for running an analysis job
       HEP-Google TIM                March 24-26, 2020               6
The words contained in this file might help you see if this file matches what you are looking for:

...Hep data format activity there is literally a flurry for some time focus on columnar formats storage or conversion dianna parquet awkward array femtocode oamap etc iris idds service x doma r d root project rdataframe others coffea google tim march public cloud many projects leverage clouds hepcloud aws htcondor iccep gcpm gcp atlas ocean other independent the synthesis why not combine all these ideas analysis using e g platform with friendly https apache org suitable efficient memory representation pandas pydata that python oriented physicists find useful we should learn quite lot how got here august informal discussion started andrew hanushevsky ross thomson september conceptualized october formalized boarded engineer guilhem tesseyre november onwards various approaches investigated and tried february physics analyst shawfeng dong slac acf goals i demonstrate use of are only addressing as language demonstration has two aspects workflow needed flow setup this usually requires running a...

no reviews yet
Please Login to review.