Complex science instruments require complicated management of ancillary data such as calibration, file list, or timing information. We have found that the widely used, light-weight, single file database SQLite can make these tasks much easier. Since the database is a file, there is no server to maintain or connect with, just a single file that can be copied anywhere it is needed (including to individual end users’ workstations) and COTS tools can be used to browse it or export tables or selections to other formats. It is also fully supported by pretty much any programming or scripting environment. We present three use cases of varying complexity along with lessons learned:–Keeping a master list of file boundaries so data isn’t "dropped" during file reprocessing–Keeping track of millions of individual telemetry files (CFDP files) for collecting input for a science data pipeline–Managing calibration information for hundreds of thousands of defined channels over multiple instrument configuration regimes


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    SQLite File Databases for Science Data Pipelines 3 Use Cases


    Contributors:


    Publication date :

    2024-03-02


    Size :

    3685383 byte




    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    sqlite-images.sql

    Barzen, Johanna / Bühler, Fabian / Leymann, Frank | DataCite | 2021

    Free access

    sqlite-schema.sql

    Barzen, Johanna / Bühler, Fabian / Leymann, Frank | DataCite | 2021

    Free access

    sqlite-data.sql

    Barzen, Johanna / Bühler, Fabian / Leymann, Frank | DataCite | 2021

    Free access

    generate-sqlite-db.sh

    Barzen, Johanna / Bühler, Fabian / Leymann, Frank | DataCite | 2021

    Free access

    generate-sqlite-db-small.sh

    Barzen, Johanna / Bühler, Fabian / Leymann, Frank | DataCite | 2021

    Free access