Complex science instruments require complicated management of ancillary data such as calibration, file list, or timing information. We have found that the widely used, light-weight, single file database SQLite can make these tasks much easier. Since the database is a file, there is no server to maintain or connect with, just a single file that can be copied anywhere it is needed (including to individual end users’ workstations) and COTS tools can be used to browse it or export tables or selections to other formats. It is also fully supported by pretty much any programming or scripting environment. We present three use cases of varying complexity along with lessons learned:–Keeping a master list of file boundaries so data isn’t "dropped" during file reprocessing–Keeping track of millions of individual telemetry files (CFDP files) for collecting input for a science data pipeline–Managing calibration information for hundreds of thousands of defined channels over multiple instrument configuration regimes


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    SQLite File Databases for Science Data Pipelines 3 Use Cases


    Beteiligte:


    Erscheinungsdatum :

    02.03.2024


    Format / Umfang :

    3685383 byte




    Medientyp :

    Aufsatz (Konferenz)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    sqlite-images.sql

    Barzen, Johanna / Bühler, Fabian / Leymann, Frank | DataCite | 2021


    sqlite-schema.sql

    Barzen, Johanna / Bühler, Fabian / Leymann, Frank | DataCite | 2021


    sqlite-data.sql

    Barzen, Johanna / Bühler, Fabian / Leymann, Frank | DataCite | 2021


    generate-sqlite-db.sh

    Barzen, Johanna / Bühler, Fabian / Leymann, Frank | DataCite | 2021


    generate-sqlite-db-small.sh

    Barzen, Johanna / Bühler, Fabian / Leymann, Frank | DataCite | 2021