diff --git a/.docs/concepts/pid.md b/.docs/concepts/pid.md index c5d769192cc561add9f9bba795e8ff78f9edd196..e58ed03ac4ac9be57963d0b009fa9ad88642781d 100644 --- a/.docs/concepts/pid.md +++ b/.docs/concepts/pid.md @@ -2,6 +2,56 @@ author: Martin Weise --- -## tbd +## Data Versioning -tbd +Data is getting bigger and so are expectations of data provisioning in regards to data availability (i.e. immediately +after quality check and not in snapshot intervals), cost-effectiveness (i.e. no duplication of data), transparent, +precise citation and many more. + +[System-versioned](https://mariadb.com/kb/en/system-versioned-tables/) tables in MariaDB are improved data structures +that keep track of historical data. For each entry in a system-versioned table, a time period is maintained that denotes +the validity time span of this tuple from its start to end. Tuples in system-versioned tables are not *actually* +modified, they are marked as (in-)valid in time periods. + +<figure markdown> + +| ID | Sensor | Temp | Start | End | +|----|--------|------|-------|-----| +| 1 | A | 23.1 | t1 | | +| 2 | B | 25.8 | t2 | | + +</figure> + +Assuming that Sensor A was calibrated wrong and an updated measurement is passed to the system-versioned table, the +table contents show that the old row with Temp 23.1 is not deleted, but marked as valid in time span (t1, t3). The +updated row with Temp 22.1 is marked as valid from time span t3 onwards. + +<figure markdown> + +| ID | Sensor | Temp | Start | End | +|----|--------|------|-------|-----| +| 1 | A | 23.1 | t1 | t3 | +| 2 | B | 25.8 | t2 | | +| 1 | A | 22.1 | t3 | | + +</figure> + +System-versioned tables are part of the SQL:2011 standard and have been adopted by many database management system +vendors: MariaDB (10.5 and higher), Google BigQuery, IBM DB2 (12 and higher), SQL Server (2016 and higher), Azure SQL, +PostgreSQL with [temporal tables extension](https://github.com/nearform/temporal_tables), etc. + +## Persistent Identifier + +Data in DBRepo always has attached metadata (stored in the [Metadata Database](../../api/metadata-db)). This metadata +is provided as machine-understandable context in various open-source formats that is available, even when the original +data is not available anymore due to e.g. a retracted dataset (hence the name **persistent**). A persistent identifier +globally, uniquely identifies a data record such as: + +* Database, +* Table whose data is continously changed in the background (nature of databases), +* View who show a denomination, joined schema or aggregated result of tables, +* Subset which precisely identifies a data record to reproduce the same dataset. + +Combining [data versioning](#data-versioning) and queries, subsets can be precisely identified by storing the query +that creates them and the time when the query was executed. We store both in a table inside the database we call the +query store. \ No newline at end of file diff --git a/.docs/concepts/search.md b/.docs/concepts/search.md index 8731100f91246b2f56091af373962d30d76796e6..ee738e0bd9e99fb89c4ddf867cfb8600a13f432a 100644 --- a/.docs/concepts/search.md +++ b/.docs/concepts/search.md @@ -2,22 +2,16 @@ author: Martin Weise --- -## Index - -tbd - ## Document -TBD - -## Query +Each database in DBRepo has their own document where the document id (the identifier of a database in OpenSearch) is +equivalent to the database id (the identifier of a database in the [Metadata Database](../../api/metadata-db)). ## Unit Independent Search Since the repository automatically collects statistical properties (min, max, mean, median, std.dev) in both the -[Metadata Database](../system-databases-metadata) and the [Search Database](../system-databases-search), a special -search can be performed when at least two columns have the same semantic concept (e.g. temperature) annotated and -the units of measurements can be transformed. +[Metadata Database](../../api/metadata-db) and the Search Database, a special search can be performed when at least two +columns have the same semantic concept (e.g. temperature) annotated and the units of measurements can be transformed. <figure markdown>  diff --git a/.docs/concepts/storage.md b/.docs/concepts/storage.md deleted file mode 100644 index fcc975fab2f20d4f3b3b6b0ebd69b02168227bc1..0000000000000000000000000000000000000000 --- a/.docs/concepts/storage.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -author: Martin Weise ---- - -## S3 - -TBD - -## Sidecar - -TBD - -## Upload - -TBD \ No newline at end of file diff --git a/.docs/migration.md b/.docs/migration.md deleted file mode 100644 index e56b88ab8b99fb4cdffc9d68c5db28703f4fdaa8..0000000000000000000000000000000000000000 --- a/.docs/migration.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -author: Martin Weise ---- - -TBD \ No newline at end of file diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 605b27a39cfd4da5bf682351ebe00dd2ba2e7669..8322e26eb1201fe2c4429c6eca6cee85d74654b1 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -371,6 +371,7 @@ release-helm: only: refs: - /^release-.*/ + when: manual except: refs: - release-latest diff --git a/README.md b/README.md index e6f58da9748fbf23f034721c7487016e042dc082..1ab1d140ed50d77dbdc7426e10fefe9e53b9eb56 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ If you have [Docker](https://docs.docker.com/engine/install/) already installed with: ```bash -curl -sSL https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/release-1.4.4/install.sh | bash +curl -sSL https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/release-1.4.5/install.sh | bash ``` ## Documentation diff --git a/mkdocs.yml b/mkdocs.yml index 7c5b7e7a6988aea67a866b9d5b2949cdb3584a4e..45e06cbe388165e317e3a520d0b5ad00a8995449 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -12,7 +12,6 @@ nav: - Help with DBRepo: help.md - Installation: installation.md - Kubernetes: kubernetes.md - - Migration Guide: migration.md - contributing.md - Concepts: - Overview: concepts/index.md @@ -22,7 +21,6 @@ nav: - Monitoring: concepts/monitoring.md - Persistent Identifier: concepts/pid.md - Search: concepts/search.md - - Storage: concepts/storage.md - User Interface: concepts/ui.md - API: - Overview: api/index.md