Skip to content
Snippets Groups Projects
Verified Commit a15e2766 authored by Martin Weise's avatar Martin Weise
Browse files

Added embargo

parent 900ebb35
No related branches found
No related tags found
1 merge request!368Dev
Expand all Show whitespace changes
Inline Side-by-side
Showing
with 165 additions and 68 deletions
.docs/concepts/authentication.md
+ 3
− 3
View file @ a15e2766
......@@ -27,7 +27,7 @@ The [User Interface](../../api/ui) for example refreshes the token on-the-fly by
of an expired access token, requests a new one without having to terminate the request. This happens only once after the
access token has expired (after e.g. 15 minutes).
## OpenID Connect (OIDC)
## OpenID Connect
We use the widely accepted authentication protocol OIDC for client authentication. Other protocols are e.g. SAML2 which
are not used by DBRepo.
\ No newline at end of file
We use the widely accepted authentication protocol OpenID Connect for client authentication. Other protocols are, e.g.
SAML2 which are not used by default in DBRepo.
\ No newline at end of file
.docs/concepts/data-versioning.md 0 → 100644
+ 39
− 0
View file @ a15e2766
---
author: Martin Weise
---
Data is getting bigger and so are expectations of data provisioning in regards to data availability (i.e. immediately
after quality check and not in snapshot intervals), cost-effectiveness (i.e. no duplication of data), transparent,
precise citation and many more.
[System-versioned](https://mariadb.com/kb/en/system-versioned-tables/) tables in MariaDB are improved data structures
that keep track of historical data. For each entry in a system-versioned table, a time period is maintained that denotes
the validity time span of this tuple from its start to end. Tuples in system-versioned tables are not *actually*
modified, they are marked as (in-)valid in time periods.
<figure markdown>
| ID | Sensor | Temp | Start | End |
|----|--------|------|-------|-----|
| 1 | A | 23.1 | t1 | |
| 2 | B | 25.8 | t2 | |
</figure>
Assuming that Sensor A was calibrated wrong and an updated measurement is passed to the system-versioned table, the
table contents show that the old row with Temp 23.1 is not deleted, but marked as valid in time span (t1, t3). The
updated row with Temp 22.1 is marked as valid from time span t3 onwards.
<figure markdown>
| ID | Sensor | Temp | Start | End |
|----|--------|------|-------|-----|
| 1 | A | 23.1 | t1 | t3 |
| 2 | B | 25.8 | t2 | |
| 1 | A | 22.1 | t3 | |
</figure>
System-versioned tables are part of the SQL:2011 standard and have been adopted by many database management system
vendors: MariaDB (10.5 and higher), Google BigQuery, IBM DB2 (12 and higher), SQL Server (2016 and higher), Azure SQL,
PostgreSQL with [temporal tables extension](https://github.com/nearform/temporal_tables), etc.
\ No newline at end of file
.docs/concepts/data-visibility.md 0 → 100644
+ 33
− 0
View file @ a15e2766
---
author: Martin Weise
---
There are several ways to set the visibility of (meta-)data in DBRepo. It is possible to set the data to public/private
and the schema to be public/private for each database and separately for each table, each view and each subset of a
database.
In total there are three possible scenarios:
## Public
!!! info "Possible use-case: data publication supplement to an open-access publication"
Where the database's data and metadata is set to be *public*. This means everything in the database (tables, views,
subsets) are visible by anyone from the public.
## Mixed
!!! info "Possible use-case: private sensor measurements with timed embargo"
Where the database's data and metadata is set to be *private*. This means everything in the database (tables, views,
subsets) are by default not visible by anyone from the public. You can however make specific views that join tables
and/or filter certain columns and apply a 14-day delay-embargo.
<figure markdown>
![Mirroring statistical properties in Metadata Database and Search Database](../images/private-embargo.svg)
<figcaption>Figure 1: Public view that joins two private tables and applies a time-embargo</figcaption>
</figure>
## Private
!!! info "Possible use-case: data storage for trusted-/virtual research environments"
\ No newline at end of file
.docs/concepts/messaging.md
+ 7
− 1
View file @ a15e2766
......@@ -22,3 +22,9 @@ JSON, a tuple looks like this in RabbitMQ:
DBRepo uses AMQP to route messages which allows for both Basic/Bearer authentication. For more information please
consult the [RabbitMQ AMQP](https://www.rabbitmq.com/tutorials/amqp-concepts) documentation.
## MQTT
:octicons-tag-16:{ title="Minimum version" } 1.5.0
DBRepo supports MQTT for IoT with Basic authentication.
\ No newline at end of file
.docs/concepts/pid.md
+ 5
− 41
View file @ a15e2766
......@@ -2,46 +2,6 @@
author: Martin Weise
---
## Data Versioning
Data is getting bigger and so are expectations of data provisioning in regards to data availability (i.e. immediately
after quality check and not in snapshot intervals), cost-effectiveness (i.e. no duplication of data), transparent,
precise citation and many more.
[System-versioned](https://mariadb.com/kb/en/system-versioned-tables/) tables in MariaDB are improved data structures
that keep track of historical data. For each entry in a system-versioned table, a time period is maintained that denotes
the validity time span of this tuple from its start to end. Tuples in system-versioned tables are not *actually*
modified, they are marked as (in-)valid in time periods.
<figure markdown>
| ID | Sensor | Temp | Start | End |
|----|--------|------|-------|-----|
| 1 | A | 23.1 | t1 | |
| 2 | B | 25.8 | t2 | |
</figure>
Assuming that Sensor A was calibrated wrong and an updated measurement is passed to the system-versioned table, the
table contents show that the old row with Temp 23.1 is not deleted, but marked as valid in time span (t1, t3). The
updated row with Temp 22.1 is marked as valid from time span t3 onwards.
<figure markdown>
| ID | Sensor | Temp | Start | End |
|----|--------|------|-------|-----|
| 1 | A | 23.1 | t1 | t3 |
| 2 | B | 25.8 | t2 | |
| 1 | A | 22.1 | t3 | |
</figure>
System-versioned tables are part of the SQL:2011 standard and have been adopted by many database management system
vendors: MariaDB (10.5 and higher), Google BigQuery, IBM DB2 (12 and higher), SQL Server (2016 and higher), Azure SQL,
PostgreSQL with [temporal tables extension](https://github.com/nearform/temporal_tables), etc.
## Persistent Identifier
Data in DBRepo always has attached metadata (stored in the [Metadata Database](../../api/metadata-db)). This metadata
is provided as machine-understandable context in various open-source formats that is available, even when the original
data is not available anymore due to e.g. a retracted dataset (hence the name **persistent**). A persistent identifier
......@@ -55,3 +15,7 @@ globally, uniquely identifies a data record such as:
Combining [data versioning](#data-versioning) and queries, subsets can be precisely identified by storing the query
that creates them and the time when the query was executed. We store both in a table inside the database we call the
query store.
## DOI
DBRepo uses the DOI system minted by DataCite, this is optional and not required to function for e.g. test deployments.
\ No newline at end of file
.docs/images/architecture.drawio
+ 46
− 1
View file @ a15e2766
<mxfile host="app.diagrams.net" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36" version="25.0.1" pages="8">
<mxfile host="Electron" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/25.0.2 Chrome/128.0.6613.186 Electron/32.2.5 Safari/537.36" version="25.0.2" pages="9">
<diagram id="mvBsv1rP8O80Qe3yGnn_" name="docker-compose">
<mxGraphModel dx="683" dy="391" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" math="0" shadow="0">
<root>
......@@ -1108,4 +1108,49 @@
</root>
</mxGraphModel>
</diagram>
<diagram id="7HywRA3nQAgvNxZjCRq2" name="private-embargo">
<mxGraphModel dx="985" dy="394" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" math="0" shadow="0">
<root>
<mxCell id="0" />
<mxCell id="1" parent="0" />
<mxCell id="n6nk3BLY6128t3IB6Ma7-8" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0.25;entryY=0;entryDx=0;entryDy=0;curved=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;" edge="1" parent="1" source="n6nk3BLY6128t3IB6Ma7-1" target="n6nk3BLY6128t3IB6Ma7-5">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="n6nk3BLY6128t3IB6Ma7-11" value="&lt;span style=&quot;text-wrap: wrap; background-color: rgb(251, 251, 251);&quot;&gt;value,loc_id&lt;/span&gt;" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];fontSize=8;" vertex="1" connectable="0" parent="n6nk3BLY6128t3IB6Ma7-8">
<mxGeometry x="0.0303" relative="1" as="geometry">
<mxPoint as="offset" />
</mxGeometry>
</mxCell>
<mxCell id="n6nk3BLY6128t3IB6Ma7-1" value="" style="shape=internalStorage;whiteSpace=wrap;html=1;backgroundOutline=1;" vertex="1" parent="1">
<mxGeometry x="250" y="170" width="80" height="80" as="geometry" />
</mxCell>
<mxCell id="n6nk3BLY6128t3IB6Ma7-2" value="&lt;b&gt;table&lt;/b&gt;: sensor (private)" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
<mxGeometry x="227.5" y="150" width="125" height="20" as="geometry" />
</mxCell>
<mxCell id="n6nk3BLY6128t3IB6Ma7-9" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.75;entryY=0;entryDx=0;entryDy=0;curved=1;" edge="1" parent="1" source="n6nk3BLY6128t3IB6Ma7-3" target="n6nk3BLY6128t3IB6Ma7-5">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="n6nk3BLY6128t3IB6Ma7-12" value="id,name,lat,lng" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];fontSize=8;" vertex="1" connectable="0" parent="n6nk3BLY6128t3IB6Ma7-9">
<mxGeometry x="0.1455" relative="1" as="geometry">
<mxPoint as="offset" />
</mxGeometry>
</mxCell>
<mxCell id="n6nk3BLY6128t3IB6Ma7-3" value="" style="shape=internalStorage;whiteSpace=wrap;html=1;backgroundOutline=1;" vertex="1" parent="1">
<mxGeometry x="430" y="170" width="80" height="80" as="geometry" />
</mxCell>
<mxCell id="n6nk3BLY6128t3IB6Ma7-4" value="&lt;b&gt;table&lt;/b&gt;: location (private)" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
<mxGeometry x="405" y="150" width="130" height="20" as="geometry" />
</mxCell>
<mxCell id="n6nk3BLY6128t3IB6Ma7-5" value="" style="shape=internalStorage;whiteSpace=wrap;html=1;backgroundOutline=1;fontSize=8;" vertex="1" parent="1">
<mxGeometry x="340" y="290" width="80" height="80" as="geometry" />
</mxCell>
<mxCell id="n6nk3BLY6128t3IB6Ma7-6" value="&lt;b&gt;view&lt;/b&gt;: validated_sensor (public)" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
<mxGeometry x="290" y="370" width="180" height="20" as="geometry" />
</mxCell>
<mxCell id="n6nk3BLY6128t3IB6Ma7-7" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;" edge="1" parent="1" source="n6nk3BLY6128t3IB6Ma7-4" target="n6nk3BLY6128t3IB6Ma7-4">
<mxGeometry relative="1" as="geometry" />
</mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>
.docs/images/private-embargo.svg 0 → 100644
+ 3
− 0
View file @ a15e2766
This diff is collapsed.
.docs/installation.md
+ 3
− 3
View file @ a15e2766
......@@ -11,7 +11,7 @@ author: Martin Weise
If you have [Docker](https://docs.docker.com/engine/install/) already installed on your system, you can install DBRepo with:
```shell
curl -sSL https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/release-1.5/install.sh | bash
curl -sSL https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/release-1.6/install.sh | bash
```
!!! bug "Default installation security disclaimer"
......@@ -38,7 +38,7 @@ SSL/TLS certificate is recommended. Follow the [secure install](#secure-install)
Execute the install script to download only the environment and save it to `dist`.
```shell
curl -sSL https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/release-1.5/install.sh | DOWNLOAD_ONLY=1 bash
curl -sSL https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/release-1.6/install.sh | DOWNLOAD_ONLY=1 bash
```
### Static Configuration
......@@ -80,7 +80,7 @@ the variable `IDENTITY_SERVICE_ADMIN_PASSWORD` in `.env`.
Update the client secret of the `dbrepo-client`:
```bash
curl -sSL "https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/release-1.5/.scripts/reg-client-secret.sh" | bash
curl -sSL "https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/release-1.6/.scripts/reg-client-secret.sh" | bash
```
Also, update the JWT key according to the
......
.docs/kubernetes.md
+ 1
− 1
View file @ a15e2766
......@@ -6,7 +6,7 @@ author: Martin Weise
To install DBRepo in your existing cluster, download the
sample [
`values.yaml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/blob/release-1.5/helm/dbrepo/values.yaml)
`values.yaml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/blob/release-1.6/helm/dbrepo/values.yaml)
for your deployment and update the variables, especially `hostname`.
```shell
......
.gitlab/gen-badge.sh
+ 2
− 2
View file @ a15e2766
......@@ -9,7 +9,7 @@ else
echo "[WARNING] Skipping badge generation, displaying default badge text: unknown"
fi
curl -fsSL "${CI_SONAR_URL}/api/project_badges/measure?project=${CI_SONAR_PROJECT_KEY}&metric=sqale_rating&token=${CI_SONAR_TOKEN}" > "./final/${DOC_VERSION}/images/maintainability.svg"
curl -fsSL "${CI_SONAR_URL}/api/project_badges/measure?project=${CI_SONAR_PROJECT_KEY}&metric=security_rating&token=${CI_SONAR_TOKEN}" > "./final/${DOC_VERSION}/images/security.svg"
curl -fsSL "${CI_SONAR_URL}/api/project_badges/measure?project=${CI_SONAR_PROJECT_KEY}&metric=sqale_rating&token=${CI_SONAR_BADGE_TOKEN}" > "./final/${DOC_VERSION}/images/maintainability.svg"
curl -fsSL "${CI_SONAR_URL}/api/project_badges/measure?project=${CI_SONAR_PROJECT_KEY}&metric=security_rating&token=${CI_SONAR_BADGE_TOKEN}" > "./final/${DOC_VERSION}/images/security.svg"
echo "[INFO] retrieved badges"
\ No newline at end of file
README.md
+ 1
− 1
View file @ a15e2766
......@@ -14,7 +14,7 @@ If you have [Docker](https://docs.docker.com/engine/install/) already installed
with:
```bash
curl -sSL https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/release-1.5/install.sh | bash
curl -sSL https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/release-1.6/install.sh | bash
```
## Documentation
......
helm/dbrepo/README.md
+ 6
− 6
View file @ a15e2766
# DBRepo Helm chart
[DBRepo](https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.5/) is a database repository system that
[DBRepo](https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.6/) is a database repository system that
allows researchers to ingest data into a central, versioned repository through common interfaces.
## TL;DR
Download the
sample [
`values.yaml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/release-1.5/helm-charts/dbrepo/values.yaml?inline=true)
`values.yaml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/release-1.6/helm-charts/dbrepo/values.yaml?inline=true)
for your deployment and update the variables, especially `hostname`.
```bash
......@@ -65,10 +65,10 @@ The command removes all the Kubernetes components associated with the chart and
### Metadata Database
| Name | Description | Value |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| ---------------------------------------- |----------------------------------------------------------------------------------------------------------------------------------------| ---------------------------------------------------------------------- |
| `metadatadb.enabled` | Enable the Metadata datadb. | `true` |
| `metadatadb.host` | The hostname for the microservices. | `metadata-db` |
| `metadatadb.extraFlags` | Extra flags to ensure the query store works as intended, ref https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.5/api/data-db/#data | `--character-set-server=utf8mb4 --collation-server=utf8mb4_general_ci` |
| `metadatadb.extraFlags` | Extra flags to ensure the query store works as intended, ref https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.6/api/data-db/#data | `--character-set-server=utf8mb4 --collation-server=utf8mb4_general_ci` |
| `metadatadb.rootUser.user` | The root username. | `root` |
| `metadatadb.rootUser.password` | The root user password. | `dbrepo` |
| `metadatadb.db.name` | The database name. | `dbrepo` |
......@@ -96,9 +96,9 @@ The command removes all the Kubernetes components associated with the chart and
### Data Database
| Name | Description | Value |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| ------------------------------------ |----------------------------------------------------------------------------------------------------------------------------------------| ---------------------------------------------------------------------- |
| `datadb.host` | The hostname for the microservices. | `data-db` |
| `datadb.extraFlags` | Extra flags to ensure the query store works as intended, ref https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.5/api/data-db/#data | `--character-set-server=utf8mb4 --collation-server=utf8mb4_general_ci` |
| `datadb.extraFlags` | Extra flags to ensure the query store works as intended, ref https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.6/api/data-db/#data | `--character-set-server=utf8mb4 --collation-server=utf8mb4_general_ci` |
| `datadb.rootUser.user` | The root username. | `root` |
| `datadb.rootUser.password` | The root user password. | `dbrepo` |
| `datadb.db.name` | The database name. | `dbrepo` |
......
lib/python/README.md
+ 6
− 6
View file @ a15e2766
......@@ -48,17 +48,17 @@ client.import_table_data(database_id=7, table_id=13, file_name_or_data_frame=df)
## Supported Features & Best-Practices
- Manage user
account ([docs](https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.5/api/#create-user-account))
account ([docs](https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.6/api/#create-user-account))
- Manage
databases ([docs](https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.5/usage-overview/#create-database))
databases ([docs](https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.6/usage-overview/#create-database))
- Manage database access &
visibility ([docs](https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.5/api/#create-database))
visibility ([docs](https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.6/api/#create-database))
- Import
dataset ([docs](https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.5/api/#import-dataset))
dataset ([docs](https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.6/api/#import-dataset))
- Create persistent
identifiers ([docs](https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.5/api/#assign-database-pid))
identifiers ([docs](https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.6/api/#assign-database-pid))
- Execute
queries ([docs](https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.5/api/#export-subset))
queries ([docs](https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.6/api/#export-subset))
- Get data from tables/views/subsets
## Configure
......
mkdocs.yml
+ 5
− 3
View file @ a15e2766
site_name: Database Repository
site_url: https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.5/
site_url: https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.6/
repo_url: https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services
repo_name: fda-services
site_author: Research Unit Data Science, Technische Universit&auml;t Wien
......@@ -17,6 +17,8 @@ nav:
- Concepts:
- Overview: concepts/index.md
- Authentication: concepts/authentication.md
- Data Versioning: concepts/data-versioning.md
- Data Visibility: concepts/data-visibility.md
- Messaging: concepts/messaging.md
- Monitoring: concepts/monitoring.md
- Persistent Identifier: concepts/pid.md
......@@ -118,9 +120,9 @@ markdown_extensions:
custom_icons:
- .docs/overrides/.icons
extra:
homepage: https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/
homepage: https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.6/
version:
default: 1.5
default: 1.6
provider: mike
social:
- icon: simple/artifacthub
......
versions.json
+ 5
− 0
View file @ a15e2766
[
{
"version": "1.6",
"title": "1.6",
"aliases": []
},
{
"version": "1.5",
"title": "1.5",
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment