Skip to content
Snippets Groups Projects
Verified Commit 96b80a21 authored by Martin Weise's avatar Martin Weise
Browse files

Updated documentation

parent 03e1a15d
No related branches found
No related tags found
No related merge requests found
Showing
with 994 additions and 1611 deletions
......@@ -28,12 +28,17 @@ A-1040 Vienna, Austria
- Ganguly, Raman
- Gergely, Eva
- Güçlü, Gökay
- Grantner, Tobias
- Karnbach, Geoffrey
- Mahler, Lukas
- Michlits, Cornelia
- Rauber, Andreas
- Spannring, Max
- Staudinger, Moritz
- Stytsenko, Kirill
- Taha, Josef
- Tsepelakis, Sotiris
- Tsepelakis, Sotirios
- Weise, Martin
Interested in contributing? Send us an e-mail!
---
author: Martin Weise
hide:
- navigation
---
# Customization
!!! info "Abstract"
On this page, we showcase the customization capabilities of DBRepo to e.g. whitelabel your deployment.
## Custom Title
The default title is "Database Repository" and can be replaced by changing the `TITLE` environment variable.
## Custom Logo
The default placeholder logo consists of the two universities that developed this software. You can replace it by
mounting your own logo with a volume.
```console
docker run -v /path/to/your_logo.png:/logo.png ...
```
In case your logo is not in PNG format, you need to change the environment variable `LOGO` accordingly, e.g. for a logo
in JPEG format, set `LOGO: "/logo.JPEG"` in the `.env` file and start the container with a volume.
```console
docker run -v /path/to/your_logo.JPEG:/logo.JPEG ...
```
<figure markdown>
![](images/custom_logo.png)
<figcaption>Custom logo which replaces the default placeholder logo</figcaption>
</figure>
## Custom Icon
The default placeholder icon can be replaced by mounting your own icon with a volume.
```console
docker run -v /path/to/your_logo.ico:/favicon.ico ...
```
<figure markdown>
![](images/custom_icon.png)
<figcaption>Custom icon which replaces the default placeholder icon</figcaption>
</figure>
In case your icon is not in ICO format, you need to change the environment variable `ICON` accordingly, e.g. for an icon
in GIF format, set `ICON: "/favicon.GIF"` in the `.env` file and start the container with a volume.
```console
docker run -v /path/to/favicon.GIF:/favicon.GIF ...
```
---
author: Martin Weise
---
# Kubernetes (Azure)
You can use our pre-built Helm chart for deploying DBRepo in your Kubernetes Cluster with Azure infrastructure
provider.
## TL;DR
```shell
helm install dbrepo "oci://dbrepo.azurecr.io/helm/dbrepo-azure" \
--version "0.1.3"
```
## Architecture
This section will be extended in the future.
<figure markdown>
![Architecture Kubernetes Azure](images/architecture-kubernetes.png)
<figcaption>Architecture of the services on Azure Kubernetes</figcaption>
</figure>
### Chart values
This section will be extended in the future.
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| replicaCount | int | `1` | Number of replicas (pods) to launch. |
| nameOverride | string | `""` | A name in place of the chart name for `app:` labels. |
| fullnameOverride | string | `""` | A name to substitute for the full names of resources. |
---
author: Martin Weise
---
# Kubernetes (OpenStack)
You can use our pre-built Helm chart for deploying DBRepo in your Kubernetes Cluster with OpenStack as infrastructure
provider.
## TL;DR
```shell
helm install dbrepo "oci://dbrepo.azurecr.io/helm/dbrepo-core" \
--version "0.1.3"
```
## Architecture
This section will be extended in the future.
<figure markdown>
![Architecture Kubernetes OpenStack](images/architecture-kubernetes.png)
<figcaption>Architecture of the services on OpenStack Kubernetes</figcaption>
</figure>
### Chart values
This section will be extended in the future.
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| replicaCount | int | `1` | Number of replicas (pods) to launch. |
| nameOverride | string | `""` | A name in place of the chart name for `app:` labels. |
| fullnameOverride | string | `""` | A name to substitute for the full names of resources. |
---
author: Martin Weise
---
# Deployment (Virtual Machine)
## TL;DR
```shell
# download auxilary files
curl -o docker-compose.yml https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/master/docker-compose.prod.yml
curl -o dbrepo.conf https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/master/dbrepo.conf
curl -o .env https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/master/.env.unix.example
# increase memory for open search
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
sysctl -p
# start
docker compose up -d
```
## Architecture
The repository is designed as a microservice architecture to ensure scalability and the utilization of various
technologies. The conceptualized microservices operate the basic database operations, data versioning as well as
*findability*, *accessability*, *interoperability* and *reuseability* (FAIR).
<figure markdown>
![DBRepo architecture](images/architecture-vm.png)
<figcaption>Architecture of the services on a Virtual Machine</figcaption>
</figure>
## Docker Images
* [docker.io](https://hub.docker.com/u/dbrepo)
* dbrepo.azurecr.io
---
author: Martin Weise
hide:
- navigation
---
# Get Started
......
docs/images/architecture-kubernetes.png

220 KiB

docs/images/architecture-vm.png

80.5 KiB

docs/images/architecture.png

112 KiB

......@@ -8,33 +8,51 @@ hide:
## Refereed
##### 2023
* Weise, M., Miksa, T., Grantner, T., Taha, J., Moser, M., Tsepelakis, S., Sanchez-Solis, B. & Rauber, A. (2023).
Research Data Infrastructure Landscape at TU Wien. [Poster]. *Austrian-Slowenian High Performance Computing
Meeting*. <br />
\[[PDF](https://ec.tuwien.ac.at/~weise/pdf/weise2023landscape_paper.pdf)\] \[[Poster](https://ec.tuwien.ac.at/~weise/pdf/weise2023landscape_poster.pdf)\]
* Weise, M., Grantner, T., Taha, J., Staudinger, M., Gergely, E., Stytsenko, K., Ganguly, R. & Rauber, A. (2023).
DBRepo: A Repository for Databases supporting Data Versioning, Schema Semantics and FAIR Principles, at *Open
Repositories Conference*, June 12-15th. (Stellenbosch, South Africa). June 12-15th, 2023.
DOI: [10.5281/zenodo.8075496](https://doi.org/10.5281/zenodo.8075496)<br />
[[PDF](https://zenodo.org/record/8075496/files/3_Martin_Weise_DBRepo.pdf?download=1)]
* Weise, M., Knees, P., Hofmann, A., Ahmedaja, A., Beitane, A. & Rauber, A. (2023).
Connecting Ethnomusicology Data Collections Using Distributed Repositories and Linked Data Technology. *Proceedings
of the 3rd Conference of the ICTM National Committee for Portugal*, (Aveiro, Portugal).
DOI: [10.34726/4323](https://doi.org/10.34726/4323)
##### 2022
1. Ekaputra, F. E., Weise, M., Flicker, K., Salleh, M. R., Rahman, N. A., Miksa, T., & Rauber, A. (2022). Towards A
* Ekaputra, F. E., Weise, M., Flicker, K., Salleh, M. R., Rahman, N. A., Miksa, T., & Rauber, A. (2022). Towards A
Data Repository for Educational Factories. *Proceedings of the 8th International Conference on Data and Software
Engineering*, pp. 149-154. DOI: [10.1109/ICoDSE56892.2022.9971958](https://doi.org/10.1109/ICoDSE56892.2022.9971958).
2. Weise, M., Staudinger, M., Michlits, C., Gergely, E., Stytsenko, K., Ganguly, R., & Rauber, A. (2022).
* Weise, M., Staudinger, M., Michlits, C., Gergely, E., Stytsenko, K., Ganguly, R., & Rauber, A. (2022).
DBRepo: a Semantic Digital Repository for Relational Databases. *International Journal of Digital Curation*,
17(1), 11. DOI: [10.2218/ijdc.v17i1.825](https://doi.org/10.2218/ijdc.v17i1.825)
##### 2021
1. Weise, M., Michlits, C., Staudinger, M., Gergely, E., Stytsenko, K., Ganguly, R. and Rauber A., 2021. FDA-DBRepo: A
* Weise, M., Michlits, C., Staudinger, M., Gergely, E., Stytsenko, K., Ganguly, R. and Rauber A., 2021. FDA-DBRepo: A
Data Preservation Repository Supporting FAIR Principles, Data Versioning and Reproducible Queries. *Proceedings of
the 17th International Conference on Digital Preservation*, Beijing, China, p.34.
DOI: [10.17605/OSF.IO/B7NX5](http://doi.org/10.17605/OSF.IO/B7NX5)
## Other
1. Weise, M. (2023). A Repository and Compute Environment for Sensitive Data. FAIRness for Closed Data,
* Weise, M. (2023). A Repository and Compute Environment for Sensitive Data. FAIRness for Closed Data,
at *EMBL Bioimaging and the European Open Science Cloud*, (Heidelberg, Germany). April, 19-20th, 2023.
DOI: [10.34726/3946](https://doi.org/10.34726/3946)
2. Staudinger, M. (2022). DBRepo: A Repository to Save Research Databases. [Online].
* Staudinger, M. (2022). DBRepo: A Repository to Save Research Databases. [Online].
URL: [https://www.tuwien.at/en/tu-wien/news/news-articles/news/dbrepo](https://www.tuwien.at/en/tu-wien/news/news-articles/news/dbrepo)
accessed 2022-04-12
2. Gergely, E. (2021). Better Support for Research: Current Cooperation Projects. [Online].
* Gergely, E. (2021). Better Support for Research: Current Cooperation Projects. [Online].
URL: [https://zid.univie.ac.at/it-news/artikel/news/cluster-forschungsdaten/](https://zid.univie.ac.at/it-news/artikel/news/cluster-forschungsdaten/)
accessed 2022-04-12
:root {
--md-primary-fg-color: #1976d2;
--md-accent-fg-color: #166abc /* darken 10% */;
--md-primary-fg-color--dark: #135fa9 /* darken 10% */;
:root,
[data-md-color-accent=indigo] {
--md-primary-fg-color: #006699;
--md-accent-fg-color: #005c8a /* darken 10% */;
--md-primary-fg-color--dark: #00537c /* darken 10% */;
}
.md-main .md-content a:not(.action-button),
......
---
author: Martin Weise
hide:
- navigation
---
# System
......@@ -15,17 +11,6 @@ hide:
We invite all open-source developers to help us fixing bugs and introducing features to the source code. Get involved by
sending a mail to Prof. Andreas Rauber and Projektass. Martin Weise.
## Architecture
The repository is designed as a microservice architecture to ensure scalability and the utilization of various
technologies. The conceptualized microservices operate the basic database operations, data versioning as well as
*findability*, *accessability*, *interoperability* and *reuseability* (FAIR).
<figure markdown>
![DBRepo architecture](images/architecture.png)
<figcaption>Architecture</figcaption>
</figure>
## Services
View the docker images for the documentation of the service.
......@@ -36,7 +21,7 @@ View the docker images for the documentation of the service.
* Ports: 5000/tcp
* Prometheus: `http://:5000/metrics`
* Swagger UI: `http://:5000/swagger-ui/index.html` <a href="/infrastructures/dbrepo/swagger/analyse" target="_blank">:fontawesome-solid-square-up-right: view online</a>
* Swagger UI: `http://:5000/swagger-ui/index.html` <a href="/infrastructures/dbrepo/latest/swagger/analyse" target="_blank">:fontawesome-solid-square-up-right: view online</a>
It suggests data types for the FAIR Portal when creating a table from a *comma separated values* (CSV) file. It
recommends enumerations for columns and returns e.g. a list of potential primary key candidates. The researcher is able
......@@ -338,7 +323,7 @@ The default credentials are:
* Info: `http://:9091/actuator/info`
* Health: `http://:9091/actuator/health`
* Prometheus: `http://:9091/actuator/prometheus`
* Swagger UI: `http://:9091/swagger-ui/index.html` <a href="/infrastructures/dbrepo/swagger/container" target="_blank">:fontawesome-solid-square-up-right: view online</a>
* Swagger UI: `http://:9091/swagger-ui/index.html` <a href="/infrastructures/dbrepo/latest/swagger/container" target="_blank">:fontawesome-solid-square-up-right: view online</a>
It is responsible for Docker container lifecycle operations and updating the local copy of the Docker images.
......@@ -350,7 +335,7 @@ It is responsible for Docker container lifecycle operations and updating the loc
* Info: `http://:9092/actuator/info`
* Health: `http://:9092/actuator/health`
* Prometheus: `http://:9092/actuator/prometheus`
* Swagger UI: `http://:9092/swagger-ui/index.html` <a href="/infrastructures/dbrepo/swagger/database" target="_blank">:fontawesome-solid-square-up-right: view online</a>
* Swagger UI: `http://:9092/swagger-ui/index.html` <a href="/infrastructures/dbrepo/latest/swagger/database" target="_blank">:fontawesome-solid-square-up-right: view online</a>
It creates the databases inside a Docker container and the Query Store. Currently, we only
support [MariaDB](https://mariadb.org/) images that allow table versioning with low programmatic effort.
......@@ -375,7 +360,7 @@ standard [NGINX](https://www.nginx.com/) reverse proxy for load balancing, SSL/T
* Info: `http://:9096/actuator/info`
* Health: `http://:9096/actuator/health`
* Prometheus: `http://:9096/actuator/prometheus`
* Swagger UI: `http://:9096/swagger-ui/index.html` <a href="/infrastructures/dbrepo/swagger/identifier" target="_blank">:fontawesome-solid-square-up-right: view online</a>
* Swagger UI: `http://:9096/swagger-ui/index.html` <a href="/infrastructures/dbrepo/latest/swagger/identifier" target="_blank">:fontawesome-solid-square-up-right: view online</a>
This microservice is responsible for creating and resolving a *persistent identifier* (PID) attached to a query to
obtain the metadata attached to it and allow re-execution of a query. We store both the query and hashes of the query
......@@ -408,7 +393,7 @@ port `3306`.
* Info: `http://:9099/actuator/info`
* Health: `http://:9099/actuator/health`
* Prometheus: `http://:9099/actuator/prometheus`
* Swagger UI: `http://:9099/swagger-ui/index.html` <a href="/infrastructures/dbrepo/swagger/metadata" target="_blank">:fontawesome-solid-square-up-right: view online</a>
* Swagger UI: `http://:9099/swagger-ui/index.html` <a href="/infrastructures/dbrepo/latest/swagger/metadata" target="_blank">:fontawesome-solid-square-up-right: view online</a>
This service provides an OAI-PMH endpoint for metadata crawler.
......@@ -420,7 +405,7 @@ This service provides an OAI-PMH endpoint for metadata crawler.
* Info: `http://:9093/actuator/info`
* Health: `http://:9093/actuator/health`
* Prometheus: `http://:9093/actuator/prometheus`
* Swagger UI: `http://:9093/swagger-ui/index.html` <a href="/infrastructures/dbrepo/swagger/query" target="_blank">:fontawesome-solid-square-up-right: view online</a>
* Swagger UI: `http://:9093/swagger-ui/index.html` <a href="/infrastructures/dbrepo/latest/swagger/query" target="_blank">:fontawesome-solid-square-up-right: view online</a>
It provides an interface to insert data into the tables created by the Table Service. It also allows for view-only,
paginated and versioned query execution to the raw data and consumes messages in the message queue from the Broker
......@@ -449,7 +434,7 @@ All requests need to be authenticated, by default the credentials `elastic:elast
* Info: `http://:9097/actuator/info`
* Health: `http://:9097/actuator/health`
* Prometheus: `http://:9097/actuator/prometheus`
* Swagger UI: `http://:9097/swagger-ui/index.html` <a href="/infrastructures/dbrepo/swagger/semantics" target="_blank">:fontawesome-solid-square-up-right: view online</a>
* Swagger UI: `http://:9097/swagger-ui/index.html` <a href="/infrastructures/dbrepo/latest/swagger/semantics" target="_blank">:fontawesome-solid-square-up-right: view online</a>
It is designed to map terms in the domain of units of measurement to controlled vocabulary, modelled in
the [ontology of units of measure](https://github.com/HajoRijgersberg/OM). This service validates researcher provided in
......@@ -464,7 +449,7 @@ Furthermore, there is a method for auto-completing text and listing a descriptio
* Info: `http://:9094/actuator/info`
* Health: `http://:9094/actuator/health`
* Prometheus: `http://:9094/actuator/prometheus`
* Swagger UI: `http://:9094/swagger-ui/index.html` <a href="/infrastructures/dbrepo/swagger/table" target="_blank">:fontawesome-solid-square-up-right: view online</a>
* Swagger UI: `http://:9094/swagger-ui/index.html` <a href="/infrastructures/dbrepo/latest/swagger/table" target="_blank">:fontawesome-solid-square-up-right: view online</a>
This microservice handles table operations inside a database that is managed by the Database Service. We
use [Hibernate](https://hibernate.org/orm/) for schema and data ingest operations.
......@@ -492,6 +477,6 @@ It provides a *graphical user interface* (GUI) for a researcher to interact with
* Info: `http://:9098/actuator/info`
* Health: `http://:9098/actuator/health`
* Prometheus: `http://:9098/actuator/prometheus`
* Swagger UI: `http://:9098/swagger-ui/index.html` <a href="/infrastructures/dbrepo/swagger/user" target="_blank">:fontawesome-solid-square-up-right: view online</a>
* Swagger UI: `http://:9098/swagger-ui/index.html` <a href="/infrastructures/dbrepo/latest/swagger/user" target="_blank">:fontawesome-solid-square-up-right: view online</a>
This microservice handles user information.
---
author: Martin Weise
hide:
- navigation
---
# Deployment
# Broker Service
!!! info "Abstract"
We modified some services and exchanged them with reviewed, open-source implementations that extend the functionality
even more from version 1.2 onwards. On this page, some of the configuration possible is summarized.
## Authentication Service
## Broker Service
### Authentication
## Authentication
The RabbitMQ client can be authenticated through plain (username, password) and OAuth2 mechanisms. Note that the access
token already contains a field `client_id=foo`, so the username is optional in `PlainCredentials()`.
......@@ -54,23 +43,3 @@ token already contains a field `client_id=foo`, so the username is optional in `
connection.close()
```
## Identifier Service
From version 1.2 onwards there are two modes for the Identifier Service:
1. Persistent Identifier (PID)
2. Digital Object Identifier (DOI)
By default, the URI mode is used, creating a PID for databases or subsets. If starting the Identifier Service in DOI mode,
a DOI is minted for persistent identification of databases or subsets. Using the DOI system is entirely *optional* and
should not be done for test-deployments.
<figure markdown>
![](images/identifier-doi.png)
<figcaption>Minting a test-DOI for a subset</figcaption>
</figure>
## Gateway Service
From version 1.2 onwards we use both HTTP and HTTPS to serve the API, especially for the Authentication Service. The Discovery
Service lists both the non-secure and secure ports.
---
author: Martin Weise
---
# Identifier Service
From version 1.2 onwards there are two modes for the Identifier Service:
1. Persistent Identifier (PID)
2. Digital Object Identifier (DOI)
By default, the URI mode is used, creating a PID for databases or subsets. If starting the Identifier Service in DOI mode,
a DOI is minted for persistent identification of databases or subsets. Using the DOI system is entirely *optional* and
should not be done for test-deployments.
<figure markdown>
![](images/identifier-doi.png)
<figcaption>Minting a test-DOI for a subset</figcaption>
</figure>
---
author: Martin Weise
---
# Overview
!!! info "Abstract"
We modified some services and exchanged them with reviewed, open-source implementations that extend the functionality
even more from version 1.2 onwards. On this page, some of the configuration possible is summarized.
Source diff could not be displayed: it is too large. Options to address this: view the blob.
Source diff could not be displayed: it is too large. Options to address this: view the blob.
Source diff could not be displayed: it is too large. Options to address this: view the blob.
......@@ -6,8 +6,16 @@ copyright: CC-BY 4.0 Technische Universit&auml;t Wien & Universit&auml;t Wien
nav:
- Home: index.md
- get-started.md
- customization.md
- Deployment:
- Virtual Machine: deployment-vm.md
- Kubernetes:
- OpenStack: deployment-kubernetes-openstack.md
- Azure: deployment-kubernetes-azure.md
- system.md
- Usage:
- usage-overview.md
- Broker Service: usage-broker.md
- Identifier Service: usage-identifier.md
- publications.md
- contact.md
extra_css:
......@@ -21,6 +29,8 @@ theme:
features:
- navigation.tabs
- navigation.tabs.sticky
- navigation.instant
- navigation.sections
- content.code.annotate
icon:
repo: fontawesome/brands/git-alt
......@@ -46,6 +56,9 @@ markdown_extensions:
- pymdownx.emoji:
emoji_index: !!python/name:materialx.emoji.twemoji
emoji_generator: !!python/name:materialx.emoji.to_svg
options:
custom_icons:
- overrides/.icons
extra:
homepage: https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/
social:
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment