Skip to content
Snippets Groups Projects
Verified Commit c370b2f0 authored by Martin Weise's avatar Martin Weise
Browse files

Updated the services documentation

parent 2778ba93
No related branches found
No related tags found
No related merge requests found
Pipeline #4855 failed
Stage: configure
Stage: build
Stage: deploy
Hide whitespace changes
Inline Side-by-side
docs/deployment.md
+ 90
− 9
View file @ c370b2f0
...@@ -29,20 +29,101 @@ it is *highly encouraged* to use your own certificate, properly issued by a trus ...@@ -29,20 +29,101 @@ it is *highly encouraged* to use your own certificate, properly issued by a trus
you can use the self-signed certificate. You need to accept the risk in most browsers when visiting the you can use the self-signed certificate. You need to accept the risk in most browsers when visiting the
[admin panel](https://localhost:8443/admin/). [admin panel](https://localhost:8443/admin/).
<figure markdown>
![](images/auth-ssl.png)
<figcaption>Google Chrome warning about the self-signed certificate</figcaption>
</figure>
Sign in with the default credentials (username `fda`, password `fda`) or the one you configured during set-up. Be default, Sign in with the default credentials (username `fda`, password `fda`) or the one you configured during set-up. Be default,
users are created using the frontend and the sign-up page. But it is also possible to create users from Keycloak, they will users are created using the frontend and the sign-up page. But it is also possible to create users from Keycloak, they will
still act as "self-sign-up" created users. Since we do not support all features of Keycloak, leave out required user actions still act as "self-sign-up" created users. Since we do not support all features of Keycloak, leave out required user actions
as they will not be enforced, also the temporary password. as they will not be enforced, also the temporary password.
<figure markdown> ### Obtain Access Token
![](images/auth-create.png)
<figcaption>Alternative user creation via Keycloak</figcaption> === "Terminal"
</figure>
``` console
curl -X POST \
-d "username=foo&password=bar&grant_type=password&client_id=dbrepo-client&scope=openid&client_secret=MUwRc7yfXSJwX8AdRMWaQC3Nep1VjwgG" \
http://localhost/api/auth/realms/dbrepo/protocol/openid-connect/token
```
=== "Python"
``` py
import requests
auth = requests.post("http://localhost/api/auth/realms/dbrepo/protocol/openid-connect/token", data={
"username": "foo",
"password": "bar",
"grant_type": "password",
"client_id": "dbrepo-client",
"scope": "openid",
"client_secret": "MUwRc7yfXSJwX8AdRMWaQC3Nep1VjwgG"
})
print(auth.json()["access_token"])
```
### Refresh Access Token
=== "Terminal"
``` console
curl -X POST \
-d "grant_type=refresh_token&client_id=dbrepo-client&refresh_token=THE_REFRESH_TOKEN&client_secret=MUwRc7yfXSJwX8AdRMWaQC3Nep1VjwgG" \
http://localhost/api/auth/realms/dbrepo/protocol/openid-connect/token
```
=== "Python"
``` py
import requests
auth = requests.post("http://localhost/api/auth/realms/dbrepo/protocol/openid-connect/token", data={
"grant_type": "refresh_token",
"client_id": "dbrepo-client",
"client_secret": "MUwRc7yfXSJwX8AdRMWaQC3Nep1VjwgG",
"refresh_token": "THE_REFRESH_TOKEN"
})
print(auth.json()["access_token"])
```
## Broker Service
### Authentication
The RabbitMQ client can be authenticated through plain (username, password) and OAuth2 mechanisms. Note that the access
token already contains a field `client_id=foo`, so the username is optional in `PlainCredentials()`.
=== "Plain"
``` py
import pika
credentials = pika.credentials.PlainCredentials("foo", "bar")
parameters = pika.ConnectionParameters('localhost', 5672, '/', credentials)
connection = pika.BlockingConnection(parameters)
channel = connection.channel()
channel.queue_declare(queue='test', durable=True)
channel.basic_publish(exchange='',
routing_key='test',
body=b'Hello World!')
print(" [x] Sent 'Hello World!'")
connection.close()
```
=== "OAuth2"
``` py
import pika
credentials = pika.credentials.PlainCredentials("", "THE_ACCESS_TOKEN")
parameters = pika.ConnectionParameters('localhost', 5672, '/', credentials)
connection = pika.BlockingConnection(parameters)
channel = connection.channel()
channel.queue_declare(queue='test', durable=True)
channel.basic_publish(exchange='',
routing_key='test',
body=b'Hello World!')
print(" [x] Sent 'Hello World!'")
connection.close()
```
## Identifier Service ## Identifier Service
......
docs/stylesheets/extra.css
+ 4
− 4
View file @ c370b2f0
...@@ -8,14 +8,14 @@ ...@@ -8,14 +8,14 @@
border: 1px solid var(--md-primary-fg-color) border: 1px solid var(--md-primary-fg-color)
} }
.md-main .md-content a, .md-main .md-content a:not(.action-button),
.md-main .md-content a { .md-main .md-content a:not(.action-button) {
color: var(--md-typeset-color); color: var(--md-typeset-color);
border-bottom: 2px solid var(--md-primary-fg-color); border-bottom: 2px solid var(--md-primary-fg-color);
} }
.md-main .md-content a:focus, .md-main .md-content a:not(.action-button):focus,
.md-main .md-content a:hover { .md-main .md-content a:not(.action-button):hover {
color: var(--md-typeset-color); color: var(--md-typeset-color);
border-bottom: 2px solid var(--md-primary-fg-color--dark); border-bottom: 2px solid var(--md-primary-fg-color--dark);
} }
docs/system.md
+ 136
− 130
View file @ c370b2f0
...@@ -7,6 +7,10 @@ hide: ...@@ -7,6 +7,10 @@ hide:
# System # System
!!! info "Abstract"
This is the full system description from a technical/developer view.
We invite all open-source developers to help us fixing bugs and introducing features to the source code. Get involved by We invite all open-source developers to help us fixing bugs and introducing features to the source code. Get involved by
sending a mail to Prof. Andreas Rauber and Projektass. Martin Weise. sending a mail to Prof. Andreas Rauber and Projektass. Martin Weise.
...@@ -20,28 +24,18 @@ technologies. The conceptualized microservices operate the basic database operat ...@@ -20,28 +24,18 @@ technologies. The conceptualized microservices operate the basic database operat
View the docker images for the documentation of the service. View the docker images for the documentation of the service.
### Discovery Service ### Analyse Service
This microservice allows service discovery and registration of containers that provide services. It configures
a [Spring Cloud Netflix Eureka Server](https://cloud.spring.io/spring-cloud-netflix/reference/html/) to discover
services.
!!! debug "Debug Information"
* Port(s): 9090
* Docker Image: [dbrepo/discovery-service](https://hub.docker.com/repository/docker/dbrepo/discovery-service)
* Swagger: not configured
### Gateway Service
Provides a single point of access to the *application programming interface* (API) and configures It suggests data types for the FAIR Portal when creating a table from a *comma separated values* (CSV) file. It
the [Spring Cloud Gateway](https://spring.io/projects/spring-cloud-gateway) to route traffic to the services. recommends enumerations for columns and returns e.g. a list of potential primary key candidates. The researcher is able
to confirm these suggestions manually. Moreover, the *Analyze Service* determines basic statistical properties of
numerical columns.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9095 * Ports: 5000/tcp
* Docker Image: [dbrepo/gateway-service](https://hub.docker.com/repository/docker/dbrepo/gateway-service) * Prometheus: `http://:5000/metrics`
* Swagger: not configured * Swagger UI: `http://:5000/swagger-ui/index.html` [:fontawesome-solid-square-up-right: view online](/swagger/analyse)
### Authentication Service ### Authentication Service
...@@ -51,113 +45,122 @@ through an encrypted channel. ...@@ -51,113 +45,122 @@ through an encrypted channel.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9097 * Ports: 8080/tcp, 8443/tcp
* Docker Image: [dbrepo/authentication-service](https://hub.docker.com/repository/docker/dbrepo/authentication-service) * Admin Console: `http://:8443/`
* Swagger UI: [/swagger/authentication](/swagger/authentication)
### Metadata Database ### Broker Service
It is the core component of the project. It is a relational database that contains metadata about all researcher databases
created in the database repository like column names, check expressions, value enumerations or key/value constraints and
relevant data for citing data sets. Additionally, the concept, e.g. URI of units of measurements of numerical columns is
stored in the Metadata Database in order to provide semantic knowledge context. We use MariaDB for its rich capabilities
in the reference implementation.
The default credentials are `root:dbrepo` for the database `fda`. Connect to the database via the JDBC connector. It holds exchanges and topics responsible for holding AMQP messages for later consumption. We
use [RabbitMQ](https://www.rabbitmq.com/) in the reference implementation. The AMQP endpoint listens to port `5672` for
regular declares and offers a management interface at port `15672`.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 3306 * Ports: 5672/tcp, 15672/tcp
* RabbitMQ Management Plugin: `http://:15672`
* RabbitMQ Prometheus Plugin: `http://:15692/metrics`
### Semantics Service ### Container Service
It is designed to map terms in the domain of units of measurement to controlled vocabulary, modelled in It is responsible for Docker container lifecycle operations and updating the local copy of the Docker images.
the [ontology of units of measure](https://github.com/HajoRijgersberg/OM). This service validates researcher provided in
units and provides a *uniform resource identifier* (URI) to the related concept, which will be stored in the system.
Furthermore, there is a method for auto-completing text and listing a description as well as commonly used unit symbols.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 5010 * Ports: 9091/tcp
* Docker Image: [dbrepo/semantics-service](https://hub.docker.com/repository/docker/dbrepo/semantics-service) * Info: `http://:9091/actuator/info`
* Swagger UI: [/swagger/semantics](/swagger/semantics) * Health: `http://:9091/actuator/health`
* Prometheus: `http://:9091/actuator/prometheus`
* Swagger UI: `http://:9091/swagger-ui/index.html` [:fontawesome-solid-square-up-right: view online](/swagger/container)
### Identifier Service ### Database Service
This microservice is responsible for creating and resolving a *persistent identifier* (PID) attached to a query to It creates the databases inside a Docker container and the Query Store. Currently, we only
obtain the metadata attached to it and allow re-execution of a query. We store both the query and hashes of the query support [MariaDB](https://mariadb.org/) images that allow table versioning with low programmatic effort.
and result set to allow equality checks of the originally obtained result set and the currently obtained result set. In
the reference implementation we currently only use a numerical id column and plan to integrate *digital object
identifier* (DOI) through our institutional library soon.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9096 * Ports: 9092/tcp
* Docker Image: [dbrepo/identifier-service](https://hub.docker.com/repository/docker/dbrepo/identifier-service) * Info: `http://:9092/actuator/info`
* Swagger UI: [/swagger/identifier](/swagger/identifier) * Health: `http://:9092/actuator/health`
* Prometheus: `http://:9092/actuator/prometheus`
* Swagger UI: `http://:9092/swagger-ui/index.html` [:fontawesome-solid-square-up-right: view online](/swagger/database)
### Search Service ### Discovery Service
It processes search requests from the Gateway Service for full-text lookups in the Metadata Database. We use This microservice allows service discovery and registration of containers that provide services. It configures
[Elasticsearch](https://www.elastic.co/) in the reference implementation. a [Spring Cloud Netflix Eureka Server](https://cloud.spring.io/spring-cloud-netflix/reference/html/) to discover
services.
The Search Service implements ElasticSearch and creates a retrievable index on all databases that is getting updated !!! debug "Debug Information"
with each save operation on databases in the metadata database. The database name can be queried with ElasticSearch
to e.g. match the term "Airquality" * Ports: 9090/tcp
* Info: `http://:9090/actuator/info`
* Health: `http://:9090/actuator/health`
* Prometheus: `http://:9090/actuator/prometheus`
* Eureka Dashboard: `http://:9090/`
```console ### Gateway Service
curl http://localhost:9200/databaseindex/_search?q=name:Airquality
``` Provides a single point of access to the *application programming interface* (API) and configures
the [Spring Cloud Gateway](https://spring.io/projects/spring-cloud-gateway) to route traffic to the services.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9200, 9600 * Ports: 9095/tcp
* Docker Image: [elasticsearch](https://hub.docker.com/_/elasticsearch) * Info: `http://:9095/actuator/info`
* ElasticSearch Index * Health: `http://:9095/actuator/health`
* Prometheus: `http://:9095/actuator/prometheus`
### Container Service ### Identifier Service
It is responsible for Docker container lifecycle operations and updating the local copy of the Docker images. This microservice is responsible for creating and resolving a *persistent identifier* (PID) attached to a query to
obtain the metadata attached to it and allow re-execution of a query. We store both the query and hashes of the query
and result set to allow equality checks of the originally obtained result set and the currently obtained result set. In
the reference implementation we currently only use a numerical id column and plan to integrate *digital object
identifier* (DOI) through our institutional library soon.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9091 * Ports: 9096/tcp
* Docker Image: [dbrepo/container-service](https://hub.docker.com/repository/docker/dbrepo/container-service) * Info: `http://:9096/actuator/info`
* Swagger UI: [/swagger/container](/swagger/container) * Health: `http://:9096/actuator/health`
* Prometheus: `http://:9096/actuator/prometheus`
* Swagger UI: `http://:9096/swagger-ui/index.html` [:fontawesome-solid-square-up-right: view online](/swagger/identifier)
### Database Service ### Metadata Database
It creates the databases inside a Docker container and the Query Store. Currently we only It is the core component of the project. It is a relational database that contains metadata about all researcher databases
support [MariaDB](https://mariadb.org/) images that allow table versioning with low programmatic effort. created in the database repository like column names, check expressions, value enumerations or key/value constraints and
relevant data for citing data sets. Additionally, the concept, e.g. URI of units of measurements of numerical columns is
stored in the Metadata Database in order to provide semantic knowledge context. We use MariaDB for its rich capabilities
in the reference implementation.
The default credentials are `root:dbrepo` for the database `fda`. Connect to the database via the JDBC connector on port `3306`.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9092 * Ports: 3306/tcp, 9100/tcp
* Docker Image: [dbrepo/database-service](https://hub.docker.com/repository/docker/dbrepo/database-service) * Prometheus: `http://:9100/metrics`
* Swagger UI: [/swagger/database](/swagger/database)
### Table Service ### Metadata Service
This microservice handles table operations inside a database that is managed by the Database Service. We This service provides an OAI-PMH endpoint for metadata crawler.
use [Hibernate](https://hibernate.org/orm/) for schema and data ingest operations.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9094 * Ports: 9099/tcp
* Docker Image: [dbrepo/table-service](https://hub.docker.com/repository/docker/dbrepo/table-service) * Info: `http://:9099/actuator/info`
* Swagger UI: [/swagger/table](/swagger/table) * Health: `http://:9099/actuator/health`
* Prometheus: `http://:9099/actuator/prometheus`
* Swagger UI: `http://:9099/swagger-ui/index.html` [:fontawesome-solid-square-up-right: view online](/swagger/metadata)
### Broker Service ### Proxy
It holds exchanges and topics responsible for holding AMQP messages for later consumption. We The NGINX reverse proxy bundles the services and enables SSL/TLS communication for all endpoints.
use [RabbitMQ](https://www.rabbitmq.com/) in the reference implementation. The AMQP endpoint listens to port 5672 for
regular declares and offers a management interface at port 15672.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9098, 5672, 15672 * Ports: 80/tcp, 443/tcp
* Docker Image: [dbrepo/broker-service](https://hub.docker.com/repository/docker/dbrepo/broker-service)
* RabbitMQ Management Plugin
### Query Service ### Query Service
...@@ -167,67 +170,70 @@ Service. ...@@ -167,67 +170,70 @@ Service.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9093 * Ports: 9093/tcp
* Docker Image: [dbrepo/query-service](https://hub.docker.com/repository/docker/dbrepo/query-service) * Info: `http://:9093/actuator/info`
* Swagger UI: [/swagger/query](/swagger/query) * Health: `http://:9093/actuator/health`
### FAIR Portal * Prometheus: `http://:9093/actuator/prometheus`
* Swagger UI: `http://:9093/swagger-ui/index.html` [:fontawesome-solid-square-up-right: view online](/swagger/query)
It provides a *graphical user interface* (GUI) for a researcher to interact with the database repository's API. ### Search Service
It processes search requests from the Gateway Service for full-text lookups in the Metadata Database. We use
[Elasticsearch](https://www.elastic.co/) in the reference implementation. The Search Service implements Elastic Search
and creates a retrievable index on all databases that is getting updated with each save operation on databases in the
metadata database.
All requests need to be authenticated, by default the credentials `elastic:elastic` are used.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 3000 * Ports: 9200/tcp
* Docker Image: [dbrepo/ui](https://hub.docker.com/repository/docker/dbrepo/ui) * Indizes: `http://:9200/_all`
* FAIR Portal
### Analyse Service ### Semantics Service
It suggests data types for the FAIR Portal when creating a table from a *comma separated values* (CSV) file. It It is designed to map terms in the domain of units of measurement to controlled vocabulary, modelled in
recommends enumerations for columns and returns e.g. a list of potential primary key candidates. The researcher is able the [ontology of units of measure](https://github.com/HajoRijgersberg/OM). This service validates researcher provided in
to confirm these suggestions manually. Moreover, the *Analyze Service* determines basic statistical properties of units and provides a *uniform resource identifier* (URI) to the related concept, which will be stored in the system.
numerical columns. Furthermore, there is a method for auto-completing text and listing a description as well as commonly used unit symbols.
!!! debug "Debug Information"
* Ports: 5010/tcp
* Prometheus: `http://:5010/metrics`
* Swagger UI: `http://:5010/swagger-ui/index.html` [:fontawesome-solid-square-up-right: view online](/swagger/semantics)
### Table Service
This microservice handles table operations inside a database that is managed by the Database Service. We
use [Hibernate](https://hibernate.org/orm/) for schema and data ingest operations.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 5000 * Ports: 9094/tcp
* Docker Image: [dbrepo/analyse-service](https://hub.docker.com/repository/docker/dbrepo/analyse-service) * Info: `http://:9094/actuator/info`
* Swagger UI: [/swagger/analyse](/swagger/analyse) * Health: `http://:9094/actuator/health`
* Prometheus: `http://:9094/actuator/prometheus`
* Swagger UI: `http://:9094/swagger-ui/index.html` [:fontawesome-solid-square-up-right: view online](/swagger/table)
## Database ### UI
This container runs a relational database engine that allows data versioning and contains the Query Store, a special It provides a *graphical user interface* (GUI) for a researcher to interact with the database repository's API.
table that stores all queries issued to the Researcher Database along with metadata. We store the queries here and not
in the metadata database level to ensure that they are preserved along with the original database for a regular backup
and archival together with the original database once the container is retired.
### Container !!! debug "Debug Information"
Currently, we only support databases with * Ports: 3000/tcp, 9100/tcp
the [MariaDB engine](https://hub.docker.com/_/mariadb?tab=tags&page=1&name=10.5&ordering=-name). * Prometheus: `http://:9100/metrics`
DBRepo creates a *root* user for managing the tables, inserting data, etc. and provides a *mariadb* user that is only * UI: `http://:3000/`
granted `select` access to all tables. The default passwords need to be changed at
[`AbstractSeeder.java`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/blob/master/fda-container-service/services/src/main/java/at/tuwien/seeder/impl/AbstractSeeder.java#L39-L51)
### Query Store ### User Service
The Query Store is a special table (`qs_queries`) that stores all queries issued to the database via the HTTP API. It This microservice handles user information.
stores meta-information about the queries directly in the database container:
<figure markdown> !!! debug "Debug Information"
| Name | Type | Constraint | Default | Comment |
|------------------|--------------|-------------|-------------------------|-------------------------------|
| id | bigint | primary key | nextval(qs_queries_seq) | |
| cid | bigint | | | Column ID |
| dbid | bigint | | | Database ID |
| created | datetime | | now() | |
| created_by | bigint | | | Creator User-ID |
| execution | datetime | | | |
| last_modified | datetime | | | |
| query | text | | | |
| query_normalized | text | | | removing *, randomness |
| query_hash | varchar(255) | | | sha256 hash of `query` field |
| result_hash | varchar(255) | | | sha256 hash of the result set |
| result_number | bigint | | | |
<figcaption>Query Store table <code>qs_queries</code> schema</figcaption> * Ports: 9098/tcp
</figure> * Info: `http://:9098/actuator/info`
* Health: `http://:9098/actuator/health`
* Prometheus: `http://:9098/actuator/prometheus`
* Swagger UI: `http://:9098/swagger-ui/index.html` [:fontawesome-solid-square-up-right: view online](/swagger/user)
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment