Skip to content
Snippets Groups Projects
Unverified Commit 7a90993d authored by Martin Weise's avatar Martin Weise
Browse files

Added more info on getting started

parent 87606873
No related branches found
No related tags found
No related merge requests found
Pipeline #2422 passed
Stage: configure
Stage: build
Stage: deploy
Show whitespace changes
Inline Side-by-side
docs/contact.md
+ 1
− 0
View file @ 7a90993d
...@@ -32,5 +32,6 @@ A-1040 Vienna, Austria ...@@ -32,5 +32,6 @@ A-1040 Vienna, Austria
- Cornelia Michlits - Cornelia Michlits
- Eva Gergely - Eva Gergely
- Kirill Stytsenko - Kirill Stytsenko
- Josef Taha
- Raman Ganguly - Raman Ganguly
- Andreas Rauber - Andreas Rauber
docs/getting-started.md
+ 50
− 11
View file @ 7a90993d
...@@ -48,19 +48,23 @@ We currently maintain 14 images on [Dockerhub](https://hub.docker.com/u/dbrepo). ...@@ -48,19 +48,23 @@ We currently maintain 14 images on [Dockerhub](https://hub.docker.com/u/dbrepo).
[//]: # (=== ":fontawesome-brands-windows: Windows") [//]: # (=== ":fontawesome-brands-windows: Windows")
[//]: # () [//]: # ()
[//]: # ( Download the latest [`docker-compose.yml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/master/docker-compose.prod.yml) ) [//]: # ( Download the latest [`docker-compose.yml`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/master/docker-compose.prod.yml) )
[//]: # ( and [`.env`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/master/.env.win.example) ) [//]: # ( and [`.env`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/master/.env.win.example) )
[//]: # ( ) [//]: # ( )
[//]: # ( $ curl.exe -o docker-compose.yml https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/master/docker-compose.prod.yml) [//]: # ( $ curl.exe -o docker-compose.yml https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/master/docker-compose.prod.yml)
[//]: # ( $ curl.exe -o .env https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/master/.env.win.example) [//]: # ( $ curl.exe -o .env https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/raw/master/.env.win.example)
[//]: # () [//]: # ()
[//]: # ( Start the services) [//]: # ( Start the services)
[//]: # () [//]: # ()
[//]: # ( $ docker-compose up -d) [//]: # ( $ docker-compose up -d)
=== ":fontawesome-brands-apple: macOS" === ":fontawesome-brands-apple: macOS"
...@@ -92,8 +96,7 @@ Activate them both in your virtualization infrastructure and the local firewall ...@@ -92,8 +96,7 @@ Activate them both in your virtualization infrastructure and the local firewall
## Development ## Development
We invite all open-source developers to help us fixing bugs and introducing features to the source code. Get involved by We invite all open-source developers to help us fixing bugs and introducing features to the source code. Get involved by
sending a mail to Prof. Andreas Rauber and Projektass. Martin Weise. Clone the repository and create a feature branch sending a mail to Prof. Andreas Rauber and Projektass. Martin Weise.
from `dev` and implement your changes.
### Software ### Software
...@@ -108,22 +111,58 @@ We develop all packages with the following software requirements: ...@@ -108,22 +111,58 @@ We develop all packages with the following software requirements:
### Building ### Building
For local development you need to install the entities from the metadata database and the general DTOs that are #### Services and GUI
exchanged between the services by installing the package:
For local development you need to clone the repository and build the services from scratch:
```console ```console
mvn -f ./fda-metadata-db/pom.xml clean install $ git clone git@gitlab.phaidra.org:fair-data-austria-db-repository/fda-services.git
$ make build
``` ```
All other services depend on this package. You have to build them later: #### Swagger Codegen
It is possible to use the Swagger documentation of the services to generate HTTP clients to interact with DBRepo.
Clients can only begenerated when DBRepo is running. Just download
the [`swagger-codegen-cli.jar`](https://repo1.maven.org/maven2/io/swagger/swagger-codegen-cli/) and start the services:
```console ```console
mvn -f ./fda-container-service/pom.xml clean package -DskipTests $ docker-compose up -d
... ```
Run the codegen tool:
```console
$ java -jar ./swagger-codegen-cli-2.x.y.jar generate -i ./api-docs.yaml -l python -o ./api-container
```
### Debugging
#### Backend
Start all containers with docker-compose:
```console
$ docker-compose up -d
$ docker container stop fda-xxxxx-service # <<< replace with the service you want to debug
$ docker container rm fda-xxxxx-service # <<< replace with the service you want to debug
$ docker-compose logs -f
```
Then start the container in your IDE and debug it.
#### Frontend
Start all containers with docker-compose:
```console
$ docker-compose up -d
$ docker container stop fda-ui && docker container rm fda-ui
$ docker-compose logs -f
``` ```
Conviniently, you can build them all through [`make`](https://www.gnu.org/software/make/): Then start the frontend in another console window:
```console ```console
make build-backend $ yarn --cwd=./fda-ui dev
``` ```
docs/system.md
+ 68
− 68
View file @ 7a90993d
...@@ -13,45 +13,6 @@ The repository is designed as a microservice architecture to ensure scalability ...@@ -13,45 +13,6 @@ The repository is designed as a microservice architecture to ensure scalability
technologies. The conceptualized microservices operate the basic database operations, data versioning as well as technologies. The conceptualized microservices operate the basic database operations, data versioning as well as
*findability*, *accessability*, *interoperability* and *reuseability* (FAIR). *findability*, *accessability*, *interoperability* and *reuseability* (FAIR).
## Database
This container runs a relational database engine that allows data versioning and contains the Query Store, a special
table that stores all queries issued to the Researcher Database along with metadata. We store the queries here and not
in the metadata database level to ensure that they are preserved along with the original database for a regular backup
and archival together with the original database once the container is retired.
### Container
Currently, we only support databases with
the [MariaDB engine](https://hub.docker.com/_/mariadb?tab=tags&page=1&name=10.5&ordering=-name).
DBRepo creates a *root* user for managing the tables, inserting data, etc. and provides a *mariadb* user that is only
granted `select` access to all tables. The default passwords need to be changed at
[`AbstractSeeder.java`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/blob/master/fda-container-service/services/src/main/java/at/tuwien/seeder/impl/AbstractSeeder.java#L39-L51)
### Query Store
The Query Store is a special table (`qs_queries`) that stores all queries issued to the database via the HTTP API. It
stores meta-information about the queries directly in the database container:
<figure markdown>
| Name | Type | Constraint | Default | Comment |
|------------------|--------------|-------------|-------------------------|-------------------------------|
| id | bigint | primary key | nextval(qs_queries_seq) | |
| cid | bigint | | | Column ID |
| dbid | bigint | | | Database ID |
| created | datetime | | now() | |
| created_by | bigint | | | Creator User-ID |
| execution | datetime | | | |
| last_modified | datetime | | | |
| query | text | | | |
| query_normalized | text | | | removing *, randomness |
| query_hash | varchar(255) | | | sha256 hash of `query` field |
| result_hash | varchar(255) | | | sha256 hash of the result set |
| result_number | bigint | | | |
<figcaption>Query Store table <code>qs_queries</code> schema</figcaption>
</figure>
## Services ## Services
### Discovery Service ### Discovery Service
...@@ -109,9 +70,9 @@ that has permissions for writing and configuring the RabbitMQ queues that feed i ...@@ -109,9 +70,9 @@ that has permissions for writing and configuring the RabbitMQ queues that feed i
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9097 * Port(s): 9097
* Swagger UI: [/swagger-ui/index.html](http://localhost:9097/swagger-ui/index.html) * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9097/swagger-ui/index.html)
* Swagger API .json: [/v3/api-docs/authentication-service](http://localhost:9097/v3/api-docs/authentication-service) * Swagger API .json: [/v3/api-docs/authentication-service](http://dbrepo.ossdip.at:9097/v3/api-docs/authentication-service)
* Swagger API .yaml: [/v3/api-docs.yaml](http://localhost:9097/v3/api-docs.yaml) * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9097/v3/api-docs.yaml)
### Metadata Database ### Metadata Database
...@@ -164,8 +125,8 @@ $ curl -X POST -d '{"cdbid":1,"tid":1,"cid":1,"uri":...}' https://dbrepo.ossdip. ...@@ -164,8 +125,8 @@ $ curl -X POST -d '{"cdbid":1,"tid":1,"cid":1,"uri":...}' https://dbrepo.ossdip.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 5010 * Port(s): 5010
* Swagger UI: [/swagger-ui/](http://localhost:9097/swagger-ui/) * Swagger UI: [/swagger-ui/](http://dbrepo.ossdip.at:9097/swagger-ui/)
* Swagger API .json: [/api-units.json](http://localhost:9097/api-units.json) * Swagger API .json: [/api-units.json](http://dbrepo.ossdip.at:9097/api-units.json)
### Identifier Service ### Identifier Service
...@@ -178,9 +139,9 @@ identifier* (DOI) through our institutional library soon. ...@@ -178,9 +139,9 @@ identifier* (DOI) through our institutional library soon.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9096 * Port(s): 9096
* Swagger UI: [/swagger-ui/index.html](http://localhost:9096/swagger-ui/index.html) * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9096/swagger-ui/index.html)
* Swagger API .json: [/v3/api-docs/identifier-service](http://localhost:9096/v3/api-docs/identifier-service) * Swagger API .json: [/v3/api-docs/identifier-service](http://dbrepo.ossdip.at:9096/v3/api-docs/identifier-service)
* Swagger API .yaml: [/v3/api-docs.yaml](http://localhost:9096/v3/api-docs.yaml) * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9096/v3/api-docs.yaml)
### Search Service ### Search Service
...@@ -192,13 +153,13 @@ with each save operation on databases in the metadata database. The database nam ...@@ -192,13 +153,13 @@ with each save operation on databases in the metadata database. The database nam
to e.g. match the term "Airquality" to e.g. match the term "Airquality"
```console ```console
$ curl http://localhost:9200/databaseindex/_search?q=name:Airquality $ curl http://dbrepo.ossdip.at:9200/databaseindex/_search?q=name:Airquality
``` ```
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9200, 9600 * Port(s): 9200, 9600
* ElasticSearch: [/databaseindex](http://localhost:9000/databaseindex) * ElasticSearch: [/databaseindex](http://dbrepo.ossdip.at:9000/databaseindex)
### Container Service ### Container Service
...@@ -207,9 +168,9 @@ It is responsible for Docker container lifecycle operations and updating the loc ...@@ -207,9 +168,9 @@ It is responsible for Docker container lifecycle operations and updating the loc
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9091 * Port(s): 9091
* Swagger UI: [/swagger-ui/index.html](http://localhost:9091/swagger-ui/index.html) * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9091/swagger-ui/index.html)
* Swagger API .json: [/v3/api-docs/container-service](http://localhost:9091/v3/api-docs/container-service) * Swagger API .json: [/v3/api-docs/container-service](http://dbrepo.ossdip.at:9091/v3/api-docs/container-service)
* Swagger API .yaml: [/v3/api-docs.yaml](http://localhost:9091/v3/api-docs.yaml) * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9091/v3/api-docs.yaml)
### Database Service ### Database Service
...@@ -219,9 +180,9 @@ support [MariaDB](https://mariadb.org/) images that allow table versioning with ...@@ -219,9 +180,9 @@ support [MariaDB](https://mariadb.org/) images that allow table versioning with
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9092 * Port(s): 9092
* Swagger UI: [/swagger-ui/index.html](http://localhost:9092/swagger-ui/index.html) * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9092/swagger-ui/index.html)
* Swagger API .json: [/v3/api-docs/database-service](http://localhost:9092/v3/api-docs/database-service) * Swagger API .json: [/v3/api-docs/database-service](http://dbrepo.ossdip.at:9092/v3/api-docs/database-service)
* Swagger API .yaml: [/v3/api-docs.yaml](http://localhost:9092/v3/api-docs.yaml) * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9092/v3/api-docs.yaml)
### Table Service ### Table Service
...@@ -231,9 +192,9 @@ use [Hibernate](https://hibernate.org/orm/) for schema and data ingest operation ...@@ -231,9 +192,9 @@ use [Hibernate](https://hibernate.org/orm/) for schema and data ingest operation
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9094 * Port(s): 9094
* Swagger UI: [/swagger-ui/index.html](http://localhost:9094/swagger-ui/index.html) * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9094/swagger-ui/index.html)
* Swagger API .json: [/v3/api-docs/table-service](http://localhost:9094/v3/api-docs/table-service) * Swagger API .json: [/v3/api-docs/table-service](http://dbrepo.ossdip.at:9094/v3/api-docs/table-service)
* Swagger API .yaml: [/v3/api-docs.yaml](http://localhost:9094/v3/api-docs.yaml) * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9094/v3/api-docs.yaml)
### Broker Service ### Broker Service
...@@ -250,10 +211,10 @@ regular declares and offers a management interface at port 15672. ...@@ -250,10 +211,10 @@ regular declares and offers a management interface at port 15672.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9098, 5672, 15672 * Port(s): 9098, 5672, 15672
* Swagger UI: [/swagger-ui/index.html](http://localhost:9098/swagger-ui/index.html) * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9098/swagger-ui/index.html)
* Swagger API .json: [/v3/api-docs/broker-service](http://localhost:9098/v3/api-docs/broker-service) * Swagger API .json: [/v3/api-docs/broker-service](http://dbrepo.ossdip.at:9098/v3/api-docs/broker-service)
* Swagger API .yaml: [/v3/api-docs.yaml](http://localhost:9098/v3/api-docs.yaml) * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9098/v3/api-docs.yaml)
* RabbitMQ Management: [/#](http://localhost:15672/#) * RabbitMQ Management: [/#](http://dbrepo.ossdip.at:15672/#)
### Query Service ### Query Service
...@@ -264,9 +225,9 @@ Service. ...@@ -264,9 +225,9 @@ Service.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 9093 * Port(s): 9093
* Swagger UI: [/swagger-ui/index.html](http://localhost:9093/swagger-ui/index.html) * Swagger UI: [/swagger-ui/index.html](http://dbrepo.ossdip.at:9093/swagger-ui/index.html)
* Swagger API .json: [/v3/api-docs/query-service](http://localhost:9093/v3/api-docs/query-service) * Swagger API .json: [/v3/api-docs/query-service](http://dbrepo.ossdip.at:9093/v3/api-docs/query-service)
* Swagger API .yaml: [/v3/api-docs.yaml](http://localhost:9093/v3/api-docs.yaml) * Swagger API .yaml: [/v3/api-docs.yaml](http://dbrepo.ossdip.at:9093/v3/api-docs.yaml)
### FAIR Portal ### FAIR Portal
...@@ -275,7 +236,7 @@ It provides a *graphical user interface* (GUI) for a researcher to interact with ...@@ -275,7 +236,7 @@ It provides a *graphical user interface* (GUI) for a researcher to interact with
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 3000 * Port(s): 3000
* GUI: [/#](http://localhost:3000) * GUI: [/#](http://dbrepo.ossdip.at:3000)
### Analyse Service ### Analyse Service
...@@ -287,5 +248,44 @@ numerical columns. ...@@ -287,5 +248,44 @@ numerical columns.
!!! debug "Debug Information" !!! debug "Debug Information"
* Port(s): 5000 * Port(s): 5000
* Swagger UI: [/swagger-ui/](http://localhost:5000/swagger-ui/) * Swagger UI: [/swagger-ui/](http://dbrepo.ossdip.at:5000/swagger-ui/)
* Swagger API .json: [/api-analyze.json](http://localhost:5000/api-analyze.json) * Swagger API .json: [/api-analyze.json](http://dbrepo.ossdip.at:5000/api-analyze.json)
## Database
This container runs a relational database engine that allows data versioning and contains the Query Store, a special
table that stores all queries issued to the Researcher Database along with metadata. We store the queries here and not
in the metadata database level to ensure that they are preserved along with the original database for a regular backup
and archival together with the original database once the container is retired.
### Container
Currently, we only support databases with
the [MariaDB engine](https://hub.docker.com/_/mariadb?tab=tags&page=1&name=10.5&ordering=-name).
DBRepo creates a *root* user for managing the tables, inserting data, etc. and provides a *mariadb* user that is only
granted `select` access to all tables. The default passwords need to be changed at
[`AbstractSeeder.java`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services/-/blob/master/fda-container-service/services/src/main/java/at/tuwien/seeder/impl/AbstractSeeder.java#L39-L51)
### Query Store
The Query Store is a special table (`qs_queries`) that stores all queries issued to the database via the HTTP API. It
stores meta-information about the queries directly in the database container:
<figure markdown>
| Name | Type | Constraint | Default | Comment |
|------------------|--------------|-------------|-------------------------|-------------------------------|
| id | bigint | primary key | nextval(qs_queries_seq) | |
| cid | bigint | | | Column ID |
| dbid | bigint | | | Database ID |
| created | datetime | | now() | |
| created_by | bigint | | | Creator User-ID |
| execution | datetime | | | |
| last_modified | datetime | | | |
| query | text | | | |
| query_normalized | text | | | removing *, randomness |
| query_hash | varchar(255) | | | sha256 hash of `query` field |
| result_hash | varchar(255) | | | sha256 hash of the result set |
| result_number | bigint | | | |
<figcaption>Query Store table <code>qs_queries</code> schema</figcaption>
</figure>
swagger/api-query.yaml
+ 4
− 4
View file @ 7a90993d
...@@ -21,7 +21,7 @@ paths: ...@@ -21,7 +21,7 @@ paths:
tags: tags:
- table-history-endpoint - table-history-endpoint
summary: Find all history summary: Find all history
operationId: getAll operationId: getAll_1
parameters: parameters:
- name: id - name: id
in: path in: path
...@@ -86,7 +86,7 @@ paths: ...@@ -86,7 +86,7 @@ paths:
tags: tags:
- table-history-endpoint - table-history-endpoint
summary: Find all history summary: Find all history
operationId: getAll_1 operationId: getAll
parameters: parameters:
- name: id - name: id
in: path in: path
...@@ -152,7 +152,7 @@ paths: ...@@ -152,7 +152,7 @@ paths:
tags: tags:
- table-data-endpoint - table-data-endpoint
summary: Find data summary: Find data
operationId: getAll_2 operationId: getAll_3
parameters: parameters:
- name: id - name: id
in: path in: path
...@@ -428,7 +428,7 @@ paths: ...@@ -428,7 +428,7 @@ paths:
tags: tags:
- table-data-endpoint - table-data-endpoint
summary: Find data summary: Find data
operationId: getAll_3 operationId: getAll_2
parameters: parameters:
- name: id - name: id
in: path in: path
......
swagger/generate.sh
+ 4
− 4
View file @ 7a90993d
...@@ -25,20 +25,20 @@ function generate () { ...@@ -25,20 +25,20 @@ function generate () {
function remove () { function remove () {
echo "... removing old python api" echo "... removing old python api"
rm -rf "${HOME}/Projects/fda-services/.jupyter/api_$1" rm -rf "${HOME}/Projects/fda-services/.gitlab/tests/api_$1"
rm -rf "${HOME}/Projects/fda-services/.demo/api_$1" rm -rf "${HOME}/Projects/fda-services/.demo/api_$1"
} }
function copy () { function copy () {
echo "... copying python api" echo "... copying python api"
cp -r "./api-$1/swagger_client" "${HOME}/Projects/fda-services/.jupyter/api_$1" cp -r "./api-$1/swagger_client" "${HOME}/Projects/fda-services/.gitlab/tests/api_$1"
cp -r "./api-$1/swagger_client" "${HOME}/Projects/fda-services/.demo/api_$1" cp -r "./api-$1/swagger_client" "${HOME}/Projects/fda-services/.demo/api_$1"
} }
function replace () { function replace () {
echo "... replacing swagger client package name and gateway" echo "... replacing swagger client package name and gateway"
find "${HOME}/Projects/fda-services/.jupyter/api_$2" -type f -exec sed -i -e "s/swagger_client/api_$2/g" {} \; find "${HOME}/Projects/fda-services/.gitlab/tests/api_$2" -type f -exec sed -i -e "s/swagger_client/api_$2/g" {} \;
find "${HOME}/Projects/fda-services/.jupyter/api_$2" -type f -exec sed -i -e "s/self.host = .*/self.host = \"http:\/\/localhost:9095\"/g" {} \; find "${HOME}/Projects/fda-services/.gitlab/tests/api_$2" -type f -exec sed -i -e "s/self.host = .*/self.host = \"http:\/\/localhost:9095\"/g" {} \;
find "${HOME}/Projects/fda-services/.demo/api_$2" -type f -exec sed -i -e "s/swagger_client/api_$2/g" {} \; find "${HOME}/Projects/fda-services/.demo/api_$2" -type f -exec sed -i -e "s/swagger_client/api_$2/g" {} \;
find "${HOME}/Projects/fda-services/.demo/api_$2" -type f -exec sed -i -e "s/self.host = .*/self.host = \"http:\/\/localhost:9095\"/g" {} \; find "${HOME}/Projects/fda-services/.demo/api_$2" -type f -exec sed -i -e "s/self.host = .*/self.host = \"http:\/\/localhost:9095\"/g" {} \;
} }
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment