Skip to content
Snippets Groups Projects
Commit 33992e84 authored by Martin Weise's avatar Martin Weise
Browse files

Merge branch '399-enable-database-dump-import' into 'dev' 

Resolve "Enable database dump import"

See merge request !220
parents bc0140bc 1987a834
Branches
Tags
4 merge requests!231CI: Remove build for log-service,!228Better error message handling in the frontend,!223Release of version 1.4.0,!220Resolve "Enable database dump import"
Expand all Show whitespace changes
Inline Side-by-side
Showing
with 2948 additions and 3025 deletions
.docs/DEVELOPMENT.md 0 → 100644
+ 81
− 0
View file @ 33992e84
# Development Guide
## Dependencies
Local development depends on the following packages:
* [Apache Maven](https://maven.apache.org/) 3+
* [Java JDK](https://openjdk.org/) 17 (LTS)
* [Docker Engine](https://docs.docker.com/engine/install/) 24+
Optional but recommended:
* [GNU Make](https://www.gnu.org/software/make/) 4+
## Getting Started
The Java-based services share some classes from the `metadata-service`. You need to install them locally as Maven
library via:
```shell
mvn -f ./dbrepo-metadata-service/pom.xml clean install -DskipTests
```
## Testing
We practice test-driven development and require contributors to test their code with at least 90% code coverage.
## Code Versioning
### CI/CD
We get compute resources in-kind from [dataLAB](https://www.it.tuwien.ac.at/en/services/network-and-servers/datalab)
to run our pipeline:
<p align="center">
<img src="../.gitlab/gitlab-runner.png" alt="Gitlab runner configuration in the cluster" width="732" height="262" /><br/>
<i><strong>Figure 1.</strong> Gitlab runner configuration in the cluster.</i>
</p>
Minikube cluster with 6vCPU and 28GB RAM. The CI pipeline is configured as follows in the `config.toml`:
```toml
concurrent = 10
[[runners]]
executor = "kubernetes"
environment = [
"FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false"
]
[runners.kubernetes]
namespace = "{{.Release.Namespace}}"
privileged = true
allowed_services = ["docker:24-dind"]
[[runners.kubernetes.services]]
name = "docker:24-dind"
command = [ "--registry-mirror=http://docker-io-mirror:80", "--insecure-registry=docker-io-mirror:80", "--registry-mirror=http://gcr-io-mirror:80", "--insecure-registry=gcr-io-mirror:80", "--registry-mirror=http://ghcr-io-mirror:80", "--insecure-registry=ghcr-io-mirror:80", "--registry-mirror=http://k8s-io-mirror:80", "--insecure-registry=k8s-io-mirror:80" ]
alias = "docker"
[[runners.kubernetes.volumes.empty_dir]]
name = "rundind"
mount_path = "/var/run/dind"
medium = "Memory"
[[runners.kubernetes.volumes.empty_dir]]
name = "tmp"
mount_path = "/tmp"
medium = "Memory"
```
For each job in the CI/CD pipeline, a pod with three containers is started:
1. `build` the main build container, you can *freely* specify any image with `image: xyz` as base
2. `helper` the default helper container.
3. `svc-0` the Docker-in-Docker sidecar (rootless executed as user `rootless`/`1000`) exposing the Docker socket to the
`build` container under `
*Note.* Only Docker-in-Docker (dind) is allowed as service in the pipeline currently. For each job, a
dind-sidecar container `svc-0` is started that exposes the Docker socket at `/var/run/dind/docker.sock` in the `build`
container you can freely configure how you want. We are aware this is not optimal as it exposes *root* privileges in the
cluster.
The full CI/CD pipeline Helm chart is documented in
the [`fda-deployment`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-deployment/-/tree/master/charts/dbrepo-devops)
repository.
\ No newline at end of file
.docs/api-analyse.yaml
+ 1
− 1
View file @ 33992e84
{"definitions":{},"externalDocs":{"description":"Sourcecode Documentation","url":"https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services"},"info":{"contact":{"email":"andreas.rauber@tuwien.ac.at","name":"Prof. Andreas Rauber"},"description":"Service that analyses data structures","license":{"name":"Apache 2.0","url":"https://www.apache.org/licenses/LICENSE-2.0"},"title":"Database Repository Analyse Service API","version":"1.3.0"},"openapi":"3.0.0","paths":{"/api/analyse/determinedt":{"post":{"consumes":["application/json"],"description":"This is a simple API which returns the datatypes of a (path) csv file","parameters":[{"description":"to-do description","in":"body","name":"body","required":true,"schema":{"properties":{"enum":{"example":true,"type":"boolean"},"enum_tol":{"example":0.1},"filepath":{"example":"/data/testdt08.csv","type":"string"},"separator":{"example":",","type":"string"}},"type":"object"}}],"produces":["application/json"],"responses":{"200":{"description":"OK"},"405":{"description":"Invalid input"}},"summary":"Determine datatypes"}},"/api/analyse/determinepk":{"post":{"consumes":["application/json"],"description":"This is a simple API which returns the primary keys + ranking of a (path) csv file","parameters":[{"description":"to-do description","in":"body","name":"body","required":true,"schema":{"properties":{"filepath":{"example":"/data/testdt08.csv","type":"string"},"seperator":{"example":",","type":"string"}},"type":"object"}}],"produces":["application/json"],"responses":{"200":{"description":"OK"},"405":{"description":"Invalid input"}},"summary":"Determine primary keys"}},"/health":{"get":{"consumes":["application/json"],"description":"This is a simple API which checks if the application is healthy","parameters":[{"description":"to-do description","in":"body","name":"body","required":true,"schema":{"properties":{"status":{"example":"UP","type":"string"}},"type":"object"}}],"produces":["application/json"],"responses":{"200":{"description":"OK"}},"summary":"Check if application is running"}}},"servers":[{"description":"Generated server url","url":"http://localhost:5000"},{"description":"Sandbox","url":"https://dbrepo2.tuwien.ac.at"}]}
{"definitions":{},"externalDocs":{"description":"Sourcecode Documentation","url":"https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services"},"info":{"contact":{"email":"andreas.rauber@tuwien.ac.at","name":"Prof. Andreas Rauber"},"description":"Service that analyses data structures","license":{"name":"Apache 2.0","url":"https://www.apache.org/licenses/LICENSE-2.0"},"title":"Database Repository Analyse Service API","version":"1.3.0"},"openapi":"3.0.0","paths":{"/api/analyse/determinedt":{"post":{"consumes":["application/json"],"description":"This is a simple API which returns the datatypes of a (path) csv file","parameters":[{"description":"to-do description","in":"body","name":"body","required":true,"schema":{"properties":{"enum":{"example":true,"type":"boolean"},"enum_tol":{"example":0.1},"filename":{"example":"sample.csv","type":"string"},"separator":{"example":",","type":"string"}},"type":"object"}}],"produces":["application/json"],"responses":{"200":{"description":"OK"},"405":{"description":"Invalid input"}},"summary":"Determine datatypes"}},"/api/analyse/determinepk":{"post":{"consumes":["application/json"],"description":"This is a simple API which returns the primary keys + ranking of a (path) csv file","parameters":[{"description":"to-do description","in":"body","name":"body","required":true,"schema":{"properties":{"filepath":{"example":"/data/testdt08.csv","type":"string"},"seperator":{"example":",","type":"string"}},"type":"object"}}],"produces":["application/json"],"responses":{"200":{"description":"OK"},"405":{"description":"Invalid input"}},"summary":"Determine primary keys"}},"/health":{"get":{"consumes":["application/json"],"description":"This is a simple API which checks if the application is healthy","parameters":[{"description":"to-do description","in":"body","name":"body","required":true,"schema":{"properties":{"status":{"example":"UP","type":"string"}},"type":"object"}}],"produces":["application/json"],"responses":{"200":{"description":"OK"}},"summary":"Check if application is running"}}},"servers":[{"description":"Generated server url","url":"http://localhost:5000"},{"description":"Sandbox","url":"https://dbrepo2.tuwien.ac.at"}]}
.docs/api-metadata.yaml
+ 2681
− 2715
View file @ 33992e84
This diff is collapsed.
.docs/api-mirror.yaml
+ 0
− 14
View file @ 33992e84
openapi: 3.0.1
info:
title: OpenAPI definition
version: v0
servers:
- url: http://localhost:9050
description: Generated server url
paths: {}
components:
securitySchemes:
bearerAuth:
type: http
scheme: bearer
bearerFormat: JWT
.env.unix.example
+ 1
− 29
View file @ 33992e84
DBREPO_CLIENT_ID="dbrepo-client"
DBREPO_CLIENT_SECRET="MUwRc7yfXSJwX8AdRMWaQC3Nep1VjwgG"
JWT_ISSUER="http://localhost/api/auth/realms/dbrepo"
JWT_PUBKEY="MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqqnHQ2BWWW9vDNLRCcxD++xZg/16oqMo/c1l+lcFEjjAIJjJp/HqrPYU/U9GvquGE6PbVFtTzW1KcKawOW+FJNOA3CGo8Q1TFEfz43B8rZpKsFbJKvQGVv1Z4HaKPvLUm7iMm8Hv91cLduuoWx6Q3DPe2vg13GKKEZe7UFghF+0T9u8EKzA/XqQ0OiICmsmYPbwvf9N3bCKsB/Y10EYmZRb8IhCoV9mmO5TxgWgiuNeCTtNCv2ePYqL/U0WvyGFW0reasIK8eg3KrAUj8DpyOgPOVBn3lBGf+3KFSYi+0bwZbJZWqbC/Xlk20Go1YfeJPRIt7ImxD27R/lNjgDO/MwIDAQAB"
JWT_CERT="MIICmzCCAYMCBgGG3GWyBTANBgkqhkiG9w0BAQsFADARMQ8wDQYDVQQDDAZkYnJlcG8wHhcNMjMwMzEzMTkxMzE3WhcNMzMwMzEzMTkxNDU3WjARMQ8wDQYDVQQDDAZkYnJlcG8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCqqcdDYFZZb28M0tEJzEP77FmD/Xqioyj9zWX6VwUSOMAgmMmn8eqs9hT9T0a+q4YTo9tUW1PNbUpwprA5b4Uk04DcIajxDVMUR/PjcHytmkqwVskq9AZW/Vngdoo+8tSbuIybwe/3Vwt266hbHpDcM97a+DXcYooRl7tQWCEX7RP27wQrMD9epDQ6IgKayZg9vC9/03dsIqwH9jXQRiZlFvwiEKhX2aY7lPGBaCK414JO00K/Z49iov9TRa/IYVbSt5qwgrx6DcqsBSPwOnI6A85UGfeUEZ/7coVJiL7RvBlsllapsL9eWTbQajVh94k9Ei3sibEPbtH+U2OAM78zAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAASnN1Cuif1sdfEK2kWAURSXGJCohCROLWdKFjaeHPRaEfpbFJsgxW0Yj3nwX5O3bUlOWoTyENwnXSsXMQsqnNi+At32CKaKO8+AkhAbgQL9F0B+KeJwmYv3cUj5N/LYkJjBvZBzUZ4Ugu5dcxH0k7AktLAIwimkyEnxTNolOA3UyrGGpREr8MCKWVr10RFuOpF/0CsJNNwbHXzalO9D756EUcRWZ9VSg6QVNso0YYRKTnILWDn9hcTRnqGy3SHo3anFTqQZ+BB57YbgFWy6udC0LYRB3zdp6zNti87eu/VEymiDY/mmo1AB8Tm0b6vxFz4AKcL3ax5qS6YnZ9efSzk="
SHARED_FILESYSTEM=/tmp
SEARCH_USERNAME=admin
SEARCH_PASSWORD=admin
METADATA_DB=fda
METADATA_USERNAME=root
METADATA_PASSWORD=dbrepo
AUTH_DB=keycloak
AUTH_USERNAME=root
AUTH_PASSWORD=dbrepo
BROKER_ENDPOINT=http://broker-service:15672/admin/broker
BROKER_USERNAME=fda
BROKER_PASSWORD=fda
KEYCLOAK_ADMIN=fda
KEYCLOAK_ADMIN_PASSWORD=fda
WEBSITE=http://localhost
LOG_LEVEL=trace # error, warning, info, debug, trace
\ No newline at end of file
DOI_URL="https://doi.org"
DATACITE_URL="https://api.datacite.org"
DATACITE_PREFIX="10.1234"
DATACITE_USERNAME="username"
DATACITE_PASSWORD="password"
USER_DB_USERNAME=root
USER_DB_PASSWORD=dbrepo
NODE_ENV="development"
\ No newline at end of file
.gitlab-ci.yml
+ 0
− 31
View file @ 33992e84
......@@ -41,15 +41,6 @@ build-data-service:
- "mvn -f ./dbrepo-metadata-service/pom.xml clean install -Dstyle.color=always -DskipTests"
- "mvn -f ./dbrepo-data-service/pom.xml clean package -Dstyle.color=always -DskipTests"
build-mirror-service:
image: maven:3-openjdk-17
stage: build
needs:
- build-metadata-service
script:
- "mvn -f ./dbrepo-metadata-service/pom.xml clean install -Dstyle.color=always -DskipTests"
- "mvn -f ./dbrepo-mirror-service/pom.xml clean package -Dstyle.color=always -DskipTests"
build-frontend:
image: node:14-alpine
stage: build
......@@ -72,7 +63,6 @@ build-docker:
- "cp .env.unix.example .env"
- "docker build -t dbrepo-metadata-service:build --target build dbrepo-metadata-service"
- "docker build -t dbrepo-data-service:build --target build dbrepo-data-service"
- "docker build -t dbrepo-mirror-service:build --target build dbrepo-mirror-service"
- "docker build ./dbrepo-log-service -t dbrepo-log-service"
- "docker compose build --parallel"
......@@ -114,25 +104,6 @@ test-data-service:
junit: ./dbrepo-data-service/rest-service/target/surefire-reports/TEST-*.xml
coverage: '/Total.*?([0-9]{1,3})%/'
test-mirror-service:
image: maven:3-openjdk-17
stage: test
needs:
- build-mirror-service
script:
- "mvn -f ./dbrepo-metadata-service/pom.xml clean install -Dstyle.color=always -DskipTests"
- "mvn -f ./dbrepo-mirror-service/pom.xml clean test verify -Dstyle.color=always"
- "cat ./dbrepo-mirror-service/report/target/site/jacoco-aggregate/index.html | grep -o 'Total[^%]*%' | sed 's/<.*>/ /; s/Total/Jacoco Coverage Total:/'"
artifacts:
when: always
paths:
- ./dbrepo-mirror-service/report/target/site/jacoco-aggregate/
- ./dbrepo-mirror-service/rest-service/target/surefire-reports/
expire_in: 1 days
reports:
junit: ./dbrepo-mirror-service/rest-service/target/surefire-reports/TEST-*.xml
coverage: '/Total.*?([0-9]{1,3})%/'
test-analyse-service:
image: python:3.9-slim
stage: test
......@@ -472,7 +443,6 @@ release-latest:
needs:
- test-metadata-service
- test-data-service
- test-mirror-service
- test-analyse-service
- test-frontend
only:
......@@ -491,7 +461,6 @@ release-version:
needs:
- test-metadata-service
- test-data-service
- test-mirror-service
- test-analyse-service
- test-frontend
only:
......
CONTRIBUTING.md 0 → 100644
+ 27
− 0
View file @ 33992e84
## Contributing to DBRepo
All contributions are welcome. Ideas, code, documentation, production experiences, bugfixes... we only need to define
some key ideas in order to make this more efficient.
### Key ideas
* Follow the object-oriented programming principles for the Java-based services
* Embrace the [GitOps](https://opengitops.dev/) principles
* Simplicity first
### Development guide
We have a [development guide](./.docs/DEVELOPMENT.md) at your disposal so you can quickly setup an environment to
develop locally.
### Issues
Creating a new issue it's generally a good way to share your ideas and get feedback. We consider creating good issues
and documentation its part of the creative process, and we don't want to interfere on it. However, it's encouraged to
provide as much context as possible. Feel free to talk about a specific use case, show the maintainers what we are
trying to solve.
Explore the available labels in order to proper categorize it and get the fastest feedback.
If the contribution it's a bugfix, a little feature or documentation improvement that could be implemented in, lets say,
a couple of days at maximum, one could go directly for a PR. It's fine.
Makefile
+ 7
− 27
View file @ 33992e84
......@@ -16,14 +16,11 @@ clean:
build: build-backend build-docker
build-backend: build-metadata-service build-analyse-service build-data-service build-mirror-service
build-backend: build-metadata-service build-analyse-service build-data-service
build-data-service: build-metadata-service
mvn -f ./dbrepo-data-service/pom.xml clean package -DskipTests
build-mirror-service: build-metadata-service
mvn -f ./dbrepo-mirror-service/pom.xml clean package -DskipTests
build-metadata-service:
mvn -f ./dbrepo-metadata-service/pom.xml clean install -DskipTests
......@@ -33,7 +30,6 @@ build-analyse-service:
build-docker:
docker build -t dbrepo-metadata-service:build --target build dbrepo-metadata-service
docker build -t dbrepo-data-service:build --target build dbrepo-data-service
docker build -t dbrepo-mirror-service:build --target build dbrepo-mirror-service
docker build ./dbrepo-log-service -t dbrepo-log-service
docker compose build --parallel
......@@ -44,7 +40,7 @@ build-frontend:
build-clients:
bash ./.gitlab/swagger/generate.sh
tag: tag-analyse-service tag-authentication-service tag-metadata-db tag-ui tag-metadata-service tag-data-service tag-mirror-service tag-log-service tag-search-db tag-search-db-init tag-search-service tag-data-db-sidecar
tag: tag-analyse-service tag-authentication-service tag-metadata-db tag-ui tag-metadata-service tag-data-service tag-log-service tag-search-db tag-search-db-init tag-search-service tag-data-db-sidecar
tag-analyse-service:
docker tag dbrepo-analyse-service:latest "${REPOSITORY_URL}/analyse-service:${TAG}"
......@@ -64,11 +60,7 @@ tag-ui:
tag-data-service:
docker tag dbrepo-data-service:latest "${REPOSITORY_URL}/data-service:${TAG}"
docker tag dbrepo-data-service:latest "${REPOSITORY2_URL}/data-service:${TAG}"
tag-mirror-service:
docker tag dbrepo-mirror-service:latest "${REPOSITORY_URL}/mirror-service:${TAG}"
docker tag dbrepo-mirror-service:latest "${REPOSITORY2_URL}/mirror-service:${TAG}"
docker tag dbrepo-data-service:latest "${REPOSITORY2_URL}/data-service:${TAG}
tag-metadata-service:
docker tag dbrepo-metadata-service:latest "${REPOSITORY_URL}/metadata-service:${TAG}"
......@@ -98,7 +90,7 @@ tag-storage-service-init:
docker tag dbrepo-storage-service-init:latest "${REPOSITORY_URL}/storage-service-init:${TAG}"
docker tag dbrepo-storage-service-init:latest "${REPOSITORY2_URL}/storage-service-init:${TAG}"
release: build-docker tag release-analyse-service release-authentication-service release-metadata-db release-ui release-metadata-service release-data-service release-log-service release-search-db release-mirror-service release-search-db-init release-search-service release-data-db-sidecar
release: build-docker tag release-analyse-service release-authentication-service release-metadata-db release-ui release-metadata-service release-data-service release-log-service release-search-db release-search-db-init release-search-service release-data-db-sidecar
release-analyse-service: tag-analyse-service
docker push "${REPOSITORY_URL}/analyse-service:${TAG}"
......@@ -120,10 +112,6 @@ release-data-service: tag-data-service
docker push "${REPOSITORY_URL}/data-service:${TAG}"
docker push "${REPOSITORY2_URL}/data-service:${TAG}"
release-mirror-service: tag-mirror-service
docker push "${REPOSITORY_URL}/mirror-service:${TAG}"
docker push "${REPOSITORY2_URL}/mirror-service:${TAG}"
release-search-db: tag-search-db
docker push "${REPOSITORY_URL}/search-db:${TAG}"
docker push "${REPOSITORY2_URL}/search-db:${TAG}"
......@@ -152,21 +140,18 @@ release-storage-service-init: tag-storage-service-init
docker push "${REPOSITORY_URL}/storage-service-init:${TAG}"
docker push "${REPOSITORY2_URL}/storage-service-init:${TAG}"
test-backend: test-metadata-service test-analyse-service test-data-service test-mirror-service
test-backend: test-metadata-service test-analyse-service test-data-service
test-data-service: build-data-service
mvn -f ./dbrepo-data-service/pom.xml clean test verify
test-mirror-service: build-mirror-service
mvn -f ./dbrepo-mirror-service/pom.xml clean test verify
test-metadata-service: build-metadata-service
mvn -f ./dbrepo-metadata-service/pom.xml clean test verify
test-analyse-service: build-analyse-service
bash ./dbrepo-analyse-service/test.sh
scan: scan-analyse-service scan-authentication-service scan-broker-service scan-gateway-service scan-metadata-db scan-metadata-service scan-search-db scan-ui scan-data-service scan-data-db scan-log-service scan-mirror-service scan-search-dashboard scan-search-service
scan: scan-analyse-service scan-authentication-service scan-broker-service scan-gateway-service scan-metadata-db scan-metadata-service scan-search-db scan-ui scan-data-service scan-data-db scan-log-service scan-search-dashboard scan-search-service
scan-analyse-service:
trivy image --insecure --exit-code 0 --format template --template "@.trivy/gitlab.tpl" -o ./.trivy/trivy-analyse-service-report.json dbrepo-analyse-service:latest
......@@ -204,11 +189,6 @@ scan-data-service:
trivy image --insecure --exit-code 0 dbrepo-data-service:latest
trivy image --insecure --exit-code 1 --severity CRITICAL dbrepo-data-service:latest
scan-mirror-service:
trivy image --insecure --exit-code 0 --format template --template "@.trivy/gitlab.tpl" -o ./.trivy/trivy-mirror-service-report.json dbrepo-mirror-service:latest
trivy image --insecure --exit-code 0 dbrepo-mirror-service:latest
trivy image --insecure --exit-code 1 --severity CRITICAL dbrepo-mirror-service:latest
scan-search-db:
trivy image --insecure --exit-code 0 --format template --template "@.trivy/gitlab.tpl" -o ./.trivy/trivy-search-db-report.json "dbrepo-search-db"
trivy image --insecure --exit-code 0 "dbrepo-search-db"
......@@ -258,5 +238,5 @@ teardown:
docs: build-docker
docker compose up -d || docker compose down
bash .docs/generate_swagger.sh || docker compose down
cd .docs && bash generate_swagger.sh || docker compose down
docker compose down
README.md
+ 3
− 73
View file @ 33992e84
......@@ -4,83 +4,13 @@
# DBRepo &mdash; A Repository for Databases
## tl;dr
```shell
docker compose up -d
docker compose logs -f
```
## Build
Build DBRepo from scratch:
```shell
make build
```
## Documentation
More documentation can be found online: https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/
## Development
### CI/CD
We get compute resources in-kind from [dataLAB](https://www.it.tuwien.ac.at/en/services/network-and-servers/datalab)
to run our pipeline:
<p align="center">
<img src="./.gitlab/gitlab-runner.png" alt="Gitlab runner configuration in the cluster" width="732" height="262" /><br/>
<i><strong>Figure 1.</strong> Gitlab runner configuration in the cluster.</i>
</p>
Minikube cluster with 6vCPU and 28GB RAM. The CI pipeline is configured as follows in the `config.toml`:
```toml
concurrent = 10
[[runners]]
executor = "kubernetes"
environment = [
"FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false"
]
[runners.kubernetes]
namespace = "{{.Release.Namespace}}"
privileged = true
allowed_services = ["docker:24-dind"]
[[runners.kubernetes.services]]
name = "docker:24-dind"
command = [ "--registry-mirror=http://docker-io-mirror:80", "--insecure-registry=docker-io-mirror:80", "--registry-mirror=http://gcr-io-mirror:80", "--insecure-registry=gcr-io-mirror:80", "--registry-mirror=http://ghcr-io-mirror:80", "--insecure-registry=ghcr-io-mirror:80", "--registry-mirror=http://k8s-io-mirror:80", "--insecure-registry=k8s-io-mirror:80" ]
alias = "docker"
[[runners.kubernetes.volumes.empty_dir]]
name = "rundind"
mount_path = "/var/run/dind"
medium = "Memory"
[[runners.kubernetes.volumes.empty_dir]]
name = "tmp"
mount_path = "/tmp"
medium = "Memory"
```
For each job in the CI/CD pipeline, a pod with three containers is started:
1. `build` the main build container, you can *freely* specify any image with `image: xyz` as base
2. `helper` the default helper container.
3. `svc-0` the Docker-in-Docker sidecar (rootless executed as user `rootless`/`1000`) exposing the Docker socket to the
`build` container under `
*Note.* Only Docker-in-Docker (dind) is allowed as service in the pipeline currently. For each job, a
dind-sidecar container `svc-0` is started that exposes the Docker socket at `/var/run/dind/docker.sock` in the `build`
container you can freely configure how you want. We are aware this is not optimal as it exposes *root* privileges in the
cluster.
The full CI/CD pipeline Helm chart is documented in
the [`fda-deployment`](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-deployment/-/tree/master/charts/dbrepo-devops)
repository.
## Contribute
Contributions are always welcome and encouraged, simply fork the repository and
Contributions are always welcome and encouraged, please read the [contribution overview](./CONTRIBUTING.md) and
contact [Andreas Rauber](http://www.ifs.tuwien.ac.at/~andi/).
## Acknowledgements
......
dbrepo-authentication-service/dbrepo-realm.json
+ 43
− 46
View file @ 33992e84
......@@ -99,7 +99,7 @@
"description" : "${default-database-handling}",
"composite" : true,
"composites" : {
"realm" : [ "modify-database-owner", "update-database-access", "create-database", "list-databases", "create-database-access", "find-database", "modify-database-visibility", "delete-database-access", "check-database-access" ]
"realm" : [ "modify-database-owner", "update-database-access", "create-database", "list-databases", "create-database-access", "find-database", "modify-database-visibility", "import-database-data", "delete-database-access", "check-database-access" ]
},
"clientRole" : false,
"containerId" : "82c39861-d877-4667-a0f3-4daa2ee230e0",
......@@ -212,6 +212,14 @@
"clientRole" : false,
"containerId" : "82c39861-d877-4667-a0f3-4daa2ee230e0",
"attributes" : { }
}, {
"id" : "e51b63c2-48dd-4bd6-95fb-d257d21b26ba",
"name" : "import-database-data",
"description" : "${import-database-data}",
"composite" : false,
"clientRole" : false,
"containerId" : "82c39861-d877-4667-a0f3-4daa2ee230e0",
"attributes" : { }
}, {
"id" : "a7ad038c-5c06-42fc-951c-15ac09d4df66",
"name" : "modify-database-owner",
......@@ -357,14 +365,6 @@
"clientRole" : false,
"containerId" : "82c39861-d877-4667-a0f3-4daa2ee230e0",
"attributes" : { }
}, {
"id" : "972badbc-ee50-4194-8352-848f1e4c5eee",
"name" : "delete-database",
"description" : "${delete-database}",
"composite" : false,
"clientRole" : false,
"containerId" : "82c39861-d877-4667-a0f3-4daa2ee230e0",
"attributes" : { }
}, {
"id" : "c047d521-cec3-4444-86c4-aef098489b7b",
"name" : "delete-maintenance-message",
......@@ -401,10 +401,7 @@
"id" : "b372f8f7-d203-4293-b991-ad93fb505917",
"name" : "escalated-database-handling",
"description" : "${escalated-database-handling}",
"composite" : true,
"composites" : {
"realm" : [ "delete-database" ]
},
"composite" : false,
"clientRole" : false,
"containerId" : "82c39861-d877-4667-a0f3-4daa2ee230e0",
"attributes" : { }
......@@ -448,14 +445,6 @@
"clientRole" : false,
"containerId" : "82c39861-d877-4667-a0f3-4daa2ee230e0",
"attributes" : { }
}, {
"id" : "8755da2b-d85a-4f40-a0bf-fe08cf8f9d75",
"name" : "delete-table",
"description" : "${delete-table}",
"composite" : false,
"clientRole" : false,
"containerId" : "82c39861-d877-4667-a0f3-4daa2ee230e0",
"attributes" : { }
}, {
"id" : "f392bfcb-0be5-4fad-9ce4-8ac6396f176d",
"name" : "export-query-data",
......@@ -676,6 +665,14 @@
"clientRole" : false,
"containerId" : "82c39861-d877-4667-a0f3-4daa2ee230e0",
"attributes" : { }
}, {
"id" : "ba1ad8f2-39aa-487d-987f-645e8a459559",
"name" : "delete-table",
"description" : "${delete-table}",
"composite" : false,
"clientRole" : false,
"containerId" : "82c39861-d877-4667-a0f3-4daa2ee230e0",
"attributes" : { }
}, {
"id" : "09f7bdb0-296f-46c8-a3a3-8f9254fb17e4",
"name" : "list-maintenance-messages",
......@@ -1062,7 +1059,7 @@
"otpPolicyLookAheadWindow" : 1,
"otpPolicyPeriod" : 30,
"otpPolicyCodeReusable" : false,
"otpSupportedApplications" : [ "totpAppGoogleName", "totpAppFreeOTPName", "totpAppMicrosoftAuthenticatorName" ],
"otpSupportedApplications" : [ "totpAppFreeOTPName", "totpAppMicrosoftAuthenticatorName", "totpAppGoogleName" ],
"webAuthnPolicyRpEntityName" : "keycloak",
"webAuthnPolicySignatureAlgorithms" : [ "ES256" ],
"webAuthnPolicyRpId" : "",
......@@ -2035,7 +2032,7 @@
"subType" : "anonymous",
"subComponents" : { },
"config" : {
"allowed-protocol-mapper-types" : [ "saml-role-list-mapper", "saml-user-property-mapper", "saml-user-attribute-mapper", "oidc-full-name-mapper", "oidc-usermodel-attribute-mapper", "oidc-usermodel-property-mapper", "oidc-sha256-pairwise-sub-mapper", "oidc-address-mapper" ]
"allowed-protocol-mapper-types" : [ "saml-user-property-mapper", "oidc-sha256-pairwise-sub-mapper", "saml-user-attribute-mapper", "oidc-full-name-mapper", "oidc-usermodel-property-mapper", "saml-role-list-mapper", "oidc-address-mapper", "oidc-usermodel-attribute-mapper" ]
}
}, {
"id" : "1849e52a-b8c9-44a8-af3d-ee19376a1ed1",
......@@ -2061,7 +2058,7 @@
"subType" : "authenticated",
"subComponents" : { },
"config" : {
"allowed-protocol-mapper-types" : [ "oidc-full-name-mapper", "oidc-usermodel-attribute-mapper", "oidc-address-mapper", "saml-user-attribute-mapper", "saml-user-property-mapper", "saml-role-list-mapper", "oidc-sha256-pairwise-sub-mapper", "oidc-usermodel-property-mapper" ]
"allowed-protocol-mapper-types" : [ "oidc-full-name-mapper", "oidc-usermodel-attribute-mapper", "oidc-usermodel-property-mapper", "oidc-address-mapper", "saml-user-property-mapper", "saml-user-attribute-mapper", "oidc-sha256-pairwise-sub-mapper", "saml-role-list-mapper" ]
}
} ],
"org.keycloak.userprofile.UserProfileProvider" : [ {
......@@ -2119,7 +2116,7 @@
"internationalizationEnabled" : false,
"supportedLocales" : [ ],
"authenticationFlows" : [ {
"id" : "7b9f9c3b-0636-406b-8ac3-c7259ef423d3",
"id" : "7b272faa-5cbf-4783-8bca-cabe292e80cc",
"alias" : "Account verification options",
"description" : "Method with which to verity the existing account",
"providerId" : "basic-flow",
......@@ -2141,7 +2138,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "70be7410-8823-4c0f-aa70-39d9d0e1c7f2",
"id" : "9d570708-d15a-4063-a3b1-d742c9c85529",
"alias" : "Authentication Options",
"description" : "Authentication options.",
"providerId" : "basic-flow",
......@@ -2170,7 +2167,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "c3bfd9c1-1e61-4789-9797-7e28867ac15b",
"id" : "2579048c-e648-4e95-9bd9-fb1fe776a64b",
"alias" : "Browser - Conditional OTP",
"description" : "Flow to determine if the OTP is required for the authentication",
"providerId" : "basic-flow",
......@@ -2192,7 +2189,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "ba6106ab-6fca-43b3-a170-c1db8352fc69",
"id" : "76197fdc-44e0-470c-b84f-7e666edda852",
"alias" : "Direct Grant - Conditional OTP",
"description" : "Flow to determine if the OTP is required for the authentication",
"providerId" : "basic-flow",
......@@ -2214,7 +2211,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "bd04da6b-71b0-42d0-a279-4402b7cf35cf",
"id" : "1c163e76-4872-4996-b7a2-937fd9827f4a",
"alias" : "First broker login - Conditional OTP",
"description" : "Flow to determine if the OTP is required for the authentication",
"providerId" : "basic-flow",
......@@ -2236,7 +2233,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "2016d479-271f-4dfd-88c3-50635d1d27ea",
"id" : "fac05c8f-328f-4bb6-ac68-e5cf0dbaf768",
"alias" : "Handle Existing Account",
"description" : "Handle what to do if there is existing account with same email/username like authenticated identity provider",
"providerId" : "basic-flow",
......@@ -2258,7 +2255,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "3294bcec-0829-405e-bc34-897eb5ecbf62",
"id" : "e9478687-f9bd-4854-83a2-768903dd06a7",
"alias" : "Reset - Conditional OTP",
"description" : "Flow to determine if the OTP should be reset or not. Set to REQUIRED to force.",
"providerId" : "basic-flow",
......@@ -2280,7 +2277,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "cdc002be-a391-4853-8c40-8eef726dcfb6",
"id" : "a6a6064c-91e3-455d-a8f7-8530f0d9f932",
"alias" : "User creation or linking",
"description" : "Flow for the existing/non-existing user alternatives",
"providerId" : "basic-flow",
......@@ -2303,7 +2300,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "11e9d0a5-743f-44fe-ab9f-59f3e730e20a",
"id" : "bf274d56-ea30-450d-9ba7-1d4206f8b9ae",
"alias" : "Verify Existing Account by Re-authentication",
"description" : "Reauthentication of existing account",
"providerId" : "basic-flow",
......@@ -2325,7 +2322,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "c5ea7e5c-c07d-48c5-ba99-4e84ed167bee",
"id" : "38133f86-344b-497e-9402-87c454d7128c",
"alias" : "browser",
"description" : "browser based authentication",
"providerId" : "basic-flow",
......@@ -2361,7 +2358,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "e18342ad-98d9-4411-bf59-30e07cf65f98",
"id" : "f644f101-4f4a-458a-990d-6ced0d1353fe",
"alias" : "clients",
"description" : "Base authentication for clients",
"providerId" : "client-flow",
......@@ -2397,7 +2394,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "ca97318e-8587-43b4-919b-5c65e2a074f3",
"id" : "a12de1f9-53eb-4563-8aa8-051e583bb144",
"alias" : "direct grant",
"description" : "OpenID Connect Resource Owner Grant",
"providerId" : "basic-flow",
......@@ -2426,7 +2423,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "7f523ba5-6d6b-4a65-9ba0-107b5c3afe7a",
"id" : "edc782a3-5dee-41a2-ab0d-715fc55b01c9",
"alias" : "docker auth",
"description" : "Used by Docker clients to authenticate against the IDP",
"providerId" : "basic-flow",
......@@ -2441,7 +2438,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "c72a0c74-0dc4-45dc-8b62-a77bc42d805c",
"id" : "358c0c7d-832d-477c-b9d0-258a786325d5",
"alias" : "first broker login",
"description" : "Actions taken after first broker login with identity provider account, which is not yet linked to any Keycloak account",
"providerId" : "basic-flow",
......@@ -2464,7 +2461,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "5939da5a-b299-49b7-8c07-c45e13d995db",
"id" : "891b4eb9-3a7f-4556-8cb8-f6a07c637dae",
"alias" : "forms",
"description" : "Username, password, otp and other auth forms.",
"providerId" : "basic-flow",
......@@ -2486,7 +2483,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "6a0341db-eec8-4962-bb04-4b8a43dced2a",
"id" : "b74eeba1-2096-41d4-bafb-940ad65ecd42",
"alias" : "http challenge",
"description" : "An authentication flow based on challenge-response HTTP Authentication Schemes",
"providerId" : "basic-flow",
......@@ -2508,7 +2505,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "525006c3-c676-4bfb-83cf-79faeafe0183",
"id" : "37ca760b-2b9f-491f-aa1c-a9896923ab61",
"alias" : "registration",
"description" : "registration flow",
"providerId" : "basic-flow",
......@@ -2524,7 +2521,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "65e9d76b-d180-47e6-ad1b-dddea414668f",
"id" : "1cf97e2c-9063-4b0e-8b99-56c463a6effc",
"alias" : "registration form",
"description" : "registration form",
"providerId" : "form-flow",
......@@ -2560,7 +2557,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "feb110ed-e018-43ba-8989-0b17a3a20de3",
"id" : "01499d7a-b0a1-4dd0-a866-62e6a28d5119",
"alias" : "reset credentials",
"description" : "Reset credentials for a user if they forgot their password or something",
"providerId" : "basic-flow",
......@@ -2596,7 +2593,7 @@
"userSetupAllowed" : false
} ]
}, {
"id" : "15efa29f-2140-4e3b-803e-64d79906c18b",
"id" : "579397ef-7c1f-4a24-93c4-7e9ea2514783",
"alias" : "saml ecp",
"description" : "SAML ECP Profile Authentication Flow",
"providerId" : "basic-flow",
......@@ -2612,13 +2609,13 @@
} ]
} ],
"authenticatorConfig" : [ {
"id" : "2c135401-6244-41b0-8c49-648f38b5e1bd",
"id" : "8f778718-7e1d-417a-b6e6-fc4788b45c44",
"alias" : "create unique user config",
"config" : {
"require.password.update.after.registration" : "false"
}
}, {
"id" : "39d00721-0854-4ff7-9f5c-7f6fa71ae37a",
"id" : "fdf2fe79-06e9-4bea-b361-dab88bd38680",
"alias" : "review profile config",
"config" : {
"update.profile.on.first.login" : "missing"
......
dbrepo-data-db/sidecar/Dockerfile
+ 1
− 1
View file @ 33992e84
FROM python:3.10-alpine
RUN apk add bash curl jq
RUN apk add bash curl jq mariadb-client
ENV PYTHONFAULTHANDLER=1
......
dbrepo-data-service/README.md
+ 23
− 1
View file @ 33992e84
# Search Startup Agent
# Data Service
## Actuator
- Info: http://localhost:9093/actuator/info
- Health: http://localhost:9093/actuator/health
- Readiness: http://localhost:9093/actuator/health/readiness
- Liveness: http://localhost:9093/actuator/health/liveness
- Prometheus: http://localhost:9093/actuator/prometheus
## Swagger UI Endpoints
- Swagger UI: http://localhost:9093/swagger-ui/index.html
## OpenAPI Endpoints
- OpenAPI v3 as .yaml: http://localhost:9093/v3/api-docs.yaml
## Build
```shell
mvn -f ../dbrepo-metadata-service/pom.xml clean install -DskipTests
mvn clean package -DskipTests
```
\ No newline at end of file
dbrepo-data-service/rest-service/src/test/java/at/tuwien/annotations/MockOpensearch.java
+ 3
− 4
View file @ 33992e84
......@@ -14,9 +14,8 @@ import java.lang.annotation.Target;
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@MockBeans({@MockBean(DatabaseIdxRepository.class), @MockBean(UnitIdxRepository.class),
@MockBean(ConceptIdxRepository.class), @MockBean(TableIdxRepository.class), @MockBean(TableColumnIdxRepository.class),
@MockBean(UserIdxRepository.class), @MockBean(ViewIdxRepository.class), @MockBean(IdentifierIdxRepository.class)})
@EnableAutoConfiguration(exclude = {OpenSearchRestClientAutoConfiguration.class, OpenSearchRestHighLevelClientAutoConfiguration.class})
@MockBeans({@MockBean(DatabaseIdxRepository.class)})
@EnableAutoConfiguration(exclude = {OpenSearchRestClientAutoConfiguration.class,
OpenSearchRestHighLevelClientAutoConfiguration.class})
public @interface MockOpensearch {
}
dbrepo-data-service/rest-service/src/test/java/at/tuwien/service/DatabaseServiceIntegrationTest.java
+ 13
− 9
View file @ 33992e84
......@@ -4,13 +4,8 @@ import at.tuwien.BaseUnitTest;
import at.tuwien.annotations.MockAmqp;
import at.tuwien.annotations.MockOpensearch;
import at.tuwien.entities.database.Database;
import at.tuwien.entities.user.User;
import at.tuwien.exception.DatabaseNotFoundException;
import at.tuwien.exception.UserNotFoundException;
import at.tuwien.repository.mdb.ContainerRepository;
import at.tuwien.repository.mdb.DatabaseRepository;
import at.tuwien.repository.mdb.ImageRepository;
import at.tuwien.repository.mdb.UserRepository;
import at.tuwien.repository.mdb.*;
import lombok.extern.log4j.Log4j2;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
......@@ -45,15 +40,24 @@ public class DatabaseServiceIntegrationTest extends BaseUnitTest {
@Autowired
private DatabaseRepository databaseRepository;
@Autowired
private LicenseRepository licenseRepository;
@Autowired
private DatabaseService databaseService;
@BeforeEach
public void beforeEach() {
TABLE_1.setColumns(TABLE_1_COLUMNS);
TABLE_2.setColumns(TABLE_2_COLUMNS);
TABLE_3.setColumns(TABLE_3_COLUMNS);
TABLE_4.setColumns(TABLE_4_COLUMNS);
/* metadata database */
userRepository.save(USER_1);
imageRepository.save(IMAGE_1_SIMPLE);
containerRepository.save(CONTAINER_1_SIMPLE);
databaseRepository.save(DATABASE_1_SIMPLE);
imageRepository.save(IMAGE_1);
licenseRepository.save(LICENSE_1);
containerRepository.save(CONTAINER_1);
databaseRepository.save(DATABASE_1);
}
@Test
......
dbrepo-data-service/rest-service/src/test/java/at/tuwien/service/QueueServiceIntegrationTest.java
+ 12
− 15
View file @ 33992e84
......@@ -5,7 +5,6 @@ import at.tuwien.annotations.MockOpensearch;
import at.tuwien.config.MariaDbConfig;
import at.tuwien.config.MariaDbContainerConfig;
import at.tuwien.exception.DatabaseNotFoundException;
import at.tuwien.exception.QueryMalformedException;
import at.tuwien.exception.TableNotFoundException;
import at.tuwien.repository.mdb.*;
import at.tuwien.service.impl.QueueServiceImpl;
......@@ -22,9 +21,7 @@ import org.testcontainers.junit.jupiter.Testcontainers;
import java.sql.SQLException;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Stream;
import static org.junit.jupiter.api.Assertions.assertThrows;
......@@ -45,13 +42,10 @@ public class QueueServiceIntegrationTest extends BaseUnitTest {
private DatabaseRepository databaseRepository;
@Autowired
private TableRepository tableRepository;
private ImageRepository imageRepository;
@Autowired
private TableColumnRepository tableColumnRepository;
@Autowired
private ImageRepository imageRepository;
private LicenseRepository licenseRepository;
@Autowired
private QueueServiceImpl queueService;
......@@ -61,15 +55,18 @@ public class QueueServiceIntegrationTest extends BaseUnitTest {
@BeforeEach
public void beforeEach() throws SQLException {
MariaDbConfig.dropDatabase(CONTAINER_1, DATABASE_1_INTERNALNAME);
MariaDbConfig.createInitDatabase(CONTAINER_1, DATABASE_1);
TABLE_1.setColumns(TABLE_1_COLUMNS);
TABLE_2.setColumns(TABLE_2_COLUMNS);
TABLE_3.setColumns(TABLE_3_COLUMNS);
TABLE_4.setColumns(TABLE_4_COLUMNS);
/* metadata database */
imageRepository.save(IMAGE_1);
userRepository.save(USER_1);
containerRepository.save(CONTAINER_1_SIMPLE);
databaseRepository.save(DATABASE_1_SIMPLE);
tableRepository.save(TABLE_1_SIMPLE);
tableColumnRepository.saveAll(TABLE_1_COLUMNS);
imageRepository.save(IMAGE_1);
licenseRepository.save(LICENSE_1);
containerRepository.save(CONTAINER_1);
databaseRepository.save(DATABASE_1);
MariaDbConfig.dropDatabase(CONTAINER_1, DATABASE_1_INTERNALNAME);
MariaDbConfig.createInitDatabase(CONTAINER_1, DATABASE_1);
}
@Test
......
dbrepo-data-service/rest-service/src/test/java/at/tuwien/service/UserServiceIntegrationTest.java
+ 0
− 12
View file @ 33992e84
......@@ -3,12 +3,7 @@ package at.tuwien.service;
import at.tuwien.BaseUnitTest;
import at.tuwien.annotations.MockAmqp;
import at.tuwien.annotations.MockOpensearch;
import at.tuwien.config.MariaDbConfig;
import at.tuwien.config.MariaDbContainerConfig;
import at.tuwien.entities.user.User;
import at.tuwien.exception.DatabaseNotFoundException;
import at.tuwien.exception.QueryMalformedException;
import at.tuwien.exception.TableNotFoundException;
import at.tuwien.exception.UserNotFoundException;
import at.tuwien.repository.mdb.*;
import lombok.extern.log4j.Log4j2;
......@@ -18,15 +13,8 @@ import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit.jupiter.SpringExtension;
import org.testcontainers.containers.MariaDBContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import java.sql.SQLException;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Stream;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
......
dbrepo-data-service/services/src/main/java/at/tuwien/service/impl/MariaDbServiceImpl.java
+ 4
− 3
View file @ 33992e84
......@@ -6,7 +6,6 @@ import at.tuwien.repository.mdb.DatabaseRepository;
import at.tuwien.service.DatabaseService;
import lombok.extern.log4j.Log4j2;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.annotation.Transient;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
......@@ -25,16 +24,18 @@ public class MariaDbServiceImpl extends HibernateConnector implements DatabaseSe
}
@Override
@Transactional(readOnly = true)
public List<Database> findAll() {
return databaseRepository.findAll();
}
@Override
@Transactional(readOnly = true)
public Database find(Long databaseId) throws DatabaseNotFoundException {
final Optional<Database> database = databaseRepository.findById(databaseId);
if (database.isEmpty()) {
log.error("Failed to find database with id {}", databaseId);
throw new DatabaseNotFoundException("could not find database with id " + databaseId);
throw new DatabaseNotFoundException("Failed to find database with id " + databaseId);
}
return database.get();
}
......@@ -45,7 +46,7 @@ public class MariaDbServiceImpl extends HibernateConnector implements DatabaseSe
final Optional<Database> database = databaseRepository.findByInternalName(internalName);
if (database.isEmpty()) {
log.error("Failed to find database with internal name {}", internalName);
throw new DatabaseNotFoundException("could not find database with internal name " + internalName);
throw new DatabaseNotFoundException("Failed to find database with internal name " + internalName);
}
return database.get();
}
......
dbrepo-data-service/services/src/main/java/at/tuwien/service/impl/UserServiceImpl.java
+ 2
− 0
View file @ 33992e84
......@@ -7,6 +7,7 @@ import at.tuwien.service.UserService;
import lombok.extern.log4j.Log4j2;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.util.Optional;
......@@ -22,6 +23,7 @@ public class UserServiceImpl implements UserService {
}
@Override
@Transactional(readOnly = true)
public User findByUsername(String username) throws UserNotFoundException {
final Optional<User> optional = userRepository.findByUsername(username);
if (optional.isEmpty()) {
......
dbrepo-metadata-db/setup-schema.sql
+ 43
− 44
View file @ 33992e84
......@@ -123,8 +123,10 @@ CREATE TABLE IF NOT EXISTS `mdb_tables`
routing_key character varying(255) NOT NULL,
tName VARCHAR(50),
tDescription TEXT,
NumCols INTEGER,
NumRows INTEGER,
num_rows BIGINT,
data_length BIGINT,
max_data_length BIGINT,
avg_row_length BIGINT,
`separator` CHAR(1),
quote CHAR(1),
element_null VARCHAR(50),
......@@ -145,19 +147,19 @@ CREATE TABLE IF NOT EXISTS `mdb_tables`
CREATE TABLE IF NOT EXISTS `mdb_columns`
(
ID bigint NOT NULL AUTO_INCREMENT,
tID bigint NOT NULL,
dfID bigint,
ID BIGINT NOT NULL AUTO_INCREMENT,
tID BIGINT NOT NULL,
dfID BIGINT,
cName VARCHAR(100),
internal_name VARCHAR(100) NOT NULL,
alias VARCHAR(100),
Datatype ENUM ('CHAR','VARCHAR','BINARY','VARBINARY','TINYBLOB','TINYTEXT','TEXT','BLOB','MEDIUMTEXT','MEDIUMBLOB','LONGTEXT','LONGBLOB','ENUM','SET','BIT','TINYINT','BOOL','SMALLINT','MEDIUMINT','INT','BIGINT','FLOAT','DOUBLE','DECIMAL','DATE','DATETIME','TIMESTAMP','TIME','YEAR'),
length INT NULL,
length BIGINT NULL,
ordinal_position INTEGER NOT NULL,
is_primary_key BOOLEAN NOT NULL,
index_length INT NULL,
size INT,
d INT,
index_length BIGINT NULL,
size BIGINT,
d BIGINT,
auto_generated BOOLEAN DEFAULT false,
is_null_allowed BOOLEAN NOT NULL DEFAULT true,
val_min NUMERIC NULL,
......@@ -314,8 +316,6 @@ CREATE TABLE IF NOT EXISTS `mdb_view`
Query TEXT NOT NULL,
query_hash VARCHAR(255) NOT NULL,
Public BOOLEAN NOT NULL,
NumCols INTEGER,
NumRows INTEGER,
InitialView BOOLEAN NOT NULL,
created timestamp NOT NULL DEFAULT NOW(),
last_modified timestamp,
......@@ -365,30 +365,29 @@ CREATE TABLE IF NOT EXISTS `mdb_view_columns`
CREATE TABLE IF NOT EXISTS `mdb_identifiers`
(
id bigint NOT NULL AUTO_INCREMENT,
dbid bigint,
qid bigint,
vid bigint,
id BIGINT NOT NULL AUTO_INCREMENT,
dbid BIGINT,
qid BIGINT,
vid BIGINT,
tid BIGINT,
publisher VARCHAR(255) NOT NULL,
language VARCHAR(2),
visibility ENUM ('SELF', 'EVERYONE') NOT NULL default 'EVERYONE',
publication_year INTEGER NOT NULL,
publication_month INTEGER,
publication_day INTEGER,
identifier_type ENUM ('DATABASE', 'SUBSET', 'VIEW') NOT NULL,
identifier_type ENUM ('DATABASE', 'SUBSET', 'VIEW', 'TABLE') NOT NULL,
query TEXT,
query_normalized TEXT,
query_hash VARCHAR(255),
execution timestamp,
execution TIMESTAMP,
result_hash VARCHAR(255),
result_number bigint,
result_number BIGINT,
doi VARCHAR(255),
created timestamp NOT NULL DEFAULT NOW(),
created_by character varying(36) NOT NULL,
last_modified timestamp,
created TIMESTAMP NOT NULL DEFAULT NOW(),
created_by VARCHAR(36) NOT NULL,
last_modified TIMESTAMP,
PRIMARY KEY (id), /* must be a single id from persistent identifier concept */
FOREIGN KEY (dbid) REFERENCES mdb_databases (id),
UNIQUE (dbid, qid),
FOREIGN KEY (created_by) REFERENCES mdb_users (id)
) WITH SYSTEM VERSIONING;
......
dbrepo-metadata-service/Dockerfile
+ 3
− 0
View file @ 33992e84
......@@ -77,6 +77,9 @@ ENV DATACITE_PASSWORD=""
ENV S3_STORAGE_ENDPOINT="http://storage-service:9000"
ENV S3_ACCESS_KEY_ID="seaweedfsadmin"
ENV S3_SECRET_ACCESS_KEY="seaweedfsadmin"
ENV MIRROR_RATE=60
ENV OBTAIN_METADATA_RATE=60
ENV DELETE_STALE_QUERIES_RATE=60
WORKDIR /app
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment