Skip to content
Snippets Groups Projects
Commit a5c97882 authored by Martin Weise's avatar Martin Weise
Browse files

Merge branch 'master' into 'release-1.4.4' 

Master

See merge request !286
parents 4fe0c58e 346bf78e
No related branches found
No related tags found
2 merge requests!287Dev,!286Master
Expand all Show whitespace changes
Inline Side-by-side
Showing
with 830 additions and 495 deletions
.docs/.swagger/api.base.yaml
+ 5
− 2
View file @ a5c97882
......@@ -10,13 +10,16 @@ components:
scheme: bearer
type: http
externalDocs:
description: Sourcecode Documentation
description: Project Website
url: https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/
info:
contact:
email: andreas.rauber@tuwien.ac.at
name: Prof. Andreas Rauber
description: The REST API
description: |
The merged REST API of DBRepo for users, developers and data stewards to be accessed publicly. Have a look at
the [source code](https://gitlab.phaidra.org/fair-data-austria-db-repository/fda-services) for non-public endpoints
that are used between the services themselves.
license:
name: Apache 2.0
url: https://www.apache.org/licenses/LICENSE-2.0
......
.docs/.swagger/api.yaml
+ 666
− 313
View file @ a5c97882
This diff is collapsed.
.docs/.swagger/swagger-site.sh deleted 100644 → 0
+ 0
− 21
View file @ 4fe0c58e
#!/bin/bash
declare -A services
#services[1080]=upload
services[4000]=search
services[5000]=analyse
services[9093]=data
services[9099]=metadata
services[3305]=sidecar
rm -f ./tmp.yaml
mkdir -p ./site/swagger
touch ./tmp.yaml
# -> build paths: map
for key in "${!services[@]}"; do
cat .docs/.swagger/api-${services[$key]}.yaml | yq .paths >> ./tmp.yaml
done
# -> merge with api.base.yaml into final api.yaml
yq ".paths *= load(\"tmp.yaml\")" .docs/.swagger/api.base.yaml > .docs/.swagger/api.yaml
.docs/api/data-db.md
+ 2
− 3
View file @ a5c97882
......@@ -11,10 +11,9 @@ author: Martin Weise
!!! debug "Debug Information"
Image: [`dbrepo/data-db-sidecar:__APPVERSION__`](https://hub.docker.com/r/dbrepo/data-db-sidecar)
Image: [`dbrepo/data-db-sidecar:1.4.4`](https://hub.docker.com/r/dbrepo/data-db-sidecar)
* Ports: 3305/tcp
* Swagger UI: `http://<hostname>:1080/swagger-ui/` <a href="../swagger/sidecar" target="_blank">:fontawesome-solid-square-up-right: view online</a>
* Ports: 8080/tcp
## Overview
......
.docs/api/index.md
+ 33
− 33
View file @ a5c97882
......@@ -5,7 +5,7 @@ author: Martin Weise
# Overview
We developed a Python Library for communicating with DBRepo from e.g. Jupyter Notebooks. See
the [Python Library](./usage-python) page for more details.
the [Python Library](./open-api) page for more details.
We give usage examples of the most important use-cases we identified.
......@@ -34,7 +34,7 @@ A user wants to create an account in DBRepo.
that your administrator has assigned.
<figure markdown>
![Create user account](images/screenshots/create-account-step-1.png){ .img-border }
![Create user account](../images/screenshots/create-account-step-1.png){ .img-border }
<figcaption>Figure 1: Create user account.</figcaption>
</figure>
......@@ -44,7 +44,7 @@ A user wants to create an account in DBRepo.
authenticated with DBRepo in Figure 2.
<figure markdown>
![Create user account](images/screenshots/create-account-step-2.png){ .img-border }
![Create user account](../images/screenshots/create-account-step-2.png){ .img-border }
<figcaption>Figure 2: Login to the user account.</figcaption>
</figure>
......@@ -54,7 +54,7 @@ A user wants to create an account in DBRepo.
identically in field :material-numeric-4-circle-outline:.
<figure markdown>
![Change user account password](images/screenshots/create-account-step-3.png){ .img-border }
![Change user account password](../images/screenshots/create-account-step-3.png){ .img-border }
<figcaption>Figure 3: Change user account password.</figcaption>
</figure>
......@@ -134,7 +134,7 @@ A user wants to create a database in DBRepo.
Login and press the ":material-plus: DATABASE" button on the top right :material-numeric-1-circle-outline: as seen in Figure 4.
<figure markdown>
![Open the create database dialog](images/screenshots/create-database-step-1.png){ .img-border }
![Open the create database dialog](../images/screenshots/create-database-step-1.png){ .img-border }
<figcaption>Figure 4: Open the create database dialog.</figcaption>
</figure>
......@@ -143,14 +143,14 @@ A user wants to create a database in DBRepo.
finally create the database, press "Create" :material-numeric-3-circle-outline: as seen in Figure 5.
<figure markdown>
![Create database form](images/screenshots/create-database-step-2.png){ .img-border }
![Create database form](../images/screenshots/create-database-step-2.png){ .img-border }
<figcaption>Figure 5: Create database form.</figcaption>
</figure>
After a few seconds, you can see the created database in the "Recent Databases" list, as seen in Figure 6.
<figure markdown>
![View the created database](images/screenshots/create-database-step-3.png){ .img-border }
![View the created database](../images/screenshots/create-database-step-3.png){ .img-border }
<figcaption>Figure 6: View the created database.</figcaption>
</figure>
......@@ -213,7 +213,7 @@ access to. This is the default for self-created databases like above in [Create
Figure 7.
<figure markdown>
![Open the import CSV form](images/screenshots/import-dataset-step-1.png){ .img-border }
![Open the import CSV form](../images/screenshots/import-dataset-step-1.png){ .img-border }
<figcaption>Figure 7: Open the import CSV form.</figcaption>
</figure>
......@@ -221,7 +221,7 @@ access to. This is the default for self-created databases like above in [Create
as seen in Figure 8.
<figure markdown>
![Basic table information](images/screenshots/import-dataset-step-2.png){ .img-border }
![Basic table information](../images/screenshots/import-dataset-step-2.png){ .img-border }
<figcaption>Figure 8: Basic table information.</figcaption>
</figure>
......@@ -238,7 +238,7 @@ access to. This is the default for self-created databases like above in [Create
provide this information in :material-numeric-6-circle-outline:.
<figure markdown>
![Dataset metadata necessary for import](images/screenshots/import-dataset-step-3.png){ .img-border }
![Dataset metadata necessary for import](../images/screenshots/import-dataset-step-3.png){ .img-border }
<figcaption>Figure 9: Dataset metadata necessary for import.</figcaption>
</figure>
......@@ -246,7 +246,7 @@ access to. This is the default for self-created databases like above in [Create
dataset file onto the field in Figure 10.
<figure markdown>
![Dataset import file](images/screenshots/import-dataset-step-4.png){ .img-border }
![Dataset import file](../images/screenshots/import-dataset-step-4.png){ .img-border }
<figcaption>Figure 10: Dataset import file.</figcaption>
</figure>
......@@ -261,7 +261,7 @@ access to. This is the default for self-created databases like above in [Create
definitions by clicking the "ADD COLUMN" button in Figure 11.
<figure markdown>
![Confirm the table schema and provide missing information](images/screenshots/import-dataset-step-5.png){ .img-border }
![Confirm the table schema and provide missing information](../images/screenshots/import-dataset-step-5.png){ .img-border }
<figcaption>Figure 11: Confirm the table schema and provide missing information.</figcaption>
</figure>
......@@ -270,7 +270,7 @@ access to. This is the default for self-created databases like above in [Create
dataset as seen in Figure 12.
<figure markdown>
![Confirm the table schema and provide missing information](images/screenshots/import-dataset-step-6.png){ .img-border }
![Confirm the table schema and provide missing information](../images/screenshots/import-dataset-step-6.png){ .img-border }
<figcaption>Figure 12: Confirm the table schema and provide missing information.</figcaption>
</figure>
......@@ -284,7 +284,7 @@ access to. This is the default for self-created databases like above in [Create
13.
<figure markdown>
![Table data](images/screenshots/import-dataset-step-7.png){ .img-border }
![Table data](../images/screenshots/import-dataset-step-7.png){ .img-border }
<figcaption>Figure 13: Table data.</figcaption>
</figure>
......@@ -432,7 +432,7 @@ A user wants to import a database dump in `.sql` (or in `.sql.gz`) format into D
clicking the "OK" button :material-numeric-4-circle-outline:.
<figure markdown>
![Setup New Connection in MySQL Workbench](images/screenshots/import-database-dump-step-1.png)
![Setup New Connection in MySQL Workbench](../images/screenshots/import-database-dump-step-1.png)
<figcaption>Figure 14: Setup New Connection in MySQL Workbench.</figcaption>
</figure>
......@@ -440,7 +440,7 @@ A user wants to import a database dump in `.sql` (or in `.sql.gz`) format into D
:material-numeric-1-circle-outline: and basic connection and version information :material-numeric-2-circle-outline:.
<figure markdown>
![Server status of the Data Database in MySQL Workbench](images/screenshots/import-database-dump-step-2.png)
![Server status of the Data Database in MySQL Workbench](../images/screenshots/import-database-dump-step-2.png)
<figcaption>Figure 15: Server status of the Data Database in MySQL Workbench.</figcaption>
</figure>
......@@ -451,7 +451,7 @@ A user wants to import a database dump in `.sql` (or in `.sql.gz`) format into D
The import starts after clicking "Start Import" :material-numeric-5-circle-outline:.
<figure markdown>
![Data Import/Restore in MySQL Workbench](images/screenshots/import-database-dump-step-3.png)
![Data Import/Restore in MySQL Workbench](../images/screenshots/import-database-dump-step-3.png)
<figcaption>Figure 16: Data Import/Restore in MySQL Workbench.</figcaption>
</figure>
......@@ -510,7 +510,7 @@ A user wants to import live data from e.g. sensor measurements fast and without
database via JDBC, you can obtain the connection string in the UI under the database info (c.f. Figure 14).
<figure markdown>
![JDBC connection information](images/screenshots/database-jdbc.png){ .img-border }
![JDBC connection information](../images/screenshots/database-jdbc.png){ .img-border }
<figcaption>Figure 14: JDBC connection information.</figcaption>
</figure>
......@@ -526,7 +526,7 @@ A user wants to import live data from e.g. sensor measurements fast and without
database via AMQP, you can obtain the connection string in the UI under the table info (c.f. Figure 14).
<figure markdown>
![AMQP connection information](images/screenshots/table-amqp.png){ .img-border }
![AMQP connection information](../images/screenshots/table-amqp.png){ .img-border }
<figcaption>Figure 14: AMQP connection information.</figcaption>
</figure>
......@@ -550,7 +550,7 @@ A user wants to create a subset and export it as csv file.
Figure 17.
<figure markdown>
![Open the create subset form](images/screenshots/export-subset-step-1.png){ .img-border }
![Open the create subset form](../images/screenshots/export-subset-step-1.png){ .img-border }
<figcaption>Figure 17: Open the create subset form.</figcaption>
</figure>
......@@ -562,7 +562,7 @@ A user wants to create a subset and export it as csv file.
Once you are confident the query covers the desired result, click ":material-run: Create".
<figure markdown>
![Subset query building](images/screenshots/export-subset-step-2.png){ .img-border }
![Subset query building](../images/screenshots/export-subset-step-2.png){ .img-border }
<figcaption>Figure 18: Subset query building.</figcaption>
</figure>
......@@ -571,7 +571,7 @@ A user wants to create a subset and export it as csv file.
View" on the top (c.f. Figure 19).
<figure markdown>
![Subset result set](images/screenshots/export-subset-step-3.png){ .img-border }
![Subset result set](../images/screenshots/export-subset-step-3.png){ .img-border }
<figcaption>Figure 19: Subset result set.</figcaption>
</figure>
......@@ -584,7 +584,7 @@ A user wants to create a subset and export it as csv file.
a csv file by clicking the ":material-download: DATA .CSV" button :material-numeric-2-circle-outline:.
<figure markdown>
![Subset information](images/screenshots/export-subset-step-4.png){ .img-border }
![Subset information](../images/screenshots/export-subset-step-4.png){ .img-border }
<figcaption>Figure 20: Subset information.</figcaption>
</figure>
......@@ -648,7 +648,7 @@ A user wants to create a subset and export it as csv file.
under the database info (c.f. Figure 20).
<figure markdown>
![JDBC connection information](images/screenshots/database-jdbc.png){ .img-border }
![JDBC connection information](../images/screenshots/database-jdbc.png){ .img-border }
<figcaption>Figure 20: JDBC connection information.</figcaption>
</figure>
......@@ -728,7 +728,7 @@ A user wants to assign a persistent identifier to a database owned by them.
the ":material-identifier: GET PID" button :material-numeric-1-circle-outline: as seen in Figure 21.
<figure markdown>
![Open the get persisent identifier form](images/screenshots/assign-database-pid-step-1.png){ .img-border }
![Open the get persisent identifier form](../images/screenshots/assign-database-pid-step-1.png){ .img-border }
<figcaption>Figure 21: Open the get persisent identifier form.</figcaption>
</figure>
......@@ -743,7 +743,7 @@ A user wants to assign a persistent identifier to a database owned by them.
:material-numeric-8-circle-outline:.
<figure markdown>
![Identifier creator fields](images/screenshots/assign-database-pid-step-2.png){ .img-border }
![Identifier creator fields](../images/screenshots/assign-database-pid-step-2.png){ .img-border }
<figcaption>Figure 22: Identifier creator fields.</figcaption>
</figure>
......@@ -753,7 +753,7 @@ A user wants to assign a persistent identifier to a database owned by them.
again :material-numeric-4-circle-outline: if they are not needed in Figure 23.
<figure markdown>
![JDBC connection information](images/screenshots/assign-database-pid-step-3.png){ .img-border }
![JDBC connection information](../images/screenshots/assign-database-pid-step-3.png){ .img-border }
<figcaption>Figure 23: Identifier title fields.</figcaption>
</figure>
......@@ -764,7 +764,7 @@ A user wants to assign a persistent identifier to a database owned by them.
publication month and publication day in Figure 24.
<figure markdown>
![Identifier description fields and publishing information](images/screenshots/assign-database-pid-step-4.png){ .img-border }
![Identifier description fields and publishing information](../images/screenshots/assign-database-pid-step-4.png){ .img-border }
<figcaption>Figure 24: Identifier description fields and publishing information.</figcaption>
</figure>
......@@ -775,7 +775,7 @@ A user wants to assign a persistent identifier to a database owned by them.
your data.
<figure markdown>
![Related identifiers, license and language of the identifier](images/screenshots/assign-database-pid-step-5.png){ .img-border }
![Related identifiers, license and language of the identifier](../images/screenshots/assign-database-pid-step-5.png){ .img-border }
<figcaption>Figure 25: Related identifiers, license and language of the identifier.</figcaption>
</figure>
......@@ -787,7 +787,7 @@ A user wants to assign a persistent identifier to a database owned by them.
language on the identifier summary page (c.f. Figure 26).
<figure markdown>
![Identifier funder information](images/screenshots/assign-database-pid-step-6.png){ .img-border }
![Identifier funder information](../images/screenshots/assign-database-pid-step-6.png){ .img-border }
<figcaption>Figure 26: Identifier funder information.</figcaption>
</figure>
......@@ -795,7 +795,7 @@ A user wants to assign a persistent identifier to a database owned by them.
result is displayed in Figure 27.
<figure markdown>
![Identifier summary page](images/screenshots/assign-database-pid-step-7.png){ .img-border }
![Identifier summary page](../images/screenshots/assign-database-pid-step-7.png){ .img-border }
<figcaption>Figure 27: Identifier summary page.</figcaption>
</figure>
......@@ -884,7 +884,7 @@ A user wants a public database to be private and only give specific users access
the dialog (c.f. Figure 28).
<figure markdown>
![Database settings for visibility and access](images/screenshots/private-database-access-step-1.png){ .img-border }
![Database settings for visibility and access](../images/screenshots/private-database-access-step-1.png){ .img-border }
<figcaption>Figure 28: Database settings for visibility and access.</figcaption>
</figure>
......@@ -893,7 +893,7 @@ A user wants a public database to be private and only give specific users access
allowing users to view the private data (c.f. Figure 29).
<figure markdown>
![Database acccess dialog](images/screenshots/private-database-access-step-2.png){ .img-border }
![Database acccess dialog](../images/screenshots/private-database-access-step-2.png){ .img-border }
<figcaption>Figure 29: Database acccess dialog.</figcaption>
</figure>
......
.docs/api/metadata-db.md
+ 29
− 1
View file @ a5c97882
......@@ -2,4 +2,32 @@
author: Martin Weise
---
TBD
\ No newline at end of file
!!! debug "Debug Information"
Image: [`bitnami/mariadb-galera:11.2.2-debian-11-r0`](https://hub.docker.com/r/bitnami/mariadb-galera)
* Ports: 3306/tcp
* JDBC: `jdbc://mariadb:<hostname>:3306`
## Overview
The metadata database is the single, central source of truth within DBRepo and holds all metadata information for
interaction between the services and displaying information in the UI.
On the first start, the schema is generated by the file `/docker-entrypoint-initdb.d/setup-schema.sql` within the
container. You can add custom files that should be executed on the first start by placing them into the
`/docker-entrypoint-initdb.d/` folder as well. For example:
```yaml
services:
dbrepo-metadata-db:
...
volumes:
- /path/to/setup-some-data.sql:/docker-entrypoint-initdb.d/setup-some-data.sql
...
```
!!! warning "Alphabetic Filename Sorting"
Beware that the init script provided by Bitnami executes files in alphabetic order! For example: the file
`setup-schema.sql` is executed **after** the file `setup-data.sql`!
\ No newline at end of file
.docs/api/open-api.md
+ 1
− 1
View file @ a5c97882
......@@ -6,7 +6,7 @@ author: Martin Weise
## tl;dr
[:simple-swagger: &nbsp;View Swagger-UI](../../swagger/){ .md-button .md-button--primary tabindex=-1 }
[:simple-swagger: &nbsp;View Swagger-UI](../../rest/){ .md-button .md-button--primary tabindex=-1 }
## Overview
......
.docs/api/python.md
+ 1
− 1
View file @ a5c97882
......@@ -6,7 +6,7 @@ author: Martin Weise
## tl;dr
[:fontawesome-solid-cube: &nbsp;View Docs](../sphinx){ .md-button .md-button--primary }
[:fontawesome-solid-cube: &nbsp;View Docs](../../python){ .md-button .md-button--primary }
## Installing
......
.docs/api/ui.md
+ 1
− 1
View file @ a5c97882
......@@ -52,7 +52,7 @@ User Interface on development.
### Example
See the [Usage Overview](../usage-overview/) page for detailed examples.
See the [API Overview](..) page for detailed examples.
## Limitations
......
.docs/concepts/authentication.md
+ 22
− 4
View file @ a5c97882
......@@ -2,14 +2,32 @@
author: Martin Weise
---
This is a short conceptional overview on the authentication mechanisms provided by Keycloak. We use Keycloak as the
single source of truth for authorization and user management.
## Basic Authentication
DBRepo supports `Basic` authentication (that is with username and password) in the REST API. When requesting a resource
from e.g. the [Metadata Service](../../api/metadata-service), the service internally retrieves a
[Bearer Token](#bearer-authentication) from the [Auth Service](../../api/auth-service) and checks if a certain role
is present to perform the desired action. This entails a small overhead on each request, since the service does not
store anything.
## Bearer Authentication
TBD
DBRepo supports `Bearer` authentication by accepting JWT tokens in the
[`Authorization`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Authorization) header of HTTP requests to
any service. There are two (major) types of tokens:
## Basic Authentication
* Access tokens who are short lived (e.g. 15 minutes) to access resources, and
* Refresh token who are long lived (e.g. 30 days) to request new access tokens without having to provide username and
password again.
TBD
The [User Interface](../../api/ui) for example refreshes the token on-the-fly by intercepting each request and, in case
of an expired access token, requests a new one without having to terminate the request. This happens only once after the
access token has expired (after e.g. 15 minutes).
## OpenID Connect (OIDC)
TBD
\ No newline at end of file
We use the widely accepted authentication protocol OIDC for client authentication. Other protocols are e.g. SAML2 which
are not used by DBRepo.
\ No newline at end of file
.docs/concepts/databases.md
+ 10
− 3
View file @ a5c97882
......@@ -4,15 +4,22 @@ author: Martin Weise
## Relational Database
TBD
DBRepo manages relational databases that store information relations in tables.
## Query
TBD
A query is the method to interact with a relational database and is used to read/write data or to create/change/delete
schema information e.g. tables. DBRepo uses a query store to store certain (important) queries that generate subsets
to restore the exact same subset at a later point.
## System Versioning
TBD
DBRepo uses a mechanism offered by SQL:2013 to version tables with the system (=server) time. When inserting a tuple
into a system-versioned table, the database engine maintains invisible `ROW_START` and `ROW_END` timestamp columns to
denote a tuple validity. When deleting a tuple, the database engine actually just marks the tuple as `ROW_END = NOW()`
and does not delete the tuple.
At a later point in time, the (historic) tuple can still be queried using system versioning.
## Data Ingest
......
.docs/concepts/messaging.md
+ 15
− 6
View file @ a5c97882
......@@ -2,14 +2,23 @@
author: Martin Weise
---
## Tuple
We use the message broker RabbitMQ and a single exchange with distributed (`quorum`) queue. Both are named `dbrepo`.
TBD
<figure markdown>
<img src="../../images/exchange-binding.png" />
<caption>Figure 1: Exchange binding in DBRepo.</caption>
</figure>
## AMQP
## Tuple
TBD
A tuple is the atomic granularity of information when transmitting data through message-queue systems. Represented as
JSON, a tuple looks like this in RabbitMQ:
## MQTT
```JSON
{"firstname": "foo", "lastname": "bar"}
```
## AMQP
TBD
\ No newline at end of file
DBRepo uses AMQP to route messages which allows for both Basic/Bearer authentication. For more information please
consult the [RabbitMQ AMQP](https://www.rabbitmq.com/tutorials/amqp-concepts) documentation.
\ No newline at end of file
.docs/index.md
+ 1
− 1
View file @ a5c97882
......@@ -29,7 +29,7 @@ Installing DBRepo is very easy or
## Who is using DBRepo?
![Logos of DBRepo adopters](../images/logos.png)
![Logos of DBRepo adopters](images/logos.png)
## How can I try DBRepo
......
.docs/publications.md
+ 2
− 2
View file @ a5c97882
......@@ -20,8 +20,8 @@ hide:
DBRepo logo in various formats:
* PNG: [bigger](./images/logo/logo.png) ([smaller](./images/logo/favicon.png))
* SVG: [bigger](./images/logo/logo.svg) ([smaller](./images/logo/favicon.svg))
* PNG: [bigger](../images/logos/logo.png) ([smaller](../images/logos/favicon.png))
* SVG: [bigger](../images/logos/logo.svg) ([smaller](../images/logos/favicon.svg))
## Refereed
......
.gitlab-ci.yml
+ 4
− 4
View file @ a5c97882
......@@ -619,10 +619,10 @@ release-docs:
- "mkdir -p ./final/${APP_VERSION}/swagger"
script:
- "make gen-lib-doc gen-docs-doc"
- "cp -r ./lib/python/docs/build/html ./final/${APP_VERSION}/sphinx" # sphinx
- "cp .docs/.swagger/api.yaml ./final/${APP_VERSION}/swagger/api.yaml" # swagger
- "cp .docs/.swagger/swagger-ui.html ./final/${APP_VERSION}/swagger/index.html" # swagger
- "cp .docs/.swagger/custom.css ./final/${APP_VERSION}/swagger/custom.css" # swagger
- "cp -r ./lib/python/docs/build/html ./final/${APP_VERSION}/python" # sphinx
- "cp .docs/.swagger/api.yaml ./final/${APP_VERSION}/rest/api.yaml" # swagger
- "cp .docs/.swagger/swagger-ui.html ./final/${APP_VERSION}/rest/index.html" # swagger
- "cp .docs/.swagger/custom.css ./final/${APP_VERSION}/rest/custom.css" # swagger
- "cp -r ./site/* ./final/${APP_VERSION}" # mkdocs
- eval $(ssh-agent -s)
- "mkdir -p /root/.ssh"
......
dbrepo-analyse-service/Pipfile.lock
+ 25
− 43
View file @ a5c97882
......@@ -167,20 +167,19 @@
},
"boto3": {
"hashes": [
"sha256:8f9c43c54b3dfaa36c4a0d7b42c417227a515bc7a2e163e62802780000a5a3e2",
"sha256:cea2365a25b2b83a97e77f24ac6f922ef62e20636b42f9f6ee9f97188f9c1c03"
"sha256:42b140fc850cf261ee4b1e8ef527fa071b1f1592a6d6a68d34b29f37cc46b4dd",
"sha256:56bec52d485d5670ce96d53ae7b2cd4ae4e8a705fb2298a21093cdd77d642331"
],
"index": "pypi",
"markers": "python_version >= '3.8'",
"version": "==1.34.119"
"version": "==1.34.123"
},
"botocore": {
"hashes": [
"sha256:4bdf7926a1290b2650d62899ceba65073dd2693e61c35f5cdeb3a286a0aaa27b",
"sha256:b253f15b24b87b070e176af48e8ef146516090429d30a7d8b136a4c079b28008"
"sha256:8c34ada2a708c82e7174bff700611643db7ce2cb18f1130c35045c24310d299d",
"sha256:a8577f6574600c4d159b5cd103ee05744a443d77f7778304e17307940b369c4f"
],
"markers": "python_version >= '3.8'",
"version": "==1.34.119"
"version": "==1.34.123"
},
"certifi": {
"hashes": [
......@@ -392,9 +391,11 @@
},
"dbrepo": {
"hashes": [
"sha256:2506475fc8fb3f4fdd722e3e92f8e6ad28d0707023c3d8ea5d6d076cef71f395",
"sha256:2bdb48c70b4c99b5044fbfc12aa653c1e9281ca8913a433cc08a1e14cb4bd2ef"
],
"path": "./lib/dbrepo-1.4.4.tar.gz"
"path": "./lib/dbrepo-1.4.4.tar.gz",
"version": "==1.4.4"
},
"events": {
"hashes": [
......@@ -408,7 +409,6 @@
"sha256:a4785e48b045528f5bfe627b6ad554ff32def154f42372786903b7abcfe1aa16"
],
"index": "pypi",
"markers": "python_version >= '3.7'",
"version": "==1.2.1"
},
"flasgger": {
......@@ -424,7 +424,6 @@
"sha256:ceb27b0af3823ea2737928a4d99d125a06175b8512c445cbd9a9ce200ef76842"
],
"index": "pypi",
"markers": "python_version >= '3.8'",
"version": "==3.0.3"
},
"flask-cors": {
......@@ -449,7 +448,6 @@
"sha256:9215d05a9413d3855764bcd67035e75819d23af2fafb6b55197eb5a3313fdfb2"
],
"index": "pypi",
"markers": "python_version >= '3.7' and python_version < '4'",
"version": "==4.6.0"
},
"frozenlist": {
......@@ -580,7 +578,6 @@
"sha256:fbfdce91239fe306772faab57597186710d5699213f4df099d1612da7320d682"
],
"index": "pypi",
"markers": "python_version >= '3.8'",
"version": "==24.2.1"
},
"greenlet": {
......@@ -645,7 +642,6 @@
"sha256:fe754d231288e1e64323cfad462fcee8f0288654c10bdf4f603a39ed923bef33"
],
"index": "pypi",
"markers": "python_version >= '3.7'",
"version": "==3.0.3"
},
"gunicorn": {
......@@ -654,7 +650,6 @@
"sha256:4a0b436239ff76fb33f11c07a16482c521a7e09c1ce3cc293c2330afe01bec63"
],
"index": "pypi",
"markers": "python_version >= '3.7'",
"version": "==22.0.0"
},
"idna": {
......@@ -710,7 +705,6 @@
"sha256:61c9170f92e736b530655e75374681d4fcca9cfa8763ab42be57353b2b203494"
],
"index": "pypi",
"markers": "python_version >= '3.6'",
"version": "==1.3.1"
},
"markupsafe": {
......@@ -931,7 +925,6 @@
"sha256:ffa75af20b44f8dba823498024771d5ac50620e6915abac414251bd971b4529f"
],
"index": "pypi",
"markers": "python_version >= '3.9'",
"version": "==1.26.4"
},
"opensearch-py": {
......@@ -940,16 +933,15 @@
"sha256:b6e78b685dd4e9c016d7a4299cf1de69e299c88322e3f81c716e6e23fe5683c1"
],
"index": "pypi",
"markers": "python_version >= '3.8' and python_version < '4'",
"version": "==2.6.0"
},
"packaging": {
"hashes": [
"sha256:2ddfb553fdf02fb784c234c7ba6ccc288296ceabec964ad2eae3777778130bc5",
"sha256:eb82c5e3e56209074766e6885bb04b8c38a0c015d0a30036ebe7ece34c9989e9"
"sha256:026ed72c8ed3fcce5bf8950572258698927fd1dbda10a5e981cdf0ac37f4f002",
"sha256:5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124"
],
"markers": "python_version >= '3.7'",
"version": "==24.0"
"markers": "python_version >= '3.8'",
"version": "==24.1"
},
"pandas": {
"hashes": [
......@@ -984,7 +976,6 @@
"sha256:eee3a87076c0756de40b05c5e9a6069c035ba43e8dd71c379e68cab2c20f16ad"
],
"index": "pypi",
"markers": "python_version >= '3.9'",
"version": "==2.2.2"
},
"pika": {
......@@ -1063,7 +1054,6 @@
"sha256:ea91b002777bf643bb20dd717c028ec43216b24a6001a280f83877fd2655d0b4"
],
"index": "pypi",
"markers": "python_version >= '3.8'",
"version": "==2.7.3"
},
"pydantic-core": {
......@@ -1245,7 +1235,6 @@
"sha256:70761cfe03c773ceb22aa2f671b4757976145175cdfca038c02654d061d6dcc6"
],
"index": "pypi",
"markers": "python_version >= '3.8'",
"version": "==2.32.3"
},
"rpds-py": {
......@@ -1395,11 +1384,11 @@
},
"typing-extensions": {
"hashes": [
"sha256:6024b58b69089e5a89c347397254e35f1bf02a907728ec7fee9bf0fe837d203a",
"sha256:915f5e35ff76f56588223f15fdd5938f9a1cf9195c0de25130c627e4d597f6d1"
"sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d",
"sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"
],
"markers": "python_version >= '3.8'",
"version": "==4.12.1"
"version": "==4.12.2"
},
"tzdata": {
"hashes": [
......@@ -1414,7 +1403,7 @@
"sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d",
"sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19"
],
"markers": "python_version >= '3.10'",
"markers": "python_version >= '3.8'",
"version": "==2.2.1"
},
"werkzeug": {
......@@ -1826,7 +1815,6 @@
"sha256:fd27d8b49e574e50caa65196d908f80e4dff64d7e592d0c59788b45aad7e8b35"
],
"index": "pypi",
"markers": "python_version >= '3.8'",
"version": "==7.5.3"
},
"docker": {
......@@ -1873,16 +1861,15 @@
"sha256:b6e78b685dd4e9c016d7a4299cf1de69e299c88322e3f81c716e6e23fe5683c1"
],
"index": "pypi",
"markers": "python_version >= '3.8' and python_version < '4'",
"version": "==2.6.0"
},
"packaging": {
"hashes": [
"sha256:2ddfb553fdf02fb784c234c7ba6ccc288296ceabec964ad2eae3777778130bc5",
"sha256:eb82c5e3e56209074766e6885bb04b8c38a0c015d0a30036ebe7ece34c9989e9"
"sha256:026ed72c8ed3fcce5bf8950572258698927fd1dbda10a5e981cdf0ac37f4f002",
"sha256:5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124"
],
"markers": "python_version >= '3.7'",
"version": "==24.0"
"markers": "python_version >= '3.8'",
"version": "==24.1"
},
"pluggy": {
"hashes": [
......@@ -1944,7 +1931,6 @@
"sha256:de4bb8104e201939ccdc688b27a89a7be2079b22e2bd2b07f806b6ba71117977"
],
"index": "pypi",
"markers": "python_version >= '3.8'",
"version": "==8.2.2"
},
"python-dateutil": {
......@@ -1961,7 +1947,6 @@
"sha256:70761cfe03c773ceb22aa2f671b4757976145175cdfca038c02654d061d6dcc6"
],
"index": "pypi",
"markers": "python_version >= '3.8'",
"version": "==2.32.3"
},
"requests-mock": {
......@@ -1970,7 +1955,6 @@
"sha256:e9e12e333b525156e82a3c852f22016b9158220d2f47454de9cae8a77d371401"
],
"index": "pypi",
"markers": "python_version >= '3.5'",
"version": "==1.12.1"
},
"six": {
......@@ -1993,7 +1977,6 @@
"sha256:54d330d085c0a11fc5da0b001af87aec4dd3e814104376bf7513e8646c77442a"
],
"index": "pypi",
"markers": "python_version >= '3.7'",
"version": "==0.0.1rc1"
},
"testcontainers-opensearch": {
......@@ -2001,23 +1984,22 @@
"sha256:0bdf270b5b7f53915832f7c31dd2bd3ffdc20b534ea6b32231cc7003049bd0e1"
],
"index": "pypi",
"markers": "python_version >= '3.7'",
"version": "==0.0.1rc1"
},
"typing-extensions": {
"hashes": [
"sha256:6024b58b69089e5a89c347397254e35f1bf02a907728ec7fee9bf0fe837d203a",
"sha256:915f5e35ff76f56588223f15fdd5938f9a1cf9195c0de25130c627e4d597f6d1"
"sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d",
"sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"
],
"markers": "python_version >= '3.8'",
"version": "==4.12.1"
"version": "==4.12.2"
},
"urllib3": {
"hashes": [
"sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d",
"sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19"
],
"markers": "python_version >= '3.10'",
"markers": "python_version >= '3.8'",
"version": "==2.2.1"
},
"wrapt": {
......
dbrepo-analyse-service/app.py
+ 10
− 12
View file @ a5c97882
import json
import os
import logging
from typing import Any, List
import os
from json import dumps
import requests.exceptions
from dbrepo.api.dto import ApiError
from flasgger import LazyJSONEncoder, Swagger
from flask_httpauth import HTTPBasicAuth, MultiAuth, HTTPTokenAuth
from flasgger.utils import swag_from
from flasgger import LazyJSONEncoder, Swagger, swag_from
from flask import Flask, Response, request
from flask_cors import CORS
from flask_httpauth import HTTPBasicAuth, MultiAuth, HTTPTokenAuth
from prometheus_flask_exporter import PrometheusMetrics
from botocore.exceptions import ClientError
......@@ -52,14 +50,14 @@ app = Flask(__name__)
cors = CORS(app, resources={r"/api/*": {"origins": "*"}})
metrics = PrometheusMetrics(app)
metrics.info("app_info", "Application info", version="0.0.1")
app.config["SWAGGER"] = {"openapi": "3.0.1", "title": "Swagger UI", "uiversion": 3}
token_auth = HTTPTokenAuth(scheme='Bearer')
basic_auth = HTTPBasicAuth()
auth = MultiAuth(token_auth, basic_auth)
metrics = PrometheusMetrics(app)
metrics.info("app_info", "Application info", version="1.4.4")
app.config["SWAGGER"] = {"openapi": "3.0.1", "title": "Swagger UI", "uiversion": 3}
swagger_config = {
"headers": [],
"specs": [
......@@ -165,7 +163,7 @@ template = {
},
"externalDocs": {
"description": "Sourcecode Documentation",
"url": "https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/__APPVERSION__/"
"url": "https://www.ifs.tuwien.ac.at/infrastructures/dbrepo/1.4.4/"
},
"servers": [
{
......@@ -185,7 +183,7 @@ app.config["JWT_ALGORITHM"] = "HS256"
app.config["JWT_PUBKEY"] = '-----BEGIN PUBLIC KEY-----\n' + os.getenv("JWT_PUBKEY",
"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqqnHQ2BWWW9vDNLRCcxD++xZg/16oqMo/c1l+lcFEjjAIJjJp/HqrPYU/U9GvquGE6PbVFtTzW1KcKawOW+FJNOA3CGo8Q1TFEfz43B8rZpKsFbJKvQGVv1Z4HaKPvLUm7iMm8Hv91cLduuoWx6Q3DPe2vg13GKKEZe7UFghF+0T9u8EKzA/XqQ0OiICmsmYPbwvf9N3bCKsB/Y10EYmZRb8IhCoV9mmO5TxgWgiuNeCTtNCv2ePYqL/U0WvyGFW0reasIK8eg3KrAUj8DpyOgPOVBn3lBGf+3KFSYi+0bwZbJZWqbC/Xlk20Go1YfeJPRIt7ImxD27R/lNjgDO/MwIDAQAB") + '\n-----END PUBLIC KEY-----'
app.config["AUTH_SERVICE_ENDPOINT"] = os.getenv("AUTH_SERVICE_ENDPOINT", "http://localhost/api/auth")
app.config["AUTH_SERVICE_CLIENT"] = os.getenv("AUTH_SERVICE_CLIENT", "dbrepo")
app.config["AUTH_SERVICE_CLIENT"] = os.getenv("AUTH_SERVICE_CLIENT", "dbrepo-client")
app.config["AUTH_SERVICE_CLIENT_SECRET"] = os.getenv("AUTH_SERVICE_CLIENT_SECRET", "MUwRc7yfXSJwX8AdRMWaQC3Nep1VjwgG")
app.config["ADMIN_USERNAME"] = os.getenv('ADMIN_USERNAME', 'admin')
app.config["ADMIN_PASSWORD"] = os.getenv('ADMIN_PASSWORD', 'admin')
......@@ -253,7 +251,7 @@ def analyse_datatypes():
if filename is None or separator is None:
return Response(
json.dumps({'success': False, 'message': "Missing required query parameters 'filename' and 'separator'"}),
dumps({'success': False, 'message': "Missing required query parameters 'filename' and 'separator'"}),
mimetype="application/json"), 400
try:
......
dbrepo-analyse-service/as-yml/analyse_datatypes.yml
+ 1
− 1
View file @ a5c97882
......@@ -2,7 +2,7 @@ tags:
- analyse-endpoint
summary: "Determine datatypes"
operationId: analyse_datatypes
description: "This is a simple API which returns the datatypes of a (path) csv file"
description: "Determines MySQL 8 datatypes of a given dataset. Requires role `table-semantic-analyse`."
consumes:
- "application/json"
produces:
......
dbrepo-analyse-service/as-yml/analyse_keys.yml
+ 2
− 2
View file @ a5c97882
tags:
- analyse-endpoint
summary: "Determine primary keys"
summary: "Determine keys"
operationId: analyse_keys
description: "This is a simple API which returns the primary keys + ranking of a (path) csv file"
description: "Determines primary keys of a given dataset. Requires role `table-semantic-analyse`."
consumes:
- "application/json"
produces:
......
dbrepo-analyse-service/as-yml/analyse_table_stat.yml deleted 100644 → 0
+ 0
− 41
View file @ 4fe0c58e
tags:
- analyse-endpoint
summary: Determine table statistics
operationId: analyse_table_stat
parameters:
- name: database_id
in: path
required: true
example: 1
schema:
type: integer
format: int64
- name: table_id
in: path
required: true
example: 1
schema:
type: integer
format: int64
security:
- bearerAuth: [ ]
- basicAuth: [ ]
responses:
202:
description: Determined statistics
content:
application/json:
schema:
$ref: '#/components/schemas/TableStats'
400:
description: "Missing parameters"
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorDto'
404:
description: "Table not found"
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorDto'
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment