Skip to content
Snippets Groups Projects
Verified Commit 1e3c4a73 authored by Martin Weise's avatar Martin Weise
Browse files

Updated docs

parent 74198a27
No related branches found
No related tags found
No related merge requests found
---
author: Martin Weise
---
## tl;dr
tbd
## Description
TBD
## Solution
TBD
## DBRepo Features
- [x] Large Dataset ≈15GiB
- [x] Subset Citation using PID
- [x] External access from Grafana Dashboard
\ No newline at end of file
......@@ -9,11 +9,34 @@ author: Martin Weise
## Description
TBD
This data base contains concentrations of hazardous substances and other water quality parameters in different
environmental compartments:
* river water (water and suspended sediments)
* ground water
* waste water (treated and untreated) and sewage sludge
* storm water runoff from combined and separate sewer systems
* atmospheric deposition
* soil
Data from many different data sources were collected, checked and combined and meta data were harmonized to allow for
a combined data evaluation.
## Solution
TBD
We imported the database from the [archive](https://doi.org/10.48436/yaecs-dgr27) repository, converted the PostgreSQL
language code to MariaDB:
* `"current_user"()` to `current_user()`
* `SEQUENCE CACHE 1` to `NOCACHE` because of MariaDB Galera distribution not being able to cache sequences
* `(0)::double precision` to `0.0`
* `character varying` to `text` because MariaDB needs sizes, `text` data type can have arbitrary size
* `!~~` to `NOT LIKE`
* `ANY ARRAY` to `<array> OR <array2>` because of different MariaDB syntax
The complex nature of the views (i.e. `SELECT-FROM-SELECT` statements) required a logical overhaul of how DBRepo obtains
view metadata. As a consequence, we removed the SQL parser that tried to parse the metadata up to a depth of 10 to a
simple query to the `information_schema` that is maintained by the database engine.
## DBRepo Features
......
......@@ -2,10 +2,27 @@
author: Martin Weise
---
## tl;dr
[:fontawesome-solid-database: &nbsp;Dataset](https://dbrepo1.ec.tuwien.ac.at/pid/14){ .md-button .md-button--primary target="_blank" }
[:simple-grafana: &nbsp;Dashboard](https://dbrepo1.ec.tuwien.ac.at/admin/grafana/d/8meGcJD4k/wiener-linien){ .md-button .md-button--secondary target="_blank" }
## Description
TBD
The Subway Transportation Data-Dataset is a comprehensive and dynamic collection of data that captures the intricate
details of the city's public transportation system. This dataset encompasses a wide array of information, including bus
and tram schedules, subway routes, ticketing details, and real-time updates on vehicle locations.
## Solution
TBD
\ No newline at end of file
We wrote an algorithm that parses open data (available) information from Wiener Linien, Vienna's public transportation
agency directly and feeds it, after some cleaning, into DBRepo on a 5-minute interval.
![Subway Transportation Data Dashboard](images/screenshots/transportation-dashboard.png)
## DBRepo Features
- [x] Dynamic data (live data)
- [x] System versioning
- [x] Subset exploration
- [x] External visualization of the database
\ No newline at end of file
.docs/images/screenshots/transportation-dashboard.png

145 KiB

......@@ -137,7 +137,7 @@ services:
DELETED_RECORD: "${DELETED_RECORD:-persistent}"
GRANULARITY: "${GRANULARITY:-YYYY-MM-DDThh:mm:ssZ}"
JWT_PUBKEY: "${JWT_PUBKEY:-MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqqnHQ2BWWW9vDNLRCcxD++xZg/16oqMo/c1l+lcFEjjAIJjJp/HqrPYU/U9GvquGE6PbVFtTzW1KcKawOW+FJNOA3CGo8Q1TFEfz43B8rZpKsFbJKvQGVv1Z4HaKPvLUm7iMm8Hv91cLduuoWx6Q3DPe2vg13GKKEZe7UFghF+0T9u8EKzA/XqQ0OiICmsmYPbwvf9N3bCKsB/Y10EYmZRb8IhCoV9mmO5TxgWgiuNeCTtNCv2ePYqL/U0WvyGFW0reasIK8eg3KrAUj8DpyOgPOVBn3lBGf+3KFSYi+0bwZbJZWqbC/Xlk20Go1YfeJPRIt7ImxD27R/lNjgDO/MwIDAQAB}"
LOG_LEVEL: trace
LOG_LEVEL: ${LOG_LEVEL:-info}
METADATA_DB: "${METADATA_DB:-dbrepo}"
METADATA_HOST: "${METADATA_HOST:-metadata-db}"
METADATA_JDBC_EXTRA_ARGS: "${METADATA_JDBC_EXTRA_ARGS:-}"
......@@ -482,7 +482,7 @@ services:
GRANT_DEFAULT_READ: "${GRANT_DEFAULT_READ:-SELECT}"
GRANT_DEFAULT_WRITE: "${GRANT_DEFAULT_WRITE:-SELECT, CREATE, CREATE VIEW, CREATE ROUTINE, CREATE TEMPORARY TABLES, LOCK TABLES, INDEX, TRIGGER, INSERT, UPDATE, DELETE}"
JWT_PUBKEY: "${JWT_PUBKEY:-MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqqnHQ2BWWW9vDNLRCcxD++xZg/16oqMo/c1l+lcFEjjAIJjJp/HqrPYU/U9GvquGE6PbVFtTzW1KcKawOW+FJNOA3CGo8Q1TFEfz43B8rZpKsFbJKvQGVv1Z4HaKPvLUm7iMm8Hv91cLduuoWx6Q3DPe2vg13GKKEZe7UFghF+0T9u8EKzA/XqQ0OiICmsmYPbwvf9N3bCKsB/Y10EYmZRb8IhCoV9mmO5TxgWgiuNeCTtNCv2ePYqL/U0WvyGFW0reasIK8eg3KrAUj8DpyOgPOVBn3lBGf+3KFSYi+0bwZbJZWqbC/Xlk20Go1YfeJPRIt7ImxD27R/lNjgDO/MwIDAQAB}"
LOG_LEVEL: ${LOG_LEVEL:-trace}
LOG_LEVEL: ${LOG_LEVEL:-info}
MIN_CONCURRENT_CONSUMERS: ${MIN_CONCURRENT_CONSUMERS:-1}
MAX_CONCURRENT_CONSUMERS: ${MAX_CONCURRENT_CONSUMERS:-5}
QUEUE_NAME: ${QUEUE_NAME:-dbrepo}
......
......@@ -44,11 +44,7 @@ nav:
- UI:
- Customization: api/ui.md
- Examples:
- COVID-19 Tweets: examples/covid19.md
- Hazardous Materials: examples/hazard.md
- Influenza Monitoring: examples/influenza.md
- Manufacturing Data: examples/manufacturing.md
- Power Usage: examples/power.md
- Transportation Monitoring: examples/transportation.md
- publications.md
- contact.md
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment