Skip to content
Snippets Groups Projects
Select Git revision
  • 318fb13c2026cb961e9fec277d2af3cf0e9d6b68
  • master default protected
  • dev protected
  • 551-init-broker-service-permissions
  • release-1.10 protected
  • 549-test-oai-pmh
  • 545-saving-multiple-times-breaks-pid-metadata
  • release-1.9 protected
  • 499-standalone-compute-service-2
  • 539-load-tests
  • hotfix/helm-chart
  • luca_ba_new_interface
  • 534-bug-when-adding-access-to-user-that-is-not-registered-at-dashboard-service
  • release-1.8 protected
  • 533-integrate-semantic-recommendation
  • feature/openshift
  • 518-spark-doesn-t-map-the-headers-correct
  • 485-fixity-checks
  • 530-various-schema-problems-with-subsets
  • release-1.7 protected
  • fix/auth-service
  • v1.10.1 protected
  • v1.10.0-rc13 protected
  • v1.10.0-rc12 protected
  • v1.10.0-rc11 protected
  • v1.10.0-rc10 protected
  • v1.10.0-rc9 protected
  • v1.10.0-rc8 protected
  • v1.10.0-rc7 protected
  • v1.10.0-rc6 protected
  • v1.10.0-rc5 protected
  • v1.10.0-rc4 protected
  • v1.10.0-rc3 protected
  • v1.10.0-rc2 protected
  • v1.10.0rc1 protected
  • v1.10.0rc0 protected
  • v1.10.0 protected
  • v1.9.3 protected
  • v1.9.2 protected
  • v1.9.2-rc0 protected
  • v1.9.1 protected
41 results

data-service.md

Blame
  • Martin Weise's avatar
    Martin Weise authored
    Signed-off-by: Martin Weise's avatarMartin Weise <martin.weise@tuwien.ac.at>
    318fb13c
    History
    author: Martin Weise

    tl;dr

    !!! debug "Debug Information"

    Image: [`registry.datalab.tuwien.ac.at/dbrepo/data-service:1.4.7`](https://hub.docker.com/r/dbrepo/data-service)
    
    * Ports: 9093/tcp
    * Info: `http://<hostname>:9093/actuator/info`
    * Health: `http://<hostname>:9093/actuator/health`
        - Readiness: `http://<hostname>:9093/actuator/health/readiness`
        - Liveness: `http://<hostname>:9093/actuator/health/liveness`
    * Prometheus: `http://<hostname>:9093/actuator/prometheus`
    * Swagger UI: `http://<hostname>:9093/swagger-ui/index.html` <a href="../../rest/" target="_blank">:fontawesome-solid-square-up-right: view online</a>
    
    To directly access in Kubernetes (for e.g. debugging), forward the svc port to your local machine:
    
    ```shell
    kubectl [-n namespace] port-forward svc/data-service 9093:80
    ```

    Overview

    The Data Service is responsible for inserting AMQP tuples from the Broker Service into the Data DB via Spring AMQP. To increase the number of consumers, scale the Data Service up.

    Data Processing

    The Data Service uses Apache Spark, a data engine to load data from/into the Data Database with a wide range of open-source connectors. The default deployment uses a local mode of embedded processing directly in the service until there exists a Bitnami Chart for Spark 4.

    Retrieving data from a subset internally generates a view with the 64-character hash of the query. This view is not automatically deleted currently.

    Caching

    The Data Service uses Caffeine, a caching solution that is used to temporarily cache the connection details from the Metadata Service such that they don't have to be queried everytime e.g. a sensor measurement is inserted. By default, this information is stored for 60 minutes. System administrators can disable this behavior by setting CREDENTIAL_CACHE_TIMEOUT=0 (cache is deleted after 0 seconds).

    Limitations

    • Views in DBRepo can only have 63-character length (it is assumed only internal views have the maximum length of 64 characters).
    • Local mode of embedded processing of Apache Spark directly in the service using a local[2] configuration.

    !!! question "Do you miss functionality? Do these limitations affect you?"

    We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
    in [contact](../../contact) with us, we happily answer requests for collaboration with attached CV and your programming 
    experience!

    Security

    (none)