Skip to content
Snippets Groups Projects

Resolve "Hotfix query execution"

Merged Martin Weise requested to merge 476-hotfix-query-execution into dev
95 files
+ 1816
2854
Compare changes
  • Side-by-side
  • Inline

Files

+ 15
0
@@ -28,8 +28,23 @@ The Data Service is responsible for inserting AMQP tuples from the Broker Servic
via [Spring AMQP](https://docs.spring.io/spring-amqp/reference/html/). To increase the number of consumers, scale the
Data Service up.
## Data Processing
The Data Service uses [Apache Spark](https://spark.apache.org/), a data engine to load data from/into
the [Data Database](../data-db) with a wide range of open-source connectors. The default deployment uses a local mode of
embedded processing directly in the service until there exists
a [Bitnami Chart](https://artifacthub.io/packages/helm/bitnami/spark) for Spark 4.
Retrieving data from a subset internally generates a view with the 64-character hash of the query. This view is not
automatically deleted currently.
## Limitations
* Views in DBRepo can only have 63-character length (it is assumed only internal views have the maximum length of 64
characters).
* Local mode of embedded processing of Apache Spark directly in the service using
a [`local[2]`](https://spark.apache.org/docs/latest/#running-the-examples-and-shell) configuration.
!!! question "Do you miss functionality? Do these limitations affect you?"
We strongly encourage you to help us implement it as we are welcoming contributors to open-source software and get
Loading