Skip to content
Snippets Groups Projects
Verified Commit f12dc032 authored by Martin Weise's avatar Martin Weise
Browse files

Fixed the resource consumption

parent aafcdac3
No related branches found
No related tags found
2 merge requests!387Wrong model,!384Wrong model
Showing with 204 additions and 14 deletions
......@@ -11,6 +11,8 @@ author: Martin Weise
#### Fixes
* Fixed a bug where listing the views in the Python library did not work.
* Fixed a wrong MariaDB configuration where the `innodb_buffer_pool_size` variable was not configured to 70% of the
available memory in the Helm chart.
## v1.6.4 (2025-02-14)
......
......@@ -23,6 +23,12 @@ helm install my-release "oci://registry.datalab.tuwien.ac.at/dbrepo/helm/dbrepo"
* Optional certificate provisioner support in the underlying infrastructure:
e.g. [cert-manager](https://cert-manager.io/) (for production use).
## Database Configuration
Note that the default configuration uses a lower memory bound (2GB) than the default MariaDB memory bound (4GB). We
consequently decreased the `innodb_buffer_pool_size` to 1430MB (70% of the available memory). You need to increase this
variable when you increase the available Pod memory for performance.
## Installing the Chart
To install the chart with the release name `my-release`:
......@@ -75,8 +81,10 @@ The command removes all the Kubernetes components associated with the chart and
| `metadatadb.galera.mariabackup.user` | The database backup username. | `backup` |
| `metadatadb.galera.mariabackup.password` | The database backup user password | `backup` |
| `metadatadb.jdbcExtraArgs` | The extra arguments for JDBC connections in the microservices. | `""` |
| `metadatadb.configurationConfigMap` | The database configuration files. | `metadata-db-config` |
| `metadatadb.extraInitDbScripts` | Additional init.db scripts that are executed on the first start. | `{}` |
| `metadatadb.replicaCount` | The number of cluster nodes, should be uneven i.e. 2n+1 | `3` |
| `metadatadb.resourcesPreset` | The container resource preset | `nano-hm` |
| `metadatadb.persistence.enabled` | Enable persistent storage. | `true` |
### Auth Service
......@@ -109,6 +117,7 @@ The command removes all the Kubernetes components associated with the chart and
| `datadb.galera.mariabackup.password` | The database backup user password | `backup` |
| `datadb.jdbcExtraArgs` | The extra arguments for JDBC connections in the microservices. | `""` |
| `datadb.replicaCount` | The number of cluster nodes, should be uneven i.e. 2n+1 | `3` |
| `datadb.resourcesPreset` | The container resource preset | `nano-hm` |
| `datadb.persistence.enabled` | Enable persistent storage. | `true` |
### Search Database
......@@ -119,9 +128,13 @@ The command removes all the Kubernetes components associated with the chart and
| `searchdb.host` | The hostname for the microservices. | `search-db` |
| `searchdb.port` | The port for the microservices. | `9200` |
| `searchdb.coordinating.resourcesPreset` | The container resource preset | `small` |
| `searchdb.coordinating.replicaCount` | The number of pod replicas. | `1` |
| `searchdb.ingest.resourcesPreset` | The container resource preset | `micro` |
| `searchdb.ingest.replicaCount` | The number of pod replicas. | `1` |
| `searchdb.master.resourcesPreset` | The container resource preset | `small` |
| `searchdb.master.replicaCount` | The number of pod replicas. | `1` |
| `searchdb.data.resourcesPreset` | The container resource preset | `medium` |
| `searchdb.data.replicaCount` | The number of pod replicas. | `1` |
| `searchdb.clusterName` | The cluster name. | `search-db` |
### Upload Service
......
No preview for this file type
[client]
port=3306
socket=/opt/bitnami/mariadb/tmp/mysql.sock
plugin_dir=/opt/bitnami/mariadb/plugin
[mysqld]
explicit_defaults_for_timestamp
default_storage_engine=InnoDB
basedir=/opt/bitnami/mariadb
datadir=/bitnami/mariadb/data
plugin_dir=/opt/bitnami/mariadb/plugin
tmpdir=/opt/bitnami/mariadb/tmp
socket=/opt/bitnami/mariadb/tmp/mysql.sock
pid_file=/opt/bitnami/mariadb/tmp/mysqld.pid
bind_address=0.0.0.0
## Character set
##
collation_server=utf8_unicode_ci
init_connect='SET NAMES utf8'
character_set_server=utf8
## MyISAM
##
key_buffer_size=32M
myisam_recover_options=FORCE,BACKUP
## Safety
##
skip_host_cache
skip_name_resolve
max_allowed_packet=16M
max_connect_errors=1000000
sql_mode=STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ONLY_FULL_GROUP_BY
sysdate_is_now=1
## Binary Logging
##
log_bin=mysql-bin
expire_logs_days=14
# Disabling for performance per http://severalnines.com/blog/9-tips-going-production-galera-cluster-mysql
sync_binlog=0
# Required for Galera
binlog_format=row
## Caches and Limits
##
tmp_table_size=32M
max_heap_table_size=32M
# Re-enabling as now works with Maria 10.1.2
query_cache_type=1
query_cache_limit=4M
query_cache_size=256M
max_connections=500
thread_cache_size=50
open_files_limit=65535
table_definition_cache=4096
table_open_cache=4096
## InnoDB
##
innodb=FORCE
innodb_strict_mode=1
# Mandatory per https://github.com/codership/documentation/issues/25
innodb_autoinc_lock_mode=2
# Per https://www.percona.com/blog/2006/08/04/innodb-double-write/
innodb_doublewrite=1
innodb_flush_method=O_DIRECT
innodb_log_files_in_group=2
innodb_log_file_size=128M
innodb_flush_log_at_trx_commit=1
innodb_file_per_table=1
# 80% Memory is default reco.
# Need to re-evaluate when DB size grows
innodb_buffer_pool_size=1430M
innodb_file_format=Barracuda
## Logging
##
log_error=/opt/bitnami/mariadb/logs/mysqld.log
slow_query_log_file=/opt/bitnami/mariadb/logs/mysqld.log
log_queries_not_using_indexes=1
slow_query_log=1
## SSL
## Use extraVolumes and extraVolumeMounts to mount /certs filesystem
# ssl_ca=/certs/ca.pem
# ssl_cert=/certs/server-cert.pem
# ssl_key=/certs/server-key.pem
[galera]
wsrep_on=ON
wsrep_provider=/opt/bitnami/mariadb/lib/libgalera_smm.so
wsrep_sst_method=mariabackup
wsrep_slave_threads=4
wsrep_cluster_address=gcomm://
wsrep_cluster_name=galera
wsrep_sst_auth="root:"
# Enabled for performance per https://mariadb.com/kb/en/innodb-system-variables/#innodb_flush_log_at_trx_commit
innodb_flush_log_at_trx_commit=2
# MYISAM REPLICATION SUPPORT #
wsrep_mode=REPLICATE_MYISAM
[mariadb]
plugin_load_add=auth_pam
## Data-at-Rest Encryption
## Use extraVolumes and extraVolumeMounts to mount /encryption filesystem
# plugin_load_add=file_key_management
# file_key_management_filename=/encryption/keyfile.enc
# file_key_management_filekey=FILE:/encryption/keyfile.key
# file_key_management_encryption_algorithm=AES_CTR
# encrypt_binlog=ON
# encrypt_tmp_files=ON
## InnoDB/XtraDB Encryption
# innodb_encrypt_tables=ON
# innodb_encrypt_temporary_tables=ON
# innodb_encrypt_log=ON
# innodb_encryption_threads=4
# innodb_encryption_rotate_key_age=1
## Aria Encryption
# aria_encrypt_tables=ON
# encrypt_tmp_disk_tables=ON
\ No newline at end of file
......@@ -17,6 +17,10 @@ These presets are for basic testing and not meant to be used in production
"requests" (dict "cpu" "100m" "memory" "128Mi" "ephemeral-storage" "50Mi")
"limits" (dict "cpu" "150m" "memory" "192Mi" "ephemeral-storage" "2Gi")
)
"nano-hm" (dict
"requests" (dict "cpu" "100m" "memory" "1024Mi" "ephemeral-storage" "50Mi")
"limits" (dict "cpu" "500m" "memory" "2048Mi" "ephemeral-storage" "2Gi")
)
"micro" (dict
"requests" (dict "cpu" "250m" "memory" "256Mi" "ephemeral-storage" "50Mi")
"limits" (dict "cpu" "375m" "memory" "384Mi" "ephemeral-storage" "2Gi")
......
......@@ -3,16 +3,8 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: metadata-db-setup
name: metadata-db-config
namespace: {{ include "common.names.namespace" . | quote }}
data:
{{- with .Values.metadatadb.extraInitDbScripts }}
{{ toYaml . | nindent 2 }}
{{- end }}
02-setup-data.sql: |
BEGIN;
INSERT INTO `mdb_containers` (name, internal_name, image_id, host, port, privileged_username, privileged_password)
VALUES ('mariadb-galera:11.3.2-debian-12-r9', 'mariadb-galera:11.3.2-debian-12-r9', 1, 'data-db', 3306, '{{ .Values.datadb.rootUser.user }}', '{{ .Values.datadb.rootUser.password }}');
COMMIT;
{{ (.Files.Glob "files/01-setup-schema.sql").AsConfig | nindent 2 }}
{{ (.Files.Glob "files/my.cnf").AsConfig | nindent 2 }}
{{- end }}
{{- if .Values.metadatadb.enabled }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: metadata-db-setup
namespace: {{ include "common.names.namespace" . | quote }}
data:
{{- with .Values.metadatadb.extraInitDbScripts }}
{{ toYaml . | nindent 2 }}
{{- end }}
{{ (.Files.Glob "files/01-setup-schema.sql").AsConfig | nindent 2 }}
02-setup-data.sql: |
BEGIN;
INSERT INTO `mdb_containers` (name, internal_name, image_id, host, port, privileged_username, privileged_password)
VALUES ('mariadb-galera:11.3.2-debian-12-r9', 'mariadb-galera:11.3.2-debian-12-r9', 1, 'data-db', 3306, '{{ .Values.datadb.rootUser.user }}', '{{ .Values.datadb.rootUser.password }}');
COMMIT;
{{ (.Files.Glob "files/my.cnf").AsConfig | nindent 2 }}
{{- end }}
......@@ -620,6 +620,9 @@
"replicaCount": {
"type": "integer"
},
"resourcesPreset": {
"type": "string"
},
"rootUser": {
"properties": {
"password": {
......@@ -1015,6 +1018,9 @@
},
"metadatadb": {
"properties": {
"configurationConfigMap": {
"type": "string"
},
"db": {
"properties": {
"name": {
......@@ -1080,6 +1086,9 @@
"replicaCount": {
"type": "integer"
},
"resourcesPreset": {
"type": "string"
},
"rootUser": {
"properties": {
"password": {
......@@ -1363,6 +1372,9 @@
},
"coordinating": {
"properties": {
"replicaCount": {
"type": "integer"
},
"resourcesPreset": {
"type": "string"
}
......@@ -1371,6 +1383,9 @@
},
"data": {
"properties": {
"replicaCount": {
"type": "integer"
},
"resourcesPreset": {
"type": "string"
}
......@@ -1391,6 +1406,9 @@
"enabled": {
"type": "boolean"
},
"replicaCount": {
"type": "integer"
},
"resourcesPreset": {
"type": "string"
}
......@@ -1399,6 +1417,9 @@
},
"master": {
"properties": {
"replicaCount": {
"type": "integer"
},
"resourcesPreset": {
"type": "string"
}
......
......@@ -56,6 +56,8 @@ metadatadb:
enabled: true
## @skip metadatadb.initdbScriptsConfigMap The initial database scripts.
initdbScriptsConfigMap: metadata-db-setup
## @param metadatadb.configurationConfigMap The database configuration files.
configurationConfigMap: metadata-db-config
## @param metadatadb.extraInitDbScripts Additional init.db scripts that are executed on the first start.
extraInitDbScripts: { }
# 03-additional-data.sql: |
......@@ -65,6 +67,8 @@ metadatadb:
# COMMIT;
## @param metadatadb.replicaCount The number of cluster nodes, should be uneven i.e. 2n+1
replicaCount: 3
## @param metadatadb.resourcesPreset The container resource preset
resourcesPreset: "nano-hm"
persistence:
## @param metadatadb.persistence.enabled Enable persistent storage.
enabled: true
......@@ -185,6 +189,8 @@ datadb:
enabled: true
## @param datadb.replicaCount The number of cluster nodes, should be uneven i.e. 2n+1
replicaCount: 3
## @param datadb.resourcesPreset The container resource preset
resourcesPreset: "nano-hm"
persistence:
## @param datadb.persistence.enabled Enable persistent storage.
enabled: true
......@@ -205,17 +211,25 @@ searchdb:
coordinating:
## @param searchdb.coordinating.resourcesPreset The container resource preset
resourcesPreset: "small"
## @param searchdb.coordinating.replicaCount The number of pod replicas.
replicaCount: 1
ingest:
## @skip searchdb.ingest.enabled The limited default payload does not require this node type.
enabled: false
## @param searchdb.ingest.resourcesPreset The container resource preset
resourcesPreset: "micro"
## @param searchdb.ingest.replicaCount The number of pod replicas.
replicaCount: 1
master:
## @param searchdb.master.resourcesPreset The container resource preset
resourcesPreset: "small"
## @param searchdb.master.replicaCount The number of pod replicas.
replicaCount: 1
data:
## @param searchdb.data.resourcesPreset The container resource preset
resourcesPreset: "medium"
## @param searchdb.data.replicaCount The number of pod replicas.
replicaCount: 1
security:
## @skip searchdb.security.enabled
enabled: false
......
dependencies:
- name: mariadb
repository: oci://registry-1.docker.io/bitnamicharts
version: 20.2.2
version: 20.2.4
- name: postgresql
repository: oci://registry-1.docker.io/bitnamicharts
version: 16.4.6
version: 16.4.9
- name: common
repository: oci://registry-1.docker.io/bitnamicharts
version: 2.29.1
digest: sha256:bc14ae7bbe7be291adc4a6329ae64835c367b09277a2678c4e10cc74b19ee491
generated: "2025-02-04T22:22:11.88596441+01:00"
digest: sha256:dbba12b36ab3b4370112da3089066f57bcef2be1651134787fd158491a2a766c
generated: "2025-02-18T09:32:41.716703325+01:00"
File deleted
File added
File deleted
File added
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment