Skip to main content

Opensearch Cluster

Log monitoring in KubeAid​

OpenSearch is a log storage and search engine in KubeAid. It is used in two ways:

  1. As Graylog's backend — stores indexed logs written by Graylog (common deployment pattern)
  2. As a standalone ELK-style stack — paired with OpenSearch Dashboards for direct log search and visualization; an alternative to Graylog

In both cases, kube-prometheus remains responsible for metrics and metric-based alerts. OpenSearch handles logs only.

ScopeLog storage, search, and (with Dashboards) visualization
Log collectionFluent Bit, Fluentd, OpenTelemetry Collector, or via Graylog
Prometheus integrationNone for logs; cluster health metrics can be scraped via ServiceMonitor
KibanaNot packaged in KubeAid — use the OpenSearch Dashboards chart instead

See Monitoring for a comparison of all log monitoring options. When used with Graylog, see also the Graylog chart docs.

The sections below cover cluster operations.

How to increase the ES index limit​

Sample issue which you see in graylog logs​

[194]: index [graylog_2], type [_doc], id [938e8b50-2d32-11ec-b77c-0ad5e9494873], message [ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]

Solution​

The issue is related to elasticseach settings and those are stored internally (in a settings index/db) and can only be adjusted via ES API request.

  • Login to the opensearch opensearch-cluster-master-0 pod

  • Verify the existing setting of the index for which it is complaining. You can do so by running

curl -u $username:$password -XGET http://localhost:9200/$index_name/_settings?pretty=true
  • Increase the limit by running
curl -u $username:$password -X PUT "http://localhost:9200/$index_num/_settings?pretty" -H 'Content-Type: application/json' -d'
 {
   "index" : {
     "mapping.total_fields.limit" : 2000
   }
 }'

NOTE: We should notice that this increases memory usage and differing field names SHOULD be avoided when possible. But with graylog, fields are generated by graylog - so not something we can do much about.

Setting users and passwords​

get a shell inside opensearch master pod, and run:

cd plugins/opensearch-security/tools
chmod +x hash.sh
./hash.sh NEWPASSWORD
# use the generate hash from above - when modifying internal_users.yml to suit your needs
vi /usr/share/opensearch/config/opensearch-security/internal_users.yml
../securityadmin.sh -cd ../securityconfig/ -icl -nhnv -cacert ../../../config/root-ca.pem -cert ../../../config/kirk.pem -key ../../../config/kirk-key.pem

Snapshot opensearch to s3​

Create a secret with template functionality

  • Copy the example files into a kubeaid-config/k8s/$cluster-name/sealed-secret/graylog/s3-backup.yaml

  • Create the secret

# kubectl create secret generic s3-backup -n graylog --dry-run=client --from-literal=username=admin --from-literal=password=xxxx -o yaml | kubeseal --controller-namespace system --controller-name sealed-secrets -o yaml --merge-into kubeaid-config/k8s/$cluster-name/sealed-secret/graylog/s3-backup.yaml

Down sizing the cluster​

Down sizing might take hours, so its not a 30 min job. so relax and enjoy :)​

  • Stop routing any shards to the node you want.

    NOTE: Always take the last node in the cluster. if you are downsizing. f.exp, if you cluster has 12 pods and start with 12th pod and put the IP in the curl command given below.

# curl -v -H 'Content-type: application/json' -XPUT 'http://admin:lolpassword@opensearch-cluster-master:9200/_cluster/settings' -d '{
  "transient" :{
     "cluster.routing.allocation.exclude._ip" : "<last-pod-in-the-cluster-ip>"
   }
}'
  • Verify if all the shards are moved
# curl -s http://admin:lolpassword@opensearch-cluster-master:9200/_cat/shards | grep <last-pod-name-in-the-cluster>

# `relocating_shards` should be **0**

# curl -s http://admin:lolpassworl@opensearch-cluster-master:9200/_cluster/health | jq
{
  "cluster_name": "opensearch-cluster",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 12,
  "number_of_data_nodes": 12,
  "discovered_master": true,
  "active_primary_shards": 3963,
  "active_shards": 3979,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}
  • Set the replicas count to n - 1 (12 pods in the cluster - 1 == 11) NOTE: Do this only when cluster is green
opensearch:
  replicas: 11
  • Sync it on the argocd and it will restart the whole cluster and remove the last pod in the cluster.

Upgrade Instruction​

  • !! TAKE A SNAPSHOT FIRST !! (opensearch s3 snapshot, this should be configured as part of installation)

    # NOTE: this might take sometime depending on the size of the cluster, around 4TB takes about 60m (so watch out)
    kubectl create job -n graylog --from=cronjob/backup-s3 opensearch-manual-backup-01

    or you can get snapshot via api (Havent tried this)

  • Verify which version of opensearch works with graylog

  • opensearch cluster are not downgradable, so please restore it from snapshot (look at opensearch helm chart readme)

Restore Instruction​

  • Delete the statefulset
kubectl delete sts opensearch-cluster-master -n graylog --cascade=orphan
  • Delete the pod one by one (opensearch pod)
  • change the PV to RETAIN and change the claimRef to diff name, so new cluster build does not take the same pv. I have did it directly from the k9s console. But be careful with this.
# Do this for all the opensearch PV's
kubectl patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
  • Once the pv is set to RETAIN and now we can delete the PVC
  • Deploy the new cluster in the same namespace
  • Add s3 repo
curl -s -u admin:admin -X PUT http://opensearch-cluster-master:9200/_snapshot/ops-s3 -d '{"type": "s3", "settings": { "bucket": "<bucket-name>" } }'
  • List snapshots
curl -s -u admin:admin -X GET http://opensearch-cluster-master:9200/_snapshot/ops-s3/_all?pretty
  • Restore from the snapshot
curl -u admin:admin -X POST "opensearch-cluster-master:9200/_snapshot/ops-s3/<snapshot-name>/_restore?pretty" -H 'Content-Type: application/json'  -d '{
  "indices": "-.opendistro_security",
  "include_global_state": false
}'

Shrink over-provisioned Kubernetes Persistent Volume Claims (PVCs) for an OpenSearch cluster without incurring data loss or downtime.​

⚠️ Prerequisites​

  • Ensure your actual, used disk space across the entire cluster will comfortably fit inside the new, smaller drives once the data is rebalanced.
  • Have your updated Kubernetes manifests ready with the new storage size (e.g., 100Gi).

Step 1: Evict Data from the Target Node​

Identify the specific pod name of the node you want to shrink (e.g., opensearch-cluster-master-1). Tell the OpenSearch cluster to securely drain all data from this node and move it to the remaining nodes.

Run this command against any active node in the cluster:

curl -X PUT "http://localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "transient" : {
    "cluster.routing.allocation.exclude._name" : "opensearch-cluster-master-1" 
  }
}'

Step 2: Monitor the Drain​

Watch the migration process in real-time. Do not proceed to the next step until the node is completely empty.

curl -X GET "http://localhost:9200/_cat/allocation?v"

Verification: Look at the row for your target node.

  • The shards column must drop to exactly 0.
  • The disk.indices column must drop to 0b (or a few negligible kilobytes of residual metadata).

Step 3: Replace the Infrastructure​

Once OpenSearch confirms the node holds no shard data, you can safely swap the underlying infrastructure.

  • Terminate the Pod: Scale down your Kubernetes StatefulSet or Deployment to terminate the empty pod.
  • Delete the Storage: Delete the old, oversized PVC.
kubectl delete pvc <name-of-the-old-pvc>
  • Recreate the Node: Apply your updated Kubernetes manifest and scale the StatefulSet back up so the pod is recreated and binds to a brand new, smaller PVC.

Step 4: Allow Data Back onto the Node​

Once the new pod boots up, initializes, and rejoins the OpenSearch cluster, you must clear the routing exclusion rule. If you forget this step, OpenSearch will refuse to put any data on the new drive.

curl -X PUT "http://localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "transient" : {
    "cluster.routing.allocation.exclude._name" : null
  }
}'

Step 5: Verify and Repeat​

Known ArgoCD Drift​

ConfigMap, Service, and PodDisruptionBudget runtime drift​

opensearch ConfigMap data (cluster settings, JVM options) and Service spec fields can be updated at runtime by the opensearch cluster. PodDisruptionBudget spec may also be modified by the cluster autoscaler or operator. These cause permanent OutOfSync without affecting runtime behaviour.

Add ignoreDifferences to your ArgoCD Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: opensearch
  namespace: argocd
spec:
  ignoreDifferences:
    - group: ""
      kind: ConfigMap
      jsonPointers:
        - /data
    - group: ""
      kind: Service
      jsonPointers:
        - /spec
    - group: policy
      kind: PodDisruptionBudget
      jsonPointers:
        - /spec
  sources:
    - repoURL: https://gitea.obmondo.com/EnableIT/KubeAid
      path: argocd-helm-charts/opensearch
      targetRevision: HEAD
      helm:
        valueFiles:
          - $values/k8s/<cluster>/argocd-apps/values-opensearch.yaml
    - repoURL: <your-config-repo>
      targetRevision: HEAD
      ref: values

📘 Graylog & MongoDB Upgrade Guide (v5.2.x to v7.1.2)​

📌 Overview​

This runbook walks you through the steps to upgrade a Graylog deployment from v5.2.6 to v7.1.2. Upgrading to Graylog 7.x introduces a strict requirement for MongoDB 7.0. Because MongoDB requires sequential major version upgrades, a large part of this guide focuses on a safe, step-by-step database upgrade ladder.

Target Versions​

  • Graylog: 5.2.6 -> 6.3.13 -> 7.0.8 -> 7.1.2
  • Graylog Helm Chart: 2.3.10 -> 3.0.21
  • MongoDB: 5.0.14 -> 6.0.16 -> 7.0.11
  • OpenSearch: Pinned at 2.19.2 (Fully compatible with Graylog 7.x)

⚠️ Phase 1: Transitioning to Graylog 6.3 & Database Cleanup​

Before touching the databases, it's best to move Graylog to the end of the 6.x release line.

  1. Bump Graylog to 6.3.13: Update the image tag in your configuration and sync your deployment (e.g., via ArgoCD).
  2. Resolve Database Schema Issues (Jackson Deserialization Crash): If Graylog enters a crash-loop on 6.3.x and logs a Jackson deserialization error, it's usually caused by legacy, unencrypted inputs lingering in the database.
    • exec into your primary MongoDB pod and start the mongo shell: 
      kubectl exec -it <mongodb-primary-pod> -n graylog -- mongosh
    • Identify and remove the malformed configuration fields (e.g., an empty tls_key_password string) from the inputs collection: 
      use graylog;
      // 1. Inspect your inputs to find the corrupted one
      db.inputs.find().pretty();

      // 2. Unset the malformed field on that specific input
      db.inputs.updateOne(
        { _id: ObjectId("<INPUT_ID>") },
        { $unset: { "configuration.<FIELD_NAME>": "" } }
      );
    • Restart the Graylog pods to pick up the clean state.

🗄️ Phase 2: The MongoDB Upgrade Ladder (5.0 -> 6.0 -> 7.0)​

MongoDB requires sequential major version upgrades—you cannot skip directly from 5.0 to 7.0. Additionally, because Kubernetes StatefulSet volumeClaimTemplates are immutable, you'll need to use a neat workaround with orphan deletes to preserve your data during each step.

Step 2.1: Upgrade to MongoDB 6.0​

  1. Update the Manifest: Bump the MongoDB Operator spec.version to 6.0.16. (Tip: Avoid including OS suffixes like -ubi8 if the operator automatically appends them, as this can trigger an ImagePullBackOff).
  2. Clear Immutable Wrappers: The operator will fail to update the StatefulSet metadata. To get around this, perform an orphan-delete on the StatefulSet so it can be recreated without dropping your valuable data PVCs: 
    kubectl delete statefulset mongodb-replica-set -n graylog --cascade=orphan
    kubectl delete statefulset mongodb-replica-set-arb -n graylog --cascade=orphan
  3. Sync & Wait: Apply your changes and wait for the mongodb-replica-set-0 pod to reach a Running and Ready state.
  4. Lock Feature Flags: Because the MongoDB Community Operator manages the cluster, you must update the Feature Compatibility Version (FCV) via the Custom Resource to ensure the engine fully transitions to 6.0: 
    kubectl patch mongodbcommunity mongodb-replica-set -n graylog --type='merge' -p='{"spec":{"version":"6.0.16", "featureCompatibilityVersion":"6.0"}}'

Step 2.2: Upgrade to MongoDB 7.0​

  1. Update the Manifest: Bump the MongoDB version to 7.0.11.
  2. Clear Immutable Wrappers: Just like before, run the orphan deletes: 
    kubectl delete statefulset mongodb-replica-set -n graylog --cascade=orphan
    kubectl delete statefulset mongodb-replica-set-arb -n graylog --cascade=orphan
  3. Sync & Wait: Let your deployment tool cycle the database pods.
  4. Lock Feature Flags: Once back up, finalize the engine transition to 7.0 by patching the Custom Resource again: 
    kubectl patch mongodbcommunity mongodb-replica-set -n graylog --type='merge' -p='{"spec":{"version":"7.0.11", "featureCompatibilityVersion":"7.0"}}'

🛠️ Phase 3: Mitigating Helm Chart & ArgoCD Drift​

Upgrading the core Graylog Helm Chart to v3.x brings some architectural changes to its subcharts. Notably, the OpenSearch subchart might attempt to disconnect existing routing services by changing default labels from graylog to opensearch.

The Fix: Pinning Legacy Labels​

To prevent OpenSearch from disconnecting from Graylog (or losing its HTTP security configurations), force the OpenSearch subchart to respect your existing labels within values.yaml:

opensearch:
  fullnameOverride: "opensearch-cluster-master"
  # Force the new chart to map correctly to your existing pods
  labels:
    app.kubernetes.io/instance: "graylog"
  
  # Ensure HTTP Security remains enabled (the new chart defaults to false)
  config:
    opensearch.yml: |
      plugins:
        security:
          http:
            enabled: true

Clearing the OpenSearch Metadata Block​

Even with the labels fixed, tools like ArgoCD will trip over the immutable PVC templates when trying to update the helm.sh/chart version strings.

To solve this, run another orphan delete on the OpenSearch wrapper before you sync for the final time:

kubectl delete statefulset opensearch-cluster-master -n graylog --cascade=orphan

🚀 Phase 4: Finalizing the Graylog 7.x Rollout​

With MongoDB safely running on v7.0 and the OpenSearch label drift handled, you are ready to bring the application tier home.

  1. Update Your Configurations:
    • Bump the Graylog image tag to 7.1.2 (or use 7.0.8 as an intermediate stepping stone).
    • Ensure your Helm chart target is set to 3.0.21.
  2. Execute the Sync:
    • Trigger a full application sync.
    • A quick heads-up: Be careful with any "Prune" features if legacy resources are transitioning out of the chart's umbrella.
  3. Validation:
    • Monitor the rollout: kubectl get pods -n graylog -w
    • Log into the Graylog Web UI and navigate to System / Nodes.
    • Confirm that both MongoDB and OpenSearch show active, green connections. You're good to go!