[[ls-k8s-stack-monitoring]]
=== Stack monitoring

WARNING: This documentation is still in development and may be changed or removed in a future release.

An important step to making your environment production ready is to configure stack monitoring. Monitoring metrics can be sent to an external resource, such as {ess} or {eck}, so that in the event that any components of your environment become unresponsive, your monitoring data is available.

To enable {logstash-ref}/monitoring-with-metricbeat.html[Stack monitoring] for {ls}, you need {metricbeat} to collect {ls} metrics, {es} to store the metrics and {kib} to view the result.

[[monitor-with-ECK]]
==== Stack monitoring with Elastic Cloud on {k8s} (ECK)

TIP: Be sure that you have ECK installed so that you can follow the example. Check out <<qs-set-up>> for set up info.

For these examples, we will be modifying the Beats stack monitoring link:https://github.com/elastic/cloud-on-k8s/blob/main/config/recipes/beats/stack_monitoring.yaml[recipe] from the ECK examples.
This example initiates a production {es} cluster, a monitoring {es} cluster, {filebeat}, {metricbeat}, a production Kibana and a monitoring Kibana. It monitors {es} and Kibana and sends metrics to the monitoring cluster.
We use {metricbeat-ref}/configuration-autodiscover.html[autodiscover] to configure monitoring for multiple {ls} instances.

* <<ls-k8s-monitor-config-metricbeat>>
* <<ls-k8s-monitor-config-ls>>
* <<ls-k8s-monitor-kibana>>

[float]
[[ls-k8s-monitor-config-metricbeat]]
===== Configure Metricbeat

To monitor {ls}, add the `Logstash` module to the recipe.

[source,yaml]
--
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: metricbeat
spec:
  type: metricbeat
  version: 8.4.2
  elasticsearchRef:
    name: elasticsearch-monitoring <1>
  config:
    metricbeat:
      autodiscover:
        providers:
          - type: kubernetes
            scope: cluster
            hints.enabled: true
            templates:
              - condition:
                  contains:
                    kubernetes.labels.app: ls <2>
                config:
                  - module: logstash <3>
                    metricsets:
                      - node
                      - node_stats
                    period: 10s
                    hosts: "http://${data.host}:9600"
                    xpack.enabled: true
...
--

<1> {metricbeat} sends metrics to `elasticsearch-monitoring` cluster.
<2> {metricbeat} scans for the pods with label `app: ls` to collect {ls} metrics.
<3> {metricbeat} logstash module calls metric endpoint of each {ls} from port `9600` for every `10` seconds.

[float]
[[ls-k8s-monitor-config-ls]]
===== Configure {ls}

Add label `app: ls` to `Deployment` for autodiscover.

[source,yaml]
--
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash
  labels:
    app: ls
...
--

After you have configured {metricbeat} and {ls}, the configurations are ready to deploy. Go to <<ls-k8s-monitor-kibana>> for info on how to confirm that everything is working.

[float]
[[kibana-metrics]]
====== Show {kib} metrics in the same {es} cluster (optional)

By default {ls} metrics will be shown in a standalone cluster. To associate data with the same cluster of {es} and {kib}, provide the `cluster_uuid` of the production {es} cluster to `monitoring.cluster_uuid` in logstash.yml.

[source,yaml]
--
apiVersion: v1
data:
  logstash.yml: |
    api.http.host: "0.0.0.0"
    monitoring.cluster_uuid: PRODUCTION_ES_CLUSTER_UUID
kind: ConfigMap
metadata:
  name: logstash-config
--

To get the `cluster_uuid`, go to {kib} > Stack Monitoring page. The URL in the browser shows the uuid in the form of `cluster_uuid:PRODUCTION_ES_CLUSTER_UUID,`.


[float]
[[ls-k8s-monitor-kibana]]
===== View monitoring data in {kib}

When everything is set, the {kib} > Stack Monitoring page will show the Logstash data.

To access {kib} by `https://localhost:5601`, set the port forwarding:

[source,sh]
--
kubectl port-forward service/kibana-monitoring-kb-http 5601
--

Get the login password:

[source,sh]
--
kubectl get secret elasticsearch-monitoring-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode; echo
--

image::./images/sm-kibana.png[Stack Monitoring screenshot]
