[[ls-k8s-troubleshooting-methods]]
=== Troubleshooting tips and suggestions

Here are some approaches that you can use to diagnose the state of your {ls} and Kubernetes system, both in the event of any problems, and as part of a day-to-day approach to ensuring that everything is running as expected.

* <<ls-k8s-checking-resources>>
* <<ls-k8s-viewing-logs>>
* <<ls-k8s-connecting-to-a-container>>
* <<ls-k8s-diagnostics>>
* <<ls-k8s-pq-util>>
* <<ls-k8s-pq-drain>>

[float]
[[ls-k8s-checking-resources]]
=== Checking resources

You can use the standard Kubernetes `get` and `describe` commands to quickly gather details about any resources in your {ls} and Kubernetes environment.

[source,bash]
--
kubectl get pod logstash-7477d46bb7-4lcnv

NAME                                    READY   STATUS        RESTARTS         AGE
logstash-7477d46bb7-4lcnv               0/1     Pending       0                2m43s
--

If a Pod fails to reach the `Running` status after a few seconds, run this command to get more insights:

[source,bash]
--
kubectl describe pod logstash-7477d46bb7-4lcnv

(...)
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  34s (x2 over 115s)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu.
--

You can check the CPU and memory resources by running this command:
[source,bash]
--
kubectl top pod logstash-7477d46bb7-4lcnv

NAME                        CPU(cores)   MEMORY(bytes)
logstash-7d5b749899-tfg4f   37m          882Mi
--

[float]
[[ls-k8s-viewing-logs]]
=== Viewing logs

{ls} Docker containers do not create log files by default. They log to standard output.

To view the log, run:

[source,bash]
--
kubectl logs -f logstash-7477d46bb7-4lcnv
--

To enable debug log, set `log.level: debug` in logstash.yml in `ConfigMap`.

[float]
[[ls-k8s-connecting-to-a-container]]
=== Connecting to a container

At times, you may need to connect directly from your command shell into {ls} and other Kubernetes resources.

[source,bash]
--
kubectl exec -it logstash-7477d46bb7-4lcnv -- bash
--

[float]
[[ls-k8s-diagnostics]]
=== Running diagnostics

Thread dumps and heap dumps can be helpful when you are debugging hard problems. Connect to the container, and then run the commands to gather the diagnostics.

==== Thread dump
[source,bash]
--
jdk/bin/jstack -l 1 > /tmp/jstack_output.txt
--

==== Heap dump
[source,bash]
--
jdk/bin/jcmd 1 GC.heap_dump /tmp/heap_dump.hprof
--

==== Extract file from the container
[source,bash]
--
kubectl cp logstash-7477d46bb7-4lcnv:/tmp/heap_dump.hprof ./heap.hprof
--

[[ls-k8s-pq-util]]
=== Running PQ utilities

In the event of persistent queue corruption, the `pqcheck` and `pqrepair` tools are available for troubleshooting.

Run {logstash-ref}/persistent-queues.html#pqcheck[pqcheck] to identify corrupted files: 

[source,bash]
--
kubectl exec logstash-0 -it -- /usr/share/logstash/bin/pqcheck /usr/share/logstash/data/queue/pipeline_id
--

Run {logstash-ref}/persistent-queues.html#pqrepair[pqrepair] to repair the queue: 

[source,bash]
--
kubectl exec logstash-0 -it -- /usr/share/logstash/bin/pqrepair /usr/share/logstash/data/queue/pipeline_id
--

[[ls-k8s-pq-drain]]
=== Draining the PQ

{ls} provides a `queue.drain: true` configuration setting to pause shutdown, when Logstash is stopped gracefully, until all messages from a persistent queue have been handled. 
Special consideration needs to be taken when using the `queue.drain: true` setting when using {k8s}. By default, a {k8s} pod has a grace period of 30 seconds to shutdown before it is closed forcefully, via a `SIGKILL`, which may cause {ls} to exit before the queue is fully drained.

To avoid {ls} shutting down before the queue is completely drained, we recommend setting the `TerminationGracePeriodSeconds` value to an artifically long period, such as 1 year, to give {ls} sufficient time to drain the queue when this functionality is required. 