[ Thanos ][ Query Frontend ] Replica missing

Revision	Date	Description
`1.0`	24.07.2024	Init Changelog

Problem

When report Incident

When all replicas are missing.

Severity	Value	Time
SEV-2	None of Thanos QF replica is not working.

Escalation

I Line	II Line	III Line
`DevOps`	`SRE`

Requires access / logs, etc.

Panel

Kubernetes API (with kubectl, k9s, etc.) access:
- Needs at least: LIST, GET, WATCH on pods, pods/logs, deployments in monitoring Namespace.
AWS Access (Management Console or CLI):
- Account: AWS Common
- Resources:
  - Amazon ElastiCache: redis/thanos

Monitoring

Dashboard:
- Number of Pods
- Last Terminated Reason

Logs

Pods:

Thanos Query Frontend

Environment

Kubernetes cluster: AWS Common

DB

Amazon ElastiCache:
- Redis cluster: thanos

Steps to take

Verify pods from thanos-query-frontend deployment and identify problem.

Last modified: 17 February 2025