[ Thanos ][ Query ] Http Request Query Error Rate High

Revision	Date	Description
`1.0`	24.07.2024	Init Changelog

Problem

When report Incident

When all replicas are missing.

Severity	Value	Time
SEV-2	None of Thanos Query replica is not working.

Escalation

I Line	II Line	III Line
`DevOps`	`SRE`

Requires access / logs, etc.

Panel

Kubernetes API (with kubectl, k9s, etc.) access:
- Needs at least: LIST, GET, WATCH on pods, pods/logs, deployments in monitoring Namespace.

Monitoring

Dashboard:
- Instant Query API: Errors

Logs

Pods:

Thanos Query

Environment

Kubernetes cluster: AWS Common

DB

None.

Steps to take

Verify pods from thanos-query deployment and identify problem.

Last modified: 17 February 2025