Ninja Docs Help

LLD001-003 - Break Glass

Introduction

Purpose

This document outlines the 'Break Glass' procedure for emergency access to Amazon Web Services (AWS) resources. It is designed to provide guidelines on how to gain access to critical systems in a controlled and secure manner during emergencies.

Changelog

Revision

Date

Description

1.0

02.07.2024

Initial document

Background

This procedure applies to all AWS environments and resources managed by T-mobile Polska.

The Break Glass procedure involves the following steps:

  • Identification of Emergency Situation

  • Approval for Break Glass Activation

  • Execution of Break Glass Protocol

  • Post-Access Review and Documentation

Procedure

Identification of Emergency Situation

An emergency situation warranting the use of the Break Glass procedure typically includes scenarios such as:

  1. Loss of access due to Cloud Shepperd or CloudIAM system.

  2. Outage of Active Directory

  3. Compromised administrator accounts.

  4. Critical system failures requiring immediate administrative intervention

Approval for Break Glass Activation

Prior to the activation of the Break Glass procedure, the following approvals must be obtained:

  1. Primary Approval: From the Chief Information Security Officer (CISO) or an equivalent role.

  2. Secondary Approval: From a member of the senior management team not directly involved with the AWS infrastructure.

Execution of Break Glass Protocol

The execution involves the following steps:

  1. Accessing Emergency Credentials: Stored in a secure and encrypted location, accessible only to approved personnel.

  2. Access IAM Account: Use credentials to log in to IAM Account

  3. Assuming Emergency IAM Role: Assume break glass role on AWS account that requires emergency action.

  4. Monitoring and Logging: All activities performed under the Break Glass procedure are to be monitored and logged for audit and review purposes.

Post-Access Review and Documentation

After the emergency is resolved, the following steps must be taken:

  1. Revoking Emergency Access: Ensure that the emergency access is revoked and the normal operation is restored.

  2. Audit and Review: Conduct a thorough audit of the actions taken and document the findings.

  3. Lessons Learned: Review the process to identify improvements for future emergency situations.

Security Considerations

  1. The Break Glass account credentials must be stored in a highly secure, encrypted, and physically secure location.

  2. Access to the Break Glass procedure should be restricted to a minimum number of personnel.

  3. Regular audits and reviews of the Break Glass procedure and its usage should be conducted.

Implementation details

Creation o Emergency IAM Role

Break Glass User

  1. Create 2 dedicated IAM users in TMPL_IAM account.

  2. Activate MFA Authentication - store device in safe place (TBD.).

  3. Store credentials in Vault.

  4. Add permission policy:

    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": [ "arn:aws:iam::*:role/a-tmpl-prod-rol-break-glass" ] } ] }

Role Creation

  1. Create IAM role a-tmpl-prod-rol-break-glass on each AWS Account.

  2. Add policy:

    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "*", "Resource": "*", "Condition": {"Bool": {"aws:MultiFactorAuthPresent": "true"}} } ] }
  3. Add Trust policy:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::IAM-ACCOUNT-ID:USER-ID1", "AWS": "arn:aws:iam::IAM-ACCOUNT-ID:USER-ID2" }, "Action": "sts:AssumeRole" } ] }

Password rotation and access revoking

  1. As a best practice, implement AWS provided solution based on Lambda function.

  2. Add manual trigger to invoke Lambda each time break glass procedure has come to the end.

Monitoring and Logging

  1. Create a new log groupa-tmpl-prod-cwgrp-break-glass.

  2. Direct CloudTrail Logs to CloudWatch Group .

  3. Create Metric Filters for activities:

    User Login: { ($.eventName = "ConsoleLogin") && ($.userIdentity.type = "IAMUser") && ($.userIdentity.userName = "break-glass-user") } Role Assumption: { ($.eventName = "AssumeRole") && ($.requestParameters.roleArn = "arn:aws:iam::*:role/a-tmpl-prod-rol-break-glass") }
  4. Create Alarms Based on the Filters.

  5. Create an SNS Topic a-tmpl-prod-sns-break-glass.

  6. Add email addresses or other endpoints to receive notifications.

Last modified: 17 February 2025