How to Recover an EC2 Instance You Can No Longer SSH Into

An instance stops answering. SSH hangs, the application is down, and the console shows it as running. There is a sensible order to work through from here, and the uncomfortable part is that two of the better options are only available if somebody turned them on months ago. The commands are short. The preparation is what decides whether you have three options at two in the morning or one.

First, read what the status checks are saying

EC2 runs four kinds of status check. System, instance and attached EBS run automatically on every instance, and application checks are opt-in.

aws ec2 describe-instance-status --instance-ids i-1234567890abcdef0

The distinction that matters immediately is between the first two. A failing system status check points at the AWS side, meaning the host, its power or its network, and for an EBS-backed instance a stop and start usually migrates it to a different host and ends the problem. A failing instance status check points at your side, so exhausted memory, a corrupted file system, an incompatible kernel or a broken network configuration. Nothing at the AWS layer helps with that, and the rest of this guide is about that case.

Both increment CloudWatch metrics, StatusCheckFailed_System and StatusCheckFailed_Instance, which are worth an alarm regardless.

Option one, the console output

This costs nothing, needs no preparation, and is where a kernel panic or a failed mount announces itself.

aws ec2 get-console-output --instance-id i-1234567890abcdef0 --latest --output text

Without --latest you get the buffered output AWS posts shortly after a state transition, which is not continuously updated. With it you get the latest serial console output during the instance lifecycle, and that option is supported only on Nitro-based instances. Only the instance owner can read it at all.

A screenshot is the other free look, and often the faster one for a Windows box sitting at an error dialog.

aws ec2 get-console-screenshot --instance-id i-1234567890abcdef0 --output text --query ImageData | base64 --decode > screenshot.jpg

Know its limits first. It does not work on bare metal instances, instances using an NVIDIA GRID driver, or instances powered by Arm-based Graviton processors, and it is unavailable in a few Regions including GovCloud. The image comes back as JPG no larger than 100 kb.

Option two, Session Manager if the agent is still alive

If the box has SSM Agent running and an instance profile with AmazonSSMManagedInstanceCore, try a session before anything more invasive.

aws ssm start-session --target i-1234567890abcdef0

This works when SSH is broken but the operating system is otherwise healthy, which covers a surprising share of real incidents. A locked-out sshd config, an untested firewall rule, a full /home. The agent dials out over 443 and does not care that port 22 is dead.

Option three, the serial console

The serial console connects to a virtual serial port outside the VPC entirely, using neither the instance security group nor the subnet network ACL, so it keeps working when the network configuration is what is broken.

Four things have to be true, and the first three cannot be arranged during an incident on an instance you cannot reach.

Account access must already be granted, per Region. By default there is none.

aws ec2 get-serial-console-access-status
aws ec2 enable-serial-console-access

A password-based OS user must exist on the instance. For Linux you log in at a password prompt, and there is no way to create that user from outside. Bake it into the AMI or set it with configuration management on every host.

The instance must be Nitro-based, which covers all virtualised Nitro instances and most bare metal types, and it must be in the running state. You cannot connect while it is stopping, stopped or pending, so the serial console disappears the moment you decide to stop the instance.

Given all that, connecting takes two commands. The public key you push is removed after sixty seconds, so run them back to back.

aws ec2-instance-connect send-serial-console-ssh-public-key \
  --instance-id i-1234567890abcdef0 \
  --serial-port 0 \
  --ssh-public-key file://my_key.pub \
  --region us-east-1

ssh -i my_key i-1234567890abcdef0.port0@serial-console.ec2-instance-connect.us-east-1.aws

The IAM action to grant is ec2-instance-connect:SendSerialConsoleSSHPublicKey, restricted to specific instances rather than left open across the account. One session per instance, about an hour long, and thirty seconds after disconnecting before a new one can start.

Option four, take the volume somewhere you can read it

When the operating system will not come up far enough for any console to help, the root volume goes to a machine that works. The modern version avoids the manual dance. A root volume replacement task swaps the root volume on a running instance while keeping instance store data, non-root EBS volumes, every network interface with its addresses, and the IAM profile.

aws ec2 create-replace-root-volume-task \
  --instance-id i-1234567890abcdef0 \
  --snapshot-id snap-0abc123def456 \
  --delete-replaced-root-volume

Two constraints decide whether this is the right tool. Only snapshots taken directly from that instance's current or previous root volumes work, not copies, and the instance is rebooted during the process. Passing --delete-replaced-root-volume destroys the old root volume permanently once the task succeeds, so leave it off if you still want to read the broken disk.

To inspect and repair rather than roll back, the classic route still applies, and it is destructive in one specific way. Stopping the instance loses everything on instance store volumes, and the automatically assigned public IPv4 address changes on the next start unless the instance uses an Elastic IP.

# 1. stop the impaired instance
aws ec2 stop-instances --instance-ids i-1234567890abcdef0

# 2. detach the root volume, noting its device name
aws ec2 detach-volume --volume-id vol-0impaired

# 3. attach it to a rescue instance in the SAME Availability Zone as a data volume
aws ec2 attach-volume \
  --volume-id vol-0impaired \
  --instance-id i-0rescue \
  --device /dev/sdf

Launch the rescue instance from the same AMI, then mount and look.

lsblk -f
sudo mkdir /rescue
sudo mount /dev/xvdf1 /rescue
cat /rescue/etc/fstab

Use lsblk -f rather than assuming the device name, because the kernel may present it differently from the name you attached it under. A bad /etc/fstab is the most common thing you will find, usually a device ID that changed or a UUID that no longer exists. AWS recommends UUIDs over device IDs for exactly this reason, and lsblk -f prints the correct one.

The automated version

A Systems Manager runbook performs that whole sequence for common connectivity failures.

aws ssm start-automation-execution \
  --document-name AWSSupport-ExecuteEC2Rescue \
  --parameters "UnreachableInstanceId=i-1234567890abcdef0"

It builds a temporary helper instance, stops the original, takes a backup AMI, attaches the root volume to the helper, runs EC2Rescue against it, puts the volume back and restarts. Two things to weigh first. Encrypted root volumes are not supported at all, and it stops your instance, with the same consequences for instance store data and the public IPv4 address as doing it by hand.

The fifteen minutes to spend on a calm afternoon

Everything above splits into what you can do during an incident and what you cannot. The second list is short.

• Run aws ec2 enable-serial-console-access in every Region you use, and grant ec2-instance-connect:SendSerialConsoleSSHPublicKey to whoever gets woken up.
• Put a password-based user on your AMIs, so the serial console has something to log into.
• Install SSM Agent and attach AmazonSSMManagedInstanceCore everywhere, so option two exists.
• Alarm on StatusCheckFailed_System and StatusCheckFailed_Instance.
• Use Elastic IPs wherever a changed address would break something downstream.

None of it takes long, and all of it is impossible to arrange once the instance has stopped answering.

If you want this prepared across a fleet rather than remembered per host, it is routine work inside our AWS cloud management and infrastructure management engagements.