How to securely access AWS resources using AWS Session Manager. No more bastion servers!

The traditional way of accessing bastion servers

A bastion host is a server whose purpose is to provide access to a private network from an external network, such as the Internet.

We usually connect to the bastion server using a corporate VPN to access or debug applications in the private subnet.

Since bastion servers are accessible from the Internet, they need to be added to the public subnet. And for the server to be accessible using SSH, we need to create and manage SSH keys.

Example architecture using bastion server

Here you can see we have created a corporate VPN; whenever a developer tries to access bastion-server, he first needs to connect to the VPN.

Local system users for every developer on the bastion-server are created based on IAM user permissions. Hence the IAM users of every developer are mapped to their respective local system users using IAM SSH keys.

We can use this solution for managing AWS EC2 SSH access using IAM → https://github.com/widdix/aws-ec2-ssh

Problems with using this approach

There is no proper solution for auditing session logs of all developers. If we create a solution, we will still need to manage it.
Even the solution for Manage AWS EC2 SSH access with IAM is now deprecated due to the shift of bastion server to new AWS solutions of SSM session manager and EC2 instance connect.
We need to depend on a VPN for access.
This solution needs to be in a public subnet over the Internet, which might bring risks if not used or set correctly.
Whenever the container/bastion-server is updated due to patches or features, then every time, developers need to update their known_hosts file to remove old identification since the remote host has changed.
Have to manage the SSH keys when accessing the bastion server over VPN.

Why use session manager, and what benefits does it provide?

Session Manager is a fully managed AWS Systems Manager capability.

With Session Manager, you can manage your EC2 instances, edge devices, on-premises servers, and virtual machines.

You can use an interactive one-click browser-based shell or the AWS Command Line Interface (AWS CLI).

Session Manager provides secure and auditable node management without the need to open inbound ports, maintain bastion hosts, or manage SSH keys.

AWS Session Manager also allows you to comply with corporate policies that require:

Controlled access to managed nodes
Strict security practices, and
Fully auditable logs with node access details while providing end users simple one-click cross-platform access to your managed nodes.

Benefits

Session Manager offers these benefits:

Centralized access control to managed nodes using IAM policies
No open inbound ports and no need to manage bastion hosts or SSH keys
One-click access to managed nodes from the console and CLI
Port forwarding
Cross-platform support for Windows, Linux, and macOS
Logging and auditing session activity (CloudTrail, S3, CloudWatch Logs, EventBridge, and SNS)
AWS PrivateLink support for managed nodes without public IP addresses

And the best part, Session Manager is completely free of cost (except for the session logs storage cost)

Replacing bastion server with AWS Systems Manager Session Manager

Our development server is now in a private subnet with this new architecture.

We have two different solutions as per developer requirements:

Remote access using SSM session manager: This is used by most of the developers who do not need Port forwarding
Port Forwarding access using Session Manager: Developers temporarily request permission for this when they want to forward development server port to local like using Jupyter Notebook

IAM Users need to belong to appropriate IAM groups to get access as per the two solutions shown above.

With this new approach, we have the below benefits:

All session activity is logged in S3 and encrypted using KMS.
No in-bound security groups or SSH keys to manage.
AWS IAM ultimately controls server access with different solutions per user requirements.

Combining SSM Session Manager with EC2 Instance Connect

SSM Session Manager also provides the additional functionality of connecting to the server using SSH. AWS provides this functionality if we want to perform SCP to copy files from our local machine to the server.

We do not recommend copying files from a local machine to a development server, but if you need it for any use case, below are steps for secure configuration. Remember that session activity is not logged when you perform SSH using SSM Session Manager.

The disadvantage of using SSM Session Manager alone for SSH

Before explaining EC2 Instance Connect let me tell you one significant disadvantage of using session manager and SSH together.

Unfortunately, as of writing this post, the session manager with SSH does not entirely resolve the issue with SSH keys. We still need SSH keys if we want to perform SCP to copy files from our local machine to the server.

SSM Systems Manager does provide connecting to the server using session manager + SSH, but the server has to have SSH keys.

So why EC2 Instance Connect, and how does it solve our issue when connecting with SSH?

Amazon EC2 Instance Connect provides a simple and secure way to connect to your Linux instances using Secure Shell (SSH). With EC2 Instance Connect, you use AWS Identity and Access Management (IAM) policies and principals to control SSH access to your instances, removing the need to share and manage SSH keys. All connection requests using EC2 Instance Connect are logged to AWS CloudTrail so that you can audit connection requests.

Using EC2 Instance Connect solves our issue of SSH keys; we do not need to create instances with SSH keys and do not need to manage and share keys.

EC2 Instance Connect helps us with connecting to our development server using SSH with a temporary SSH key generated by our script every time a user requests access using SSH. This access is verified by the IAM policy to allow or deny the user.

{
  "Sid": "SendSSHPublicKey",
  "Effect": "Allow",
  "Action": "ec2-instance-connect:SendSSHPublicKey",
  "Resource": "arn:aws:ec2:<region>:<account>:instance/*",
  "Condition": {
    "StringEquals": {
      "aws:ResourceTag/Name": "<development-sever-name>",
      "ec2:osuser": "${aws:username}"
    }
  }
}

Combining it with SSM Session Manager

Since we have a solution to the disadvantage we faced with session manager + SSH, we can combine the two for a more secure and managed solution.

We can update the IAM policy we used above to include permissions for accessing the document AWS-StartSSHSession.

{
  "Sid": "StartSSHSession",
  "Effect": "Allow",
  "Action": "ssm:StartSession",
  "Resource": "arn:aws:ssm:*:*:document/AWS-StartSSHSession"
},
{
  "Sid": "SendSSHPublicKey",
  "Effect": "Allow",
  "Action": "ec2-instance-connect:SendSSHPublicKey",
  "Resource": "arn:aws:ec2:<region>:<account>:instance/*",
  "Condition": {
    "StringEquals": {
      "aws:ResourceTag/Name": "<development-sever-name>",
      "ec2:osuser": "${aws:username}"
    }
  }
}

Using this policy, a user can create a temporary SSH key using EC2 Instance Connect and use this key to access the development server using SSM Session Manager. This process is automated by using a script.

This solves our issue of performing SSH or SCP to copy files from the local machine to the development server and vice-versa.

Solution advantage:

SSH / SCP access using Instance Connect + Session Manager: Developers temporarily request permissions for this when they want to copy files from their local machine to the development server

BEWARE: It is not recommended to use SSH or SCP with Session Manager because developers who have access will be able to copy any files from their local machine to/from the server. These activities will not be logged in S3.

Conclusion

By replacing our bastion server with Systems Manager Session Manager, we can make our access more secure, easily auditable, and centrally managed using IAM.

The two solutions: Remote access and Port-Forwarding, provide developers a way to access the development server without requiring them to connect to a VPN and maintain any SSH keys.