A Lowdown on Managing SSH Keys in the Cloud

Introduction

SSH keys help machines to identify, authenticate and communicate with each other over a secure channel. However, this channel is only as secure as the keys guarding it. SSH keys are like weapons that help you defend against attacks. But, when those weapons are compromised, they can become the threat instead of the shield.

Over 84 percent of today’s enterprises use a form of SSH protocol in their infrastructure.1 Yet, the importance of properly managing this underlying infrastructure is rarely understood until after a breach. Using just one stolen SSH key, hackers can gain access to an enterprise’s server, search for more keys and then access the remaining servers, thus using a single key to attack the entire infrastructure.

Unlike digital certificates, SSH keys were never designed to expire, leaving enterprises more susceptible to this kind of detrimental breach. Instead, these keys must be checked periodically and manually retired to avoid permanently opening back doors to sensitive data.

Unfortunately, our recent survey at the RSA event revealed that over 47% of our respondents did not deploy this crucial key rotation.1 In conducting additional research we found that the primary reason our respondents did not enforce key rotation was lack of visibility across all keys.

Enterprises want to keep their systems safe using SSH keys, but by failing to keep those keys secure, they are ultimately more at-risk. Without protecting their SSH keys, enterprises are as vulnerable as ever to circumstances like the one’s Booz Allen Hamilton – one of the country’s top defense contractors–saw. In this particular scenario, the private SSH keys and various plain text passwords belonging to employees and contractors with Top Secret Facility clearance were left unsecured (and without a password) on a public Amazon server.

Evolution of ADC to Cloud

Creating a key pair is fairly easy, but mapping their trust relationships (or who can access what) at a later stage is becoming increasingly difficult. And also the growing adoption of hybrid and multi-cloud environments has inevitably resulted in a proliferation of SSH keys in an environment.

Here’s an example. Due to an inherent vulnerability, RSA-2048 bit keys are deemed unreliable. You are tasked with identifying all SSH keys in your infrastructure with that particular vulnerability. You open your key inventory spreadsheet to begin, only to find it is outdated. So, you plan on rebuilding the inventory from scratch. For an accurate count of all your SSH keys, you must manually log into each machine, map their trust relationships and then terminate vulnerable keys from each one. Now, imagine logging into each of your thousands of servers to gain access to anywhere from 10 to 200 keys per server. ²

Is your head spinning yet?

To make matters worse, you can make a mistake at any point in this exercise.

And herein lies the shortfalls of manual SSH key management.

It is:

• Unreliable – the documentation is not automated and some entries can be missed.
• Inaccurate – anyone can tamper with the data.
• Error-prone – the manual methodology has many layers of human intervention.
• Inefficient – a small error can undo any progress and leave critical systems vulnerable

Manual management shortfalls aside, the most important part of any security system is visibility. And with current SSH key management practices, there is less to no visibility across your SSH keys. We looked at one upgrade scenario above. Now let’s look at four additional examples of management practices that risk falling short of your expectations.

1) Monitoring the SSH Environment – An SSH environment is riddled with thousands of keys. With some keys granting root access and other keys guarding valuable data, it is important you are in control of who has access to what servers. You need control over where each key can be used and what commands can be executed using each key. Maintaining complete visibility over these keys can prevent misuse and also flag unused, orphaned and defunct keys before it’s too late.

2) Rotating SSH Keys – Unlike SSL certificates, SSH keys do not have expiration dates. Instead, SSH keys continue to provide access to your application unless explicitly removed. Key rotation, i.e., changing every authorized key (and corresponding identity keys) regularly, is an important security measure that prevents hackers from misusing compromised keys. As a best practice, you should rotate all your keys every 60 days, which is almost impossible without proper visibility.

3) Key Compliance – Compliance requirements put forth by agencies like SOX, PCI, and FISMA require enterprises to properly terminate access to critical systems. But in the case of key-based access mechanisms, the access is essentially permanent, leaving it in direct violation of these compliance requirements. To properly regulate access to critical systems, users need visibility, which traditional methods fail to provide.

4) De-provisioning Departing Users – Unlike with SSH key-based access, users are not permanent. Some may leave the organization and some may switch departments. What is going to happen to all their keys post-departure? Their access must be revoked. Instead of manually checking each server and removing their accounts, users should consider removing unnecessary accounts from all servers with a single click, using full visibility mode.

SSH Key Management in a Multi-Cloud Environment

Each cloud vendor – whether it be Amazon AWS, Google Cloud Platform or Microsoft Azure – allows you to generate a new key pair or import an existing key pair from your inventory. However, they can fundamentally differ in the way they manage the addition of users and SSH access in the cloud. While AWS allows you to push a key directly into the VM during instance creation, Google Cloud Platform only allows cloud administrators to push a key up to the instance metadata level. With AWS and Azure, users are not allowed to push new key pairs through the console once the instance is up and running, while GCP allows its users to create a key on-demand, even if that instance is already active. And, similar to any on-prem device, users can also log into each cloud vendor’s VM independently and push new keys to provide access to necessary users. In these scenarios, it is crucial that the cloud vendor’s console accurately represent the SSH key count within its key pair inventory and its many VMs. But none of these consoles can ever display an accurate count of its SSH keys when the keys are independently pushed into the VMs. Additionally, users must also factor in the differences in SSH key management practices among cloud vendors which require explicit training for in-house expertise. With all these difficulties in the current system, do you still think manually managing SSH keys across on-prem and multi-cloud environments is the way to go?

Simplify SSH Key Management in the Cloud with AppViewX

When using AppViewX’s Certificate Lifecycle Automation solution, it doesn’t matter if your keys are in the cloud or on-premises, the management practice remains the same across all environments and is flexible to conform to each specific vendor on the platform. Now, let’s look at some of the AppViewX Platform features that make SSH management both effortless and errorless.

Discovery and Inventory:

You can discover keys from multi-vendor, hybrid network infrastructures – like servers, ADCs, client devices, cloud instances and VMs– on an on-demand basis. The AppViewX Platform allows you to keep your inventory updated every day with an option to schedule the key sync each night.

Once the keys are discovered, they are stored in an inventory that gives you centralized visibility of all SSH keys across hybrid and multi-cloud environments. No more logging into each VM or on-prem machine to identify the number of SSH keys present in them – just by clicking “Discover,” the AppViewX Platform provides you a full view of the keys.

Automated Key Management:

When your inventory becomes cluttered, our holistic view graphically represents information tied to each key, such as associated hosts and accounts, to make management simpler. A work order based mechanism applies the necessary checks during key creation or modification to avoid key proliferation. With the AppViewX Platform, keys are created using best-in-class encryption algorithms with passphrase protection, then pushed to the required hosts automatically – not just to the cloud instances, but also to respective VMs within them. You can choose to reuse keys on your on-prem devices or in the cloud to avoid importing key pairs through the cloud’s console. And, by automatically rotating SSH keys, unauthorized users, with wrongful access to critical systems are permanently removed.

Role-Based Access Control:

All keys are not created equal. Some protect access to mission-critical application systems while others protect access to less-important testing environments. With the AppViewX Platform, keys can be grouped based on functionality, and required policies (such as recommended cryptographic techniques and workflows) can be mapped appropriately to make management simpler. And, our RBAC’s tight integration with your LDAP ensures that the necessary teams have a granular view of key groups, as well as the ability to monitor policy violations and unauthorized key usage. Using this granular access, you can rest assured that no individual has more privileges than they should ideally have.

Conclusion

The secure communication that the SSH protocol provides is sure to promote adoption. Given it is easier to create keys at will than identifying existing ones in the environment, there is a chance that more than 90 percent of our existing SSH keys are inactive². Hence, the sheer number of SSH keys in an already complex infrastructure will make it even more difficult to manage without the proper tools. AppViewX helps you achieve the visibility required to manage your SSH keys more efficiently and without the errors incurred by manual intervention. With essential role-based access control and simple work-order based task executions, managing your SSH keys will never be easier. To find out how AppViewX can help you automate SSH key management in the cloud and on-prem, please visit https://www.appviewx.com/solutions/certificate-lifecycle-automation/

Technology type	Vendors
ADC	F5 Networks, HA PROXY
CLOUD	Amazon Web Services, *Microsoft Azure, *Google Cloud Platform
LINUX	CentOS, Debian, Fedora Server, FreeBSD, Linux Mint, Opensolaris, Redhat, Suse, Ubuntu