Find and Fix Potential Etcd Data Exposure with Dome9

Over the weekend, a cybersecurity researcher wrote about thousands of servers running the popular etcd service that were exposing sensitive information — passwords, credentials and configuration settings to the world.

What Happened?

Etcd is an open-source, distributed key-value store that stores the configuration parameters required for different servers and applications, and this can include service auth credentials. As apps read/write data, Etcd allows for a more efficient process for distributing passwords and reconfiguring servers and networks, making it a popular choice for managing container clusters.

By default, earlier versions of the etcd service returned stored key-value pairs without requiring any authentication.

A simple Shodan query revealed servers that exposed the service port 2379 to a wide scope, and the following GET request retrieved the key-value pairs from these servers:

GET http://<ip address>:2379/v2/keys/?recursive=true

This exposure is ironic, because a service that is used to store credentials for other services such as databases was itself running unprotected. There were two simultaneous security lapses that resulted in this exposure:

1. Running etcd in default setting without authentication enabled

2. Leaving the service port open to the public

This is not the first time that an unauthenticated open-source service was exposing data because of insecure default settings (anyone remember MongoDB or ElasticSearch data exposures?)

What Can We Do About It?

This vulnerability could have been avoided by following simple security best practices to start with:

– Adopt a closed-by-default network security posture: Unless you have a valid reason to keep a service running, turn it off. Open it up to just the right level of scope (e.g., through security group settings on AWS or tags in GCP). If possible, implement security measures to open the service port for just the time that it is needed.

– Implement a microsegmentation architecture that segments the application and controls which parts of the application can talk to which others, and which parts are allowed to talk to the public.

– Never open a service to the outside world until and unless you have authentication enabled. As a corollary to this, understand the default settings of the services that you run, including (and especially) open-source services. It is your responsibility to Read The Fun Manual and configure the service appropriately.

How Can Dome9 Help?

Dome9 can help you protect your servers by making sure that no security groups have TCP port 2379 open to a wide scope. The Dome9 Compliance Engine gives you a simple automated way to check if any of your instances have opened TCP port 2379 to a wide scope.

You can run this rule at any point of time as well as get an automated alert to your email or some other way. This simple security policy rule written using the Governance Specification Language (GSL) covers this:


Instance should not have inboundRules contain [port <= 2379 and portTo >= 2379 and protocol in ('TCP','ALL') and scope='0.0.0.0/0']


You can inspect an exposed instance using our entity explorer:

[wp_colorbox_media url=”https://dome9.com/wp-content/uploads/2018/03/etcd-1-960×469.png” type=”image” hyperlink=”https://dome9.com/wp-content/uploads/2018/03/etcd-1-960×469.png” alt=””]

You can also fix the security group misconfiguration from within the Dome9 platform in Full Protection mode:

[wp_colorbox_media url=”https://dome9.com/wp-content/uploads/2018/03/active-etcd-960×469.png” type=”image” hyperlink=”https://dome9.com/wp-content/uploads/2018/03/active-etcd-960×469.png” alt=””]

Dome9 also offers an active protection capability called dynamic access leases. These are just-in-time ACL’s that let you lock down servers+ports by default. Admins and ops teams can get access to specific services for a pre-defined period of time, after which the ports are closed again. Though etcd should normally not be exposed to the public, if for some reason a human admin wants to connect for a limited amount of time, then access leases can come handy.

[wp_colorbox_media url=”https://dome9.com/wp-content/uploads/2018/03/etc-960×71.png” type=”image” hyperlink=”https://dome9.com/wp-content/uploads/2018/03/etc-960×71.png” alt=””]

As customers transition to the public cloud, the traditional IT “perimeter” has evolved and an edge firewall isn’t enough. With the dynamic nature of cloud assets and the threat landscaping evolving, you need a combination of network security and access control to truly provide a holistic infrastructure security and protect against such attacks.

Learn how Dome9 provides powerful active protection capabilities that serve as guard rails to protect your assets in the cloud.

Sign up for a free trial @ Dome9!