A Real Project Experience With AWS VPC Endpoints — When, Why, What, and Caveats

8 min readNov 10, 2023

Hi! As my first Medium article, I would like to share with you a real case experience that I had with AWS VPC endpoints. I wish that I had read this article when I was dealing with AWS VPC endpoints.

The main goals of this article are:

Discuss When and Why to Utilize VPC Endpoints
Define What Are VPC Endpoints
List Caveats and Tips To Avoid Problems

When and Why to Utilize AWS VPC Endpoints?

A new demand has arrived: my team needed to restrict, as maximum as possible, the communication of a given set of applications deployed on AWS (due to legal reasons, I can not share specific details about the applications). One of the restrictions was related to making sure that the applications were not communicating via internet gateway, but in a private connected way to the supported AWS services.

And that answers the first question: when to use AWS VPC endpoints? If it is a requirement to not expose the application through any public internet communication (via Internet Gateway), you would achieve this, on AWS, by using AWS VPC endpoints.

But why do we need to be concerned about it? Concise answer: security worries and billing costs.

If your context involves a security need and you do not want to impose availability risks or bandwidth constraints on your network traffic, you might want to use it. And also, to cut costs, given that it is not required to have a deployed internet gateway, network address translation (NAT) device, Virtual Private Network (VPN) connection, or AWS Direct Connect connection to use the VPC endpoint. [1]

What Is a VPC Endpoint?

Before I properly define it, I would like to show two images that address the problem:

Figure 1 — Architecture without VPC endpoint

From Figure 1, we can see that the communication from the Lambda to S3 goes through a NAT Gateway and then goes to the internet, to finally reach the S3. There are two main downsides with this approach:

In terms of security, the traffic is leaving the VPC and the AWS Cloud, and then it comes back. This approach was not a desired behavior given our security requirements.
In terms of costs, you are hourly charged with this approach, due to the use of NAT Gateway.

Is there any way to make it safer and also reduce the costs? Let us reference the second image:

Figure 2 — Architecture with VPC endpoint

Notice that the traffic is occurring only within the AWS Cloud, not going to the internet. The data flows from the Lambda, to the interface VPC endpoint and then to the S3.

In this case, the data does not go through the NAT Gateway. So, it would be possible to delete it from this architecture, saving money.

Therefore, the Figure 2 approach allowed us to meet the requirements regarding security and also save money.

A Formal Definition

According to Amazon:

VPC endpoints are virtual devices. They are horizontally scaled, redundant, and highly available Amazon VPC components that allow communication between instances in an Amazon VPC and services without imposing availability risks or bandwidth constraints on network traffic. [2]

There are two types of VPC endpoints: gateway endpoints and interface endpoints.

Gateway Endpoints

A gateway endpoint is designed to direct traffic to specific IP routes within an Amazon VPC route table, using a prefix-list format that is managed by AWS. This routing mechanism is employed for directing traffic bound for Amazon DynamoDB or Amazon Simple Storage Service (Amazon S3) services. It is important to note that gateway endpoints do not provide support for AWS PrivateLink. [2]

Besides, there are no additional charges for using gateway endpoints. [3]

Interface Endpoints

According to Amazon:

The interface endpoints facilitate connectivity through AWS PrivateLink to various services, encompassing AWS managed services, services hosted by other AWS customers and partners within their Amazon VPCs (referred to as endpoint services), and compatible AWS Marketplace partner services. [2]

Under the hood, the interface endpoint uses AWS PrivateLink, a highly available and scalable technology that enables you to privately connect your VPC to services like AWS API Gateway, AWS AppSync, AWS S3, and other AWS and third party services, as if they were in your VPC.

An interface endpoint comprises one or more elastic network interfaces, each assigned a private IP address, functioning as the entry point for directing traffic toward a supported service.

Presently, interface endpoints offer support for numerous AWS managed services. And also, they support the use of security groups.

Caveats and Tips

Now, I would like to share with you some lessons learned from the use of VPC endpoints. Here is the list of topics in this section:

“Enable DNS name” Option
Certain Services Do Not Work with Disabled “DNS name” Option
Turning API Gateway and AppSync APIs into a private API

“Enable DNS name” Option

When creating a new interface VPC endpoint, there will be a configuration called “DNS name”, that is turned on by default. When choosing the VPC for your interface VPC endpoint, right below the VPC drop down, there are “Additional settings”, as shown in Figure 3 with a red highlight :

Figure 3 — Selecting VPC when creating an interface VPC endpoint

When expanded, you may find the default “Enable DNS name” option turned on, as shown in Figure 4:

For example, you have a Lambda running with Node.js and you want to access S3.

If you create an interface VPC endpoint with the discussed configuration turned on, in order to guarantee communication between the services, you need to make sure that:

Lambda’s security group allows outbound traffic to port 443, type HTTPS and protocol TCP to the security group attached with the S3 interface VPC endpoint
The interface VPC endpoint’s security group must allow inbound traffic from Lambda’s security group on port 443
“Enable DNS hostnames” and “Enable DNS support” attributes are enabled for your VPC

Figure 5 details the security groups rules:

Figure 5 — Detailed security groups rules

But, what if you do not want the “Enable DNS name” option turned on? Right now, you might be thinking: why do we need to complicate things?!

In my team’s case, we were dealing with a highly coupled infrastructure that existed for a long time, and we did not have the time needed to properly decouple. So, we did not want to apply private communication to all components of our infrastructure, given it could affect working applications out of the demand’s scope. That was why we needed to turn it off.

In this case, you must specify the URL to be used when creating the S3 client. Go to the AWS Console, then VPC -> Endpoints and find the S3 interface VPC endpoint. Click to show the details. You would find something like shown in Figure 6:

Your URL will be the one showed in “DNS names”, with a little adjustment: replace ‘*’ with ‘https://bucket’. The final URL for this example would be: “https://bucket.vpce-1a2b3c4d-5e6f.s3.us-east-1.vpce.amazonaws.com”

Now, go to the Lambda’s code and specify that URL when creating a new S3 client, like this:

import { S3Client } from '@aws-sdk/client-s3';
...
const S3_ENDPOINT_URL = 'https://bucket.vpce-1a2b3c4d-5e6f.s3.us-east-1.vpce.amazonaws.com';
const s3Client = new S3Client({
      apiVersion: API_VERSION,
      endpoint: S3_ENDPOINT_URL,
    });
...

For more information, check out [4].

To summarize, the first approach involves the adjustment of security groups rules, to guarantee the communication between the services, and this affects all the resources that consume the VPC endpoint’s specified type (S3, API Gateway, SQS…). The second approach requires the adjustment of the security groups as well, and also the code adjustment.

Certain Services Do Not Work with Disabled “DNS name” Option

At the time of writing this article, AWS does not provide us another way of using the following services without the “Enable DNS name” option enabled:

com.amazonaws.region.ecr.dkr
com.amazonaws.region.ecr.api

Although it is possible to create them with the option disabled, you will not be able to use them correctly unless the option is activated [5].

I wonder why they still allow that creation, even though the documentation states that it should be turned on…

Turning API Gateway and AppSync APIs Into a Private API

We were dealing with an EC2 that communicated with a public API Gateway and AppSync. At that time, the EC2 had a security group with a too permissive outbound rule: it was allowing HTTPS traffic to 0.0.0.0/0. One of our requirements was related to the removal of the 0.0.0.0/0 rules. By doing so, how did we adjust the configurations to keep the communication working?

API Gateway Changes

We needed to change the existing public API, from API Gateway, into a private one. Then, we attached an interface VPC endpoint of type com.amazonaws.{region}.execute-api to it. Subsequently, we allowed outbound traffic from EC2 to the VPC endpoint security group. And also, allowed the inbound traffic in the VPC endpoint security group from the EC2, like shown in Figure 7:

Figure 7 — Detailed EC2 and private API from API Gateway communication

One important thing to mention is that turning a public API Gateway into a private one does not change the resource ID of it. So, other applications accessing it would only need to change the security groups rules, but they would be able to use the same API URL.

AppSync Changes

To tackle this problem, we needed to have a private API on AWS AppSync. Unlike the API Gateway, it is not possible to change an existing public API to a private one. It is necessary to create a new one, configured as private, therefore, there will be a resource ID change. So, the applications that once referred to an old API URL (the public URL) now need to reference the new one.

We created a new interface VPC endpoint, of type com.amazonaws.{region}.appsync-api.

The security groups rules are similar to the ones mentioned earlier, shown in Figure 8: