7

Some background:

I've created a moderately complex network using Amazon's vpc. It's a three-tiered network across two availability zones. Each layer has a subnet in zone-a and zone-b. The presentation layer is at the top, there’s an application layer in the middle, and a core layer at the bottom.

All of the security groups and ACL's for the subnets are currently allowing ALL inbound and outbound traffic to help me reduce surface area of the issue.

The presentation layer’s routing table is pointing all traffic to an internet gateway. The NAT gateway is in a segregated subnet also pointing all traffic to the internet gateway.

My application has two components, a UI (React.js) and an API (Node/Express). They are deployed as docker images. In front of each is a classic load balancer.

The UI-ELB is internet facing and resides in the presentation layer, routing traffic from 80/443 to port 8080 and is associated with my app-ec2 that is placed in the application-layer subnet.

My API has an internal load balancer in front of it. The API-ELB is in the application layer (in the same subnet as the app-ec2), and takes traffic on port 80/443 and routes it down to the api-ec2 in the core on port 3000.

Both load balancers are offloading the certificate before passing traffic to their instances.

I have both my load balancers associated as alias's in Route53, and referenced in the applications by their pretty url (https://app.website.com). Each load balancer passes the defined healthchecks and reports all ec2 instances in use.

Lastly, on the API I have enabled cors using the cors nodejs package.

Here's a quick and dirty diagram of my network.

The problem:

The APP-ELB successfully routes me to the application. However, when the app tries to send a GET request to the API-ELB, it first sends an OPTIONS request that tims out with the error code 408.

Where it gets weird

Some of the weirdest things I've encountered while debugging are:

  1. I can SSH into the app-ec2 instance and can run a successful curl against the API-ELB. I’ve tried many, and they all work. A few examples are: curl -L https://api.website.com/system/healthcheck and curl -L -X OPTIONS https://api.website.com/system/healthcheck. It always returns the desired information.
  2. I've moved the entire application out of my network into a public default vpc and it works as it's supposed to.
  3. I have the api-ec2 writing all network requests to the console. While it shows the healthcheck requests, it does not show any requests from the app-ec2. This leads me to believe traffic is not even reaching the api.

Really the biggest thing that has me at a complete loss is that curling the internal api elb works, but the axios request to the same exact url does not. This doesn't make sense to me at all.

What I've tried

I originally spent a lot of time playing with ACL rules and security groups thinking I did something wrong. Eventually I just said, "screw it", and opened everything up to try and take that piece out of the equation.

I've spent way to much time playing with Cors on my api. Eventually landing on the configuration I have now, that is the default app.use(cors()) callback provided by the cors node package. I've also included the app.options('*', cors()) that is recommended in the documentation.

I've google everything under the sun, but specifically whether I need to define some special custom headers with the elbs? But can't seem to find anything. Plus, when I moved my app out of the network it worked just fine.

I'm sure I've tried many other things, but these seem to be the most pertinent. What am I missing? I realize this is potentially a very vague and broad issue, and an enormous post, but I appreciate any insight and your time in reading in it!

  • Thats two subnets per zone: a presentation layer, an application layer, and a core layer. That's three. Can you clarify that? I have the route table for the presentation layer and NAT drive routing all traffic through an internet gateway. I have both the presentation and core layers routing all traffic through the nat drive. That seems self-contradictory. If the presentation layer is routing through the NAT (drive?) (Gateway?), then it isn't also routing through the Internet gateway. Which of your layers are on which of your subnets, and what's the default route for each subnet? – Michael - sqlbot Jun 5 '17 at 20:16
  • 1
    ...Specifically, your external-facing ELB must be on subnets whose default route points to the Internet Gateway, which almost always means it is not correct to place it on the same subnet(s) as the instances it is balancing traffic towards. Those instances would be on a subnet whose default route is the NAT Gateway... and the NAT Gateway itself, in turn would not be on any of the same subnets as the instances for which it is providing outbound services but might be on the same subnet as the ELB. – Michael - sqlbot Jun 5 '17 at 20:20
  • Yeah, sorry about that, started getting some words mixed up. I have two subnets (one in zone-a and one in zone-b) in each of the three layers. The presentation layer is routing through the internet gateway, as is the nat. One thing I did fail to mention was that NAT is in it's own seperate subnet. Then the app and core route through the nat gateway. – David Meents Jun 5 '17 at 20:21
  • 1
    You may want to thoroughly review the question and clarify accordingly -- it's a good question, but there are some difficult convolutions to follow, and many things to check. When you say "The application-UI is timing out on the options request when making an api call" who sees this error? The external caller? curl -X OPTIONS 127.0.0.1... on app-ec2? Only OPTIONS is broken? The ELBs are "Classic" not "Application," correct? Are all of the instances able to correctly access the Internet via the NAT, e.g. curl ipv4.icanhazip.com? (Yes, I ask for a reason that may seem obscure.) – Michael - sqlbot Jun 5 '17 at 20:36
  • 1
    Unless I'm totally mistaken, react.js apps run in the browser and need to contact the API server, your frontend server just serve html and js files and does no routing/proxying of request toward the API – Tensibai Jun 6 '17 at 15:52
7

So what you have actually is this:

Schema of OP's architecture

As your API ELB is in a private zone it can't be accessed from the internet.
Your frontend in React.js just run in User's browser and not on the UI servers, those server just serve static files.

You have two options, configure your frontend servers to redirect API calls to the API ELB or just update the API ELB to be internet facing.

The usual pitfall of JavaScript apps is forgetting they run inside user's browser and not on the frontend servers as a JEE application would.

1

This sounds like an asymmetric or n-path routing issue. Here is what is probably happening:

Machine A at IP address 192.168.1.1 makes initiates a [SYN] request through the LB at 192.168.1.10. the LB then proxies the payload to Machine B at 192.168.1.2, so the payload now has source: 192.168.1.1 and has has destination: 192.168.1.2 (which used to be 192.168.1.10).

So what happens now when 192.168.1.2 responds with a [SYN, ACK]? What should happen is that Machine B should respond to Machine A through the load balancer - typically because of a default route or gateway on the server that routes traffic through the LB. In this case however, the machine is on the same subnet, so the route/gateway is not used and the routing table ignored by the server. This means that when the server responds, the [SYN,ACK] appears to Machine A to come from an IP different than the IP that Machine A initiated the request with - it was expecting a source IP of 192.168.1.10 (the LB) but is seeing a [SYN,ACK] coming from 192.168.1.2 (machine B) and thus the LB is unable to establish a connection with the machine B in this scenario because the response went to the wrong device.

The reason this works for external traffic is because of your default route - the responses to everyone else are routed through the ELB. The ELB sees that it was initiating a connection and automagically intercepts the response and swaps the source of 192.168.1.2 back to 192.168.1.10.

So, for one solution to this issue, you could implement one-armed load balancing (also known as a load balancer on a stick). What this will do is use a Source NAT on the inside interface of the load balancer (so assume you had outside interface 192.168.1.10 on your load balancer and 192.168.1.11 on the inside interface). This will make all traffic appear to be coming from 192.168.1.11 from the perspective of Machine B which should solve your connection issue.

It appears however, that your AWS ELB doesn't support SNAT, so you will either need to put your hosts and ELB on different subnets or use something that supports SNATs like F5's Virtual Edition which comes in hourly or BYOL flavors. Beware connection limitations with SNATing though - if you need over about 30k simultaneous connections you will run into SNAT port exhaustion and need to start using a SNAT pool..

Hence, you best solution (for cost and to prevent future issues) would be to make sure the client and server are on different subnets.

The best way to confirm would be to use tcpdump on the connecting host and/or back-end server and look for responses coming directly to/from the back-end server instead of going through the load balancer. You can then load your dump file into WireShark to figure out exactly what is going on.

  • ELB doesn't forward packets. It makes new TCP connections and forwards payloads. Route asymmetry is one thing the problem can't be. – Michael - sqlbot Jun 5 '17 at 19:02
  • So do F5's and they still suffer from asymmetric routing issues. Even with a full proxy architecture and a separate, new TCP connection, an F5 load balancer will by default the source address of the connecting client, so the issue still occurs exactly as described above. I am assuming the ELB works in a similar manner. I know A10s do behave in the same manner. – James Shewey Jun 5 '17 at 19:13
  • They don't, ELBs have a seperate IP. The – Robo Jun 6 '17 at 9:58
  • 1
    AWS ELB can act as a reverse proxy, not only as a TCP load balancer. As OP said the ELB is doing the SSL offloading, it can't be a TCP balancer and have to be a HTTP reverse proxy. Your answer is not applicable to the context and ELB are never used for outgoing packets (they are not routers at all). Moreover, if you try to proxy with an F5 with 2 interfaces and setting the same subnet on each interface you're indeed creating a problem, solving it with SNAT is just a bad solution. – Tensibai Jun 6 '17 at 15:16
  • 1
    Well, here what the machine B from your example see is the ELB IP, the client IP will be in the X-Forwarded-Port header. The client could be aside the server that won't be a problem. When in HTTP mode an ELB doesn't act as a F5 with SSL termination. (even in TCP mode, it's still an nginx like load balancer, nothing remotely comparable at all). What I think you are hung on is "proxying", we're indeed talking about proxying packets and not forwarding packets. I can send you a tcpdump of an ELB within the same subnet as 2 machines if you want, it works. – Tensibai Jun 6 '17 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.