VHost specific requests for load balanced services using Ethon
With the POODLE SSLv3 vulnerability which was found a couple of weeks ago basically everyone turned off SSLv3. So did we.
Unfortunately this caused a few other problems for us.
We are running multiple Ruby on Rails applications on multiple servers which all listen on the same port but different vhosts.
<VirtualHost *:443>
ServerName api.example.com
...
</VirtualHost>
<VirtualHost *:443>
ServerName api-sandbox.example.com
...
</VirtualHost>
The server uses the domain to select the correct vhost to then forward the request to the right application.
In front of those servers there is a load balancer which then tries to evenly distribute the requests between each all the servers running instances of the application.
This is a setup which works quite well for us. Multiple requests can come in and in case of too much load we can just deploy an application on another server and add it to the load balancing.
After a successful deployment we run a couple of smoke tests to make sure that the correct version was deployed and responds in the way we would expect it to. All of this is part of our general deployment process and has been automated as far as possible.
When deploying the application on multiple servers at the same time we have to make sure to check that each instance of the application got smoke tests running against it. For that reason we have to bypass the load balancer and make direct requests to each individual server.
For SSLv3 this actually was not much of an issue. We would just make a basic HTTP GET request to the server IP and set the Host header. This would allow the server to correctly pick the application and thus respond as expected.
Since we are doing almost all of our development in Ruby the deployment scripts are written in Ruby (Capistrano and a few custom scripts for the smoke tests). A request which does the trick looks like this:
require 'net/http'
require 'uri'
require 'openssl'
url = "https://10.10.10.10/path"
host = "example.com"
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
http.ssl_version = :SSLv3
request = Net::HTTP::Get.new(uri.request_uri)
request['Host'] = host
response = http.request(request)
While having SSLv3 active this works like a charm.
Unfortunately this does not work for TLSv1. So just replacing :SSLv3 with :TLSv1 would not do the trick.
It actually took me quite a while to figure out the best way of making the smoke tests work again.
I could not find anything in the Ruby standard library which helped me accomplish calling a very specific application on a specific server (or maybe I just did not look hard enough).
The issue here was that TLSv1 and higher are using SNI which is a TLS extension that requires providing the domain name already during the handshake process.
At first I tried to find a similar approach of calling the IP and providing some kind of HTTP header.
This does not work because the information about which application should be called is not available at the handshake. Since the server cannot select the correct vhost it also does not know which SSL certificate to return.
In order to keep using TLS and avoid having to add a workaround (like a second host specific vhost) another approach was required.
I tried a couple of different Ruby HTTP clients but could not quite get it to work. After a while a colleague pointed out the --resolve flag for curl.
This actually solved the issue. just call
curl https://example.com/path --resolve "example.com:443:10.10.10.10"
and the request will skip using DNS and just assume that example.com for requests on port 443 is located at 10.10.10.10. The server will know which virtualhost is called as it does on any normal request which was forwarded by the load balancer, since the domain is actually given in the request uri.
Well, now there is a solution. Running a console command directly from within the Ruby code is usually quite ugly (in my opinion). This is especially the case since I would have to manually parse all of the output and make sure that I get the correct response code, response body and content type.
Therefore I kept looking to find a solution which would integrate well into our existing code.
We are using the Ruby gem Typhoeus a lot and it is even based on libcurl. I figured there should be a way of passing the resolve flag along and getting the same result as I did using curl. Unfortunately this is not the case or maybe I just did not look hard enough - again.
Either way, in the end I decided to take a look at Ethon. Ethon is the libcurl wrapper which is also used by Typhoeus. I figured, without the abstraction it should be easy to map any curl request directly to a call with Ethon.
Turns out I was right. I had some issues of figuring out how exactly Ethon wanted the resolve flag to be passed in though.
My first attempts at just passing a plain string failed miserably. I guess I expected too much abstraction at a wrapper gem which is the basis for another great gem (which usually adds the abstraction and ease of use I like)
e = Ethon::Easy.new(url: "https://example.com/foo", :resolve => "example.com:443:10.10.10")
results in
Ethon::Errors::InvalidValue: The value: example.com:443:10.10.10.10 is invalid for option: resolve
Not being able to figure out how to pass it in right away and not being willing to invest enough time to eventually figure it out on my own, I decided to open up an issue on github and ask the developer directly.
Who better to give you precisely the information you need on how to use a library than the person who probably knows all of the ins and outs?
Well, as it turns out I was right. I got the information I needed to come up with a solution for our smoke tests very quickly.
The following snipped is what I ended up with to just fit right in where I took the net/http part out.
require 'ethon'
require 'ostruct'
require 'uri'
r = "example.com:443:10.10.10.10"
uri = "https://example.com"
resolve = Ethon::Curl.slist_append(nil, r)
e = Ethon::Easy.new(url: uri, resolve: resolve, timeout: 4)
e.perform
e.response_headers =~ /^Content-Type: (.*)$/
content_type = $1
response = OpenStruct.new(
:body => e.response_body,
:code => e.response_code.to_s,
:content_type => content_type
)
and for POST requests:
require 'ethon'
require 'ostruct'
require 'uri'
r = "example.com:443:10.10.10.10"
uri = "https://example.com"
resolve = Ethon::Curl.slist_append(nil, r)
e = Ethon::Easy.new(timeout: 4, resolve: resolve)
b = "" # your post body
e.http_request(uri, :post, body: b)
e.perform
e.response_headers =~ /^Content-Type: (.*)$/
content_type = $1
response = OpenStruct.new(
:body => e.response_body,
:code => e.response_code.to_s,
:content_type => content_type
)
The resolve flag is actually not just useful for issues with TLS and SNI but rather allows you to make any request on a server despite a specific domain not actually pointing to the server you are trying to call.
This will certainly come in handy for setting up a server and testing the vhosts before updating the DNS records.