System tracing with AWS X-Ray

Knowing your application is running fast or slow is one thing, understanding what makes up those characteristics is another. This is not a new problem, APM (Application Performance Monitoring) tools like New Relic and AppDynamics will happily take your money to make this problem easier and have done for a while. However, this problem gets even more complicated in a serverless environment where integration points are not always totally under your control. This is a frustration of mine as AWS Lambda can often be an opaque beast.

AWS X-Ray is a lightweight entrant into this field, with the extent of the configuration being a tick box in the ‘advanced settings’ section of the ‘config’ tab. The feature switch will even add the required permissions for you.

With this enabled, each request will now produce a trace.

Now for the interesting bits, we can see that this Lambda was cold and took a while to start executing. We can also see where Amazon have called out some of their ‘initialization’ time.

Slow Lambda execution

My Lambda includes a call to a RESTful API but the default X-Ray setup won’t show this. I want to call this out especially in my monitoring because this external dependency has the possibility of affecting my service. To do this I need to create a sub-segment, Amazon as ever have thought of everything and have created a decorated version of HttpClient with their own X-Ray integration. It’s literally a drop in replacement with no further configuration. With this addition, I now get a much clearer view of the profile of my service. The remote API call I’m calling makes up a large percentage of the total time taken.

Lambda execution with HttpClient augmented

Amazon does support creating your own custom sub-segments which is something I want to try out soon and maybe write about later.

The other interesting feature I’ve not yet tried but is also on the backlog for investigation is attaching metadata to the segments and sub-segments. I can imagine this being really useful by adding pertinent data to traces, this could then be used in investigating themes of slowness for example.

Finally, another freebie from Amazon: out of the box, a service map is created linking together any associated resources it can find from your traces. The green around each service is a ring chart of % success, these turn yellow if there are problems.

I could imagine a slightly more sophisticated version of this being a fantastic DevOps dashboard.

linked services

Using AWS Certificate Manager and ELB with WordPress

Over the last couple of weeks, I’ve been using this website as a way to learn more about Amazon Web Services. I’ve always found topics easier to learn when I have something practical to apply them to.

During some time away from work, I started looking into making this site HTTPS. Security is something which interests me in and out of work, so it seemed like a good idea.

The plan was to add HTTPS to my WordPress EC2 instance, but I didn’t know much more than that.

I knew that Amazon had Certificate Manager, so I started there.

Here is the first piece of AWS awesomeness, but with a cost: they will give you a free certificate, but currently you can only use it with their Elastic Load Balancing. On first consideration, I thought this was going to be a hassle and increase the work required, but now on reflection, I think this is a good approach.

I can add ELB in front of my EC2 instance and use that to terminate my SSL connections. Now my application doesn’t really have to change; it can be oblivious to the security layer I want to add.

Two different ELB styles

To reduce complexity I went with the Classic Load Balancer instead of the Application Load Balancer. I didn’t feel I needed any of the application level features.

One thing to watch out is that the default health check uses “/index.html”. The install of WordPress I’m using produced an error for that URL and the ELB took my instance out of its pool, this effectively took my site down. Changing the health check to “/” was easy and brought me back online.

Out-of-the-box ELB comes with some really useful monitoring baked into its console UI. This was great for identifying and resolving the problem, a common theme on this platform.

ELB Metrics

I choose to maintain both HTTP and HTTPS for my website, mapping both the HTTP on my instance. Again my instance doesn’t know anything about the ELB.

mapping http and https