Just writing up a quick ‘gotcha’ that we faced recently in our work with AWS that we found very little documentation available, and no real posts indicating how to resolve, so thought we’d put this out there in the hopes that someone finds it useful.

We’ve just been building an app that is a node front end that interacts with both SNS (for messaging/events) and Aurora (as a data store).

This is our first foray into Aurora, so IAM permissions across the board hadn’t been granted. We decided to follow this approach to securing our app (which would eventually live in ECS), so that we could tie down exactly what this app was doing at the ECS task level.

The node app was relatively straight forward to build (simple CRUD API for the most part), and the tokenisation was easy to setup, and then things stopped with the following error:

Our code looked (roughly) like this:

We went through a number of iterations on this, and found a lot of guidance around the initial handshake needing to be sent over in clear text when using the tokenisation RDS signer, which is an important step to include like so (we use sequalize):

We had a user setup within IAM that had the appropriate Aurora role awarded (interestingly, the AWS console wasn’t happy at all with the role that came with that guide above:

The AWS console reported that ‘rds-db:connect’ didn’t exist, you can’t select it in any of the profile builders, and that this profile wouldn’t have any impact at all on permissions – not exactly confidence building! We lost a lot of time investigating this simply because we wondered if the AWS guide was out of date etc. – turns out it wasn’t, the console just wasn’t aware of the setting.

Lightbulb moment

One of the developers wondered if the AWS.config.update call in the above was actually working, and instead changed his local AWS default profile (the one that lives in your local .aws/credentials file) away from our standard dev profile (which did not have any Aurora permissions) into using the key/secret key of the user that we had setup in AWS which did have aurora permissions. Bang! Straight through, connecting no bother.

Naturally, we weren’t going to run the ECS task under a role (we were attaching a role to the ECS task), but locally we run as a shared development account.

We switched back to the developer account, and attached the appropriate Aurora role to it, and all things continued to work as expected.

So, the gotcha for us really was the AWS.config.update – it seemed to ignore this, and use instead our default profile. With more experience, we’d have realised this sooner, though we’re at least at the bottom of it and thought we’d circulate for a wider audience.

Important note around IAM policy changes and final gotchas

Probably something that most are aware of when working with IAM, but policy updates aren’t immediate – we had a number of cases where a change didn’t appear to work because we were testing immediately on the application of the policy – at worst, they were taking 5mins or so to come into play, so it looks like policy updates are applied on a regular schedule. If it doesn’t work immediately, wait!

Worth also noting, we struggled with the signer initially until we ran it in async – turns out that when you use role based authentication, you need to run the signer in async mode.

Note: You must ensure that you have static or previously resolved credentials if you call this method synchronously (with no callback), otherwise it may not properly sign the request. If you cannot guarantee this (you are using an asynchronous credential provider, i.e., EC2 IAM roles), you should always call this method with an asynchronous callback.

Reference

Our original stackoverflow post on the subject too.