It is time for some new final tuning of my small certificate signing service. In previous parts, I showed you what AWS Lambda service is and how to import simple Python application into serverless microservice. I also connected Lambda function to S3 storage service where I put certificates and key files. Then I added a trigger to the function, so Lambda function will execute automatically every time someone uploads new CSR file with certificate request to S3 bucket. Now I will show you not only how to make this function serverless but also storageless using API Gateway. It is not standard approach but in some scenarios might be interesting. So we will connect API Gateway and Lambda without S3 backend for keys and certificates.

Create new API Gateway

In the previous post How to display HTML page using only AWS API Gateway, I described what API Gateway is showed you how you could use it to display static HTML pages. That is not common usage but worth to know you can do that. We will use that concept in this example – API gateway will be responsible for providing the frontend webpage with the form where a user will paste CSR file content. Clicking the Submit button will invoke Lambda function, and upon the success, we will display signed certificate on the same web page.

For this example, we need to create new API. On API Gateway dashboard click Create API button to start the creator. In the first step, you need to specify a name of an API and optionally some description. Also, select New API option because we need it to be blank. Then click another Create API button.

Our API can have multiple resources with methods or we can place them directly in the root. Let’s do it nicely and create new resource sign by selecting New Resource from Action menu button.

 

Add static web form to API Gateway

Now define the GET method, change the code as described in the article How to display HTML page using only AWS API Gateway so only allowed return type be text/html and provide the code of the web page which I made available on GitHub. Then deploy the API in prod stage.

Let’s take a quick look at the code. It contains JavaScript function and pure HTML form code. The function is called when user click Send button. It does three things:

• Read value provided in the <textinput> area and store it as a JSON pair using the description as the key assigned to the read value. We use JavaScript JSON.stringify() function for the conversion. As its input, we provide variable data where we manually define keys and assigned the values from form fields.
• Below the form there is <div> section named post-result. Its content will be updated if the function is executed correctly. I will describe data parsing in another paragraph.
• Parameters to execute POST method on defined API. We provide API URL as URL variable. You already have this URL from GET method deployment. We also set values of default HTML headers like method type, content-type, etc. and add additional header field named x-api-key – replace the string in code available on GitHub to your key. We will create it soon.
• If defined POST method is not executed proper error message will appear as dialog box.

We used JQuery template to submit the form. Why can’t we just call the method directly from the pure HTML code? There are two reasons. Primary we want to receive JSON structure because it is easier to process in AWS and pass to Lambda function. It is not required but simplifies future coding. A secondary reason is that our POST method will require some form of authentication. So we need to pass the API key in the x-api-key header of HTTP request. You cannot modify this header from HTML code and JQuery is one of the simplest ways I know. That key is the basic security perimeter for invoking our Lambda function. You may create POST method without authentication according to documentation, but it is insecure, might cause high bill if someone keeps executing your Lambda this way and to be honest it did not work for me. Let’s stick to more secure way.

If our configuration is proper, we should see following simple web form

 

Remember this way to create form and provide API key is not entirely secure as the key is visible in the source of the returned web page, but it will fit our example. Also, I found out that if I do not use default IAM policy and role for API Gateway and Lambda the API key is required even if it is disabled in method configuration. Honestly, I do not know if this is the bug or the feature.

 

Create POST method for API Gateway

Before we follow with POST method definition, let’s create empty Python2.7 Lambda function and name it CertSigningAPIGW. This functions should just return the event variable. Also, don’t assign any triggers.

def lambda_handler(event, context):
 # TODO implement
 return event

When Lambda function is ready, we need to get back to our API Gateway configuration and define POST method under our sing resource. Our POST method must have Integration type set to Lambda Function. We also need to provide region name where our Lambda is defined and function name.

We had to create the function before method configuration. If we try to connect method to the function that does not exist in selected region, the configuration will not be accepted, and an error message will appear on API console. The creator is helpful because it will list functions available in selected region when we move the cursor to text input field.

When we click Save button, the wizard will ask us if we want to give API Gateway permission to invoke your Lambda function and will display its ARN. If we do not manage any specific policies just accept it by clicking OK, otherwise you need to create a proper configuration using IAM console.

One more thing you should do it enabling the requirement to provide the API key to each POST method call. By default it is not required but as I mentioned before API Gateway may need it in some circumstances so better enable it in configuration. The option we need to edit we will wind Method Execution properties under the name of API key required. Change default value false to true there and accept clicking on check icon next to it.

We can now deploy updated configuration to prod stage and let’s give it a try.

Let’s try if our API and its integration with Lambda is working. First, select our method from Resource group and click the test icon with a thunderbolt in Customer object. This way we can test the method call internally even before we deploy API to any stage. For POST we can provide the request body content, put there just “false” string and press Test button. When API complete the call then on right side of the pane, you will see three fields – response header and response body that API create using our configuration and return value from Lambda function. The third text box will have logs from method execution. Remember that right now our Lambda function returns event variable. When you use API Gateway with no modification this variable contains a copy of POST method body, in our case, it is just string “false.” Response header field provides information returned by API – you can see there default values as we did not change them. Logs text box gives us information about method execution but not about the Lambda function – we need to debug it separately using CloudWatch or X-Ray.

Let’s now try to execute POST method externally using curl or Postman or REST API plug-in to your browser or any other tool you prefer. I am recently working mostly on Postman, but in the past, I preferred browser plugin more often. I provide the same body content as previously, the URL and select POST method. The response we received is not what we expected

We received 403 error code which says that we are not authorized to execute the method. Remember that response codes are defined in HTTP standard. What we are missing here is API key. You may ask why we did not have to provide it when we tested method internally. The reason is simple – we were testing resource configuration, not the full API call itself.

 

Create API key

To authenticate the client calling the method, we will provide API key in HTTP header. The key must be recorded using header field named x-api-key but first, we need to create the key itself. To do this go to API Gateway dashboard and select API Keys from the list on the left side. Each key has its name (it is used internally only to differ the keys), the value which has to be at least 20 characters long and optionally the description. To create new key select Create API key from Action button menu, provide the name and leave API key option to Auto Generate. When you click Save the new key will appear on the list. Now you can display its properties and the key value itself in API key field – it is hidden by default, so you need to click Show link.

You must assign the key to Usage Plan. The plan defines thresholds of each API usage. If you are afraid that someone may generate unwanted cost by invoking an open method or you just want to set limits, this is the place.

Select Usage Plan from the left side menu and then click Create button. Then assign a name to your plan and set very low throttling and quota levels. Of course, you can disable those limits, but that is not the recommended approach. After you create the plan you must define which APIs and which stages of those APIs will use it. Also in API Key tab assign previously created key to this data plan. You can edit the configuration of Usage Plan any time but remember that it created the connection between your API stage and API key.

We can get back to testing from the external source. The API key we connected to our API Gateway stage have to be provided in HTTP header under name x-api-key (it is safer to use lower case letters only). In Postman just click Headers then add a new key and assign value.

If you call POST method now, you will not see the Forbidden error message but output returned by function

Let’s now connect our form to POST method, and display body passed with POST method to Lambda, so we know how to handle it later. First edit the GET response body code defined in Integration Data step of GET method and update URL of the method and API key values if you have not done it yet. Remember to redeploy API at the end.

Open the form again and put any text in the textbox and click Send button. You should now see the dialog window that form has been correctly sent to API method. However, where is the response? It was sent back to the browser, but our website doesn’t know yet how to process it. So there are two ways to see the reply returned by our POST method. First one is to use developer tool built into any browser. You can find the response received by the browser. In Chrome it will look like this

So we received JSON structure with just one element where the key is ‘description’ and its value is the text we put in form text area. In Headers tab we can check all HTTP headers that were used by each side when we called the GET and POST methods. Alternatively, we can enable CloudWatch logging for the API and check the logs there. Remember – this is the JSON that our Lambda will receive from Gateway API in event variable.

 

Capture POST response

We know that we receive the response from API Gateway when we call the POST method. We now need to update our code to process it in a web browser. As with every problem, it can be solved in multiple ways, but I will pick up the easiest one. We already use JQuery library in our form for parsing the input, converting it to JSON and adding HTTP headers required for API call. In function submitToAPI() which we execute by clicking the Submit button, we already have a piece of the code. The code is assigned to “success” event:

 success: function () {
 // clear form and show a success message
 alert('CSR has been sent for signing');
 },

We just display a dialog window with the message. Let’s modify the code a little:

  success: function (response) {
 // clear form and show a success message
 alert('Returned body from API: ' + JSON.stringify(response));
 },

Now we refer to the variable called response from JQuery library. We know it is JSON object so we can use stringify() method to display its content.

Let’s modify the code a little more and put the value of description key into text the area below the form. In the HTML code returned by GET method I’ll add horizontal line and <div> field below the <form> section.

<hr>
<div id='post-result'></div>

We will update this object if POST method is executed properly. To do this we need to append JavaScript code a little

 success: function (response) {
 $('#post-result').append(JSON.stringify(response));
 },

 

Remove S3 dependency from Lambda function

Our API Gateway is almost ready so time to change Lambda function a little. I will use code from AWS Lambda guide part III – Adding S3 trigger in Lambda function where the S3 event was a trigger to invoke the function. The goal is not to use S3 service anymore.

The certificate request is provided by the user in the form and sent to API Gateway using POST method. The script in form converts the input into JSON structure. API Gateway then passes it to Lambda as event variable. All we need to do is read the content from the specific key in JSON.

try:
    crt_req = crypto.load_certificate_request(crypto.FILETYPE_PEM, event['description'])
except ClientError as e:
    print(e.message)

Next are the signing certificate and key files stored on S3. Every problem can be solved in multiple ways. Let’s look at few possibilities how we can pass the content of those two files to Lambda function:

• Use other storage services – that’s probably what most of the readers thought first. Just move the files somewhere else. It can be on-premise service that you can reach over the Internet, it can be EC2 instance allowing access via HTTPS, storage service from Azure or other cloud providers. Of course, we won’t use S3 anymore but will still be dependent on another storage service.
• Provide them as Lambda parameters from API – you can pass additional variables to Lambda from API Gateway. How to do this is nicely described in this AWS Documentation section. If data is sensitive additional protection can be added as described here. The only limit we may face is that variables cannot exceed 4KB in size in total. I didn’t test that approach but looks like it’s doable.
• Store key and signing certificate on the ephemeral drive used by Lambda. So just put it in the .zip file and access from Python code like from local drive. Each Lambda function receives 500MB of non-persistent disk space in its own /tmp directory. We move a little further and will just store them with source code.

To make last option working add the signing-ca.crt and root-ca.key files to root folder of ZIP archive for Lambda. Remember that we keep Python source file in the root of the archive. We need to modify the code a little – instead of using S3 service we just read the file from a local folder as an I/O operation.

try:
    ca_cert_file = open('signing-ca.crt', 'r')
    ca_cert = crypto.load_certificate(crypto.FILETYPE_PEM, ca_cert_file.read())
    ca_cert_file.close()
except ClientError as e:
    print(e.message)

try:
    ca_key_file = open('root-ca.key', 'r')
    ca_key = crypto.load_privatekey(crypto.FILETYPE_PEM, ca_key_file.read(), b'test')
    ca_key_file.close()
except ClientError as e:
    print(e.message)

 

Return signed certificate in Lambda function

Last modification we need to add is what data our Lambda function will return back to API Gateway. It has to be the JSON so we can process it later using JavaScript on our page. Let’s return JSON with two keys. First one will be crt_text so we can see the certificate parameters as text, second will be crt_pem where we put PEM type of signed certificate. We are not doing complicate operations on JSON structure so we will use json library that is available by default on Lambda.

import json

If we want to create JSON object first we need to store values in a variable of Dictionary type. We declare it this way

CRTOutput = {}

The Dictionary variable consists of key-value pairs. Almost the same as in JSON. The names we use for keys in the variable will be used as keys in JSON. We need to dump the content of the crypto object and assign the value to Dictionary variable.

msgCertificateParameters = crypto.dump_certificate(crypto.FILETYPE_TEXT, new_crt)
CRTOutput['crt_text'] = msgCertificateParameters.decode('utf-8')
msgCertificateParameters = crypto.dump_certificate(crypto.FILETYPE_PEM, new_crt)
CRTOutput['crt_pem'] = msgCertificateParameters.decode('utf-8')

In previous chapters, main Python function wrote the output into Lambda log window and in also put it as a new S3 object in our bucket. Our main function returned no value. Now we will return JSON object created via dump() function from json library using Dictionary variable as input

return json.dumps(CRTOutput)

 

Parse the Lambda output and display via API Gateway

Lambda function returned JSON object to API Gateway. We cannot display it raw but parse the values assigned to the keys. The response is provided to our JQuery code in the response variable. We need to first call the parseJSON function. It takes a well-formed JSON string and returns the resulting JavaScript value. We can then easily access each key using the key name. The <div id=’post-result’> container is already defined but is empty. We can easily update it using append() function from the standard JQuery library.

 success: function (response) {
 response = $.parseJSON(response)
 $('#post-result').append(response.crt_text + "<br><br>");
 $('#post-result').append(response.crt_pem + "<br><br>");
 },

 

We have all pieces in place so connect via web browser to created API and paste whole content of CRS files into the text area field and click Send button. If everything works as expected, you will see the signed certificate on screen first as user readable description and then in PEM format.

If you experience problems you should start troubleshooting from checking the CloudWatch logs (remember to enable logging first for your API and Lambda function) and by using Postman application to check what data is exchanged between the client and the gateway.

This code does not contain extensive exception handling. Things, like checking the input from the web form or displaying error message based on error type, should be implemented in any application that is not meant for demo purposes.