Basic Serverless Applications- Python Back End
Prototype Serverless Application- The Back End (Python Version)
After completion of the front end and environment portions of the application, web pages are loadable and there is a user pool that can be authenticated against, but it is not an application yet. For this prototype, a database, and the ability to access it is required. This requires three major services working together:
1. A database to store information in tables and process SQL statements (Amazon RDS in this example).
2. The ability to execute code on the server side against this database (Amazon Lambda).
3. A means to access the functions from the client-side web applications (AWS API Gateway).
These capabilities implement the middle portion of the architecture as shown below:
1. Security- determines what privileges it takes (either configured IAM security roles or user access privileges from Cognito) to connect to the API and for each service to work with each other. Security is never fun to work with (it either works or it does not with no clue as to what it does not like typically but this is necessary to avoid security breaches).
2. Services- these are the enablers to the back end that you must configure, but you do not have to code or install the functionality on a server that you own.
3. Application- this is the end goal of the prototype, to deliver the application functionality that you want.
The security portion has three checks:
1. User Privileges Check- is the API open to the public where anyone with the URL could run it or do you want to restrict it to people who have logged into a specific Cognito user pool?
2. Network and API Role Access Check- does the API (or other services) have the correct privileges and connectivity (network connection) to access the serverless function?
3. Database Role Access Check- does the serverless function have the correct privileges to access a particular database?
There are 3 services involved in this back end:
1. API Access (API Gateway)- this allows you to access the API and functions that it calls. In this prototype an HTTPS request to a specific URL is used to pass parameters to the API which then calls the serverless function and returns the result in JSON format.
2. Serverless Function (Lambda)- this software (Python for this prototype) takes the parameters passed in through the API gateway and uses them to connect to the appropriate database and execute SQL statements.
3. Serverless Database (RDS)- the database allows you to store information and retrieve it using the API and serverless functions.
There are 2 pieces of the application that are implemented in the back end of this prototype:
1. Application Serverless Function- this is the custom software that allows you to ask for information from a specific database and return the results in JSON format to the requestor.
Download from: https://jbgreene.the-greenes.com/uploads/protolambda-python.zip
2. Application Database Objects- this is a simple table that stores the information that you want to access.
Note- vendors may change their screens around over time as they update their products so you may have to adapt to changes in on-screen names and flows to get the job done.
Note- most of the unique application functionality is on the client side versus server to take advantage of large numbers, power and minimize network and server load.
Serverless Database (RDS, step 8)
Starting at the base of the stack, the database provides a relational data store to hold, manipulate and retrieve information for the application. The choice for this prototype is the RDS product, specifically using the MySQL compatibility option to select data from a table by running a SQL statement. The application is built to store all the information needed to make the connection and execute a SQL statement in the config.js file.
The first step in configuring the back end is to create the database instance to hold the database objects:
1. From the AWS Management Console, select the RDS product link.
2. Click on the Create Database button in the middle of the screen.
3. On the create database page:
a. Choose the Standard create tile for Database Creation Method.
b. Choose Amazon Aurora for Engine Option.
c. For Edition, choose the Amazon Aurora with MySQL compatibility radio button.
d. Choose Serverless for Capacity Type.
e. Choose MySQL 5.7 for Version.
f. Enter a name such as pCluster for DB Cluster Identifier.
g. Enter admin for Master User Name.
h. Enter (and re-enter) a password of at least 8 characters.
i. Accept default Capacity Settings (this is just a prototype).
j. Under Connectivity:
i. Select Default VPC in the drop-down list and note it on the settings sheet.
ii. Select the default for the Subnet Group and note it on the settings sheet.
iii. For VPC security group, select existing one if you have one else select create new one and enter a name for that group (i.e., pDbSec).
iv. Expand the Additional Configuration section and click in the radio box marked Data API (this will let you use the built-in Query Editor to create database objects).
k. Accept defaults for the remaining parameters (again, this is just a prototype and these parameters can be useful for real applications at times).
l. Click on the Create Database button at the bottom (it takes perhaps a few minutes to create).
m. Refresh the Databases page until the status is available for the database cluster just created.
n. Click on the link for the cluster created under the DB Identifier column.
o. Note the endpoint (host) and port for the database created.
Application Database Objects (step 9)
This configures and creates objects in the environment that are used by the prototype web application via the APIs.
1. From the AWS Management Console, select the RDS product link.
2. On the resources screen, select the DB Clusters link.
3. Select the pCluster link.
4. On the left menu, select Query Editor.
a. Select the pCluster link from the Database Instance or Cluster drop down list.
b. Select Add New credentials if the admin user name is not already in the drop down list.
c. Select the admin user name.
d. Enter the admin user password.
e. Click on the Connect to Database button.
f. Show list of databases by typing the following command and clicking on the Run button:
show databases
g. Create a database (container within the cluster for objects) by running the following command and clicking on the Run button (note you will not get any rows of data back, instead you look in the output section at the bottom of the page for a Status of Success or any error messages to know if the command worked):
create database protodb
h. Type the following command to make the protodb the default database to work with and click the Run button:
use protodb
i. Type the following to show existing tables in protodb and click the Run button:
show tables in protodb
j. Type the following to create the prototab table and click the Run button:
create table protodb.prototab
(protokey int not null auto_increment primary key,
proto1 varchar(100),
proto2 varchar(100))
k. Type the following to show existing tables in protodb which should now include prototab and click the Run button:
show tables in protodb
l. To verify the columns in prototab, enter the following and click the Run button:
show columns from protodb.prototab
m. Insert a row of data into the prototab table that can be queried by the application:
insert into protodb.prototab
(proto1,proto2)
values ('first','row of data')
n. Insert another row of data into the prototab table that can be queried by the application:
insert into protodb.prototab
(proto1,proto2)
values ('second','row of data')
o. Now verify that the data that you put in looks correct by running the following command and clicking the Run button (note you will see the protokey column which is just a unique reference for the row):
select * from protodb.prototab
p. Create a user and give it access to this content by running the following commands and clicking the Run button:
create user ‘testuser’ identified by ‘testpw’
grant all on protodb.* to testuser
q. To test the new user:
i. Click the change database button
ii. For database instance or cluster, select pCluster
iii. Select Add New Database Credentials for DB Username drop down.
iv. Enter testuser for user name
v. Enter testpw for password
vi. Click the Connect to Database button.
vii. Run the following command to verify you can execute queries:
Select * from protodb.prototab
Database Role Access Check (IAM, step 2b)
The data for an application typically requires security to keep out those who are unworthy. In the AWS world, Lambda applications need an IAM role with permissions to access the database. For this application, you will set up full privileges for database access for the Lambda function although the role will show some of the lesser privileges that you might want to consider for your applications.
Set up the IAM role to Access RDS from Lambda
1. From the AWS Management Console, select the IAM product link to get the IAM Management Console.
2. Select Roles from the menu on the left.
3. Click on the Create Role button.
4. Select the AWS Service tile at the top of the screen.
5. Under Use Case, click on the Lambda link and click on the Next: Permissions button.
6. Under the Filter Policies search field, enter the following policy names and then click the check box to the left of the policy found to select them (some are duplicates or subsets of others, I just list them to give you a feel of what options might be available if you do a non-prototype application):
a. AmazonRDSDataFullAccess
b. AmazonRDSFullAccess
c. AmazonRDSReadOnlyAccess
d. AmazonVPCFullAccess
e. AWSLambdaVPCAccessExecutionRole
7. Click on the Next: Tags button.
8. Add any tags (just searchable key value pairs if you want to categorize your applications that you want) and click on the Next: Review button.
9. Enter Role Name and any description that you want (i.e., pRdsRole) and click on the Create Role button.
Serverless Function (Lambda, step 10)
An API function is software written to accomplish a purpose. For this prototype, it is a Python function that takes configuration parameters and executes SQL that you provide.
To configure the serverless function for this prototype:
1. From the AWS Management Console, select the Lambda product link to get the Lambda Management Console.
2. Select Functions on the left menu.
3. Click the Create Function button.
a. Choose the Author From Scratch tile.
b. Under Basic Information, enter function name (i.e., pFunction).
c. From the Runtime drop-down list, select Python 3.8.
d. Under Permissions, choose Use Existing Role.
e. For Existing Role, select the role created under IAM (i.e., pRdsRole).
f. Under Advanced Settings, choose the VPC selected for the database earlier.
g. Choose all Subnets associated with the database created earlier.
h. For Security Groups, choose the security group used for the database.
i. Click the Create Function button.
j. Wait until the Creating the Function banner at the top indicates that it is completed.
k. You now have an empty function.
Application Serverless Function (step 11)
For the client side to interact with services or content in the serverless environment, software is needed. The Lambda service is set up to run Python applications that perform tasks such as execute queries against the database. To load and configure the prototype Lambda function:
1. Select Functions on the left menu.
2. Select the pFunction function.
3. Select Upload a Zip File under the Actions drop-down button.
a. Click the Upload button.
b. Find the zip file downloaded for the Lambda function in your file system (protolambda_python.zip).
c. Click the Save button.
4. Click the test button at the top.
5. Choose Create New Test Event.
a. Enter an Event Name (i.e., pTest).
b. There is no need to configure test parameters as the prototype application relies on URL query parameters.
c. Click on the Create button.
d. Select the Test Event and run a test. It will have errors saying no query string. This is OK since the function is designed to run from the API Gateway.
API Access (API Gateway, step 12a)
The function engine (Lambda) is designed to execute pieces of software. The next piece is the puzzle is a way for you to access that software, in this case it is an Application Programming Interface Gateway. It could be a job scheduling tool or even another function. For this API, the goal is to construct a way for a web HTTPS request to cause the function to execute and return HTML/JSON to the client for display and processing.
To set up the API Gateway linked to Lambda function:
1. From the AWS Management Console, select the API Gateway product link to get the API Gateway Management Console.
2. Click on the Create API button.
3. Under the Rest API section, click on the Build button.
4. Click OK on the welcome popup if this is the first time.
5. Select the New API radio button.
a. Enter an API name (i.e., pApi).
b. Enter a Description if you like.
c. For Endpoint Type, choose Regional (note- you have 600 available for basic accounts).
d. Click the Create API button.
6. On the Actions drop down button in the middle, select Create Method.
a. Choose the GET option and click the check mark next to it.
b. Select the Lambda Function radio button for integration type.
c. Select your region for the Lambda function.
d. Enter/select your Lambda function name (pFunction).
e. Click the Save button.
f. Click OK to add the permission for the API Gateway to execute the Lambda function.
g. Select Resources on the menu on the left.
h. Click on the Actions drop down button and select Enable CORS.
i. Click on the Enable CORS and Replace Existing CORS headers. Confirm by clicking the Yes, Replace Existing Values button.
i. To add the ability to pass URL query parameters through the API Gateway to the Lambda function, set up resource mapping for the API.
i. Click on the Resources link on the left side menu
ii. Click on the GET method link.
iii. Click on the Integration Request link.
iv. Open the Mapping Templates section.
v. Select the When There are No Templates defined radio button.
vi. Click on the Add Mapping Template link.
vii. Enter application/json and click on the check mark under Content Type.
viii. Click on the application/json link.
ix. Choose the Method Request Passthrough option on the Generate Template drop-down list.
x. Click Save.
j. Click on the Actions drop down button and select Deploy API.
i. On the Deployment Stage list box, select New Stage.
ii. Enter a name for the new stage such as Release.
iii. Enter descriptions if desired.
iv. Click on the Deploy button.
k. Click on Stages on the menu on the left and find the invoke URL at the top.
i. Select the Release stage link.
ii. The format of this URL is:
https://<api id>.execute-api.<region>.amazonaws.com/<stage name>
iii. Note the API ID, URL and stage name for later reference.
iv. Click on the URL link to open a new window and test access to the API (should have access to the prototype Lambda function showing no results as you have not passed in the parameters which will occur in the next section).
7. You now have access to the database via the Lambda function and the API Gateway. It is not yet secure yet but can be tested with the prototype web pages.
8. To test query parameter passing, added the following to the URL:
?db_host=hostname&db_port=port&db_user=user&db_password=pw&db_sql=sql
Network and API Role Access Check (step 2c)
Working with the theme of requiring permission to access services within the back end applications, the next role needed is the security role for the API Gateway. I have included this as a reminder although you already gave permission for the prototype API to access the prototype Lambda function when you created the API.
Test API Access (Before Cognito Authentication)
To test the back end functionality, you must configure the Amplify and database settings into the config.js file for this prototype web application. I would not recommend putting database password information into a client-side file normally, but for this application the goal of the settings page is to show what configuration information is needed to make it work. Lambda allows you to read configuration files on the server or even configure parameters into its setup. To update the settings to work with this prototype for the back end:
1. Open the config.js file from earlier in Visual Studio or a text editor.
2. Edit the file to insert the settings for Amplify and Cognito noted earlier in this article for your prototype (see Parameters/Settings to Remember table at the end of this article):
a. apiClientId
b. apiBaseUrl
c. apiStage
d. dbHost
e. dbPort
f. dbUser
g. dbPass
3. Save your changes to config.js.
4. Using the tools on your workstation, zip all the files (not just config.js, be sure to include the uploads directory) into a new zip file (perhaps call it protoweb2.zip). I used the Windows File Explorer, highlighted the files, right clicked on Send To and then Compressed (Zipped) Folder option. The other zip utilities that I tried under windows did not handle sub-directories in a way that worked for Amplify.
5. Go to the Amplify Management Console URL (which is on the AWS Links tab of the prototype application or can be accede from the AWS Management Console).
6. Select the tile for this prototype application.
7. Select the Choose Files button near the bottom of the page.
8. Select the zip file that you just created and click the open button (titles for that button may vary based on your operating system and web browser combination).
9. AWS should deploy the new web pages and give you the Deployment Successfully Completed message.
To test, you must load the application web page in a new browser window to get the updated parameters:
1. Close out all browser windows and then open a new web browser window.
2. Go to the prototype web site.
3. Go to the API calls tab.
4. You will now see the API and database configuration parameters.
5. Click on the API in Window (see the JSON).
6. Click on the Call API Non-Secure button and see JSON and parsed output.
Cognito Access Authorization (Cognito Resource Server, step 7b)
At this point, the prototype API is available to anyone with a web browser. To complete the security model, authorization is required to ensure that anyone accessing the API has successfully logged into the appropriate Cognito User Pool that was previously created. The first step is to create a server (connection point) in Cognito to respond to requests to validate access tokens authorizing access.
Create a Resource Server for the Cognito User Pool:
1. From the AWS Management Console, select Cognito to access the Cognito Management console.
2. Select the Manage User Pools button.
3. Select the user pool for this prototype (i.e., pPool).
4. Click on the Resource Servers link on the left menu.
5. Click on the Add a Resource Server link.
a. Enter a name for the resource server (i.e., pServer).
b. Enter an Identifier (i.e., pId).
c. Enter a Scope Name (i.e., pScope).
d. Enter a description that makes sense to you.
e. Click on the Save Changes button.
Configure the Cognito User Pool to use the Custom Scope:
1. From the AWS Management Console, select the Cognito to access the Cognito Management Console.
2. Select the Manage User Pools button.
3. Select the Cognito User Pool created for this prototype.
4. Select App Client Settings on the left menu under App Integration.
5. Click the check box for the custom scope for this prototype for the resource server created.
6. Click the Save Changes button.
User Privileges Check (API Gateway Authorizer, step 12b)
The next step in the security process for the API is to configure the API to check with the Cognos server before it grants access to the API. This is a two-step process- create an Authorizer within the API Gateway and then configuring the API’s request to require it.
Create an Authorizer in API Gateway:
1. From the AWS Management Console, select the API Gateway to get the API Gateway Console.
2. Select the API That you created for this prototype.
3. Select Authorizers from menu on the left click the Create New Authorizer button.
4. Enter a name for the authorizer (i.e., pAuth).
5. Select the Cognito radio button under Type.
6. Select your region under Cognito User Pool and select an available region on the drop-down list.
7. Select the Cognito User Pool for this prototype.
8. For Token Source enter the word Authorization.
9. Click the Create link.
10. Note: the test button will not work if you choose to use access tokens for Cognito.
11. Exit all browser windows and open a new browser window.
Configure API Gateway to use Cognito Scope:
1. From the AWS Management Console, select the API Gateway to get the API Gateway Console.
2. Select your API created for this prototype.
3. Select Resources from the menu on the left.
4. Under the Resources column, select the GET resource.
5. Click on the Method Request panel, click on the Method Request link.
a. Click on the pencil icon next to Authorization.
b. On the Authorization drop down, select the Authorizer created for the Amazon Cognito User Pool.
c. Save settings with the check mark icon for the Authorization.
d. Click on the pencil next to OAuth Scopes.
e. Type in the scope and enter the scope full name configured (i.e., pId/pScope).
f. Click the check mark icon next to OAuth Scopes to save the changes.
g. On the Action drop-down, select Enable CORS.
h. On the Action drop-down, select Deploy API.
i. Select the Deployment Stage and then click on the Deploy button.
j. Wait a couple of minutes for the API to deploy completely.
Note: At runtime, if the scope specified matches the scope of the incoming token then the call succeeds. Otherwise, you will just get a 401 Unauthorized response.
To test the API call using the new authorizer scope, obtain an access token by logging into Cognito and use a URL with the following format:
https://<api id>.execute-api.<region>.amazonaws.com/<stage name>
Note the access token has to be passed in the header of the request which is done using the xmlhttp.setRequestHeader Javascript function.
Testing the Full Prototype
To test, you must load the application web page in a new browser window to get the updated parameters:
1. Close out all browser windows and go back to the URL in a new browser window.
2. Go to the prototype web site.
3. Log into the prototype.
4. Go to the API calls tab.
5. Click on the Call API Non-Secure button and you will get an error as it now needs the Cognito token for access.
6. Click on the Call Secure API button and see JSON and parsed output.
Settings to Keep Track Of
When working in a hosted and serverless environment, you are not writing code to do all of the functions and instead rely on services provided by the vendor. As such, much of the work is configuring these services where you must keep track of configuration settings in one service so that you can connect another service to it. Here is a list of configuration settings to note as you go through this process:
1. API Client ID: __________
2. API Base URL: __________
3. API Stage: __________
4. DB Host: __________
5. DB Port: __________
6. DB User: __________
7. DB Password: __________
Comments
Post a Comment