AWS EC2 Scheduler: On-demand

Consider a situation where you have a fleet of EC2 instances and you want to schedule their start and stop automatically for cost saving. For example — you want to start a group of instances everyday at 9:00 am in the morning and stop at 6:00 pm in the evening. So, you want a scheduler kind of thing for your EC2 instances which will do the job.

When you need this kind of scheduler for different groups of EC2 instances with different start/stop timing settings then you would need different schedulers. In this article I will discuss how we can achieve this by using Amazon CloudWatch and AWS Lambda.

---

You can also read this article on Medium.

Solution Overview

There could be different approaches to overcome this challenge. AWS has it’s own solution AWS Instance Scheduler which can automate start-stop of EC2 as well as RDS instances.

In this article I will discuss about a much simpler version of the solution by leveraging Amazon CloudWatch Events and AWS Lambda functions.

Solution Description

In CloudWatch Events, we can create Rules that trigger based on an automated schedule using cron expressions. For each Scheduler (for one set of EC2 instances) we will create two scheduled rules — one to start and the other to stop the EC2 instances. These rules can invoke AWS Lambda functions as target.

So, we need two Lambda functions which will perform the job of starting and stopping the EC2 instances respectively.

So, conceptually a Scheduler will be a logical entity which will contain two separate CloudWatch Events Rules. Every time somebody wants to have a new Scheduler, we have to create two such rules each pointing to the Lambda functions (back-end functions) for starting and stopping the EC2 instances. We can automate the process of creating these rules by another Lambda function (front-end function).

Solution Implementation

All the source code and configuration details, discussed here, are available at the Github repository. You can use the CloudFormation template for one-click-deployment of all the resources.

Note

STEP 1: Create back-end Lambda functions for performing core tasks such as starting and stopping EC2 instances

I’ve used Python for creating the Lambda functions. You can use any other supported programming languages.

I’ve created two Lambda functions start-ec2-instances and stop-ec2-instances.

STEP 2: Associate required IAM Roles to back-end Lambda functions

The back-end Lambda functions require permissions to perform actions on EC2 instances. We need IAM Roles with appropriate Permissions for these Lambda functions.

I’ve created two IAM Roles — lambda-ec2-start-instances & lambda-ec2-stop-instances and attached these roles to the two back-end Lambda functions respectively.

These roles require appropriate permissions. I created two IAM Policies — ec2-start-instances & ec2-stop-instances and attached to the two IAM Roles respectively.

STEP 3: Attach common permissions to IAM Roles associated to back-end Lambda functions

The Lambda functions also require CloudWatch Logs permission to generate logs. I’ve created a policy cloudwatch-log-message and attached to the two IAM Roles.

Optional: At this point, we have successfully created two back-end Lambda functions for performing core tasks — starting and stopping EC2 instances. We can perform some testing to make sure these work.

To do that, invoke each of the Lambda functions separately with parameters as mentioned in start-stop-ec2-instances.json.

Your EC2 instances will be started or stopped based on which Lambda function you invoked.

STEP 4: Create front-end Lambda functions for managing CloudWatch Events Rules

Now that we have successfully created and tested the back-end Lambda functions, let’s now create the front-end Lambda functions which will perform operations such as creating a scheduler, deleting a scheduler etc.

As I already explained, a Scheduler is a logical entity with two CloudWatch Events Rules which invoke back-end Lambda functions as targets.

So, let’s first create the Lambda function create-ec2-scheduler for creating a Scheduler.

Additional Operations: There are some other operations that can be performed on a Scheduler such as deletion, enable/disable of a Scheduler. I have created three more Lambda functions which are delete-ec2-scheduler, enable-ec2-scheduler and disable-ec2-scheduler.

STEP 5: Associate required IAM Roles to front-end Lambda functions

Similar to STEP 2, we need to create IAM Roles with appropriate permissions and attach to the front-end Lambda functions.

I’ve created four IAM Roles — lambda-ec2-create-scheduler, lambda-ec2-delete-scheduler, lambda-ec2-enable-scheduler & lambda-ec2-disable-scheduler and attached these roles to the four front-end Lambda functions respectively.

These roles require appropriate permissions. I’ve created four IAM Policies — eventbridge-create-rule, eventbridge-delete-rule, eventbridge-enable-rule & eventbridge-disable-rule and attached to the four IAM Roles respectively.

Attach Additional Permissions: The delete-ec2-scheduler function also requires permissions to remove targets from CloudWatch Events Rules. So, I’ve created eventbridge-remove-targets policy and attached to the lambda-ec2-delete-scheduler role.

STEP 6: Attach common permissions to IAM Roles associated to back-end Lambda functions

Lambda functions also require CloudWatch Logs permission to generate logs. Similar to STEP 3, I’ve attached the policy cloudwatch-log-message to the four IAM Roles.

STEP 7: Associate required permissions to back-end functions

At the end of STEP 3, we tested that the back-end Lambda functions are working. However, that is only when the identity invoking the Lambda functions has required permission to invoke the functions.

In our solution, the back-end Lambda functions will be invoked by CloudWatch Events Rules. So, we need to associate right permission policies to the back-end functions so that CloudWatch Events Rules can invoke those functions.

Execute the commands add-permission-to-start-ec2-instances.cmd and add-permission-to-stop-ec2-instances.cmd to add required permission to start-ec2-instances and stop-ec2-instances functions respectively.

STEP 8: Perform Final Testing

Now, we’ve completed the solution implementation and it’s time to perform the final testing.

Create a Scheduler: Invoke create-ec2-scheduler function with parameters as mentioned in create-ec2-scheduler.json to create a Scheduler named my-scheduler.

You’ll find two CloudWatch Events Rules — my-scheduler-start and my-scheduler-stop got created.

These rules point to the corresponding back-end Lambda functions as targets. You can check that by going to the details page of any of these rules.

The Cron expression times are in UTC format. So, convert your local time to UTC and use it for creating Scheduler.

Note

Disable a Scheduler: Invoke disable-ec2-scheduler function with parameters as mentioned in enable-disable-delete-ec2-scheduler.json to disable my-scheduler. You’ll find both the rules my-scheduler-start and my-scheduler-stop are now disabled.

Enable a Scheduler: Invoke enable-ec2-scheduler function with the same parameters as mentioned above to enable my-scheduler. Both the rules will be enabled now.

Delete a Scheduler: Similarly, invoke delete-ec2-scheduler function with the same parameters to delete my-scheduler which in-tun will delete both the rules.

One-Click Solution Deployment

I’ve put all these inside a AWS CloudFormation template so that all the resources (AWS Lambda functions, IAM Roles, IAM Policies etc.) can be provisioned in a single click. This can save significant time & resources and might interest you if you are lazy 😜 !!!

Important Notes

The create-ec2-scheduler function internally calls AWS EventBridge service’s PutRule API method. By definition, PutRule method creates or updates the specified rule. So, we don’t need a separate Lambda function for updating an existing Scheduler.

For example — you want to update the start and stop timing or the set of instances an existing scheduler is currently targeting to. In such cases, you can invoke the same create-ec2-scheduler function with new parameter values. Your existing scheduler will be updated accordingly.

Bottom Line

As I already mentioned there could be different approaches to address the problem, I just wanted to make it simple and kept each and every components separate based on their purposes.

One approach could be to combine multiple components into a single one. For example — let’s consider the back-end functions where we can combine start-ec2-instances and stop-ec2-instances into a single function say, start-stop-ec2-instances. In that case, the function will require another parameter to define the start/stop action. 

This will also reduce the no. of roles and policies used. Because instead of lambda-ec2-start-instances and lambda-ec2-stop-instances now we can have a combined role say, lambda-ec2-start-stop-instances which will get attached to the function start-stop-ec2-instances. 

As a result, instead of two policies — ec2-start-instances and ec2-stop-instances now we can have a combined policy say, ec2-start-stop-instances attached to the role lambda-ec2-start-stop-instances. And this process continues for front-end functions as well.

Another Solution — EC2 Scheduler: Central

If you are looking for a scheduler solution which is centralise in nature, and you want to manage your EC2 instances’ start/stop scheduling just by simple tagging, then read this article where I explained the solution in details.

---

Do you have any thoughts? Feel free to share in the comments below.