Automation is the end goal for many software developers. It's the ultimate work-smarter-not-harder adage applied.

Because why not?

The purpose of automation is to free up your time and brainpower. It's supposed to give you space to do something else, no need for manual work and repetitive tasks. But while there's a general advocacy for automation in software and engineering, it's not always the best solution.

In my past projects, sometimes automation did not weigh up against its purported benefits, resulting in unnecessary time sinks. Over the years of implementing successful and failed automations, I've developed an assessment matrix to help my teams determine if automation is necessary.

Just because it sounds good doesn't mean that it's the right thing to do in every situation.

What's the time profit?

Before we jump into the task of automating, we need to assess our time profit.

Why is this important?

Because not all automations are worth the same. It's also easy to lose yourself in the task of creating the automations. Time is a commodity in dev work. Sometimes, making automations takes up more time than it gives back.

Here's the equation I throw at my teams before they pitch the task:

(task time * duration without automation) - (time to automate * resource) = time profit

Here's how the equation works:

• Task time: How long does the repeated task take? Ten minutes? Twenty-five minutes? An hour?
• Duration without automation: How long is the task expected to be repeated? Is it indefinitely? Three weeks? Six months? Until marketing finally makes a decision?
• Time to automate: How long will it take to create the automation? Thirty minutes? Three working days? Two working days?
• Resource: How many people does this task require? One developer? Two developers? The entire team?

Your time profit is the result of the above equation. Does the automation create a sizable and impacting surplus? In the event where the manual task is expected to go on indefinitely, the automation is certainly worth it.

However, if it's only expected to be for a week and costs you three days to complete, then it's not that efficient for that particular user case – obviously.

I use this equation, especially with new junior developers, because they often fall into the trap of "automation enthusiasm." While automation is not a bad thing, it also needs to be weighed up against the situation and never taken out of context.

There are also different types of automations, which impacts on the time profit.

The smallest automation jobs can be most efficient

There are several types of automations, starting with infrastructure orchestration and all the way to programmatic processes.

One of the easiest and highest time-profit automations is anything that can be ported to a cron job, a time-based scheduler in Unix-based systems. Cron typically handles manual tasks that need to run at a certain time or interval. These tasks tend to already exist in the form of scripts that a developer needs to run in order to keep business operations going.

For me, a typical cron job setup takes about 15 minutes on the test server, and another 15 to replicate it in production. This is on condition that the automated scripts already exists. I usually also add a buffer to the time it takes to complete the job, mainly used for waiting because things can take time to boot up or give connection access. (That time adds up!)

If we apply the time-profit equation to a task that's expected to occur for the next three months, it looks something like this:

// Time profit equation
(task time * duration without automation) - (time to automate * resource) = time profit
 
 // Our automation in question
 (5 [mins] * 90 [days] ) - (45 [mins] * 1 [dev]) = 405 mins time profit --> or 6 hours and 45 mins

Six hours is almost a day saved. And that's not accounting for the inevitable, incalculable human error, like someone losing track of the task and running it later than expected. When I first joined one of my dev teams, we had to manually run reports every morning. Every now and then, one of the devs on the roster would forget or have a meeting run over. This would often cause the deadline to run over and the report would be delivered late. This caused delays in other departments, and in some cases it trickled down to impact the customer in a negative way. When cron jobs were introduced, it solved the ongoing issue and removed reliance from the dev team to press a button.

Most of the time, you can create a cron job directly on your server. However, the idea of cron jobs can also be extended to cloud providers such as AWS, where you can create scheduled tasks via services such as CloudWatch Events and Lambda.

Let's start with a custom cron job on your server.

Custom cron job on server crash course

1. Log into the server that you want your cron job to run via SSH.
2. Use crontab -e to open your crontab file.
3. It might prompt you for an email address and/or editor. Nano is easiest if you're not familiar with Vim.
4. A blank crontab file will appear. There are two parts to creating a cron job - the frequency and what to run. Here's an example of what a simple cron job looks like:

# custom cron job
MAILTO="some@user.com"
12 15 * * * php /home/user/capacity-report.php

Here's a reference chart of what each 'block' in a cron job schedule represents:

minute hour day_of_month month day_of_week

So, if we wanted to automate a report that runs every Sunday at 9am, the configuration would look something like this:

0 9 * * 0 php /home/user/report.php

Once you're done, you can save your configuration and exit out. You will see the following response:

crontab: installing new crontab

‍Creating AWS CloudWatch Events on Lambda

AWS lambda is a micro cloud service that lets you attach code to a machine without the need to set anything up. You can just push your scripts over to a Lambda and use it as needed.

The great thing about Lambda is that you only have to pay for what you use. So you can have hundreds of scripts sitting on the service and pay almost next to nothing on them. A way to leverage this service is to create small automations that can be triggered through another AWS service called CloudWatch Events.

In a way, CloudWatch Events on Lambda is a form of cron job – but in the cloud.

Here are the steps on how to automate your Lambda scripts with CloudWatch Events.

1. Open CloudWatch console and navigate to 'Events, Create rule'.
2. Chose 'Schedule' as  the 'Event source'. Specify the frequency. Add your target. You will need to create the IAM role for it to give CloudWatch Events permission to send events to targets.
3. Configure the final details such as name and description for the rule. Save it by choosing 'Create rule'.

And that's basically it. There isn't really much to attaching an automated trigger to your lambda function in AWS.

Easy schedulers can still lead to a time sink

From past projects, a major time cost was the initial investment in figuring how Lambda works and setting up the pipeline processes. For me, it took about two days to figure out. But once that hurdle was achieved, the subsequent scripting setup was limited to the complexity of the script.

For example, an automation project I worked on required values from a legacy database with over normalized structures across 50+ tables and 50k rows of data took about three days to unravel. Another two days was spent optimizing the process and three to five more days of back-and-forth feedback from Commercial.

All in all, the automation project took up about two weeks of my time. Most of it went towards building the thing I wanted to automate.

If you plug this into our time profit equation, the automation would have to make up for 288,000 minutes of manually running a task. Assuming it's replacing a 30-minute task every day, we're talking about having to let it run for many years before that upfront work pays for itself.

For some of the work I've was assigned to, the process can stop at the manual generation step. Why? Because the time cost to create the full process was more than the manual process. This often happens when the thing required is a one-off. Time-wise, sometimes it's not efficient to transform the manual process into something compatible with Lambda or wherever your script is going to reside.

Your calculations

Is task automation really worth the effort? Or is it a time-sink trap?

Sorry, but we have to go with a traditional engineering answer here: It depends. But I would hop that, by arming yourself with the info we went over above, you can hopefully make better informed decisions against misplaced "automation enthusiasm."

There are some other points to consider.

For one, it's true that learning to create initial automations can be the most time-consuming part of the process – especially if it's an unfamiliar territory. But sinking an inequitable amount of time into one project can be repaid if it produces learnings and foundations that can be applied in future automation processes. (And it could often be up to you, the dev, to sell to your team or managers why this time investment is worth it.)

Another point is that the cost analysis for automation can turn into a question of future efficiencies – and we're not just talking about the usefulness of a particular script continuing or ending. For example, cron jobs and pipelines are transferrable automations that can be easily put to work elsewhere when necessary. On the other hand, infrastructure orchestration automations are often not transferrable. That means a simple but proprietary automation for a prototype application that may or may not be scrapped can easily result in wasted work.

Creating automations is a balance against efficiency and what the business needs. Not everything needs to be automated – only the things that will add long-term value to your ability to protect your time.