Dear 22-Year-Old stupid self,

Congratulations on starting your first software engineering job! Keep in mind that $60k a year is really low even for an entry level engineer. Give it a couple of years, and you can easily earn a 6-figure salary. Also, take pride in the time you spent in school grinding away on difficult problems instead of partying.

Trust me when I say that you will enjoy coding more in 10 years than you do now. I know that right now, you count down the hours till the work day is done so that you can go home and do things that you think you want to do. In 10 years, you’ll enjoy your work so much that at the end of the day, you’ll wonder where all the time went.

I probably shouldn’t say this, but a $500k salary is possible for you. Yes, I know your career goal is $100k. That’s fine for now. But you’ll find out that there are certain companies that pay far more than the legacy company you work at now. Will you get to $500k by 33? That’s a secret you’ll have to find out, but you won’t get there without some hard work.

Eventually you’ll consider getting a masters degree by working on it part time while working full time. This will help you a little bit, but not as much as mastering AWS will. Get those certifications. Get that experience. You’ll eventually leave this certifications behind, but that is OK. They played an important role in your growth.

At some point, you’ll get into management. Try not to hate yourself too much. You’ll have the opportunity to lead some incredible people during this time. You probably won’t be their best mentor, but do your best.

Your big break won’t come until you have about 8-9 years of experience. A horrible virus will kill millions of people and cause companies (including Big Tech companies) to begin hiring remote employees. You’ll join a team at AWS. It will be the best opportunity of your life up until that time. It won’t end well, but that’s OK!

Don’t listen to anyone who insists you’ll burn out of the industry. Yes, this happens to a lot of people, but there’s no reason it should happen to you. As a remote software engineer, you don’t have to commute, get to spend more time with your wife and kids, and get to live in a nice home that you couldn’t afford in more expensive parts of the country. So what if you hit a salary ceiling? If you do, it will be high enough to continue living how you want for your whole life.

One thing you’ll discover at AWS and other big tech companies is that you’re not as smart as you think you are. But that's OK! Intelligence is just one variable in your success and not the most important one. You’ll find that many very intelligent people job hop too much to become truly effective at any other companies. Some are very intelligent, but leverage their intelligence to coast and don’t actually work very much.

One of the people at AWS you look up to the most was promoted to Senior Principal engineer. That’s not quite a distinguished engineer, but it’s pretty close! You won’t know all the details, but you suspect a big part of their promotion had to do with solving organizational problems that their leaders (management) were ill equipped to deal with.

Maybe you’ll go out on your own someday. If you do, find other people to run most of the business while you focus on the software.

I’ve been in the industry for over a decade now. I use AI every day, and it hasn’t robbed me of my joy when coding. In fact, it makes the frustrating parts of coding much less frustrating!

Have a nice day.

The Bombshell Announcement

In September 2024, Andy Jassy dropped the bombshell: Amazon employees would soon be back in the office five days a week. No more hybrid. No more halfway. This wasn’t entirely unexpected—Amazon had been signaling a shift since February 2023—but the official announcement still landed like a gut punch for hybrid employees. This post reflects on what that shift—“RTO” for short—meant for me, others, and the industry.

My Dream Job

When I first got my offer from AWS in early 2022, it was a thrilling opportunity that felt almost too good to be true. Tech giants like Amazon were hiring beyond their usual hubs, and I got in at just the right time. I passed the interview loop and received a Senior SDE offer that nearly doubled my total compensation. Those early days were incredible: I soaked up wisdom from big-tech veterans while mentoring younger engineers. The pandemic-era remote setup gave me a unique shot at growth, balance, and fulfillment. At least a third of my 100+ person org lived nowhere near an Amazon office. We thrived under the remote model.

The Quiet Push

The first sign of change came quietly: a “soft” announcement about a three-day return-to-office requirement. It was buried in an internal news post, and I remember the sinking feeling I got reading it. It reminded me of the moment years ago when I was informed of a layoff. Details were sparse. People were skeptical. An internal Slack channel sprang up to advocate for remote work policies. Many colleagues dismissed it as a blip, something Amazon would inevitably reverse. We all hoped. We were wrong.

Andy Jassy’s Message

Months later at a company meeting, Andy Jassy delivered this clear directive: “If you can’t disagree and commit, it’s probably not going to work out for you at Amazon because we are going back to the office at least three days a week.” While definitive, the word “probably” stood out to me. Could incredible performance override the relocation demand? I clung to that sliver of ambiguity.

A Looming Deadline

Relocation was no longer a question of “if” but “when,” unless I received an exception. Would the demand come right before a vest date? During peak hiring season? These questions weighed heavily on me, forcing me to evaluate not just my career but my life priorities. I had some factors in my favor: I was a high performer in my org, and we had a major release planned for re:Invent. The timing was critical—how could I deliver on such a pivotal project while navigating relocation logistics? I hoped leadership would recognize the stakes and prioritize the project’s success. Thankfully, they did, or at least they didn’t demand relocation during this critical period.

Living with Uncertainty

RTO didn’t occur in isolation. During this same period, Amazon implemented extensive cost-cutting measures, including layoffs, reorganizations, and the elimination of entire divisions. Leadership tried to keep these developments under wraps, but leaks surfaced—sometimes weeks in advance—on platforms like Blind. I thought AWS might be immune, but eventually, layoffs hit us too.

Most days, I carried on as if nothing had changed. But every so often, the reality blindsided me: “Why am I pouring my soul into this when my future here feels so uncertain?” For months, I worked with the nagging dread that I’d be forced to relocate or leave. That unease never fully went away. The lead-up to re:Invent 2023 was especially grueling. Long hours were spent racing to meet the release deadline, all while knowing the company could demand relocation the moment the dust settled. Fortunately, re:Invent came and went without that ultimatum. I exhaled with relief, having crossed the two-year mark and secured a brief window to plan my exit. In June 2024, I was gone.

I didn’t realize how much stress I was carrying until I left. The constant uncertainty—the questioning, the worrying—had taken its toll. After leaving, the fog lifted. I could breathe again. That clarity was a gift, and it reminded me of the importance of aligning work with the life I wanted to live.

Reflections on Amazon Culture

Despite everything, I truly enjoyed Amazon’s culture. The emphasis on ownership, the high standards, the energy—it was challenging, but rewarding. Amazon pushed me to grow in ways I hadn’t expected, and I’ll always be grateful for that. If conditions were right—if remote work or local office options became feasible again—I’d absolutely consider returning.

For me, going into an office wasn’t the problem. The problem was relocation. There wasn’t an Amazon office near my home, and uprooting my family wasn’t on the table. If I were younger, childless, and more eager for adventure, I might have taken the leap. But that wasn’t my reality.

What’s Next for the Industry and AWS?

Amazon and AWS are not going anywhere. When the RTO announcement was originally made back in February 2023, the $AMZN stock price averaged under $100 for the month. As I write this, the price is around $230—higher than it has ever been. Since announcing RTO, Amazon has also implemented numerous cost-cutting measures such as layoffs and deprecations. Their business is becoming more focused (a good thing). While I still think eliminating remote work was the wrong decision, I hold no illusions that it will visibly harm Amazon’s trajectory.

Some might argue that Amazon will lose its best engineers and struggle to attract top talent because of this decision. However, this was likely an intentional calculation by leadership. Their conclusion appears to be that Amazon doesn’t necessarily need the most talented engineers—it needs diligent and driven ones. Historically, Amazon has never been regarded as the most employee-friendly company, and the removal of remote work has dispelled any lingering perceptions of it aspiring to be the world’s best employer.

Amazon’s move to implement RTO was a boon for the leadership of other companies across the industry. Before this shift, competing with the largest software engineering employer that embraced remote work posed a significant challenge. Now, that pressure has eased. Following Amazon’s lead, many companies have rolled out their own RTO policies with newfound confidence. However, companies still offering remote opportunities now have access to some of the best talent in the market.

Realistically, remote work isn’t going away, but there won’t be as many remote jobs available as there were during COVID. I do think there’s more available than there were pre-COVID.

For the Remaining Remote Workers at Amazon

The reality is that many people at Amazon are still working remotely. They’ve likely managed this due to fortunate circumstances or because their leaders have prioritized other concerns over enforcing RTO. A lucky few may have even secured long-term exceptions. However, my advice to those in this position is clear: most of you should start planning to either relocate or leave. Even if you manage to extend your remote arrangement for years, the trade-offs can be steep. You may feel like a second-class employee—excluded from opportunities for advancement. Promotions will likely be off the table, yet the effort you’ll need to exert will remain as demanding as if you were striving for one.

For most of you, this means exiting right after a vest date with another job lined up. Consider yourself fortunate (as I do) if you’re able to find a position with comparable compensation, but you might not find that.

In the past few weeks, a number of X (formerly known as Twitter) users in the tech community discovered that some AWS services are being deprecated (or something, it’s not entirely consistent or clear). These services are identified by Jeff Barr in the following Tweet (or post or whatever it is now):

Additionally, my understanding is the QLDB (not mentioned) is also deprecated.

Of this list, most of the deprecations make sense. I’ll go through each one with my thoughts. Note that even with my recent employment at AWS, I don’t really have any inside info on this. These are my own thoughts.

Why is AWS deprecating services?

The leaders at AWS must believe that they can discontinue these (I assume unprofitable) services without losing customer trust. For each of these services, one of more of the following is likely true:

• not many customers use it

• the customers have other options that are relatively straightforward to migrate with minimal financial impact (or they are allowed to stay)

CodeCommit

CodeCommit is inferior to alternatives like GitLab, BitBucket and GitHub. At some point in the distant past, AWS chose not to make CodeCommit a competitive service compared to these other Git providers. So, deprecation (or whatever this is) makes sense to me. Interestingly, I have friends who work for government companies and contractors that use CodeCommit extensively in AWS GovCloud. I am curious to see how that will play out. I’m betting that CodeCommit will continue to exist in GovCloud pretty much indefinitely.

Data Pipeline

I’ll admit I’m not super familiar with Data Pipeline. I’ve used it in the past to back up DyanmoDB tables, but that use case is no longer necessary as DDB natively supports backups. If I wanted to perform ETL using native services today, I would NOT involve data pipeline. Instead, I would use modern (and Serverless) services like Glue, S3 and Lambda (and potentially step functions). So, this deprecation makes sense to me. It does seem like the kind of thing though that if a company was using Data Pipeline, migration off of it might not be very straightforward.

ForeCast

What is ForeCast? I vaguely recall it being announced. I have no idea whether this service deserves to be deprecated or not because I know nothing about it.

SimpleDB

SimpleDB is inferior to DynamoDB which has replaced it. SimpleDB customers should be able to move to DynamoDB. It is interesting that Jeff had this in the list. My understanding is that “new signups” for SimpleDB have been discontinued for a very long time.

Cloud9

At some point in the past, AWS must have decided it did not want to seriously compete with Visual Studio Code. So, the deprecation of Cloud9 makes sense with that in mind. That said, customers of this product liked it a lot. I’m sure it will be sad for them to see it go.

CloudSearch

I don’t think I’ve ever heard of anyone using CloudSearch. I think what customers really want is a Serverless search service. OpenSearch supposedly has that, but I don’t consider a service to be truly Serverless unless it scales to $0 when not being used. Think of things like an empty DynamoDB table or an empty S3 bucket.

QLDB

Again, this is a service I don’t understand. How can there be a crypto service? I thought the whole point of crypto was decentralization? Unlike the other services, it sounds like AWS plans to actually delete the data in this services. That tells me there are very few customers using it.

S3 Select

Of every service mentioned, S3 Select is the biggest surprise. S3 itself is a service that will outlive everyone reading this, so it’s odd seeing a feature of it being discontinued. That said, I don’t think much is lost here because AWS Athena is in every a way a more powerful service that can do everything S3 Select could do and much more (correct me if I’m wrong). I’m honestly shocked that the implementation of S3 Select isn’t just Athena on the backend.

I guess that S3 Select works without much configuration whereas Athena requires some configuration to be setup. So at most, this will just annoy the few people that were using it.

What do AWS Deprecations mean for me?

These deprecations mean that AWS services that are unprofitable for AWS or lack users or meet whatever criteria AWS sets aren’t guaranteed to be around forever. It means that any shiny new service announced at re:invent might not be all that it is marketed to be. QLDB was announced at re:invent 2019. AWS Forecast went GA in 2019 as well. S3 Select went GA late in 2018. These services aren’t really all that old, but AWS doesn’t feel the need to continue with them. We can expect more deprecations to be announced in the future, even for products being released in 2024 and beyond.

Hopefully these deprecations lead new product teams at AWS to be more careful about what products are being built.

As with all things where people are involved, engineering performance tends to land on a bell curve. This post will generally be directed toward people who are currently in the middle of the curve, and are looking for tips getting further to the right. My guidance is also biased towards engineers working in a corporate type of environment as opposed to a startup environment (though the tips should be helpful regardless).

On the left, you have the lowest performers in your organization. In my experience, people on the left of the curve tend to think they should be somewhere in the middle and are often clueless about their poor performance. In one situation, an engineer on my team came from a company where little to nothing was expected of him. He carried that mindset into my team. In the worst situations, I’ve seen engineers with 0 accomplishments during their entire tenure with a company. Engineers far to the left generally don’t receive raises and are often subject to performance improve plans and terminations. My observation is that people tend to be on the left due to laziness instead of incompetence. In other words, incredibly smart and knowledgable people can end up there. Don’t be on the left!

On the right, you have the highest performers on your team or in your organization. These engineers are highly reliable, deliver independently, and have a long list of accomplishments. These are the people your management hopes to promote quickly.

And of course, most people fall in the middle of the curve. But even the middle has a left side and a right side. At high performing companies, these distinctions are important. Being in the middle-left might be similar to being in the far left if your company is looking to reduce headcount. Being in the middle-right might be equivalent to being on the far right if you’re on a high-performing team.

Getting further to the right

Your first step is figuring out where you are on the curve. Your company might have an official mechanism or tool for this. If so, you can use that to determine approximately where you are. Otherwise, you should speak with your manager and potentially your skip level manager to find out where you land. If your manager for some reason won’t tell or you think they might not be as forthcoming as you’d like, try to estimate where you compared to the rest of your team. (note: managers tend to be more forthcoming with their highest performers). Do you ship more or fewer features than the rest of your team? Does your manager go to you or others when they have important work to be done? Are you able to help others on the team more than you require help from others?

Remember, it’s OK if others on the team are stronger in some or many ways than you. You generally want to be on a team where you can learn from and be inspired by others.

Ask your manager what you need to do differently to get a higher performance rating. If you’re in the middle of the curve, and you ask this of your manger then you might be surprised at their response. At my first full-time job as an engineer, I thought “writing a lot of code” was the most important metric. It turned out that anyone on the team could write a lot of code, but only a couple of people on the team were trusted to release the code.

At another job, I was on a team with an engineer who was a pull request machine. He would find the easiest tasks in the sprint and quickly pump out high quality pull requests until there were no more tasks. However, another engineer on the team was promoted before him because they were better at supporting engineers from other teams. In other words, their non-coding skills like “feature design”, “prioritization”, and “building rapport” were weighted higher than coding ability. Additionally, positive feedback for an engineer from outside of your immediate team is often more strongly weighted than positive feedback from engineers within your team. The incredible coder on my team lacked positive feedback from outside of the team.

Do what your manager told you to do. Once you’ve identified what it is you need to do to improve your performance, make a habit of doing it!

Getting to the far right of the curve

Depending on where you work, your company likely has at least some extraordinary engineers. If that is the case, then simply following your manager’s guidance might not be enough to get you to the far right of the curve.

Here are some qualities of top engineers I’ve observed:

• Their leadership goes to them for help. Every manager or director in software has engineers that they trust for help and advice. These people tend to be the top performing engineers on the team or organization.

• They quickly recognize bad decisions based on past experience. For example, I presented a design to an incredible engineer who I worked with at the time. The design included a short-term solution that was a bit of a workaround until a long-term solution could be identified. The incredible engineer immediately told me that I should assume the short-term solution would be around forever. He was right! This totally changed the way I approached the problem and sent me back the drawing board.

• They are incredibly detail oriented. If you present them with a big idea that you’ve thought through already, they’ll recognize quickly that the idea is fine and dive into areas of the implementation or execution where you might run into trouble. For example, an engineer I was working with put together a design and asked 1me to review it. In the design, the engineer assumed they could access some data via an API. They were correct, but the API was more of a control plane API (pretty low traffic) whereas the API he was designing was clearly a data plane API (high traffic). Thus, his design needed to include ensuring the existing control plane API could be made ready to handle the additional volume.

• They think in terms of trade-offs, including business trade-offs. Even well-accepted best practices in software engineering should not always be followed in certain circumstances. Top performing engineers understand this and can articulate to leadership and engineers when conventionally good ideas should not be implemented. A good example of this is understanding when a process should be automated vs. having the manual steps documented. As someone who used to manage a DevOps teams, I believed for a while that everything should be automated, but this isn’t actually the case. For instance, if a process is only performed once a year and can be reliably performed by reading some instructions, then it likely isn’t worth the time spent automating it. How many times does the automation need to be run to cover the cost of the automation development effort? Additionally, you want to think about the opportunity cost of having an engineer write the automation. If you’re automating something that rarely runs when there’s innovation work needed, then maybe you’re doing the wrong thing.

Closing thoughts

One thing I think about is the situation where almost everyone on a team is pretty equally high performing (toward the right). This can actually happen and does happen. I have actually seen teams formed by taking top engineers from other teams and placing them all on the same team. If you put these people on the curve, they might all end up in the middle. If they are then compared with lower performing team, then the comparison will not be accurate or fair. In these cases, the manager of the high performing team should put the engineers toward the right and justify that placement with their management. The team may need to have many accomplishments to justify this sort of placement, but it’s the correct thing to do.

As an engineer, if you have a choice to be on a team with many other high performing engineers or a more “curve fitting team”, then you should try to be on the high performing team. You might not standout on your team as much, but you’ll improve faster than you would on the more normal team. Additionally, if your company is good, they will recognize your team as exemplary and place engineers (hopefully) according to how I described in the preceding paragraph.

Why it makes sense

I would argue that whether this trend is true or not, there are reasons for it that make sense. The tooling available to full-stack engineers has never been better. For instance, a full-stack engineer can use tools like React Native or Google Flutter to build applications that be deployed to every major platform using the same code base. Furthermore, the server side code can be written using TypeScript and deployed on Node. Coupled with React Native, this means that the full-stack engineer need only master a single language.

Where it falls apart

Writing non-trivial front-end applications targeted for many platforms is tricky and time-consuming. Furthermore, designing complex backend applications for modern applications is also tricky and time-consuming for different reasons. With that in mind, it makes sense to keep the disciplines separate. However, this same logic applied to development and operations. Many companies have found success combining development and operations into a single role. They likely see the same opportunity with front-end and back-end work.

My advice

If you’re a backend engineer, you’d serve yourself well to begin delving into front-end tools using a framework like React or React Native. There are numerous available resources to begin learning.

If you’re a frontend engineer, learning backend design is less straightforward (imo) since it is less about jumping into code and more about developing “systems thinking”. With front-end code, you tend to have a single user using a given instance of the application at a time. With back-end, a single application instance should serve thousands of concurrent users. The database and app must gracefully handle concurrent writes. To learn more, start watching systems design videos on YouTube.

Versatility a driving factor?

Companies and leaders (even smart ones) are obsessed with treating every engineer as equivalent. They want to create an environment where skillset need not be considered when reassigning engineers to other teams and projects. Of course, this sounds crazy to any sane engineer. However, requiring that every engineer be a full-stack engineer drives reality closer to this management utopia.

Full-stack might make most sense for established teams

If your application or application component is mature, then full-stack engineers might make a lot of sense. When is your application mature? Let’s just say that once you are at the point where new features require some modifications to a mostly complete back-end codebase and an established front-end codebase that deploy to a running app, then we’ll consider it mature within the context of this post.

Microservices Scenario: You are tasked with implementing a new feature for a service consisting of 20 microservices divided amongst 5 software engineering teams (Teams A, B, C, D, E). Of the 20 microservices, you find that 8 of them need development work in order to complete the feature. However, the 8 services belong to 3 of the teams. Team A owns 4. Team B owns 2, Team C has 2. You are on Team A. How do you proceed?

A 10x engineer/code ninja might proceed by quickly completing all of the work in the 4 microservices that their team (Team A) owns and then just wait for the other teams to complete their work. This is a valid path forward for a junior engineer sometimes. However, it often isn’t feasible. What if Team A’s code changes cannot be completed without dependencies from Team B or Team C first? If that is the case, then you cannot complete the work any quicker than it takes Team B and Team C to complete theirs (even if you are a 1000x engineer).

A better way to proceed would be to put together a design document detailing the changes required by each team and micro to complete the feature. Present the design to your leadership and the other teams to make sure you have buy-in. Answer any questions and fill in design gaps that the other teams point out (they know their services better than you do after all).

Now contrast the above scenario with this one.

Monolith Scenario: You are tasked with implementing a new feature for a service consisting primarily of a monolith and a few supporting micros for things like auth and notifications. You have access to all of the code for the monolith and the few micros. How do you proceed?

In this scenario, a 10x engineer/code ninja quickly implementing the feature is almost certainly the proper course of action. A design document is likely not needed.

At this point, you might be thinking I’m advocating monoliths over microservices. I’m not. A strong case can be made for either and is dependent on context. Realistically, a strong programmer will likely be more productive in a microservices system when they are implementing features that only require code changes to microservices that their team owns because microservices consist of inherently less complex code bases than an equivalent monolith. The key is the team separation that hinders the impact of strong coders.

For services and products that are distributed amongst microservices across many teams, the key area of expertise needed shifts from the code base to the design of the system and how the various pieces interact. In other words, the design you put together on the whiteboard and diagrams and then formalize in the separation of teams and microservices becomes more important to the productivity of the organization than any single engineer’s strong coding ability.

So, being a 10x engineer for a microservice product is less dependent on your coding ability and more dependent on your ability to harmonize the various microservices and the teams that own them.

As a quick reminder, Site Reliability Engineering approaches software operations as software engineering problems and seeks to solve operational problems using software. This is distinct from DevOps which is where the software engineering team for a particular service is additionally responsible for running that service in production. And as always, this is muddied by the fact that there are “DevOps” teams which would be better named “SRE” teams because they don’t actually do any software engineering for a service.

Anyways, getting back to Google. I still have not been able to find whether the rumor is true or not, but I suspect that given the current market conditions and their layoff earlier in the year that Google might be taking a deeper look at how they run their software.

Much of what Site Reliability Engineering originally set out to accomplish can now be achieved using various services, platforms, and practices. The container image has made packaging software standardized (mostly). Kubernetes has standardized container orchestration. CloudFormation/CDK/Terraform/etc. has made cloud infrastructure more manageable. There are a million managed observability (logging, tracing, metrics, alarms, etc) services to choose from as well. Even if you are running in a data center and use few managed services, you can still use open source tools like Kubernetes/ElasticSearch/Grafana and have what feels like modern managed platform (at least for software engineering teams).

I suspect Google (if the rumor is true) intends to shift the operations responsibilities fully to the software engineering teams. Many successful software companies already operate that way. I am speculating here, but I also suspect that Google may have identified they would get more value by having their SRE teams focused on producing cloud services in the observability realm.

I do not think that SRE (including misnamed DevOps teams) will go away in most of the industry. A typical “SRE/DevOps Engineer” is well-versed in the services/tools/practices I mentioned above whereas many software engineers prefer to solves problems primarily in their application codebase. However, many companies do tend to follow Google’s lead, so we’ll see.

Everyone has their favorite tool, so I have no doubt that this will make some people upset. In this post, I’ll discuss some of my reasoning.

Software Engineers prefer programming languages

My main opinion around IaC is that that software engineers should prefer a full programming language and framework for doing infrastructure as code. IaC Frameworks that allow for this include:

• AWS CDK

• Pulumi

• Terraform CDK

Of these tools, I recommend AWS CDK if you’re on AWS as it is covered by AWS Support Plans and is actively developed and used internally by Amazon. For other clouds, I prefer Pulumi over Terraform CDK because Pulumi is more mature. In a couple of years, if HashiCorp keeps up its commitment to Terraform CDK, I expect it to become more compelling.

Less skilled programmers should NOT use a full programming language

If your team doesn’t include at least a couple of skilled programmers, then I would not advise picking an IaC that uses a full programming language. You’re better off using an IaC that uses a DSL/JSON/YAML like

• CloudFormation

• Terraform

• Bicep

• GCP Config Connector

If your team has a mix of strong programmers and less skilled programmers, then it’s probably fine to use a programming language IaC.

Take advantage of Support Plans

If you are using IaC at a large corporation, then you very likely have an enterprise level support plan with one or more cloud providers. You should strongly consider selecting IaC tools that are covered by these plans. Should you run into issues with the covered IaC, then your cloud provider is “on the hook” to provide support.

These tools include

• AWS CloudFormation

• Azure Bicep

• AWS CDK

• GCP Config Connector?

Pulumi and Terraform both have Enterprise Plans that you can pay for. Of course, you would generally pay for these separately from your cloud provider support plan.

Single-Cloud

Each big Cloud Provider has robust IaC using a domain specific language (DSL) offering for single-cloud if you choose their Cloud. AWS has CloudFormation. GCP has Config Connector. Azure has Bicep. Terraform works well for all of these as well.

The tricky part is that teams with programmers should prefer an IaC tool that uses a programming language. This is easy in AWS because you can just use CDK. However, as far as I know, Config Connector for GCP is not supported by any programming languages. This leads me to prefer Pulumi for GCP (Terraform CDK would also work).

Multi-Cloud

Most of the cloud specific tools provide little or no support for third party resources. Bicep is just for Azure. Config Connector pretty much just supports GCP.

CloudFormation has public and private registries now. I highly recommend using them for companies that are already deeply embedded in AWS CloudFormation.

However, no tool has better multi-cloud and third-party functionality than Terraform and by extension Pulumi since it can make a Terraform provider into a Pulumi provider. Both tools are open source so custom providers can be written.

Hybrid Cloud

For Hybrid Cloud cases, I believe that Ansible is likely the best tool though I wouldn’t say Ansible is strictly an IaC tool (it really isn’t one at all). However, it can be combined with IaC tools like Terraform.

Reality is that if you’re doing Hybrid cloud, the most difficult thing you’ll be doing is data center automation. Various hardware components have very specific tools for configuring them. These tools can almost always be configured by Ansible. None of the other tools mentioned in this post would support this.

DSL > JSON/YAML

I prefer IaC tools that use a DSL for the configuration language over tools that use JSON or YAML. JSON is difficult to write because of quote escaping and multi-line hell. In YAML, it is easy to make indentation errors. Furthermore, a truncated YAML file is often still valid YAML which can lead to issues should the file be corrupted.

DSLs like HashiCorp Configuration Language for Terraform or Azure Bicep are better suited for IaC. They are much more readable than YAML/JSON and much more writeable YAML/JSON. They still can be prone to truncation issues, but not to the extent YAML is.

Of course, as a programmer using AWS CDK, Terraform CDK, or Pulumi you don’t have to care too much about this. A CDK that produces JSON is a good model.

Serverless IaC

If you are deploying Serverless apps, then much of this discussion isn’t highly applicable because there are IaC options specific to Serverless that do a great job. I may do a separate post on those.

Kubernetes

Like Serverless, I would have to do another post on IaC for Kubernetes. Overall, I am underwhelmed by some of the popular tools for Kubernetes IaC like Helm which I view as total garbage. That said, most of the tools described in this post actually do a pretty good job deploying to Kubernetes so much of this post could still apply.

Conclusion

Choosing the right IaC is an important decision and often not one that is easy to change. I’ve only touched on some of the dimensions used in making a decision. Other dimensions include cost, developer community, and leveraging your existing experience.

I like how this looks. It’s clean. Each component should be able to evolve and scale independently of the others.

Except that’s not really how it works .

In reality, each micro is heavily constrained by their callers (upstreams) or their dependencies (downstreams) or both. Let’s redraw the diagram to see what it really looks like.

Every single one of these micros likely has to interact with the user and tenant management service. Every request will need to be authenticated and authorized. The caller context has to be verified as it flows through every service. Each service will have to figure out how to ship their data to the Search micro. The various business logic services likely have dependencies on each other to avoid duplicated effort.

In general, these pains are worth it. More micros means a smaller blast radius if a single micro is breached, more fault tolerance in case one of the micros becomes unavailable (unless it’s auth or user management), better resource utilization for heavily used services (like auth) and rarely used micros.

However, the one alleged “benefit” that I completely find ridiculous is the idea that micros evolve independently. I have never found this to be the case. Let’s look at this from the perspective of callers (upstreams) and dependencies (downstreams).

The Upstream Problem

By upstreams I mean services that are early in the call flow. The client (frontend/Web browser) is the most upstream on this diagram (the client isn’t a micro though). API Gateway is second. Auth is likely third. User/tenant management is probably fifth. Everything else is after that.

The issue for upstream services is that downstream micros often do not yet provide sufficient APIs to enable your features. Why? Usually, this is because those services are owned by other teams which have differing priorities from your team. This creates overhead in that leadership and the two teams need to negotiate priority. However, if the downstream team is unable to provide the needed functionality in a timely manner, then the upstream might implement it themselves or find a clunky workaround.

This is often necessary, but it is problematic because it has potential to create technical debt that is difficult to unwind because micros end up having functionality built into them that logically should be elsewhere.

For example, what would happen if the Users/Tenants team failed to deliver their data to the Search team. Thus, the Search team cannot provide search functionality containing user data. However, Business Logic 1 service needs this functionality. Business Logic 1 finds a way to maintain a list of users, so instead of using the Search team, they use their own data. To make matters worse, Business Logic 2 service decides to invoke Business Logic 1 to search user data because the functionality is ready.

Months later, the Search to finally is able to get the data they need from the Users/tenants team. Search builds the search functionality for user data. However, Business Logic 2 team does not adopt it because they already have the functionality they need.

The Downstream Problem

By downstream I mean services that are late in the call flow. The User/Tenants service in the above diagrams serves as our example.

The issue for downstreams is that changing the interface is hard. For one, you MUST never break your callers/upstreams (except under rare circumstances like an active breach in which case it might be necessary). For instance, you own the user service and have an API for retrieving a user by social security number that is invoked over HTTP like the following:

HTTP GET /users?ssn=333-33-3333

A few years after your product launch, your leadership realizes that querying for user data by social security number is a bad idea. In fact, a compliance agency threatens to remove one of your certifications when they discover this API. The decision is made to remove it as quickly as possible.

But you can’t remove it immediately. The API is used by nearly every other micro. The best you can do is allow it to optionally take in a second field called id like the following:

HTTP GET /users?ssn=333-33-3333&id=abcde-fghi-jklmnopq

Your upstream micros can then invoke you either using ssn or id. This gives them a migration path off of the API. But you are stuck with ssn as a parameter until the last caller switches to using id. Removing ssn prior to that would break the upstream services and cause the product to lose service.

This is the downstream problem. Fixing your technical debt often requires your callers to change. If your clients are external, then moving your customers onto the non-deprecated feature could take years. This is why companies like Amazon and Microsoft very rarely remove APIs and take the interface very seriously.

Solving these Problems

The management and leads of each micro must periodically meet to align on their mission. Teams must commit to adhering to the principles of the architecture except as a last resort. Should the principles be compromised, teams must commit to resolving once a principled solution is ready. Teams must strive to rarely deprecate APIs or change the APIs that require a difficult migration.

Many of these problems shows up less often by increasing the number of micros owned by each team or simply combining those micros into a single service. This empowers teams to more often unblock themselves.

To sum it up, the benefits of microservices include

• easy to divvy up the work to a lot of teams

• better fault tolerance

• low blast radius

• independent scaling

The drawbacks include

• technical debt is often shared across many micros

• migrations involving multiple teams have to be carefully orchestrated

• latency has to constantly be dealt with

Here’s a few types of tests that are common in software development:

1. Unit Tests test a class or function without testing its dependencies. No connecting to databases, the file system, or making real network calls. If your function does those things, you need to find a way to mock those dependencies. Otherwise, a unit test isn’t possible.

2. Integration Tests test a class or function and its dependencies (though not necessarily the dependencies’ dependencies). These tests do not connect over the network to any dependencies (though this rule can be bent).

3. System Tests test the running software connected to all dependencies including databases, other APIs, etc. System tests test the interface that the end users or clients interact with. So for a Web site, a proper System test would use a Web browser to click buttons and make sure proper things happen. To system test an API, you would invoke it and then use other APIs to make sure it did what it was supposed to do.

Pretty much all programmers are familiar with unit tests, but system and integration tests are often less familiar to newer programmers. We’ll focus on system tests in this post.

System Testing Command Line Tic-tac-toe

Below we have a code block written in Go that is a simple game of tic-tac-toe based on the command line. The game assumes that the two players are sharing a terminal. The users input a number between 1 and 9 where 1 is the upper left hand square, 5 is the middle square, and 9 is the bottom right hand square. It’s pretty simple.

package main

import (
  "fmt"
)

var board [3][3]string

func init() {
  for i := 0; i < 3; i++ {
    for j := 0; j < 3; j++ {
      board[i][j] = "-"
    }
  }
}

func displayBoard() {
  for i := 0; i < 3; i++ {
    for j := 0; j < 3; j++ {
      fmt.Print(board[i][j])
      if j != 2 {
        fmt.Print("|")
      }
    }
    fmt.Println()
  }
}

func didPlayerWin(player string) bool {
  for i := 0; i < 3; i++ {
    if (board[i][0] == player && board[i][1] == player && board[i][2] == player) ||
      (board[0][i] == player && board[1][i] == player && board[2][i] == player) {
      return true
    }
  }

  if board[0][0] == player && board[1][1] == player && board[2][2] == player {
    return true
  }

  if board[0][2] == player && board[1][1] == player && board[2][0] == player {
    return true
  }

  return false
}

func main() {
  var player = "X"
  var move int
  var row, col int
  var gameOver = false
  var winner string
  numMoves := 0
  for !gameOver && numMoves < 9 {
    displayBoard()
    fmt.Print("Player ", player, ", enter move (1-9): ")
    fmt.Scan(&move)
    row = (move - 1) / 3
    col = (move - 1) % 3
    if board[row][col] == "-" {
      board[row][col] = player

    } else {
      fmt.Println("Invalid move, try again.")
      continue
    }
    if didPlayerWin(player) {
      winner = player
      gameOver = true
    }

    if player == "X" {
      player = "O"
    } else {
      player = "X"
    }

    numMoves = numMoves + 1

  }
  displayBoard()
  if winner != "" {
    fmt.Println("Player", winner, "wins!")
  } else {
    fmt.Println("It's a tie")
  }
}

Here is a simple (and very incomplete) unit test for the didPlayerWin function:

package main

import "testing"

func TestDidPlayerWin(t *testing.T) {

  board[0][0] = "X"
  board[0][1] = "X"
  board[0][2] = "X"

  board[1][0] = "O"
  board[1][0] = "O"

  if !didPlayerWin("X") {
    t.Fail()
  }
}

In this test, it simply checks if a player won. In the tested case, the X player won by connecting 3 in a row on the top row. It’s pretty straightforward.

The rest of the code is not easily unit testable because it is a monolithic function (though for a trivial code base like this, it is possible).

Writing system tests for this game is easy and can be done with a simple bash script. Here is an example system test for the game:

echo "1 2 3 5 2 9 8 4" | go run main.go | grep -q "Player O wins\!" && echo SUCCESS || echo FAILED | grep "SUCCESS"

If you run the above code, then the result will be “SUCCESS” printed to the screen. If change the grep statement to check if Player X wins then the result will be that “FAILED” will print.

How does this snippet work? Let’s break it down:

1. echo "1 2 3 5 2 9 8 4"

   1. Simply prints the string "1 2 3 5 2 9 8 4" to the console

   2. This gets piped to the next command

2. go run main.go

   1. Runs the tic tac go program which prompts for input

   2. Receives the “1 2 3 5 2 9 8 4” string from the previous command

   3. Writes “Player 0 wins!” (and other stuff) to stdout (or it should if the code works!)

   4. Pipes the output to the next command

3. grep -q “Player O wins\!” && echo SUCCESS || echo FAILED

   1. Checks if the output from the previous command contains “Player O wins”. Writes “SUCCESS” to stdout if so other writes writes “FAILED

   2. The -q flag suppresses the output from showing up in stdout. This isn’t necessary, but it makes the test cleaner

4. grep “SUCCESS”

   1. Checks to make sure the previous command writes “SUCCESS” to stdout or not.

   2. I simply added this so that the proper error code would show up in $?. This allows the code to be run in a larger script and know when the test is failing

We can add many more tests like the following:

# check if O wins
echo "1 2 3 5 2 9 8" | go run main.go | grep -q "Player O wins\!" && echo SUCCESS || echo FAILED

# check if X wins
echo "1 4 2 5 3" | go run main.go | grep -q "Player X wins\!" && echo SUCCESS || echo FAILED

# check for a tie
echo "1 2 4 7 9 5 8 6 3" | go run main.go | grep -q "tie" && echo SUCCESS || echo FAILED

You can just keep adding tests like this. Unlike our unit test, it tests all of the code.

Now you might be thinking “well, can’t we just write more unit tests”? Yes of course, and we should! However, much of the code is not very easily unit testable.

Let’s refactor the code to be more unit testable. We’ll also put everything into a struct because why not.

package main

import (
  "fmt"
)

var game *TicTacToe

func init() {
  game = &TicTacToe{
    board:         [3][3]string{},
    currentPlayer: "X",
    numMoves:      0,
    over:          false,
    winner:        "",
  }
  for i := 0; i < 3; i++ {
    for j := 0; j < 3; j++ {
      game.board[i][j] = "-"
    }
  }
}

type TicTacToe struct {
  numMoves      int
  board         [3][3]string
  currentPlayer string
  over          bool
  winner        string
}

func (game *TicTacToe) overMessage() {
  if game.winner != "" {
    fmt.Println("Player", game.winner, "wins!")
  } else {
    fmt.Println("It's a tie")
  }
}

func (game *TicTacToe) setCurrentPlayerAsWinner() {
  game.winner = game.currentPlayer
}

func (game *TicTacToe) promptInput() int {
  var move int
  fmt.Print("Player ", game.currentPlayer, ", enter move (1-9): ")
  fmt.Scan(&move)
  return move
}

func (game *TicTacToe) convertInputToCoordinates(move int) (int, int) {
  row := (move - 1) / 3
  col := (move - 1) % 3

  return row, col
}

func (game *TicTacToe) isOver() bool {
  if (game.numMoves) > 8 {
    game.setIsOver()
  }
  return game.over
}

func (game *TicTacToe) displayBoard() {
  for i := 0; i < 3; i++ {
    for j := 0; j < 3; j++ {
      fmt.Print(game.board[i][j])
      if j != 2 {
        fmt.Print("|")
      }
    }
    fmt.Println()
  }
}

func (game *TicTacToe) didCurrentPlayerWin() bool {
  for i := 0; i < 3; i++ {
    if (game.board[i][0] == game.currentPlayer && game.board[i][1] == game.currentPlayer && game.board[i][2] == game.currentPlayer) ||
      (game.board[0][i] == game.currentPlayer && game.board[1][i] == game.currentPlayer && game.board[2][i] == game.currentPlayer) {
      return true
    }
  }

  if game.board[0][0] == game.currentPlayer && game.board[1][1] == game.currentPlayer && game.board[2][2] == game.currentPlayer {
    return true
  }

  if game.board[0][2] == game.currentPlayer && game.board[1][1] == game.currentPlayer && game.board[2][0] == game.currentPlayer {
    return true
  }

  return false
}

func (game *TicTacToe) move(row, col int) {
  game.board[row][col] = game.currentPlayer
  game.numMoves++
}

func (game *TicTacToe) switchPlayer() {
  if game.currentPlayer == "X" {
    game.currentPlayer = "O"
  } else {
    game.currentPlayer = "X"
  }

}

func (game *TicTacToe) isValidMove(row, col int) bool {
  if row < 0 || row > 2 {
    return false
  }

  if col < 0 || col > 2 {
    return false
  }

  return game.board[row][col] != "-"
}

func (game *TicTacToe) setIsOver() {
  game.over = true
}

func main() {

  for !game.isOver() {
    game.displayBoard()
    move := game.promptInput()
    row, col := game.convertInputToCoordinates(move)
    if game.isValidMove(row, col) {
      fmt.Println("Invalid move, try again.")
      continue
    }

    game.move(row, col)

    if game.didCurrentPlayerWin() {
      game.setIsOver()
      game.setCurrentPlayerAsWinner()
    }

    game.switchPlayer()
  }
  game.displayBoard()
  game.overMessage()
}

This code is much easier to unit test. Here are some unit tests:

package main

import "testing"

func TestDidCurentPlayerWin(t *testing.T) {

  game.move(0, 0)

  game.move(1, 0)
  game.move(0, 1)
  game.move(1, 0)

  game.move(0, 2)

  if !game.didCurrentPlayerWin() {
    t.Fail()
  }

}

func TestIsOver(t *testing.T) {
  game.numMoves = 0
  if game.isOver() {
    t.Fail()
  }

  game.numMoves = 9
  if !game.isOver() {
    t.Fail()
  }
}

func TestConvertInputToCoordinates(t *testing.T) {
  row, col := game.convertInputToCoordinates(1)
  if row != 0 || col != 0 {
    t.Fail()
  }

  row, col = game.convertInputToCoordinates(5)

  if row != 1 || col != 1 {
    t.Fail()
  }
}

func TestSwitchPlayer(t *testing.T) {
  game.currentPlayer = "X"
  game.switchPlayer()
  if game.currentPlayer != "O" {
    t.Fail()
  }
}

One thing to note is that I had to modify the original unit test because the previous function no longer existed. What modifications did I have to make to the system tests? Well…

System Tests Enable Large Refactors

So we just refactored the original code to be more testable. This required the original unit test to be rewritten (or at least heavily modified). Guess what changes I had to make to the system tests? NONE. They are all still valid. (In fact, I used the system tests to validate that the new code works). Why do the system tests still work? They work because the client interface did not change. In this case, the client interface is text streams via stdin and stdout.

The same holds true for software where the interface is an API. As long as the API interface does not change, you can perform a large refactor on the code base and still be confident in your system tests.

In fact, I would argue that without comprehensive system tests, a large refactor is likely not possible in a reasonable timeframe.

Unit Tests are still important

The fact that system tests are superior to unit tests is no excuse to skip writing unit tests. Unit tests will generally find bugs earlier in the software lifecycle process than system tests. They can also easily be written during development whereas that isn’t always possible for system tests (though it was for our tic tac toe example). Unit tests also force you to write code that is unit testable which is a plus. So you should write both types of tests.

So why point out that system tests are more important if both types of tests should always be written? Let me put it this way. Consider the scenario where I start a new job on a new product. I look at the codebase and find that it isn’t very well tested. Maybe it has 50% unit test code coverage and one system tests that only tests the most common scenario. Where should my time be spent? Easy. I’m going to write system tests. A codebase that is in production in this situation probably mostly works. System tests will give more confidence at release time and allow faster iteration. Writing additional unit tests would give me more confidence that the code works as it is, but gives me little additional confidence as the code evolves. This is because evolving code often means evolving unit tests. The system tests will always be valid so long as the interface does not change.

The vast majority of software engineering teams combine on-call responsibilities with support. Hopefully, you’ll spend the vast majority of your on-call shift working support tickets and assisting other teams with their questions.

However, responding to alarms and outages takes precedence over all support activities. If you have to choose between keeping your service running versus helping Don from R&D figure out why his proof-of-concept project can’t access your service, then always choose to keep the service running. Don deserves your attention, but he can wait.

Make sure you keep your computer close to you. If you go out, make sure you understand how quickly you are expected to respond to a page. If it’s within minutes, you might need to find another engineer to cover for you while you go out.

Transient issues will alarm you, potentially often. A synthetic test might unluckily fail a few times in a row or latency might spike for a few minutes before resolving itself. These types of things happen on the Internet. In general, you should respond to the page when these issues happen and make sure that they get resolved. If it is 2:00 am and the issue resolves itself, then go back to bed and figure out the cause in the morning. Otherwise, it is good to check if you can determine what happened. Common causes for transient issues include:

• Hardware failure if this happens, then your service should automatically spin up new virtual machines or containers on new hardware. However, your service might be under provisioned during this time which can lead to increased fault rates and latency.

• Eventual consistency. Many workloads make changes that are eventually consistent. This can cause synthetic tests to sometimes fail. The reason for this is because a common pattern for a synthetic test is 1) do something 2) make sure that something was done. If the do something is eventually consistent, then making sure it is done might be done with polling. However, you don’t want to poll forever, so eventually you need to timeout. All of this can lead to you getting paged in the middle of the night.

• Deployments/updates. Maybe your service or one of your dependencies does not gracefully handle a deployment. For instance, it is possible that your deployment system might take too many nodes out of service at a time. This can lead to increased fault rates and increased latency because of constrained resources.

• Service Provider disruption. In this case, your cloud provider such as AWS or GCP or hosting provider might have a disruption. If you’re using AWS, maybe one of the availability zones where your service is partially deployed is disrupted. Your workload/service nodes should shift to a different zone. However, in the process you will likely get paged.

If a a severe service disruption occurs, and it isn’t apparent how to make the service recover within several minutes, then you should start an outage bridge or join the outage bridge that someone else created. As the on-call engineer, your primary focus is not to figure out why the service is disrupted though sometimes this will be either necessary or become apparent as the outage goes on. Your primary focus is to make the service recover from the disruption.

Your troubleshooting sequence should vary depending on what your service does and what the disruption is. However, this is the sequence I usually follow, and it generally works well:

1. Check if the application itself is working by signing in and testing the disrupted part of the service.

2. Go to monitoring dashboard. Does anything obvious jump out? Do differing metrics spike or dip around the same time?

   1. Note that a massive traffic spike can indicate a denial of service attack. Most engineers are not equipped to deal with this alone. If you suspect a Denial of Service attack, then you need to engage your network engineers and your cloud provider.

3. Check the metrics that caused the alarm. Do they show signs of recovery?

4. Did your team recently deploy a change that correlates with the time the issue started surfacing? If so, you should immediately begin the rollback process if it is safe to do so. If you are unsure, page the engineer that made the change and have them perform the rollback. Remember, recovering is more important than figuring our precisely why the disruption is occurring.

5. Query the logs for ERROR logs. Is the frequency of ERROR logs higher than usual? Most logging systems display a histogram which should make that apparent.

   1. Do the ERROR logs indicate that a database is disrupted? If the database is disrupted, contact your DBA (if you have one) to help troubleshoot. If you don’t have a DBA, then it is likely your team’s job to troubleshoot the database.

   2. Is a downstream service disrupted? If so, make sure that team is represented on the outage bridge. If they recently deployed a change that correlates with the outage, they should roll it back. Make sure the incident manager on the bridge understand the impact the downstream service’s disruption is having on your service. One thing you can do to possibly assist that team is provide them with the RequestIDs of requests to them that are failing.

   3. If the ERROR logs indicate that a third party service (like a payment system) is disrupted, then engage their support immediately. Create a support ticket with all the details necessary. Get them on the outage bridge if possible as well.

6. Check the primitive metrics such as CPU, Memory, and Disk usage. Additionally, check how many VMs/containers are active for your application. Was there a recent scale up or down?

7. At this point, if a path for recovering from the disruption is not apparent, then it is time to bring in other engineers from your team or even others teams to help troubleshoot.

If your service is completely disrupted and there’s not an apparent fix, then it might be time to enter what I call the danger zone. These are actions that you generally should NOT do as they might disrupt your service, but your service is already disrupted so it is slightly “safer” to do them. You shouldn’t do them alone. Make sure another engineer or manager has eyes on the actions. Here are some ideas:

1. Replace all the existing VMs/containers with new ones (for stateless services). This can usually be done pretty safely. Modern systems will drain existing connections and roll out the new nodes before the existing ones shutdown. This can fix your service because:

   1. The underlying hardware might be disrupted in a somewhat invisible way. Restarting nodes has a strong chance that most of the new nodes are placed on different hardware.

2. Restart the machines (for stateful services). You probably can’t easily just replace the nodes your database is running on. Instead, you can restart the machine. If you believe it isn’t safe to restart the machine, consider restarting the services running on the machine.

3. Touch the cloud infrastructure. One of the worst outages I have ever been a part of was the result of a cloud managed load balancer being partially deployed on bad hardware. We never figured this out during the incident but it became apparent later on. We got lucky. I recommended we change a configuration on the load balancer (without knowing the LB was the problem), and that solved the issue. Why? Not at all because of the configuration changes! It was because the LB was transitioned to new hardware during the process.

4. Emergency hot fix. Consider pushing out a code change that adds more logging or metrics that you believe might make the issue more apparent. Or, if a developer has a hunch on a fix that might fix the issue, consider pushing that out.

There’s no denying that outages are stressful. I hope some of these tips help you deal with them.

If you’re reading this post, it is possible you are about to start your first on-call shift or perhaps your previous on-call shift was rough. Maybe you’ve done on-call before, but you’ve never felt comfortable when working those shifts. You’ve come to the right place. If you follow these tips, you will be prepared for your next on-call shift!

Here’s what you you need to do before your first shift:

Review previous incident/outage logs. Most companies have their on-call engineers and incident managers write incident logs or an incident report whenever a major service disruption happens. Read through several of these. How did the on-call engineers solve the problem? How long did it take the service to recover? What was determined to be the root cause? Did your service come into play? Who was most responsible for correcting the issue? Which services tend to be “problematic”?

Perhaps your company doesn’t officially collect these logs. If that is the case, see if you can find the chat thread on Slack where it was discussed.

Talk to engineers involved in solving outages. Often times, the incident/outage logs don’t do a great job of spelling out how the engineer who solved the issue figured it out. So talk directly to the engineer who solved it! The vast majority of engineers will be happy to share this information with you because they will want others to be able to solve such problems. Consider asking these types of questions:

• How did you solve the latest outage?

• What are the most common types of outages at our company?

• What are the most important tools you use for diagnosing the service during an outage?

• What are the company expectations for escalating or starting a bridge?

• Do you have any general tips/advice?

As with anything, make sure you thank them for their time.

Shadow an on-call engineer. Whenever an incident/outage strikes, ask the current on-call engineer to immediately bring you into a call so that you can watch how they diagnose and resolve the issue. Hopefully, they’ll be able to explain what they are doing as they do it, but keep in mind that outages can be stressful so they might not be able to do that. That’s OK! Take what you can get. Keep the following in mind as you observe:

• How were they notified an incident has happened? Was it the alarms or something else?

• Which tools do they open first? Do you have access to those tools? You should.

• What do their Web browser bookmarks for on-call look like? Would it be useful to copy their structure?

• At what point do they open an outage bridge?

Join a few outage bridges. An outage bridge is a conference call where on-call engineers and incident managers meet to attempt to quickly recover from an outage (and other things, but those are beyond the scope of this post). Generally speaking, only people actively involved in correcting the outage should join an outage bridge. You don’t want too many cooks in the kitchen. However, joining for the purpose of learning how the process works is a valid exception. Join a bridge and just listen. Join another bridge and see if you can find the root cause before the other people on the bridge do.1 Don’t feel the need to speak unless you have something useful to say.

Know your Tools

Your tools assist you in troubleshooting an issue. Know what you have available to you before you begin your first shift.

Logs

Logs are human readable messages that your service emits whenever it does something. Logs are generally categorized by levels like INFO, WARN, and ERROR. INFO is something you expect to happen. WARN is something that you should maybe be concerned about, but also maybe not. ERROR means that something went wrong.

Usually, ERROR logs are the most helpful logs to look at during an outage. Before you begin your on-call shift, ensure that you know how to search for ERROR logs within a specific time window. Make sure you are comfortable working with date/time in UTC and a 24 hour time format.

You should hopefully have a log management system to help you search the logs. Most log management systems use SQL or a SQL-like syntax to query logs. You should study up on this. Put together a “querying cookbook” with example snippets of how to search your logs. Amazon CloudWatch Logs Insights has a good example here. You don’t want your first time querying the log management system to be during an outage!

RequestID

Logs should always include a RequestID. After you’ve searched for ERROR logs, look at the RequestID for one of them and query the log system to find all the logs associated with that RequestID. This will show you what the system actually did before it failed.

Make sure you know how to do this before your on-call shift begins. This will often lead you to the cause.

The Source Code

If you’re a software engineer, then you obviously have the of source code for your services available to you. Once you’ve looked at some logs, check in your code at the entry point and follow the call stack all the way down (or up depending how you visualize a stack in your mind). Are there any logs that aren’t getting emitted that should? Do you see perhaps where you’re not catching an evil null pointer exception?

If you’re an operations, DevOps, or Site Reliability Engineer then you might not have the source code readily available. I don’t agree with this approach, but many companies feel that for compliance and security reasons, this is a proper approach.

Metrics

Whereas Logs are human readable messages, metrics are human readable numbers that can be displayed. In operations, we have what are known as the four golden signals which are some of the most important metrics. I’ll cover three of them here:

• Latency measures how long it takes requests to complete. If you see that your requests are taking a long time, then that is cause for concern!

• Traffic measures how many requests your service receives. There are legitimate spikes in traffic like all your users waking up and using your service in the morning and nefarious spikes in traffic like a DDoS attack. However, even legitimate spikes in traffic can take down your service.

• Faults rate measures how often requests fail for unexpected reasons. This is distinguished from requests failing for acceptable reasons like passing incorrect parameters. Any fault is concerning. Faults are also known as errors or 500s.

Other metrics to collect are CPU, Memory, Disk, and VM/Instance usage. This is especially important if your application is running in a data center or on a legacy hardware. However, in modern systems these are generally the last place to look when an issue occurs. This is because modern systems are designed for hardware failure and will automatically terminate VMs/Containers that are misbehaving.

Before beginning your on-call shift, make sure you know how to view Latency, Traffic, and Fault for all of your services. Ideally, this information should be on a Dashboard.

Alarms

In the on-call world, the worst way to be notified of an incident or outage is by one of your customers telling you about it. This is why operations use alarms.

The metrics we covered in the previous section are very useful in and of themselves. However, they become much more useful if you hook up alarms to them when they cross a threshold. For instance, if you know that your service should only take 50 milliseconds to complete a request on average, then you should set an alarm to go off if the average latency stays above 60 milliseconds for a few minutes.

Make sure your alarms do not generate many false positives. In other words, don’t set them up to page your team for a blip. An exception to this could be if you have 24 hour coverage across many time zones, but even then you want the on-call to focus on actual problems and not transient issues.

Synthetic Tests

These test your service the same your consumers do and do so continuously, typically every minute. For a Web application, synthetic tests simulate user clicks to browse the Web applications and perform actions like logging in or purchasing an item. For an API, synthetic tests simply call the API and verify the results.

Synthetic tests should hook up to alarms. If a synthetic test fails (or fails a certain number of times above a threshold), the alarm should page the on-call.

Note that many services/teams do not have synthetic tests that run continuously. If your team doesn’t have them, then I highly recommend implementing them.

Paging System

Alarms should trigger a paging system that pages the on-call. Additionally, the paging system allows you to page your co-workers if you need assistance. Common paging systems include PagerDuty and Opesgenie.

Traces

Traces are a more advanced topic for on-call and will be covered more in-depth in a future post.

Your Application!

One of the first things I do when I get a page is to login to my application and see if I can reproduce the issue. For instance, if I am alarmed for high latency, I login to check if the application feels slow. If a certain feature is alarmed because it supposedly does not work, then I verify that.

The reason to do this is that sometimes the issue isn’t with your application. Instead, the issue is with the monitoring system! A non-working application that you have verified yourself is also a strong signal to start an outage bridge.

Dashboard

A dashboard displays your most useful metrics, logs, and alarms all in once place. If an alarm is triggered, then that should be apparent on your dashboard (usually it will be red and marked with an X instead of a green checkmark). So, make sure you know how to access your dashboards before starting a bridge.

If your team does not have a dashboard, you can create what I would call a “makeshift dash” by having handy links to each of your important metrics and alarms organized using bookmarks. Here’s a good bookmark hierarchy:

• Dashboard/

  • ServiceA

    • ApplicationURL

    • Latency

    • Traffic

    • Errors

    • Logs

  • ServiceB

    • …repeat

Chances are that other team members who do on-call might have this setup already in their bookmarks. Ask them if they could export their bookmarks!

Dashboards are also useful for correlating data. For instance, let’s say your dashboard contains both a time series chart for request latency and for traffic (number of requests). You get alarmed for request latency. You check the dashboard and see that both latency and traffic have spiked.

Chat Software

If your company discusses outage in a chat product like Slack, go back a few months and read every message up until the current date. There might be an #incidents channel or something similar to that you can check out. You’ll likely learn a few things from this.

Know your architecture and infrastructure

Is your application running in the Cloud or in a data center? If it is in the cloud, which public cloud are you using? If there is an issue with the underlying cloud services or data center, are you responsible for engaging the provider/vendor or is there a dedicated team for that? Does your team manager your application on Virtual Machines? Is your application containerized? Is it running on a Serverless compute engine like AWS Lambda?

Database

Are you using a Relational Database or a NoSQL database or both? Do you have database administrators that can assist during on-call? If there is a database issue, do you have metrics or logs that would indicate that?

Dependencies

Do your services interact with other services owned by other teams? Those are you dependencies! Make sure you know how to contact/page the other teams in case an outage is caused by them. Additionally, if your service is called by other teams, make sure they know how to contact you.

Do your services interact with 3rd parties that are in your application’s critical path? A good example of a common, critical 3rd party dependency is using a service like Stripe to handle payments. If so, make sure you know how to contact their Support Team in case they bring your service down.

Good luck!

On-call can be rough. However, going into your shift prepared makes it less painful. I hope this post was helpful for you!

Most companies do not have on-call responsibilities for their software engineers. Development teams and operations teams are often intentionally kept separate. However, top performing software companies do tend to have on-call responsibilities for software engineers. This is often the case even if those companies additionally have dedicated SRE teams (like Google).

If you’ve never been on-call and have no plans to join a team with on-call responsibilities, you might be tempted to think this post isn’t for you. However, this post is precisely for you. I hope to motivate you to join an on-call team or appreciate the one you are on and what it does for you.

Let’s distinguish between support shift and on-call. While most software engineering teams do not have on-call, many of them do have support shifts. When assigned to support, the engineer is responsible for working tickets issued to their team from other teams. The difference between a support shift and on-call is that support shifts do not page engineers in the middle of the night when the service is down.

You should consider joining an on-call/True DevOps team for these reasons.

On-call software engineers write more operationally efficient code. The working relationship between an operations org and a development org often ends up in a vicious cycle. Operations teams are frustrated because they are constantly up all night dealing with poorly planned upgrades and outages, and development teams are frustrated from being pestered with requests from ops.

A software engineering team that is responsible for all of their on-call responsibilities is a true DevOps team. The engineers on these teams write more operationally efficient code simply because they don’t want to be woken up in the middle of the night to respond to a page.

Tips for writing operationally efficient code will be covered in a later post.

On-call will improve your troubleshooting skills. This will be the topic of a future post, but it is inherent that software engineers who wrote the code are best equipped to troubleshoot it when things go wrong. Beyond that, on-call will expose you to metrics, alarms, monitoring, log diving, etc.

On-call gives you exposure throughout your organization via the outage bridge. If you’re an entry or mid-level engineer, then it is possible that the outage bridge could be the primary two-way interaction you have with your upper management. Furthermore, it is likely that people from adjacent teams will be on the bridge as well. This probably sounds scary (and it can be), but it is a huge opportunity. If you can be the engineer who people depend on during these bridges, then that could very well lead to advancement or improved opportunities.

It could even make you less likely to be laid off or fired. Often times, specific layoff decisions are made by management in a group setting. If your name comes up, management probably isn’t going to cut the people best at keeping the service running even if you’re one of the weaker coders on the team. A different manager might even vouch for you solely because of that.1

Top software companies tend to have an on-call rotations for software engineers. The reason for this is that true DevOps/Engineers on-call is considered a best practice (for many of the reasons in this post actually). Top software organizations and startups tend to pay well, so it is worth becoming comfortable with on-call responsibilities if you wish to join one.

On-call makes software engineers more familiar with IT Concepts tangental to coding. I’m not going to lie, before joining on-call teams I had very little exposure to many IT concepts and tools that I now consider to be vital to my career. Things like practical computer networking, deployment automation, load balancers, cloud, Linux, database optimization, configuration management, firewalls, etc were things that I had only superficial knowledge about. Understanding the environment my software runs in greatly affects the way I write code.

On-call isn’t for everyone. I’m not sure I’ve ever met anyone who loves being on-call. Getting paged at 2:00 am is only the start of the pain. On-call makes our profession more similar to that of being an ER Nurse/Doctor trying to quickly fix a problem without making it worse in the process (except usually people’s lives are not on the line in our profession). Outage bridges are undeniably stressful, especially when your leadership joins and asks difficult questions. Fixing a live production service disruption that you caused is more humbling than receiving a bug ticket the following morning when your Ops counterpart runs into it. Maybe you have a strict sleep schedule that can’t be disrupted.

There are ways to alleviate some of these hardships that I’ll cover in a future post.

When my daughter was born, I had to slow down my Sheep Code posts. My development team was also working toward a product launch, so I had very little mental space to allow me to write. However, with the New Year here, I am ready to pick things back up at full speed. Here’s what to expect this year.

Name Change/Rebrand

The name “Sheep Code” was never meant to be the permanent name of this newsletter. I plan on rebranding it to a new name this year. However, I still have not thought of a name! If you think of anything, please reach out.

Series

To make my posts a bit more navigable, I’m going to categorize posts into series. Names are pending, but here is what I have so far.

• DevLife - these will be short posts about my work as a software engineer and other non management roles (like DevOps). I’m going to lean heavily into my individual experience on these posts.

• ProCode - infrequent, longer posts focused on exemplary software engineering. Some posts will contain code and diagrams.

• TheMgmt - these posts will focus on management at a tech company. The primary audience is for managers, people seeking to be in management, or those curious about what managers do at tech companies.

Posts in a series will have one of those as the start of the title followed by a counter integer. For instance, the first DevLife posts might be titled “DevLife #0 My First Internship”.

Free posts

The vast majority of my posts will not require a paid subscription. Any content I write specifically for beginners or people looking to get into tech will always be free. DevLife posts will almost always be free. ProCode posts will usually be at least partially free.

Paid posts

I do plan to write posts that require a paid subscription to access the full content. Posts where managements/executives are the primary audience are more likely to be paid. If a post requires a paid subscription I will not send an email to the free subscribers about it unless the post has free content in it that is useful. I don’t want to spam your inbox!

What I won’t write

I will never write posts that are overly critical of specific people or companies. Any negative examples I use will be annonymized. I don’t write gossip pieces either. I do appreciate other writers that ethically criticize companies, but that isn’t what this newsletter is about.

Should you become a paid subscriber?

At this point, I have not written any paid content. Thus, you should only become a paid subscriber now if you wish to support me and have the financial means to do so. My paid content will be directed at professionals and management. If you’re a beginner, please don’t feel like you need to become a paid subscriber! If my posts help you break into the industry, then once you get a job perhaps consider paying then.

Sneak Peak

I have a good number of unpublished drafts that I plan to finish this year and send. Here are some of them (titles will likely change):

• DevLife #?: How does on-call work?

• DevLife #?: Getting the most out of 1-1s

• DevLife #?: Navigating the dreaded outage bridge

• DevLife #?: Balancing Security and Risk

• DevLife #?: Devastating Angry Interteam Dynamics

Why do you want to learn to code? Do you have time to learn how to code? Do you have any friends who know how to code? Or do you have friends that also want to learn how to code? Does your current employer have coding needs where you can get practical experience?

Most people who ask me about learning to code are ultimately seeking to gain employment in IT/Tech. They usually know at least one person who knows how to code (they know me after all), but often don’t know anyone else looking to learn (sometimes they do). Most of them are not working in jobs where it wouldn’t be easy to apply their coding skills. Almost all of them dramatically underestimate the level of effort most people need to learn programming.

Most them don’t know that IT/Tech consists of a whole lot more than just programming/software engineering. But there are other ways to get into tech than just knowing how to code! I firmly believe that coding is the most important skill for anyone in IT/Tech, but a systems administrator doesn’t need the same level of proficiency as a software engineer. A support engineer doesn’t necessarily need to have strong coding skills. A DBA mostly uses SQL. A product manager often doesn’t need to know how to code at all. A cloud administrator probably spends more time reading documentation and writing short scripts than full software. etc.

So before I recommend a programming language to start with, I often recommend taking a different path in tech than getting straight into software engineering. I encourage them to get into the industry in a role where coding skills are secondary and grow from there.

But if they’ve determined that they want to learn to code, here is some additional information I tell them.

The most important aid you can get in learning to code is a mentor. Or even better find several mentors. You should reach out to your mentor when you need help. There’s a science to asking for help though. I’ll write a post later about this, but it basically boils down to “state your goal” and “try to be specific” when stating your problem. The reason for this is that your mentor is busy. Open ended questions take time to answer, but answering a specific question can often be written in 10 minutes or less and sent over email. On the other hand, a specific question with no context can be frustrating to answer because maybe the wrong question is being asked.

More importantly, a mentor can help you direct your growth. Most technical knowledge is valuable, but some knowledge is way more valuable for you than other knowledge. A mentor can help you choose the best path.

The second most important aid you can get in learning to code are learning peers. These are people that are on the journey with you. These people will help keep you sane during the process. One thing to remember though is everyone learns at a different pace. If your peers are ahead of you or picking up stuff quicker than you, do not let that bother you. In fact, peers who are ahead of you can become mentors. That’s a good thing!

Completing practical projects is the best way to learn programming. Notice I didn’t say this should be the only method. It also shouldn’t even be your first method. Before trying to tackle a project, you should first go over the introductory tutorial for your chosen programming language. You should then dive deeper into the documentation. Test what you learn with small snippets of code. Watch a Twitch stream of someone coding with your chosen language. After you’ve done all this, then it is time to pick a project and see it to completion.

At this point, you might notice that this method of learning suspiciously resembles school/college. Well, yeah it does. College is a good (but expensive in both time and money) way to learn. Self-driven learning is much cheaper in terms of money, but not everyone can do it.

Unfortunately, many people looking to get into IT/Tech don’t have friends that are already in the industry or looking to get into the industry. So you might think much of the above doesn’t apply to you. But it still does!

Find online communities and influencers1 that you enjoy and provide value to you. Find virtual peers and mentors this way. Follow tech bloggers like me. Twitter/Slack/Discord/Telegram groups that focus on IT/Tech are great resources. Traditional forums are often great too.

So with all this said, I feel that I am ready to sort of answer the question of which programming language to start with. If know someone who is willing to mentor you, pick a language they know well because that is what they’ll be able to best help you with. If your have a peer group that leans toward a particular language, pick that language! If you find an online community that you find compelling, choose a programming language that community prefers.

Subscribe now

If you still haven’t figured out which language to pick, here are some suggestions sorted by focus. I also list some languages I recommend beginners to avoid.

Data Science

Choose one of these languages if you work with datasets.

• Python - Great language for data science. Numpy and Pandas are your friends and so is Jupyter notebooks. I would recommend Python over R because Python is used widely in the industry and not just for data science. Honestly, Python could show up in every category.

• R - statisticians love R and use it. RNotebooks is pretty cool too. I used it a good bit while I was studying at Georgia Tech. Ultimately, if you’re getting into data science, you’ll need to know R.

Software Engineering

We write large software products and services.

• Java - boring but reliable OOP language that is EVERYWHERE. Java has a ton of language features, so mastering it will teach you almost everything you need to know about programming (except for pointers). Will also teach you to fear Null. Once you’ve become competent with Java you should consider moving on to Kotlin.

• Go - simple language that will force you to think about error handling and introduce you to the dangers of pointers. Excellent language for learning the basics of programming because it doesn’t give you as many shortcuts that languages like Python do. Go is probably my favorite language, and I miss it.

• SQL - all software engineers need to know how to work with databases. Sure, NoSQL databases are an option these days but SQL can even be used with DynamoDB. SQL is a great second language to learn.

• Avoid C/C++ people will hate me for this, but you shouldn’t start with these languages. A vast amount of incredibly useful software is written in these languages, and you should consider learning them if you want to become an open source contributor to incredible projects like the Linux kernel or Git. But these languages are hard.

Game Development

Game developers create video games. I actually don’t know much about game development, so I’ll do my best with this one.

• yiiiikes. C++ is apparently the most common language used for game programming. I am so sorry. Maybe choose Web development instead?

• C# - a good alternative and similar to Java. C# is used for programming in Unity. It is much easier to write than C++. C# was also the first programming language I learned, so it holds a special place in my heart. Please start with C# instead of C++.

Linux Systems Administrator

A linux systems administrator usually is writing scripts that are under 10,000 lines long. Often, they are much shorter.

• BASH - the bourn again shell. Anyone who wants to administer Linux systems should learn bash. Looping is painful, but piping commands together is awesome.

• Python - you’ll use Python for any task too frustrating to complete with BASH (or at least that is what I do).

• Avoid Perl. I don’t know how this language happened, but it is ugly.

DevOps Engineer

A DevOps Engineer deploys, updates, and monitors live software services. They also tend to get paged at 2:00 am.

• BASH - All DevOps engineers need to know some BASH.

• Ansible - not really a language, but it sort of is. Lets you make your BASH scripts run on thousands of machines.

• Go - Terraform, Docker, and Kubernetes are both written in Go. Chances are, you’ll extend one of these and doing so will require you to write Go. Go is also simple enough that learning it is much easier than complicated languages like Java.

• Python - seeing a pattern yet?

Web Development

Web developers make Websites. This includes the code that runs on the frontend and on the server. A couple of languages are widely used for both.

• JavaScript - all the code running in your browser is JavaScript. Additionally, JavaScript is widely used for server side development as well.

• TypeScript - makes JavaScript better and safer. It’s probably better to start with JavaScript though and learn TypeScript as a second language. Anything JavaScript can do, TS can do better!

Database Administrator

A DBA works with relationship database management systems like Postgresql or MySQL.

• SQL - has many variants, but the base language is generally the same or similar. Watch out for how different SQL variants use quotes.

I hope you enjoyed reading these suggestions. Python of course appears the most on the list. If you want to learn a language and be effective with it fast, then Python is probably a good bet. However, there are many programming language features and concepts that Python will not teach you. Things like pointers, memory allocation, static types etc just don’t exist in Python. A language like Java would give you a more complete experience.

One thing to remember is that you don’t have to wait until you master a language before learning another one. Instead, focus on becoming proficient. Once proficient, there’s no reason to not try picking up another language. I guarantee you the second one will be easier than the first (unless you started with Python and then try C).

Once you’ve picked up a few language, you’ll want to start learning about Design Patterns2. Or, you might read up on why many Design Patterns are now much, much easier to implement due to functional programming concepts becoming so ubiquitous.

I’ve known Forrest for a decade or so now. He has something for everyone.

Looking to get into IT/Tech? Check out the Cloud Resume Challenge. I have worked with several really outstanding individuals who got into DevOps by completing the challenge.

Do you need a good laugh? Go look at his cartoons.

Are you longing to listen to someone make all of the AWS services into a song? He’s done that too.

He also writes serious posts sometimes, but you can just follow me for those, right?

For brevity, when I refer to AI, in this post I mean large language models like ChatGPT.

Subscribe now

Here are my predictions:

AI assisted programming will result in increased efficiency for programmers who embrace it. AI will bootstrap new projects. It will help you find bugs. It will implement algorithms for you. It will write code much faster than you can. It will document your code. It will explain your code. It will write tests for you. It will improve your code. etc. Amazon CodeWhisperer and Github Copilot were just the beginning.

AI assisted programming will not reduce the need for programmers, but it will reduce the demand for programmers who refuse to use AI. Programmers who utilize AI will have advantages over those who don’t. If an AI can generate unit tests for your application in seconds what would take you an hour, then you should take advantage of the AI. I personally do not think there will be many (if any) types of programming where AI will not be of great assistance. Think of it this way. When high level languages like FORTRAN were released, did it kill the demand for assembly programmers? Yes and no. Very few companies these days have any need for assembly programmers, but some do and pay very well. Meanwhile, pretty much every software company needs programmers who write in high level languages.

AI won’t be able to deliver complex end-to-end applications for a long time but when it can, product managers and designers will become programmers. By end-to-end, I mean applications that can be fully written and incrementally improved by AI with very minimal human programmer intervention. By complex, I mean anything more complicated than a CRUD application with a frontend slapped onto it.

Fully managed platforms for AI generated applications will become common. This is the shortcut to AI being able to deliver end-to-end applications quickly. Companies will design platforms with composable parts with a natural language interface to create applications and run them on the platform. The tech is already to the point where this should be possible so I expect to see it very soon.

AI competition will be fierce. OpenGPT will not be the only player. There will be a community of AIs with many tuned to specific domains and at differing price points. I imagine annual competitions will AI will compete on problem sets specifically designed to be challenging for AI.

Utilizing AI will be a skill. The existence of AI won’t mean that everyone will get the same value out of it. Think of it this way. Learning how to efficiently get accurate information on the Internet is a skill that takes time to develop. Google made that easy, but even crafting a Google search query that will give you the results you want is a skill. You’ll need to recognize when the AI is wrong. You’ll need to understand where AI is better than you and where its limitations are. Additionally, selecting which AI to use for certain tasks will be important.

Educators will adapt to AI availability. Just as Google search made it much easier for students to cheat, so will AI. That said, educators will develop teaching methods that both allow students to learn from AI and prevent students from using AI to cheat. Will verbal exams become a thing again? As an aside, I graduated from Georgia Tech’s OMSCS program back in 2018 (I think!). Ironically, one of the most difficult courses I took there was “Artificial Intelligence”. The final exam in this class was a take home exam that took me a week to complete. However, I am confident that OpenGPT could have solved all of the problems for me. So I think that AI will present challenges to online education.

I hope you enjoyed this post!

For instance, I visited the following pages every day:

• https://aws.amazon.com/new/

• https://cloud.google.com/release-notes

• https://www.hashicorp.com/blog

In my current role, I have less need to visit these pages every day, but I still do every once in a while. Today, on GCP’s release notes page, I saw this change:

These Cloud SQL updates GCP made stuck out to me because the previous functionality was a source of frustration for my team. Let me explain.

Previously, deleting a Cloud SQL Instance did not release the Instance name from whatever GCP’s backend name pool is. Why? I have no idea, but I suspect GCP might wisely keep your database around for a while after you “delete it” just in case one of their big customers has second thoughts1. Maybe GCP wants to be the hero when they get the following message from a customer:

“Hello Google, we didn’t actually mean to delete our production user database that we forgot to take backups on. Is there anything you can do? Also, our spend is $5 million a month with you”.

So if this is wise of GCP then why was it frustrating for my team?

Consider the case where you are a cloud engineer. Part of your job is developing automation and testing that it works. This might involve running and re-running Terraform commands such as plan, deploy, and destroy. You might run this on the following snippet to create a database instance called receipts:

resource "google_sql_database_instance" "main" {
  name             = "receipts"
  database_version = "POSTGRES_14"
  region           = "us-central1"

  settings {
    tier = "db-f1-micro"
  }
}

Straightforward and not so bad! However, once you destroy it and run it again, you’d get something like this message2 after running Terraform plan/apply:

Error: database name "receipts" already exists

Do you see our source of frustration now? Once you use a name, it sticks around for a while after being deleted which means you have to pick a new name each time you fully test the automation! To workaround this frustration, we’d do stuff like this:

resource "random_id" "name" {
  byte_length = 8
}

resource "google_sql_database_instance" "main" {
  name             = "receipts-${random_id.name.dec}"
  database_version = "POSTGRES_14"
  region           = "us-central1"

  settings {
    tier = "db-f1-micro"
  }
}

This makes the code only slightly more complicated and your database instance names won’t be as clean. Like, I would rather look at receipts in the console than receipts-37563856!

But now GCP has fixed this…two years late to make my life simpler, but it’s good to see they fixed it.

So why write about this frustration of mine? I hope you’ll keep reading…

One of major differences between an experienced engineer and someone new to the field is that the experienced engineer will have gone through problems and frustrations like the one above. And of course they find workarounds to those problems. You really can’t easily learn these things without experiencing them for yourself because they are rarely present in the official documentation or training materials. It is often simple to Google the answers to these problems but not always. This is why real world experience is better than certifications and training.

I’ll also add that finding a workaround to a problem will help you solve similar problems in the future or avoid them altogether. From our example, the “workaround lesson” is that uses suffixes in your cloud resource names can help you avoid problems!

Anyways, your troubles will vary based on whatever technology you use. Here are some from my experience that you might find fun or had the misfortune of experiencing yourself:

• Kubernetes - if you’ve used Kubernetes, then you’ve almost definitely been burned by the now deprecated PodSecurityPolicy. This monster caused us SO many headaches at my roles where we used Kubernetes.

• Terraform - ever tried looping over a provider?

• AWS CDK - Stack Exports for inter-stack references are magic until you have to remove one.

When interviewing an engineer for a cloud engineering role, one of my favorite questions is “what frustrations have you had with <insert-tool-here>”? If an engineer has no frustrations with a tool, then they probably don’t have much experience with it. It’s also feels pretty validating when I hear people have had the same problems I have had but maybe that’s selfish.

I’m not going to go over study tips in this post. There are plenty of prep resources elsewhere. You can see above the general order in which I took the exams though. One frustration I experienced through my certification journey was the “moving target”. My goal was to obtain ALL AWS certifications which I think did a couple of times, but AWS kept releasing new certifications! I eventually moved on to obtaining Kubernetes and GCP certifications. You can see many of my certifications (and past certifications) here if you’re interested.

So yeah I’m one of those obnoxious1 10x cloud certified people you see on LinkedIn from time to time (or at least I used to be)!

So was the cloud certification hype real?

For my compensation, the hype was real. After achieving 8 or so AWS certifications (I eventually got 10), my next salary (from switching jobs) increased by 50%. I had multiple comparable offers like this. Additionally, one company I worked for started offering large bonuses (5k-10k) for each additional certification achieved. So getting certified can definitely improve your compensation though it is not guaranteed. You still have to successfully interview!

For my knowledge, the hype was real. Before beginning my AWS certification journey, I worked with AWS everyday while wearing multiple hats (Admin/Dev/DevOps/etc). My team administered a non-production AWS account, and we also worked on our company’s CI/CD tooling. I already knew a lot about AWS! But getting certified forced me to learn more. This is especially true if you take certifications that are less related to your role. For instance, I’ve never worked in a Big Data role so getting the “Big Data” certification (rip2) forced me to learn about Hadoop.

For job hunting, certifications can be helpful. Many companies take cloud certifications very seriously (while some do not care at all). I was actually shocked by how easy some of the interviews were. Interviewers often assumed I was cloud competent so they didn’t really ask technical questions and instead focused on whether I was a cultural fit or not (this is scary btw! As an interviewer don’t do this!). On a side note, cloud consulting companies take certifications more seriously than most anyone else because having certified employees is often part of their brand.

Certifications can streamline an interview process at some companies. Think of it this way. DevOps/Cloud Center of Excellence/Cloud Engineering teams require engineers with broad experience and set of skills. One of the difficulties interviewers face is figuring out how to fairly interview a candidate based on their resume. For instance, a Cloud Center of Excellence team needs people with software engineering, linux administration, cloud knowledge, DevOps, CI/CD, infrastructure as code, IT knowledge, etc. Most candidates will only have some of those qualifications. As an interviewer, it is more fair to focus on what the candidate claims to know for these types of roles. When an interviewer sees certifications on a resume, they immediately can ask questions that a certified individual should know. This makes it easier for an interviewer to interview you!

Certifications do not replace experience. I would usually prefer to hire a candidate with experience than a certified candidate with no experience. There’s no replacement for actually using the technology. However, you CAN get experience outside of work by doing personal projects or challenges! I highly recommend the Cloud Resume Challenge if you are in a situation where you have certifications but no real world experience. Not all employers will count this as relevant experience, but some will (and should IMO).

I’m personally “done” with certifications

Regardless of all these benefits, I do not plan to renew most of my cloud certifications. Why? Keep reading.

Recertification sucks. Since I achieved most of my certifications within a few months of each other, I basically had to re-certify most of them at around the same time. I did choose to re-certify the AWS Professional exams. The rest I let lapse or am going to let lapse. My GCP Professional Architect certification expired this past month in fact! For me, re-taking an exam every few years that I have already passed in the past (heh) is uninteresting. I would prefer to focus my time on my work and family than taking exams.

I’ve reached diminishing returns. While pursuing more certifications could improve my knowledge, it isn’t likely to improve my compensation. Furthermore, any additional IT knowledge I need, I can simply gain by reading the docs and trying out the tech.

Beware of fraudulently certified people

I have interviewed hundred of candidates throughout my career for software engineering, cloud engineering, DevOps engineering, Cloud Architect etc. Often, I interviewed candidates that fall into one or more of the following buckets:

• claim to be certified but can’t prove it

• are certified but can’t answer basic cloud questions

Unfortunately, cheating is rampant for certification exams. Cheaters either get someone else to take the exam for them or study “exam dumps” that have exact questions that another test taker copied directly from the exam. Either case can lead to an individual being fraudulently certified.

However, it is pretty easy to identify these people during an interview by asking basic cloud questions. For instance, every AWS certified individual should know the following:

• The difference between an AWS region and an availability zone

• Be able to simply explain what S3 is

• What IAM Roles are used for

A certified individual should know a lot more than these, but if they can’t answer these then that is an indicator that they could possibly be fraudulently certified.

AWS Certifications for an SDE at AWS

So is being AWS certified important for working as a software development engineer at AWS?

Being AWS certified did not directly help me become an SDE at AWS. The interview process for an SDE is focused around problem solving with code, system design, and leadership principles. The interviewers do not assume the candidates know anything about AWS and rightfully so because not every software engineer has experience with AWS!

The knowledge gained from being AWS certified has tremendously benefited me at AWS. The reality is that I was able to onboard and become a (hopefully!) productive member of my team much quicker because of the certifications I achieved. Many teams at Amazon (including mine) leverage AWS services extensively. Services/tools such as CloudFormation, EC2, CDK, DynamoDB are very important to know. I would go as far to say that SDEs at AWS who don’t learn the core services that their team use will struggle to be effective members of their team.

Miscellaneous Confessions

These are fun.

You might be able to pass some exams with “minimal prep”. AWS used to have an “Alexa Certified Skill Builder” exam. A few of my friends and I signed up for it on the first data of its beta. Instead of doing extensive prep for it, we crammed for it the night before and shared notes before taking it.

We all passed.

In reality, the “Alexa Skill Builder” certification could have been called the “Serverless” certification + Alexa. We passed because we used Serverless at work extensively. So you could say that our work experience was the true prep.

I tried doing the same thing with the Machine Learning AWS exam. That didn’t go so well, but fortunately Amazon gives a free retake if you fail a beta exam :D (or at least they did at the time).

Adding a certification badge to your Outlook signature might make people respect you more. Or it might not. I did this after getting my first few certifications, and I feel like I got less pushback whenever asking engineers to tighten up their IAM permissions. However, I also was in a meeting where someone rolled their eyes at this, so some people might find it obnoxious. I guess just know your company culture before adding the signature.

I usually think I failed at the end of every exam. I don’t know how this can be since I usually get pretty good scores. But the moment I hit the “submit” or “end test” button, this moment of dread fills my soul. Did I just waste $300 of my employer’s cash? What if I fail and someone asks me how it went? And speaking of which…

Surveys at the end of exams before scores are shown are HELL. Please stop doing this test administrators! I know people are more likely to take the survey if they can’t see their score without it. However, my heart rate is faster while taking those surveys than during the final minute of Bama’s football game vs the Texas Longhorns a couple weeks go.