SES Template Drift: When Git and Live Don't Match

The first time most teams notice drift is when somebody redeploys.

The story is always the same. An engineer ships a small unrelated change to a template. CI runs. UpdateTemplate fires. The next morning, the team gets a customer email asking why a fix that "you told me was deployed three weeks ago" is suddenly broken again.

It turns out a different engineer had hand-edited that template in the SES Console during an incident two weeks earlier. The Console fix wasn't in Git. The redeploy from main overwrote it. The customer-facing bug is back.

This is template drift. Drift is the silent failure mode of "we manage templates as code" when the AWS Console remains an editing surface. It's almost universal in teams that started with the Console and added Git-based workflows later. Most teams don't realize how much drift they have until they go looking.

What drift is, and the typical first cause

Drift is the gap between two things you expect to be the same: the template definition in your source of truth (Git, IaC, a control layer) and the template definition currently live in SES.

The first cause is usually defensible. A customer reports a typo at 4pm on a Friday. The engineer on call opens the SES Console, fixes the typo, and saves. The fix is live in 90 seconds. The engineer plans to "open a PR on Monday to backfill the change in Git." Monday becomes Tuesday becomes never.

Drift can also start with even less drama. A new hire is exploring the Console. They click into a template, change a word, and save. They didn't realize it was the production template. Nobody else notices. The change persists.

The trigger is human. The mechanism is structural: the Console is a fully-functional editing surface for a resource you said your CI pipeline owned, and AWS does not enforce single ownership.

Why drift compounds

Drift would be a minor annoyance if it stayed local. It doesn't. It compounds.

The second engineer doesn't know the first engineer's fix exists. They open the file in Git, see the typo'd version, and assume Git is canonical. They make their change against the typo'd version, open a PR, get it reviewed, and merge. CI redeploys. The first engineer's fix is gone.

Now drift has escalated. Not only is Git out of sync with SES, but the team has lost trust in their own process. The next engineer who fixes a template in the Console will be even less inclined to "trust the pipeline" — they've seen pipelines silently overwrite real fixes. A culture of out-of-band edits sets in. Every change becomes a coordination problem.

The deeper failure is that drift erodes the assumption that lets engineering work in the first place: that the source of truth is the source of truth. Once the team has felt drift bite once, every change to a template carries an unspoken question — "are we sure this matches what's in production?" — and the only honest answer is "let me go check."

Detecting drift

You can't fix what you can't see. The first investment is detection.

The simplest detection is a scheduled job that compares Git to SES.

import boto3
import json
import pathlib
import sys

ses = boto3.client("sesv2")
templates_dir = pathlib.Path("templates")
diffs = []

for path in templates_dir.glob("*.json"):
    expected = json.loads(path.read_text())
    name = expected["TemplateName"]

    try:
        live = ses.get_email_template(TemplateName=name)["TemplateContent"]
    except ses.exceptions.NotFoundException:
        diffs.append((name, "missing in SES"))
        continue

    if live.get("Subject") != expected["Subject"]:
        diffs.append((name, "subject differs"))
    if live.get("Html") != expected["Html"]:
        diffs.append((name, "html differs"))
    if live.get("Text") != expected["Text"]:
        diffs.append((name, "text differs"))

if diffs:
    for name, problem in diffs:
        print(f"DRIFT: {name} - {problem}")
    sys.exit(1)

Run this nightly against production. Have it post to a Slack channel when it finds drift. Don't run it as a hard CI gate on every PR — that creates incentive to dismiss it. Run it as a daily report and treat each drift event as worth investigating.

The signal you want is twofold: count of drifted templates (how big is the problem) and time-to-detect (how long between a Console edit and the drift report catching it). The second number is the more important one. Drift caught within 24 hours can be reconciled. Drift caught three months later is a small archaeology project.

Preventing drift, structurally

Detection alone is reactive. Prevention is the goal.

There are three serious approaches, and you almost certainly want a combination.

Make the Console read-only in production. This is the single highest-leverage change. Use IAM and SCPs to deny ses:UpdateTemplate and sesv2:UpdateEmailTemplate to every principal in the production account except the deploy role. SSO users, including admins, get read access only. Break-glass admin access is gated behind a documented procedure with retroactive review.

This is the structural fix. Once humans cannot edit production templates from the Console, the most common drift mechanism is impossible by construction.

The objection is always: "but what about incidents?" The answer is: your incident response procedure includes break-glass access that is logged, time-bound, and reviewed. The Friday-afternoon Console fix becomes "open a PR with the fix, mark it [hotfix], get a quick review, merge, deploy." If your CI takes 20 minutes and you can't tolerate that during an incident, that's a CI problem, not a drift problem.

Reconcile automatically. Some teams choose to live with the possibility of Console edits and have CI continuously reconcile SES to match Git. The reconciliation runs every 30 minutes — if SES drifts away from Git, the next run pushes Git's version back. This treats Git as authoritative without forbidding Console edits.

The downside is real: a hotfix in the Console gets silently overwritten 30 minutes later, which is the failure mode I described in the opening. Reconciliation must be paired with a notification — the moment SES drifts from Git, somebody is told, before the reconcile runs. This way, the engineer who hotfixed in the Console gets a Slack message saying "your edit will be reverted in 14 minutes unless you backfill to Git."

Move the editing surface. A control layer like Sovy eliminates drift by removing the Console as an editing surface entirely. The control layer is the source of truth. It owns the SES API call. The Console becomes a read-only view of what the control layer has published, and humans don't edit there because there's nothing to edit. Drift cannot occur because there's only one writer to SES, and that writer's state is what the control layer holds.

This is the cleanest structural answer for teams that want non-engineers to be able to edit templates. Git-as-source-of-truth requires non-engineers to learn Git or accept being unable to make changes. A control layer can offer non-engineers an editing UI while still maintaining single-writer discipline.

Recovering from existing drift

If you're reading this and you suspect (or know) you have drift, the recovery has a specific shape.

Step 1: Inventory. Run the detection script. Capture every drifted template. Don't try to fix anything yet.

Step 2: Identify the truth for each one. For each drifted template, the question is: which version is correct — Git or SES? The answer depends on the template's history. Look at recent CloudTrail events for the template. Look at recent commits to the file in Git. Most often, the SES version is correct because it includes a hotfix that was never backfilled.

Step 3: Backfill Git from SES, not the other way around. If you redeploy from Git in this state, you will overwrite real fixes that exist only in production. The right move is to update the Git file to match the current production SES content, with a commit message that explains "backfill from production: includes hotfix from <date>." This converges the two without losing the fix.

Step 4: Get review on the backfill. A backfill PR is the right place to also evaluate the hotfix's quality. It was rushed; it might have unintended side effects; it might be the right place to apply a more durable fix.

Step 5: Then deploy normally. Once Git matches production, your normal pipeline is safe to use again.

Don't shortcut this process. Teams that try to "just redeploy from Git and be done with it" reintroduce the bugs that caused the original hotfixes.

The cultural side

Drift isn't just a technical problem. It's a cultural one.

Teams that have drift have, at some point, made it acceptable to edit production templates outside the documented process. The incident might have been small. The reason might have been good. But the precedent gets set. The next person to hotfix in the Console isn't breaking new ground — they're following local custom.

The fix is partly structural (read-only Console, single source of truth, single writer) and partly cultural. The cultural part is making it normal that template changes go through the same pipeline as code changes, including during incidents. Including on Friday afternoons. Including when the typo is embarrassing.

A team I worked with had a useful ritual: every drift report that fired in Slack triggered a small retrospective. Not a blameful one — just "what happened, what was the urgency, would we make the same call again, and is there a process improvement that would have made the right call easier?" Most of the time, the answer was that the CI pipeline was slow, or the PR template required too much ceremony for a one-line copy fix, or the on-call engineer wasn't sure they had permission. Those are fixable problems. None of them are "the engineer was reckless."

The drift count went down. Not to zero — but to a level where each occurrence was noteworthy and discussed. That's the steady state to aim for.

A useful checklist

Before you ship anything else this quarter, walk through these:

• Do you have an automated check that compares Git (or your IaC, or your control layer) to live SES, and runs at least daily?
• When the check finds drift, who gets notified, and within how many hours?
• Is the SES Console read-only for production templates for non-break-glass principals?
• Do you have a documented break-glass procedure for production template hotfixes that produces a retroactive ticket and review?
• When was the last time you ran the drift check and got a clean report? If you don't know, run it now.
• Is your CI pipeline fast enough that engineers don't feel pressure to bypass it during incidents?
• Do non-engineers (marketing, support, lifecycle) have a sanctioned path to edit copy that doesn't involve the Console?

The last item is where most teams fail. They lock down the Console, build a tight Git workflow, and then hand a Slack message to engineering every time the lifecycle team wants to change a string in a renewal email. After three months of friction, someone gets quietly given Console access "just for marketing." Drift returns.

If your answer to "how does marketing edit copy?" doesn't include a path that isn't the Console and isn't engineering bottlenecking, you don't have a complete drift prevention story. That's the gap a control layer fills, and it's the gap that hand-rolled GitOps doesn't.

---

Sovy is a control layer for Amazon SES templates. It eliminates drift by being the single writer to SES, while offering non-engineers a sanctioned editing surface. If you have a drift problem you've stopped trying to fix, we'd like to hear from you.