JSTGTECH

Cloud roundup: macOS Screen Sharing bug now under attack

Sun, 16 Aug 2026 00:00:00 GMT

Two security items today, both worth checking against your own fleet rather than waiting for the next patch cycle, plus an AWS monitoring feature that quietly kills a homegrown workaround a lot of us are running.

macOS Screen Sharing flaw is being exploited to plant Monero miners

CVE-2026-65400, a bug in macOS's built-in Screen Sharing (VNC over TCP 5900) that let a network attacker connect without valid credentials, is now under active exploitation. The Netherlands' NCSC says it's seen abuse on multiple systems with port 5900 open to the internet — in every case the attacker landed root and dropped a Monero miner (BleepingComputer). Apple actually fixed this back on August 6 in macOS Tahoe 26.6.1, Sequoia 15.7.9, and Sonoma 14.8.9, but patch uptake on Macs sitting outside MDM is always spotty, and this is exactly the kind of service that gets enabled once for a support session and never turned off. If you manage any Macs — dev workstations, kiosk boxes, build machines — check whether Screen Sharing is on (System Settings → General → Sharing) and either disable it or confirm the update is installed. A crypto miner is the visible symptom; unauthenticated root access is the actual problem.

ShieldBreak: an unpatched Windows Defender bypass with a public PoC

Researcher Nightmare Eclipse published a technique called ShieldBreak that defeats a fix Microsoft shipped for an earlier Defender flaw (RoguePlanet), using a user-mode callback hook to tamper with file contents mid-scan via the Cloud Filter API (BleepingComputer). It affects Windows 10, Windows 11 25H2, and Windows Server, the PoC reportedly hits 100% on tested systems, and there's no CVE or patch yet — Microsoft says it's still investigating. It requires Defender to be enabled and running, so this isn't remote-code-execution-from-nothing, but it is a working local privilege-escalation path with no fix on the horizon. Nothing to patch today, but worth flagging to whoever owns endpoint security on your Windows fleet so it's on their radar before a patch lands, not after.

EC2 gets built-in application health checks

Amazon EC2 now has Application Status Checks — instance-level monitoring that periodically hits an HTTP/HTTPS port and path you define and confirms it returns the response code you expect, on a 60-second interval (AWS). It sits alongside the existing infrastructure status checks and plugs straight into Auto Scaling, which can replace an instance whose app-level check fails even if the instance itself looks perfectly healthy — a stopped web server, a crashed Docker daemon, a broken app dependency. If you've ever bolted a sidecar or a custom CloudWatch alarm onto an ASG just to catch "the instance is up but the app is dead," this replaces that homegrown setup with a native, tag-or-instance-ID-scoped check. Available in every commercial region plus GovCloud, no extra infrastructure required.

Also worth a look

AWS Direct Connect has had degraded connectivity since 13:17 UTC yesterday for customers connected at the Equinix FR5 facility in Frankfurt — a co-location partner facility issue, not an AWS-side fault, with AWS recommending VPN failover in the interim. If you've got a DX connection through FR5 and no redundant path through another location, that's worth checking on today.

Bottom line

Neither security item needs a fire drill, but both are worth five minutes: confirm Screen Sharing is off or patched on any internet-reachable Mac, and make sure Windows Defender's ShieldBreak gap is on your endpoint team's watchlist. The EC2 health check feature is the one to actually go try — it's a straightforward win if you're currently monitoring app health with anything homegrown.

Cutting NAT gateway costs with VPC endpoints that actually help

Sun, 16 Aug 2026 00:00:00 GMT

A NAT gateway bills $0.045/hour just to exist (about $32/month per AZ) plus $0.045/GB processed, in every region, for every private subnet that needs outbound internet access. Most of that traffic, on a typical workload, isn't actually going to the internet — it's a Lambda function calling S3, an ECS task writing to DynamoDB, or an EC2 instance pulling a Secrets Manager value, all routed out through NAT and back into AWS because nothing told the VPC a shorter path existed. VPC endpoints are that shorter path. They don't replace NAT gateways outright, but on most accounts they eliminate the majority of the traffic NAT was ever pushing.

Two different mechanisms, and it matters which one you reach for

"VPC endpoint" covers two unrelated implementations that happen to share a name:

Gateway endpoints — S3 and DynamoDB only. A gateway endpoint is a route table entry, not a network interface. Traffic to the service's address range is routed directly within AWS's network instead of out through the internet gateway or NAT. There's no hourly charge and no per-GB charge — it's free.
Interface endpoints (AWS PrivateLink) — everything else: DynamoDB (as an alternative), Secrets Manager, SSM, SQS, SNS, ECR, CloudWatch Logs, Bedrock, and most other AWS services. An interface endpoint provisions an elastic network interface with a private IP in your subnet, and AWS gives it a DNS name that resolves in place of the public service endpoint (when you enable private DNS). These cost $0.01/hour per AZ you deploy into, plus $0.01/GB processed — not free, but usually far cheaper than the NAT traffic it replaces once you look at the actual GB numbers.

The pricing gap is why the order of operations matters: turn on gateway endpoints for S3 and DynamoDB first, always — there's no cost trade-off to evaluate, they're strictly free traffic that used to route through paid NAT. Interface endpoints are the ones that need a per-service cost comparison before you add them.

Adding a gateway endpoint

Gateway endpoints attach to specific route tables, not the whole VPC, so private subnets need the association explicitly:

aws ec2 create-vpc-endpoint \
  --vpc-id vpc-0123456789abcdef0 \
  --service-name com.amazonaws.us-east-1.s3 \
  --route-table-ids rtb-0a1b2c3d4e5f6a7b8 rtb-1a2b3c4d5e6f7a8b9 \
  --vpc-endpoint-type Gateway

In Terraform:

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.us-east-1.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = [for rt in aws_route_table.private : rt.id]
}

Check that it actually took effect with a route table describe — you're looking for a pl- (prefix list) destination pointing at the endpoint's ID, not a 0.0.0.0/0 route to NAT for S3 traffic:

aws ec2 describe-route-tables --route-table-ids rtb-0a1b2c3d4e5f6a7b8 \
  --query 'RouteTables[0].Routes[?DestinationPrefixListId!=`null`]'

Adding an interface endpoint

Interface endpoints need a subnet placement (one ENI per AZ you list) and a security group, since they're a real network interface that other resources connect to:

resource "aws_vpc_endpoint" "secretsmanager" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.us-east-1.secretsmanager"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private[*].id
  security_group_ids  = [aws_security_group.vpc_endpoints.id]
  private_dns_enabled = true
}

resource "aws_security_group" "vpc_endpoints" {
  vpc_id = aws_vpc.main.id
  ingress {
    from_port       = 443
    to_port         = 443
    protocol        = "tcp"
    security_groups = [aws_security_group.app.id]
  }
}

private_dns_enabled = true is what makes this transparent to application code: the SDK still calls secretsmanager.us-east-1.amazonaws.com, but Route 53 Resolver answers with the endpoint's private IP instead of the public one, inside the VPC. Nothing in your application config changes — which is also why it's easy to add one and not notice it's doing nothing, covered below.

Where NAT still has to stay

VPC endpoints only cover AWS service APIs that have a PrivateLink or gateway implementation. NAT (or an internet gateway with a public IP, for public subnets) is still required for:

Calls to non-AWS third-party APIs — Stripe, Datadog, npm/PyPI registries during a build, any SaaS webhook target. There's no PrivateLink endpoint for the general internet.
AWS services without a PrivateLink endpoint in your region yet — check the AWS PrivateLink service list per-region before assuming coverage; newer or smaller services lag.
Cross-region calls to AWS services — an interface endpoint is regional; a us-east-1 subnet calling an S3 bucket in eu-west-1 through a gateway endpoint still needs a path out, because the endpoint only covers same-region traffic patterns for most services (S3 gateway endpoints are a partial exception via cross-region access points, but don't assume it works until you've checked the specific service).

Don't decommission your NAT gateway because you added endpoints for your top three services — audit what's actually calling out first (see below), because the leftover traffic is usually smaller but never zero.

Finding out what's actually costing you, before you guess

Don't add endpoints speculatively for every service AWS offers — at $0.01/hour/AZ each, a dozen unused interface endpoints across 3 AZs is $26/month for nothing. VPC Flow Logs tell you what's actually flowing through NAT right now. Enable them on the NAT gateway's ENI (or the whole VPC) and query the destination:

fields dstAddr, bytes
| filter srcAddr like /^10\./
| stats sum(bytes) as totalBytes by dstAddr
| sort totalBytes desc
| limit 20

Cross-reference the top destination IPs against AWS's published IP address ranges for your region — jq '.prefixes[] | select(.region=="us-east-1") | .service' ip-ranges.json gives you the service name per CIDR block. If a destination IP resolves to DYNAMODB or S3, that traffic is a candidate for a free gateway endpoint right now. If it resolves to SECRETSMANAGER or ECR and represents a meaningful share of your NAT bytes total, run the $0.01/GB endpoint cost against what that GB volume currently costs at NAT's $0.045/GB — the endpoint usually wins by a wide margin, but a low-traffic service isn't worth the flat hourly charge across every AZ, since that's a fixed cost you pay whether or not it's used, unlike NAT's per-GB-only marginal cost for that same traffic.

The gotcha: adding the endpoint doesn't guarantee it gets used

An interface endpoint with private DNS enabled changes DNS resolution, but only for resolvers that actually query the VPC's Route 53 Resolver. Three ways this silently fails to redirect traffic, leaving you paying for both the endpoint and unchanged NAT usage:

A custom DNS server configured in the VPC's DHCP options set that doesn't forward to the AWS-provided .2 resolver — the private DNS record never gets seen, so the SDK still resolves the public IP and routes out through NAT anyway.
A hardcoded regional or FIPS endpoint URL in application config (https://s3.dualstack.us-east-1.amazonaws.com or a client explicitly configured with a non-default endpoint) bypasses the standard hostname the private DNS record matches.
Cached negative DNS lookups from before the endpoint existed — some container base images or runtimes cache resolver failures more aggressively than successes; a task that started before the endpoint went live may need a restart, not just time, to pick up the new record.

Confirm it's actually working, don't assume it: dig the service hostname from inside a resource in the private subnet and check the answer is a private (10.x/172.16.x/192.168.x) address, then watch the endpoint's own CloudWatch metrics (BytesProcessed on the interface endpoint) climb while the NAT gateway's BytesOutToDestination correspondingly flattens. If the endpoint's traffic metric stays near zero after deployment, something in the DNS resolution path above is still routing around it.

What to actually do

Turn on S3 and DynamoDB gateway endpoints everywhere, immediately — they're free and there's no scenario where they make things worse. For everything else, pull a week of VPC Flow Logs against the NAT gateway, resolve the top destination IPs against AWS's IP ranges, and add interface endpoints only for the services showing real GB volume, checking each one actually took over the traffic afterward rather than assuming the Terraform apply was enough. On a typical account this drops NAT gateway processing charges sharply without touching NAT's role for genuine third-party internet traffic, which is the traffic it was actually built for.

Cloud roundup: S3 finally names the policy that denied you

Fri, 14 Aug 2026 00:00:00 GMT

Quieter news day, no fresh KEV entries or breaking CVEs in the last 24 hours — so today's roundup is AWS-only, and it's a genuinely useful batch: a debugging quality-of-life fix, an infra-as-code-friendly VPN client, and Bedrock picking up dedicated offensive-security models.

S3 access-denied errors now name the exact policy ARN

Amazon S3 now includes the specific IAM or AWS Organizations policy ARN in HTTP 403 responses for explicit-deny cases — across SCPs, RCPs, identity-based policies, session policies, and permission boundaries (AWS). If you've ever gotten a bare "Access Denied" and had to manually walk every SCP and permission boundary in the chain to find the one deny statement, this removes that entire step — the error just tells you which policy did it. No opt-in, no cost, live in every region including GovCloud and China. This is the kind of change that saves real debugging time without anyone having to change a line of code.

AWS Client VPN ships a scriptable CLI and centralized policy controls

The Client VPN desktop app got a rebuild (v6.0.x, OpenVPN3-based) that adds a CLI with full feature parity to the GUI, so VPN connections can be scripted into CI pipelines and IaC workflows instead of requiring a human to click through the app (AWS). It also adds admin controls to scope VPN profiles to specific users or push a global profile to every device, plus faster connection establishment on Windows, macOS, and Linux. It's a free upgrade, backward-compatible with existing endpoints. Worth grabbing if you've ever had to walk a remote contractor through manually configuring VPN client settings — that's now something you can push as a managed profile instead.

Bedrock adds OpenAI's Daybreak Red and Blue for authorized security work

AWS added two purpose-built cybersecurity models to Bedrock: Daybreak Blue (GPT-5.6 Sol) for defensive workflows like vulnerability discovery, detection engineering, and incident response, and Daybreak Red (GPT-5.6 Cyber) for advanced authorized work — vulnerability research, exploit reproduction, mitigation development (AWS). Access isn't self-serve — it requires enrolling in Daybreak access through OpenAI or your AWS account team, and it's US East (N. Virginia) only for now. Inference data isn't used for training and isn't shared back to OpenAI by default. If your security team has been prototyping LLM-assisted detection engineering or authorized pentest tooling with general-purpose models, this is worth evaluating once you can get enrolled — it's a narrower, more accountable access model than "give the whole team an API key."

Bottom line

Nothing urgent to patch today. The S3 error message change is the one to actually notice — it'll quietly save you time the next time you're chasing down a deny across a dozen SCPs.

Service spotlight: Amazon Cognito user pools vs identity pools

Fri, 14 Aug 2026 00:00:00 GMT

Half the confusion I've seen around Cognito isn't about how it works — it's that "Cognito" is the marketing name for two separate services that happen to share a console tab. Get the two confused and you either build an auth flow that can't touch AWS resources, or wire up AWS credentials for users who were never actually authenticated.

What it actually is

User pools are the authentication half: a managed user directory that handles sign-up, sign-in, password policies, MFA, and email/SMS verification, and issues JWTs (ID, access, and refresh tokens) on successful login. They can front their own hosted UI or federate to social IdPs and SAML/OIDC providers, so "login with Google" and "login with your corporate Okta" both terminate in the same user pool.

Identity pools (Cognito Federated Identities) are the authorization half: given a token — from a Cognito user pool, a social IdP, SAML, or even an unauthenticated "guest" request — an identity pool exchanges it for temporary AWS credentials via STS, scoped by an IAM role. That's the piece that lets a mobile app upload directly to S3 or call DynamoDB without a backend in the middle, using credentials that expire instead of a long-lived API key baked into the app.

The two compose but aren't interchangeable: a user pool alone gives you "who is this person," not "what can they touch in AWS." An identity pool alone will happily hand out credentials to unauthenticated guests if you let it — it doesn't do authentication itself, it just brokers whatever token you hand it.

Where it earns its keep

You skip building and hardening your own auth service. Password hashing, MFA enrollment, account recovery, and JWT issuance are the kind of thing that's deceptively easy to get 90% right and dangerously easy to get the last 10% wrong (timing attacks on password comparison, token replay, session fixation). Cognito's user pools cover that surface area so it's not your team's to maintain.
Direct-to-AWS access without a backend proxy. A static site or mobile app can let an authenticated user write straight to an S3 prefix scoped to their identity (${cognito-identity.amazonaws.com:sub} in the IAM policy) instead of routing every upload through an API Gateway + Lambda pair whose only job is forwarding bytes.
Federation is handled once, centrally. Add a SAML IdP or a social login provider to the user pool and every app using it gets that login option — you're not reimplementing OAuth handshakes per client.

Where it goes wrong in practice

The tier structure is the sharpest edge. Cognito's current pricing splits user pools into Lite, Essentials, and Plus feature plans, and features that look like they should be table stakes — SAML/OIDC federation, advanced security features like compromised-credential and risk-based adaptive authentication — only exist on Essentials or Plus, billed per monthly active user (MAU). A team that prototypes on Lite because the free allowance looks generous, then adds "sign in with our IdP" for an enterprise customer, discovers that single feature moves the entire pool's MAU count onto the paid tier, not just the new users using it. Check which tier a feature needs before you promise it in a sprint.

The other recurring mistake is treating identity pool credentials as a substitute for real authorization. STS credentials scoped to an identity-pool IAM role are still full IAM credentials — if the attached role is too broad, an authenticated (or worse, guest-enabled) unauthenticated identity can reach far more than "upload your own profile picture." Audit identity pool roles the same way you'd audit any IAM role handed to untrusted clients, with Condition blocks scoping access to the caller's own identity ID, not just a wide S3 prefix.

A practical tip

Before adding a Cognito feature to a design doc, check the feature plan comparison for the tier it actually requires — "advanced security" and third-party federation are the two that most often get assumed as free defaults and aren't. And if you're issuing AWS credentials via an identity pool, run aws sts get-caller-identity with a test unauthenticated identity's credentials before shipping, to confirm guest access is scoped as tightly as you think it is rather than as tightly as you meant it to be.

The Capital One breach: an SSRF bug into 100M records

Fri, 14 Aug 2026 00:00:00 GMT

In March 2019, an attacker used a server-side request forgery (SSRF) bug in a misconfigured web application firewall to reach a single internal URL — http://169.254.169.254/ — running behind a Capital One-hosted application in AWS. That one request was enough to pull temporary IAM credentials for a role with broad S3 read access, and from there exfiltrate more than 100 million credit applications and 140,000 Social Security numbers (DOJ indictment). No malware, no phishing, no zero-day in AWS itself — just a web app that could be tricked into fetching an internal URL on the attacker's behalf, and an internal URL that handed over live credentials to anyone who asked. Capital One paid an $80 million OCC fine and settled a $190 million class action over it (Reuters). Seven years later, IMDSv1 — the version that made this possible — is still the default reachable endpoint on plenty of EC2 instances, because "enabled" is not the same as "enforced."

Root cause

Every EC2 instance can reach a link-local address, 169.254.169.254, which serves the Instance Metadata Service (IMDS) — instance ID, AMI info, user-data, and critically, temporary security credentials for whatever IAM role is attached to the instance. It's how an EC2 instance gets AWS credentials without anyone hardcoding a key. In the original version of that service, IMDSv1, the endpoint answers any plain HTTP GET request with no authentication step at all — if a request reaches that IP from the instance, it gets an answer. That design was fine as long as nothing running on the box could be tricked into making arbitrary outbound requests on an attacker's behalf. But web applications routinely do exactly that: fetch a URL from a query parameter, proxy a request, render a remote image, validate a webhook. The former engineer charged in the case had reportedly discovered a misconfigured ModSecurity WAF rule in front of a Capital One web application that could be coerced into requesting an arbitrary URL and returning the response — a textbook SSRF (indictment). Point that SSRF at 169.254.169.254/latest/meta-data/iam/security-credentials/<role-name> and IMDSv1 hands back an access key, secret key, and session token for whatever role the instance was running as — no extra proof that the request came from a legitimate, non-hijacked process.

Blast radius

The stolen role's permissions, not the SSRF bug itself, set the ceiling on the damage. In this case the role had read access to S3 buckets used for a range of Capital One's credit-card and loan-application data, compiled from six years of applications going back to 2005 — 106 million individuals across the US and Canada, including Social Security numbers, bank account numbers, and credit scores (Capital One's disclosure). Stolen IMDS credentials are functionally indistinguishable from legitimate ones to everything downstream — CloudTrail logs the calls as coming from the role, IAM enforces exactly the policy attached to that role, and nothing about the request signature reveals it originated from an SSRF payload rather than the application itself. That's the general shape of every IMDS-credential-theft incident: the SSRF or RCE is the entry point, but the actual damage is bounded entirely by how over-permissioned the instance role was. A role scoped to one bucket prefix limits an attacker to that prefix; a role with s3:* across the account hands over the account's data.

Remediation

AWS's direct answer, shipped in November 2019 a few months after this breach became public, is IMDSv2 — it requires a session to be established with a PUT request that returns a token, and every subsequent metadata request must carry that token in a header (AWS). Critically, IMDSv2's PUT requires HTTP with no redirects followed and sets a configurable hop limit (default 1) on the response's TTL — which means a proxy or SSRF payload relaying a simple GET typically can't complete the handshake or forward the token past one network hop, while a process running natively on the instance can. That single change would have broken the Capital One attack path as described. Enforcement is the part teams skip: IMDSv2 has been opt-in at the instance level since launch, so an unpatched fleet stays exploitable indefinitely unless someone acts. Three concrete steps: set HttpTokens: required (not just optional) on every instance and in every launch template, ideally via an AWS Config rule (ec2-imds-v2-check) that flags drift; use an **SCP or Config rule to deny new instance launches without IMDSv2 enforced**, since a one-time fleet fix doesn't stop the next Terraform apply from reintroducing IMDSv1; and independently, scope every instance role to least privilege — the hop-limit defense is layer two, not a replacement for making sure a stolen credential is only ever worth as much as the narrowest policy you could get away with attaching.

The bigger lesson

This wasn't an AWS vulnerability — IMDS behaved exactly as designed, and the WAF misconfiguration was Capital One's, not Amazon's. But it's the canonical case study for why cloud metadata endpoints deserve the same scrutiny as any credential store: anything that can make an outbound HTTP request from inside your VPC is a potential path to 169.254.169.254, and IMDSv1's design assumed that path would never exist. IMDSv2's token-and-hop-limit model closes the SSRF-relay case specifically, but it only helps the instances where someone actually flipped HttpTokens to required — which is why the Config rule and the SCP matter as much as the setting itself. If your fleet still allows HttpTokens: optional, the difference between "we have IMDSv2 available" and "we're actually protected" is one unenforced launch template away from mattering.

Cloud roundup: 153GB LiteLLM breach exposes cloud secrets

Thu, 13 Aug 2026 00:00:00 GMT

The big story today isn't a new breach — it's the bill coming due on an old one. A researcher got hold of the actual haul from March's LiteLLM supply-chain attack, and the scale is worse than the original disclosure suggested. On the AWS side, two changes worth planning around: more control over your EKS control plane, and a pricing shift that quietly doubles some OpenSearch costs.

153GB of stolen credentials surface from the March LiteLLM attack

Back in March, attackers compromised the Trivy scanner and used it to slip two malicious versions of LiteLLM onto PyPI for about 40 minutes — long enough for a .pth file to start harvesting secrets from every CI run that installed it. Hudson Rock has now obtained and analyzed the actual stolen archive: 153GB, 433,909 files, with 118,829 CI runner dumps attributable to 2,488 corporate domains — including AWS, Cisco, Samsung, Salesforce, Microsoft, and dozens of other large enterprises (Help Net Security). The dumps include AWS secret access keys, Salesforce client secrets, Slack signing secrets, and Azure environment variables captured mid-pipeline. The uncomfortable part: any credential that was live in a CI runner during that 40-minute window is still valid until someone actively rotates it — removing the malicious package did nothing to invalidate what it already stole. If you had LiteLLM anywhere in a CI/CD dependency tree back in March, this is worth a credential audit today, not a "we'll get to it" item — check for AWS keys, Slack tokens, and cloud provider secrets that predate March 19 and haven't been rotated since.

Amazon EKS now lets you tune scheduler, controller manager, and API server parameters

AWS added support for configuring Kubernetes control plane parameters directly on EKS — scheduler behavior, controller-manager settings, and API server options that used to require self-managing the control plane to touch (AWS). The scheduler example AWS calls out is a good one: switching the node resource fit strategy from the default LeastAllocated (spread pods across nodes) to MostAllocated (pack nodes tightly) can meaningfully cut node count for workloads where you don't need the spare headroom. It's available in every region EKS runs in, no migration required. Worth a look if you've ever wanted more control over pod-placement or autoscaling responsiveness on EKS without giving up the managed control plane.

OpenSearch/Elasticsearch Extended Support surcharge is about to double

AWS extended the security-patch window for older Elasticsearch (1.5–7.8) and OpenSearch (1.0–1.2, 2.3–2.9) versions through November 7, 2027 — but from November 7, 2026, the Extended Support surcharge jumps to equal your instance cost, effectively doubling the price of running those old versions (AWS). If you've got domains sitting on a pre-7.9 Elasticsearch or early-2.x OpenSearch version because upgrading was never urgent, this is the forcing function: budget for either the upgrade or the doubled bill before November.

Bottom line

The LiteLLM fallout is the one to act on today — a credential rotation check costs you an hour and closes a door that's been open since March. The AWS items are both "plan now, act before the deadline" — EKS control plane tuning whenever you get to it, OpenSearch versions before the November surcharge hits.

Cloud roundup: Metabase CVSS 10 SQLi confirmed exploited

Wed, 12 Aug 2026 00:00:00 GMT

Another security-heavy day: the Metabase bug I flagged Monday as a zero-day with no CVE now has one and is confirmed under active exploitation, Cisco's VPN appliances picked up an exploited DoS flaw, and Microsoft's August Patch Tuesday landed with an already-exploited zero-day of its own.

Metabase's CVSS 10 SQL injection now has a CVE and is in CISA KEV

CVE-2026-72898 is an unauthenticated SQL injection in Metabase's password-reset endpoint (POST /api/session/reset_password) that lets an attacker inject arbitrary SQL into the application database and walk away with admin access — no auth, no user interaction required, CVSS 10.0 (Bishop Fox). CISA added it to the Known Exploited Vulnerabilities catalog on August 11 (CISA). It affects a wide version range — 0.58 through 0.63.4 and the matching Enterprise 1.x builds — so if you're running self-hosted Metabase anywhere, including a "just for internal dashboards" instance, patch it now and don't wait for the next maintenance window. Admin takeover on a BI tool usually means access to every data source it's connected to, which for most of us is production databases and warehouses.

Cisco ASA and FTD hit with an actively exploited VPN DoS flaw

CVE-2026-20349 (CVSS 8.6) is a heap inspection bug in Cisco Secure Firewall ASA and FTD's Remote Access SSL VPN service — a crafted, unauthenticated HTTP request can crash the device and force a reload (The Hacker News). Cisco confirmed active exploitation and CISA added it to KEV in the same August 11 batch as the Metabase bug, with a federal remediation deadline of August 14. If you've got ASA/FTD devices with SSL VPN listeners facing the internet, this is a "patch this week" item — a device that keeps rebooting under attacker control is its own kind of outage, on top of whatever else the attacker is probing for while it's down.

Microsoft's August Patch Tuesday: ~400 fixes, one already exploited

Microsoft's August Patch Tuesday shipped fixes for roughly 400 vulnerabilities including three zero-days, and confirmed one — CVE-2026-68820, a use-after-free elevation-of-privilege bug in the Windows Ancillary Function Driver for WinSock — is already being exploited in the wild (BleepingComputer). AFD.sys bugs are a recurring privilege-escalation path once an attacker has any foothold, so this is the one to prioritize on your Windows fleet — EC2 Windows instances, jump boxes, and anything else running Windows — even though the initial-access vector is elsewhere.

Also worth a look

Amazon Connect added a performance dashboard for Cases, giving managers case-volume, resolution-trend, and SLA-attainment views without building custom reporting on top of the Cases API — a small but welcome addition if you're running support workflows through Connect.

Bottom line

Two unauthenticated, actively-exploited bugs landed in KEV on the same day — Metabase SQLi and the Cisco ASA/FTD DoS — so triage internet-facing instances of either today. The Windows AFD zero-day is a "patch it in the normal cycle, but don't push the normal cycle out" item.

Cloud roundup: New ransomware exploits N-central bug

Tue, 11 Aug 2026 00:00:00 GMT

If you're still triaging the N-central saga from earlier this week, there's a new wrinkle: it's now ransomware. Otherwise it's a lighter AWS day — two operational upgrades worth bookmarking rather than dropping everything for.

StormEncryptor ransomware ties back to the N-central auth bypass

Microsoft disclosed that Storm-1175, a financially motivated China-linked actor that previously ran Medusa ransomware, has moved to a new strain called StormEncryptor — and the likely entry point is CVE-2026-18577, the N-able N-central authentication bypass CISA added to KEV earlier this week (The Hacker News). Microsoft assesses that CVE-2026-18577 is actually a patch bypass for the earlier CVE-2026-18556, meaning the first fix didn't fully close the door. Once in, Storm-1175 moves fast — AnyDesk or SimpleHelp for persistence, Advanced IP Scanner for discovery, Mimikatz for credential dumping, then exfiltration and encryption within days of initial access. If you're an MSP, or downstream of one running N-central, this is the concrete "why" behind all the patch-immediately warnings from earlier this week: confirm you're on the latest hotfix (not just the first one), and hunt for AnyDesk/SimpleHelp installs you didn't put there yourself.

EC2 gets application-level health checks, not just instance-level

AWS launched Application Status Checks for EC2 — you configure a protocol, port, path, and expected response code, and EC2 polls it every 60 seconds, feeding the result into the same status-check signal Auto Scaling already watches (AWS). It's live in all commercial regions plus GovCloud. The gap this closes: instance and system status checks tell you the VM is up, not that nginx crashed or the app inside stopped responding — that's traditionally been a custom health-check script or an ALB target group doing double duty. If you run EC2 outside an ALB, or want faster detection than an ALB's own health-check interval, this is a built-in way to get unhealthy instances replaced without writing your own watcher.

AWS DRS now preserves UEFI boot mode on Linux failover

AWS Elastic Disaster Recovery now carries UEFI boot mode through to recovered Linux instances automatically, instead of dropping them into legacy BIOS mode and leaving you to fix boot config after a failover (AWS). No configuration change needed, no extra cost. It's a small thing until the day you actually fail over and discover your UEFI-dependent app won't boot — worth knowing about before you need it, not during an actual DR event.

Also worth a look

AWS brought U7in-24TB high-memory EC2 instances to São Paulo for SAP HANA/Oracle-scale workloads, and bumped OpenSearch Serverless from 1,500 to 10,000 collections per collection group — useful if you're running dense multi-tenant search and were bumping into the old ceiling.

Bottom line

The N-central story just got a lot more concrete: it's now actively being used to deploy ransomware, so if you haven't confirmed your hotfix level, do that today. On the AWS side, the two operational items — EC2 app-level health checks and DRS's UEFI fix — are both worth turning on now, quietly, before you need them.

Cloud roundup: LoadMaster RCE hits its KEV deadline today

Mon, 10 Aug 2026 00:00:00 GMT

If you run a Progress/Kemp LoadMaster load balancer, today's the deadline. Beyond that, it's a quieter day — another max-severity zero-day to know about, a bad update on last week's N-central saga, and AWS chipping away at supply chain risk.

Progress LoadMaster command injection hits its remediation deadline today

CISA added CVE-2026-8037 (CVSS 9.6), an unauthenticated command injection flaw in Progress (Kemp) LoadMaster, to the KEV catalog on August 7 after eSentire and watchTowr Labs both reported active exploitation attempts — 792 reported attempts by one count (The Hacker News). The bug lives in a function called escape_quotes() and lets an unauthenticated attacker send unsanitized input to a LoadMaster API endpoint and get arbitrary command execution on the appliance. Progress shipped the fix back in June (GA 7.2.63.2 / LTSF 7.2.54.18), so this is really about finding the load balancers your org forgot to patch. Federal civilian agencies have until today, August 10, to remediate under BOD 26-04 — if you run LoadMaster anywhere with its management API exposed, that's your action item regardless of whether you're a federal shop.

A CVSS 10 zero-day in Metabase, no CVE yet

Metabase — a popular open-source BI/dashboarding tool a lot of teams point at production databases — has a maximum-severity, unauthenticated SQL injection zero-day being actively exploited in the wild, including against Metabase Cloud itself (The Hacker News). An attacker can inject SQL through the app database with no credentials and land admin access, which means the credentials for every database Metabase is connected to are exposed too. It affects the 1.58 through 1.63 release lines; patched builds are out (1.58.24 through 1.63.5). If you can't patch immediately, block the /api/session/reset_password endpoint at your edge as a stopgap, then rotate the DB credentials Metabase held once you're patched — assume they were seen.

N-able N-central: the first patch wasn't enough

An update to the N-central story from earlier this week: N-able shipped Hotfix 2 after finding attackers who'd exploited CVE-2026-18577 were registering Cloudflare Tunnel connections on compromised managed endpoints to keep access even after the N-central server itself got locked down (The Hacker News). If you patched with Hotfix 1 and called it done, it's worth another look — N-able is now shipping expanded IOCs and a detection template, and explicitly warning that a clean scan doesn't guarantee you weren't hit. Good reminder that "authentication bypass" bugs on management platforms deserve a persistence hunt, not just a patch-and-move-on.

AWS Security Hub Extended adds supply chain security

AWS added supply chain security as the tenth category in Security Hub Extended, with Chainguard and Socket as the curated partners, aimed at catching malicious code in open-source dependencies before it gets built into your app (AWS). It's pay-as-you-go with no commitment, and findings land in the same OCSF-normalized dashboard as everything else in Security Hub. If you're already paying for Extended and doing dependency scanning with a separate standalone tool, worth comparing — consolidating that signal into the same place as your other findings is generally a win for whoever's triaging.

Bottom line

Patch LoadMaster today if you run one exposed. If you're on Metabase, patch and rotate credentials — assume compromise until proven otherwise. And if you touched N-central last week, go back and hunt for persistence rather than trusting the first patch.

CI/CD for a static site: build, sync, and invalidate

Mon, 10 Aug 2026 00:00:00 GMT

Deploying a static site sounds like it should be a solved problem: build it, copy the files to a bucket, done. In practice the pipeline has three separate failure modes that don't show up until production — stale HTML served from cache, orphaned objects left in the bucket after a rename, and an invalidation bill that surprises you if you get cache-control wrong. This site's own deploy pipeline (.github/workflows/deploy.yml) hits all three concerns in about 20 lines, so I'll use it as the working example rather than a hypothetical.

The three stages, and why order matters

A static-site deploy is really three distinct jobs glued together:

Build — turn source into a dist/ directory of static files.
Sync — reconcile that directory with what's in S3, including deleting anything that's no longer there.
Invalidate — tell CloudFront's edge caches to stop serving the old versions of whatever changed.

They have to run in that order and each has to fully succeed before the next starts — a partial sync followed by an invalidation just serves a broken mix of old and new files faster. Keep them as separate steps (not one giant shell script) so a failure at step 2 shows up clearly in the Actions log instead of buried in a 40-line run: block.

- run: npm run build

- name: Sync to S3
  run: aws s3 sync ./dist "s3://my-bucket" --delete

- name: Invalidate CloudFront
  run: |
    aws cloudfront create-invalidation \
      --distribution-id "$DISTRIBUTION_ID" \
      --paths "/*"

That's the naive version. It works, but it's wrong in two specific ways — neither obvious until you look closely at cache-control and --delete together.

Sync: cache-control headers aren't optional

aws s3 sync doesn't set a useful Cache-Control header by default — S3 serves objects with no explicit caching directive, which browsers and CloudFront interpret conservatively. For a static site built by a bundler (Astro, Vite, Next static export, etc.), you want two very different caching policies in the same deploy:

Fingerprinted assets (/_astro/chunk-a1b2c3.js, hashed CSS, images) — the filename changes when the content changes, so it's safe to cache these forever. public, max-age=31536000, immutable.
HTML, sitemap, RSS — the URL stays the same (/blog/index.html) but the content changes on every deploy. These need max-age=0, must-revalidate so a browser or CDN edge always re-checks before serving a cached copy.

You can't set one blanket --cache-control flag for the whole sync and get both right, so split it into two sync calls with complementary --exclude/--include filters:

- name: Sync to S3
  run: |
    aws s3 sync ./dist "s3://$BUCKET" \
      --delete \
      --exclude "*.html" \
      --exclude "*.xml" \
      --cache-control "public,max-age=31536000,immutable"

    aws s3 sync ./dist "s3://$BUCKET" \
      --delete \
      --exclude "*" \
      --include "*.html" \
      --include "*.xml" \
      --cache-control "public,max-age=0,must-revalidate"

This is exactly what this site's deploy.yml does. The --exclude/ --include pairs are inverses of each other on purpose — every object in dist/ is claimed by exactly one of the two calls, never both, never neither.

The `--delete` gotcha: it respects filters, both ways

Here's the part that isn't obvious from the CLI docs' one-line description. aws s3 sync --delete removes destination objects that aren't present in the source and match the command's own filters. It does not delete everything in the bucket that the source lacks — it only considers objects within the scope defined by --exclude/--include for that specific invocation.

That's exactly why running the two-pass sync above is safe: the first call (non-HTML) only ever deletes stale non-HTML objects, and the second call (HTML/XML) only ever deletes stale HTML/XML objects. Neither pass can accidentally delete the other's files, because each treats them as excluded and therefore invisible.

The gotcha is what happens if your two filter sets aren't exact complements — say you add a new file extension to the build output (a .webmanifest, a .txt) and forget to add it to either pass's include list. It won't get deleted when removed (each --delete ignores it), but it also won't get the cache-control header you intended on either pass — it silently falls through both filters. Test a rename/removal locally against a scratch bucket (aws s3 sync --dryrun) whenever you touch the exclude patterns, not just when you add new ones.

Invalidate: scoped paths vs. wildcard, and the pricing surprise

The instinct once you've fixed caching is to invalidate narrowly — pass the exact paths that changed instead of /*, on the theory that a full wildcard invalidation is expensive because it touches every object at the edge. That instinct is backwards for CloudFront specifically, and it's worth knowing why before you build a diff-based "only invalidate what changed" step.

CloudFront invalidation pricing is **per path string submitted in the request, not per object matched**. The first 1,000 paths per month are free; after that it's $0.005 per path. --paths "/*" is one path as far as billing is concerned, regardless of how many thousands of objects it actually clears. Compare that to a "smart" pipeline that diffs the build and submits one path per changed file — a typical content update touching 15 files costs the same order of magnitude as 15 separate wildcard deploys, and if you ever invalidate per-object on a big rebuild (hundreds of pages) you can burn through the free tier in a single deploy.

For a low-traffic personal site or portfolio, /* on every deploy is both simpler and, counter-intuitively, usually cheaper than trying to be clever about scoping. It's the right default. The one place scoping earns its complexity is a high-frequency deploy pipeline (many deploys per hour, e.g. a CMS with instant-publish) where wildcard invalidations would otherwise queue up and a --paths "/blog/*" "/index.html" pattern targeting only the collections that actually changed keeps the queue from backing up — CloudFront processes invalidations from a single distribution somewhat serially, and a backlog of full-site wildcards delays the one that matters.

- name: Invalidate CloudFront
  run: |
    aws cloudfront create-invalidation \
      --distribution-id "$DISTRIBUTION_ID" \
      --paths "/*"

If you do scope it, don't hand-roll the diff from git diff --name-only — map source paths to routes, not files. A change to a shared layout component invalidates every page that uses it, not the one file that changed; a naive file-based diff will under-invalidate and leave stale pages in cache with no error to tell you it happened.

Rolling this out

Add the two-pass sync and cache-control split first, deploy once, and check response headers with curl -I against a fingerprinted asset and against / to confirm the immutable/must-revalidate split landed correctly before you touch the invalidation step. Then switch the invalidation from whatever ad-hoc scoping you had to /* and watch a billing cycle — for most personal and small-business traffic levels you'll stay inside the free 1,000-path tier for months. Only reach for path-scoped invalidations once you have actual evidence (a CloudFront invalidation queue backing up, or genuinely exceeding the free tier) rather than optimizing against an assumption about cost that, for this specific service, runs the opposite direction from most people's intuition.

CloudFront invalidations without nuking your AWS bill

Mon, 10 Aug 2026 00:00:00 GMT

If your deploy pipeline ends every push with `aws cloudfront create-invalidation --paths "/*"`, it works, but it's the CDN equivalent of clearing your entire browser cache because one tab was stale. It's slow to propagate, it costs money past a fairly small free tier, and — the part people miss — it doesn't actually need to exist for most of your assets if you name your files right. This site's own deploy (deploy.yml) does a scoped invalidation of exactly two paths, not a wildcard, and here's the reasoning and the mechanics behind that choice.

How invalidation pricing actually works

CloudFront gives you **1,000 free invalidation path requests per month, per account**. Past that, each additional path costs $0.005. The part that trips people up is what counts as "one path":

/* — a single wildcard path — counts as one path, no matter how many objects it matches at the edge. Nuking your entire distribution with one wildcard call costs the same as invalidating one specific file.
An explicit list of paths in a single create-invalidation call is billed per path in the list. aws cloudfront create-invalidation --paths "/index.html" "/blog/index.html" "/rss.xml" bills three paths, not one call.

So a single /* is actually the cheapest way to invalidate by path-count — the trap isn't cost from the wildcard itself, it's what a wildcard does operationally: it forces CloudFront to revalidate every object in the distribution against the origin on the next request, which means a burst of origin requests to S3 right after every deploy, and it invalidates objects that never changed. On a low-traffic personal site that's harmless. On anything with real traffic or a slow/rate-limited origin, a full-distribution wildcard after every deploy is the thing that actually costs you — in origin load and in cache-miss latency for visitors who hit the edge in the seconds after the invalidation lands, not in the $0.005-per-path line item.

The free tier resets monthly and is shared across all distributions in the account, so if you run several sites or a multi-tenant setup off one AWS account, a chatty pipeline on one distribution eats the free allowance for all of them.

The real fix: stop invalidating, start fingerprinting

Invalidation is a workaround for a caching mistake: telling CloudFront to cache a URL for a long time when the content behind that URL can change. Fingerprinted (content-hashed) filenames remove the mistake instead of compensating for it. Astro's build already does this for you — run npm run build and look at dist/_astro/:

_astro/client.a1b2c3d4.js
_astro/index.e5f6a7b8.css

The hash is derived from the file's content. Change one character of source, the hash changes, the URL changes. Because the URL is different, there's nothing to invalidate — the old URL still exists at the edge serving old content (fine, nothing references it anymore), and the new URL is a cache miss exactly once, everywhere, the first time each edge location requests it. This is why you can safely set:

Cache-Control: public, max-age=31536000, immutable

on everything under _astro/ (or /assets/, however your bundler names it). A year-long max-age plus immutable tells both browsers and CloudFront "never revalidate this, ever" — and that's true, because the filename itself guarantees the content can't change out from under that URL. This is the single biggest lever for cutting invalidation traffic to near zero: the majority of a static site's bytes (JS, CSS, hashed images) never need an invalidation call in their entire lifetime.

Set this at the S3 origin via object metadata (Astro/most bundlers don't set Cache-Control on upload themselves — your sync step has to), or override it at the CloudFront cache behavior level with a policy scoped to the hashed asset path pattern:

aws s3 sync ./dist s3://my-bucket/ \
  --exclude "*" --include "_astro/*" \
  --cache-control "public, max-age=31536000, immutable" \
  --metadata-directive REPLACE

Run this pass before the pass that uploads everything else, so the hashed-assets rule doesn't get clobbered by a broader default Cache-Control applied later in the sync.

Where you still need surgical invalidation

Fingerprinting only works for files whose name changes when their content changes. Two categories of file don't get that treatment, and those are the only things worth invalidating:

index.html and any other non-hashed HTML entrypoint. The URL /blog/index.html (or /blog/ via the pretty-URL rewrite) has to stay stable — it's what's in every bookmark, backlink, and search index entry — but its content changes every time you publish. Give these a short max-age (or no-cache so CloudFront/browsers always revalidate against origin) and invalidate them explicitly on deploy.
Non-hashed static assets you can't rename, like favicon.ico, robots.txt, sitemap-index.xml, or rss.xml — anything a spec or a client expects at a fixed path.

For this site, that means the deploy step invalidates a short, explicit list, not a wildcard:

aws cloudfront create-invalidation \
  --distribution-id "$CF_DISTRIBUTION_ID" \
  --paths "/index.html" "/blog/*" "/rss.xml" "/sitemap-index.xml"

/blog/* is doing real work here — every post and tag page under /blog/ is HTML with a stable, non-hashed URL, so a scoped wildcard on just that subtree is the right call. It's still one billed path (wildcards always are), but more importantly it only forces revalidation on the part of the site that actually changes on every deploy, leaving the immutable hashed assets alone.

The gotcha: invalidations aren't instant, and they queue

create-invalidation returns immediately with a status of InProgress — the CLI call succeeding does not mean the content is gone from every edge location yet. Full propagation across all of CloudFront's edge locations typically completes within a few minutes, but there's no SLA guaranteeing a specific time, and it's not uncommon to see stale content served from one edge location after another has already updated. If your deploy pipeline runs a post-deploy smoke test that curls the live URL immediately after create-invalidation returns, don't assert on content freshness right away — poll get-invalidation for Status: Completed first, or just accept some propagation lag in the check:

aws cloudfront get-invalidation \
  --distribution-id "$CF_DISTRIBUTION_ID" \
  --id "$INVALIDATION_ID" \
  --query 'Invalidation.Status'

The second, less obvious gotcha: **invalidation requests queue per distribution**, and each request can list at most 3,000 paths (or 15 with wildcards). If a script fires invalidations in a tight loop — say, a bulk content migration that invalidates per-file instead of batching — later requests sit InProgress behind earlier ones rather than running in parallel, so a burst of small invalidations can take noticeably longer to fully clear than one batched call with the same total path count. Batch your paths into as few create-invalidation calls as the file-list limits allow, rather than looping a call per file.

What to actually do

Fingerprint everything your bundler can fingerprint and set max-age=31536000, immutable on it — that's most of your bytes and it needs zero invalidation calls, ever, for the life of the file. For the small set of non-hashed entrypoints (HTML, robots.txt, feeds), use a short max-age and a scoped explicit-path invalidation on deploy, not /*. You'll stay comfortably inside the free 1,000-path monthly allowance even with several deploys a day, and — the bigger win — your origin only gets hit for the handful of objects that actually changed, not your entire distribution.

Ditch IAM access keys: GitHub Actions OIDC to AWS

Mon, 10 Aug 2026 00:00:00 GMT

If your GitHub Actions workflows still authenticate to AWS with a stored AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY pair, you're carrying a long-lived credential that can leak from a log, a fork's pull_request_target run, or a compromised dependency — and it keeps working until someone remembers to rotate it. OpenID Connect (OIDC) federation gets rid of that entirely: GitHub mints a short-lived, workflow-scoped identity token, AWS STS trades it for temporary credentials, and there's nothing sitting in your repo secrets for an attacker to steal. This is the same pattern I use for this site's own deploy pipeline, so what follows is the working setup, not just the AWS docs paraphrased.

How the trust actually works

GitHub Actions exposes an OIDC provider at https://token.actions.githubusercontent.com. Every workflow run can request a signed JWT from it (via id-token: write permission) with claims describing the repo, branch, and workflow.
You register that provider as an IAM OIDC identity provider in your AWS account, once.
You create an IAM role whose trust policy says "I'll accept tokens from that provider, but only if the sub claim matches this specific repo and ref."
The aws-actions/configure-aws-credentials action exchanges the JWT for temporary STS credentials scoped to that role, valid for the run only.

No secret ever leaves GitHub's control plane. Nothing to rotate, nothing to revoke except the role's trust policy.

Step 1: create the OIDC provider

Do this once per AWS account (Terraform, since you're presumably managing the rest of your IAM this way too):

resource "aws_iam_openid_connect_provider" "github_actions" {
  url             = "https://token.actions.githubusercontent.com"
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"]
}

That thumbprint is GitHub's OIDC endpoint CA thumbprint. Gotcha: AWS actually ignores this field for GitHub's provider now (it validates via the standard TLS CA bundle instead), but the argument is still required by the resource — don't spend time trying to keep it "current," it's a legacy requirement AWS kept for backward compatibility.

Step 2: write a trust policy scoped tighter than you think you need

This is where most setups go wrong. The sub claim format is repo:<org>/<repo>:<qualifier>, and it's tempting to wildcard it into uselessness:

{
  "Effect": "Allow",
  "Principal": {
    "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
  },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
    },
    "StringLike": {
      "token.actions.githubusercontent.com:sub": "repo:my-org/jstgtech-web:ref:refs/heads/main"
    }
  }
}

Two things to get right:

Always set the aud condition. Without it, any GitHub Actions run anywhere that requests a token for sts.amazonaws.com audience and happens to match your sub pattern can assume the role. aud is cheap insurance and AWS's own quickstart includes it — don't skip it because the console wizard makes it feel optional.
Scope sub to the exact ref, not just the repo. repo:my-org/my-repo:* lets a PR from a fork-turned-branch, a tag push, or an environment: deployment all assume the same role your production deploy uses. If a workflow only needs to deploy from main, pin sub to repo:my-org/jstgtech-web:ref:refs/heads/main. If you use GitHub Environments for a manual-approval gate, scope to repo:my-org/jstgtech-web:environment:production instead — that ties the AWS role to the same approval gate protecting your environment secrets.

For a PR-only workflow (say, terraform plan on pull requests, no write access), use a separate, more restrictive role with sub matching repo:my-org/jstgtech-web:pull_request and a read-only policy attached — don't reuse your deploy role's trust policy with a broader condition "just for now."

Step 3: use it in the workflow

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-site-deploy
          aws-region: us-east-1

permissions: id-token: write is not optional — it's what lets the runner request the JWT in the first place, and GitHub defaults this to none at the org or repo level on newer accounts. If you get `Error: Not authorized to perform sts:AssumeRoleWithWebIdentity`, check this before anything else; it's the single most common cause, ahead of trust-policy typos.

Trade-offs worth knowing before you migrate

Session duration is capped by the role, not the token. The GitHub JWT is short-lived by nature, but your role's MaxSessionDuration still governs how long the assumed credentials last. Keep it at the default (1 hour) or lower for deploy roles — there's no reason a CI job needs an 8-hour session.
Cross-account deploys need per-account providers. The OIDC provider and trust policy live in the AWS account being deployed into, not in some central account. If you deploy to three accounts (dev/stage/prod), you need the provider registered in each, with role names and trust conditions matched to that account's own audience.
Self-hosted runners change the token issuer. If you ever move a workflow to self-hosted runners inside your own VPC, the token still comes from token.actions.githubusercontent.com (GitHub issues it, not the runner), so this setup doesn't need to change — but it's worth confirming if you're debugging a runner migration and OIDC suddenly stops working for an unrelated reason (usually a network path to GitHub's OIDC endpoint from the runner).
You still need least-privilege on the role's permissions policy. OIDC only fixes how the workflow authenticates, not what it's allowed to do once authenticated. A perfectly scoped trust policy attached to a role with AdministratorAccess is still one compromised Action away from a bad day — scope the permissions policy to exactly the S3 bucket, CloudFront distribution, or Terraform state path the workflow touches.

Rolling it out without a big-bang cutover

If you're migrating an existing pipeline off access keys, run both in parallel for one deploy cycle: add the OIDC role, switch configure-aws-credentials to role-to-assume, watch a real deploy succeed, then delete the IAM user and its access keys. Don't delete the old credentials in the same PR that introduces the new role — if the trust policy's sub condition is wrong, you want a fallback for the next deploy instead of a broken pipeline and no way back in until you fix IAM by hand.

IAM Identity Center: kill per-account IAM users for good

Mon, 10 Aug 2026 00:00:00 GMT

If you're running more than one or two AWS accounts, you've probably got the same mess I inherited: a pile of IAM users, one per human, duplicated in every account they need access to, each with its own password, its own MFA device, and — if you're unlucky — its own long-lived access keys sitting in someone's ~/.aws/credentials. Nobody remembers to deprovision the ex- contractor's user in the staging account. Nobody's rotated the access keys on the prod-readonly user since 2023. IAM Identity Center (the renamed AWS SSO service) fixes this by moving identity out of individual accounts entirely: you log in once, in one place, and get temporary credentials into whichever accounts and roles you're assigned to. This is the setup I run across a management account plus workload accounts, with the actual Terraform and the parts that don't work the way the docs imply.

What Identity Center actually replaces

Per-account IAM users have three structural problems: identity is duplicated per account (N users × M accounts to manage), credentials are usually long-lived (passwords, access keys), and there's no single place to see "what can this person get into." IAM Identity Center centralizes all of that:

One identity source — either Identity Center's own built-in directory, or federated from an external IdP (Okta, Entra ID/Azure AD, Google Workspace, or on-prem AD via AD Connector) over SCIM.
Permission sets — reusable IAM policy bundles (think "IAM role templates") that get provisioned as actual IAM roles in target accounts when you assign them.
Account assignments — a mapping of (user or group) × (permission set) × (account), managed centrally from the Identity Center console/API in your management account or a delegated administrator account.

The end state: a human logs into the Identity Center portal URL once, sees a tile for every account/role combination they've been granted, clicks one, and gets temporary STS credentials. No IAM user, no password in that account, no access key to leak.

Setting it up

Identity Center is enabled once per AWS Organization, in one region (it's free-tier, but the instance and its permission sets live in a single home region even though assignments apply org-wide). I run mine out of us-east-1 in the management account, alongside the org's other org-wide resources.

1. Choose your identity source

If you already run Okta or Entra ID, use it — don't stand up a second identity directory to maintain. Enable SCIM provisioning in Identity Center (Settings → Identity source → Automatic provisioning), which gives you an SCIM endpoint URL and a bearer token. Paste those into your IdP's SCIM app config, and your IdP becomes the source of truth for users and groups; Identity Center just mirrors them.

If you don't have an external IdP yet, the built-in Identity Center directory is fine to start with — you can migrate to an external IdP later without re-doing your permission sets or assignments, since those reference group IDs, not the identity source itself.

2. Create permission sets

A permission set is either a collection of AWS managed policies, a custom inline policy, or both, plus a session duration. I manage mine in Terraform using the aws_ssoadmin_permission_set resource — this is the piece people skip because the console makes it feel like a one-off, and then six months later nobody remembers what's in the Billing-ReadOnly permission set or why.

data "aws_ssoadmin_instances" "this" {}

resource "aws_ssoadmin_permission_set" "read_only" {
  name             = "ReadOnlyAccess"
  instance_arn     = tolist(data.aws_ssoadmin_instances.this.arns)[0]
  session_duration = "PT4H"
}

resource "aws_ssoadmin_managed_policy_attachment" "read_only" {
  instance_arn       = tolist(data.aws_ssoadmin_instances.this.arns)[0]
  permission_set_arn = aws_ssoadmin_permission_set.read_only.arn
  managed_policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess"
}

resource "aws_ssoadmin_permission_set" "billing_admin" {
  name             = "BillingAdmin"
  instance_arn     = tolist(data.aws_ssoadmin_instances.this.arns)[0]
  session_duration = "PT1H"
}

resource "aws_ssoadmin_permission_set_inline_policy" "billing_admin" {
  instance_arn       = tolist(data.aws_ssoadmin_instances.this.arns)[0]
  permission_set_arn = aws_ssoadmin_permission_set.billing_admin.arn
  inline_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["aws-portal:*Billing", "aws-portal:*Usage", "ce:*"]
      Resource = "*"
    }]
  })
}

session_duration is an ISO 8601 duration, and it's the actual cap on credential lifetime for anyone assigned this permission set — not overridable at assumption time. I keep BillingAdmin at PT1H and ReadOnlyAccess at PT4H since read-only browsing sessions are lower risk than anything that touches spend controls.

How permission sets map to account assignments

This is the part that trips people up coming from single-account IAM thinking: a permission set is a template, not a role. Nothing exists in a member account until you create an account assignment — at that point Identity Center provisions an actual IAM role in the target account, named AWSReservedSSO_<permission-set-name>_<hash>, with a trust policy that allows the Identity Center service to assume it. You'll see these roles show up in IAM → Roles in every account you've assigned; don't hand-edit them, they're managed by Identity Center and your edits will get silently reverted on the next provisioning sync.

Assignment is a three-way link — group, permission set, account:

resource "aws_ssoadmin_account_assignment" "readonly_staging" {
  instance_arn       = tolist(data.aws_ssoadmin_instances.this.arns)[0]
  permission_set_arn = aws_ssoadmin_permission_set.read_only.arn

  principal_id   = data.aws_identitystore_group.engineers.group_id
  principal_type = "GROUP"

  target_id   = "222233334444" # staging account
  target_type = "AWS_ACCOUNT"
}

Walkthrough: one group, three accounts

Say I want the Engineers group to have ReadOnlyAccess in dev, staging, and prod, but a separate PowerUser permission set only in dev. First, look up the group by its SCIM-synced display name:

data "aws_identitystore_group" "engineers" {
  identity_store_id = tolist(data.aws_ssoadmin_instances.this.identity_store_ids)[0]

  alternate_identifier {
    unique_attribute {
      attribute_path  = "DisplayName"
      attribute_value = "Engineers"
    }
  }
}

locals {
  readonly_accounts = {
    dev     = "111122223333"
    staging = "222233334444"
    prod    = "333344445555"
  }
}

resource "aws_ssoadmin_account_assignment" "readonly" {
  for_each = local.readonly_accounts

  instance_arn       = tolist(data.aws_ssoadmin_instances.this.arns)[0]
  permission_set_arn = aws_ssoadmin_permission_set.read_only.arn
  principal_id       = data.aws_identitystore_group.engineers.group_id
  principal_type     = "GROUP"
  target_id          = each.value
  target_type        = "AWS_ACCOUNT"
}

resource "aws_ssoadmin_account_assignment" "poweruser_dev" {
  instance_arn       = tolist(data.aws_ssoadmin_instances.this.arns)[0]
  permission_set_arn = aws_ssoadmin_permission_set.power_user.arn
  principal_id       = data.aws_identitystore_group.engineers.group_id
  principal_type     = "GROUP"
  target_id          = local.readonly_accounts.dev
  target_type        = "AWS_ACCOUNT"
}

Apply that and everyone in Engineers sees three tiles for ReadOnlyAccess (one per account) and one extra tile for PowerUser in dev, in the Identity Center portal. If you'd rather click through the console: Identity Center → Permission sets → select one → nothing happens there for assignment; you actually go to Multi-account permissions → select account(s) → select the permission set → select the group. It's easy to land on the wrong page first time because assignment lives under the account list, not the permission set page.

CLI usage: `aws sso login`

Once assigned, configure a named profile pointing at the SSO session:

# ~/.aws/config
[sso-session my-org]
sso_start_url = https://my-org.awsapps.com/start
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[profile staging-readonly]
sso_session = my-org
sso_account_id = 222233334444
sso_role_name = ReadOnlyAccess
region = us-east-1

aws sso login --profile staging-readonly
aws sts get-caller-identity --profile staging-readonly

aws sso login opens a browser, you authenticate against your IdP (or the built-in directory), and the CLI caches temporary credentials locally, scoped to the permission set's session_duration. When they expire, you just re-run aws sso login — no key rotation, nothing to revoke in the account itself, because there was never a long-lived credential there to begin with.

Gotchas learned the hard way

Permission set edits don't auto-propagate. If you change a permission set's policy (add a managed policy, edit the inline JSON), existing account assignments keep the old IAM role permissions until you re-provision. In the console this is the "Reprovision" button on the permission set's Accounts tab; via API/Terraform it's aws sso-admin provision-permission-set triggered automatically by terraform apply for aws_ssoadmin_permission_set changes — but if you're editing an inline policy attached via a separate resource, double check the apply actually touched the permission set resource itself, not just the attachment, or the role in the target account can silently stay stale for hours.
SCIM sync lag. A new group or user added in your IdP doesn't appear in Identity Center instantly — SCIM sync typically runs every 15-40 minutes depending on the IdP. If you just created a group in Okta and your aws_identitystore_group data source in Terraform can't find it, that's not a Terraform bug, it's sync lag. Check Identity Center → Settings → Automatic provisioning for the last sync timestamp before you start debugging your HCL.
Deleting a group in the IdP doesn't clean up assignments. SCIM deprovisioning removes the user/group from the Identity Store, but I've seen orphaned aws_ssoadmin_account_assignment state entries hang around if the group was deleted out-of-band from Terraform. terraform plan will complain it can't find the referenced group ID. Clean up assignments in code before deleting the group upstream, not after.
Session duration is a hard cap, not a suggestion. Unlike an IAM role you can AssumeRole into with a custom DurationSeconds up to the role's max, the permission set's session_duration is what you get — there's no per-login override. If your CI needs longer sessions than your interactive permission set allows, that's a sign CI shouldn't be using an Identity Center human-login flow at all; use a workload identity (OIDC role assumption, like I covered in the GitHub Actions post) instead.
The management account itself is a special case. You generally don't want broad permission sets assigned in the Organizations management account — keep assignments there to a small break-glass set, and do real work through member/workload accounts. It's easy to accidentally grant AdministratorAccess in the management account because it's first in the account picker.

Trade-offs vs per-account IAM users

Single point of login, single point of failure. If Identity Center or your IdP has an outage, nobody can get interactive access to any account through SSO. This is why you keep a small number of emergency IAM users (with hardware MFA, credentials in a sealed vault process, not routine use) for break-glass — Identity Center replaces routine human access, not disaster recovery access.
No more per-account password/MFA sprawl, but you've now got a much bigger blast radius on the identity source itself — securing your IdP (and its MFA policy) matters more than it used to, because it's now the gate to every account at once.
Auditing gets dramatically better. aws sts get-caller-identity and CloudTrail both show the assumed-role session name, which Identity Center populates with the actual user's identity — so AssumedRole events in CloudTrail are attributable to a person, not a shared IAM user ARN that three people know the password to.
Programmatic/service access still isn't Identity Center's job. Permission sets and aws sso login are for humans. Machine-to-machine auth (CI/CD, Lambda, EC2) should stay on IAM roles with OIDC federation or instance profiles — don't try to shoehorn a service account into an Identity Center permission set just to avoid having two auth patterns in your org.

Rolling it out

Don't flip every account to Identity Center-only in one PR. Stand up Identity Center, get SCIM sync working, create permission sets, and assign them in one low-risk account (I used a scratch sandbox account) first — confirm people can actually log in via the portal and get the access you expect. Then extend account assignments outward account by account, and only after everyone's confirmed they can get in through SSO do you go back and deprovision the old per-account IAM users and their access keys. Keep a short overlap window rather than a hard cutover date; the failure mode of "we deleted the IAM user before confirming the SSO group assignment actually worked" is a locked-out engineer and an emergency root login, not a fun afternoon.

Diagnosing and fixing Lambda cold starts that matter

Mon, 10 Aug 2026 00:00:00 GMT

"Just use provisioned concurrency" is the answer I hear most often when someone complains about Lambda cold starts, and it's usually wrong — it fixes the symptom, costs money every hour whether or not you're invoked, and skips the part where you find out why your function is slow to initialize in the first place. Most cold start problems are fixable for free: a leaner deployment package, a runtime that doesn't need to JIT-warm, or removing a VPC attachment that was never actually necessary. This is the diagnostic process I use before reaching for the checkbook.

Measure it before you touch anything

Every Lambda invocation writes a REPORT line to CloudWatch Logs. On a cold start, it includes an Init Duration field that the warm-start version doesn't have:

REPORT RequestId: 8f3e...  Duration: 412.33 ms  Billed Duration: 413 ms
Memory Size: 512 MB  Max Memory Used: 98 MB  Init Duration: 621.47 ms

That Init Duration is the number to chase — it's time spent on the execution environment bootstrapping the runtime, running module-level code outside your handler, and (if configured) resolving your VPC ENI. It is not included in billed duration for most runtimes, so it won't show up as a cost spike, only as added latency your caller feels. Don't confuse it with Duration, which is your handler's own execution time and is what you're billed for.

To pull this across many invocations instead of eyeballing one log line, use CloudWatch Logs Insights:

fields @timestamp, @initDuration, @duration
| filter ispresent(@initDuration)
| stats count(*) as coldStarts,
        avg(@initDuration) as avgInit,
        pct(@initDuration, 95) as p95Init
  by bin(1h)

The filter ispresent(@initDuration) line is what isolates cold starts — warm invocations don't emit that field at all, so this query effectively gives you a cold-start rate and latency distribution per hour, for free, with no X-Ray required.

X-Ray is worth turning on (Tracing: Active in the function config) when you need to see where inside init time is going — module imports, SDK client construction, secrets/config fetched at startup — rather than just how long it took. In the trace timeline, the Initialization segment sits before your first subsegment; if it's dominated by something like a boto3 client build or a config file fetched from Secrets Manager at import time, that's your target, not the runtime itself.

Runtime choice moves the floor, not just the average

Interpreted runtimes (Python, Node.js, Ruby) have low inherent init overhead — a few hundred milliseconds — because there's no compilation step, just interpreter startup and module loading. Compiled/JIT runtimes (Java, C# on .NET, and to a lesser extent Go, which compiles to a static binary ahead of time) trade that off: Go's cold start is often the fastest of all because there's no runtime to boot at all, while JVM- and CLR-based functions pay for class loading and JIT warm-up on every cold start, frequently 1-3 seconds for anything beyond a trivial handler.

If you're stuck on Java for ecosystem reasons, Lambda SnapStart is the single biggest lever available: it takes a pre-initialized, encrypted snapshot of your execution environment's memory and disk state after your static initializers and any registered beforeCheckpoint hooks run, then resumes from that snapshot on cold start instead of re-running init from scratch. In practice this takes Java functions from multi-second cold starts down to sub-200ms for many workloads. It's opt-in per function (SnapStart: ApplyOn: PublishedVersions), only applies to published versions (not $LATEST), and anything non-deterministic in your static init — random values, UUIDs, timestamps, opened network connections — needs to be regenerated in a beforeCheckpoint/afterRestore hook, or you'll ship the same "random" value to every restored environment. That's the gotcha that bites people first: a cached DB connection captured in the snapshot resumes in a stale, sometimes already-closed state on the other side.

Package size is the free win everyone skips

Init duration scales with how much code Lambda has to unzip and load before your handler is reachable, and it's rarely your own code that's the problem — it's the dependency tree. A Node function that pulls in the entire AWS SDK v2 (aws-sdk) when it calls one S3 method drags in tens of megabytes it never touches at runtime.

# See what's actually contributing to package size
du -sh node_modules/* | sort -rh | head -10

# Node: import only the client you use (SDK v3 is modular by design)
npm uninstall aws-sdk
npm install @aws-sdk/client-s3

For Python, the equivalent is trimming requirements.txt to what's imported at module scope and pushing anything only used inside rarely-hit code paths to a lazy import inside the function body — module-level imports run during init, so a heavy library imported "just in case" costs every cold start, not just the invocations that use it. For any runtime, moving large, rarely-changing dependencies into a Lambda Layer doesn't reduce unzip-and-load time by itself, but it does let you avoid re-uploading (and Lambda re-validating) a multi-hundred-MB deployment package on every code change, which matters more for deploy latency than cold starts.

Container-image Lambdas deserve a specific warning here: they're pulled from ECR and have historically had noticeably worse cold starts than zip packages at larger image sizes, though Lambda's own image-caching layer has narrowed that gap significantly since launch. If you're on container images purely out of habit and your image is small, a zip package with a layer is very likely faster to cold-start.

VPC attachment: mostly a solved problem, still worth checking

Before 2019, attaching a Lambda to a VPC meant provisioning an ENI per concurrent execution environment, which could add 10 seconds or more to a cold start. AWS's Hyperplane-based networking model eliminated most of that by sharing ENIs across functions in the same VPC/subnet/security-group combination, and current VPC-attached cold starts are typically within tens to a couple hundred milliseconds of non-VPC ones. If you're still carrying a workaround from that era — a warm-up cron job, an oversized provisioned-concurrency pool sized for the old ENI cost — it's worth re-measuring with the Logs Insights query above before assuming you still need it. The overhead that remains is small but non-zero, so don't attach a function to a VPC it doesn't need just because a sibling function does; scope VPC config per function, not per stack.

When provisioned concurrency actually earns its cost

Provisioned concurrency pre-initializes a pool of execution environments and keeps them warm, billed hourly whether invoked or not — it doesn't reduce init duration, it just makes sure fewer invocations ever hit it. It's worth the always-on cost when:

You have a synchronous, latency-sensitive caller (API Gateway, an ALB, or a user-facing request path) where p99 latency is a product requirement, not a nice-to-have.
Traffic is spiky rather than steady — a steady high-volume function naturally stays warm from its own invocation rate and rarely cold-starts regardless.
You've already trimmed package size and picked the leanest viable runtime, and the remaining init duration is still unacceptable — provisioned concurrency should be the last lever, not the first.

It's overkill for async, batch, or event-driven functions (S3/SQS/EventBridge triggers) where an extra few hundred milliseconds on an occasional invocation is invisible to anyone. Application Auto Scaling can scale provisioned concurrency on a schedule (business-hours-only) or target-tracking policy, so if you do need it, don't just set a flat number sized for peak — that's paying peak-capacity prices around the clock for a curve that isn't flat.

Rollout: measure, trim, then buy

Start with the Logs Insights query above running against production traffic for a week to get a real cold-start rate and p95 Init Duration baseline — don't optimize against a guess. Fix package bloat and runtime choice first; both are free and often cut init duration by half or more on their own. Only reach for provisioned concurrency once you've re-measured after those changes and still have a latency-sensitive path that misses its SLA — and even then, size it with Application Auto Scaling against your real traffic curve, not a number that felt safe.

Scoping IAM Policies with Tag, IP, and MFA Conditions

Mon, 10 Aug 2026 00:00:00 GMT

Most IAM policies I inherit from a team are scoped by resource ARN and nothing else: "this role can ec2:* on these instances." That's a start, but it leaves gaps a determined or careless caller can walk through — an engineer's laptop credentials reaching production from a coffee shop, a role that can touch every EC2 instance in the account regardless of which team owns it, a sensitive action that only checks "are you authenticated" rather than "did you actually use MFA to get here." IAM's condition keys close those gaps without adding a second system to manage. They live right inside the policy document, they're evaluated by the same engine, and once you know the handful that matter, they cover almost every real-world scoping requirement. Here's how I use them in practice, on real production policies.

Tag-based scoping: aws:ResourceTag and aws:RequestTag

If your account has more than one team's resources in it, tag-based scoping is the highest-leverage condition you can add. Instead of enumerating ARNs (which breaks the moment someone launches a new instance), you scope by tag and let your tagging discipline do the work:

{
  "Effect": "Allow",
  "Action": ["ec2:StartInstances", "ec2:StopInstances", "ec2:RebootInstances"],
  "Resource": "arn:aws:ec2:*:123456789012:instance/*",
  "Condition": {
    "StringEquals": {
      "aws:ResourceTag/team": "platform"
    }
  }
}

aws:ResourceTag checks the tag already on the resource being acted on — use it for read/modify/delete actions. For actions that create a resource, there's no tag on it yet at evaluation time, so you need aws:RequestTag instead, checking the tag the caller is trying to apply:

{
  "Effect": "Allow",
  "Action": "ec2:RunInstances",
  "Resource": "arn:aws:ec2:*:123456789012:instance/*",
  "Condition": {
    "StringEquals": {
      "aws:RequestTag/team": "platform"
    }
  }
}

Gotcha: this only stops people who use the console or API correctly — it does nothing if the tag can simply be omitted or changed later. Pair it with a Deny that blocks untagged creation and, separately, blocks ec2:DeleteTags / ec2:CreateTags on the team key for anyone outside your platform-admin role. Otherwise "scoped by tag" degrades into "scoped by tag until someone removes the tag," which isn't a security boundary at all — it's a labeling convention people are trusting each other to respect.

Network scoping: aws:SourceIp and aws:VpcSourceIp

aws:SourceIp restricts API calls to a CIDR range — your office IP, your VPN egress, or (less usefully) 0.0.0.0/0. It's most valuable on IAM users or roles that still authenticate outside of AWS-managed network paths (a CI runner with a static egress IP, a break-glass admin user):

{
  "Effect": "Deny",
  "NotAction": ["iam:ChangePassword", "iam:GetUser"],
  "Resource": "*",
  "Condition": {
    "NotIpAddress": {
      "aws:SourceIp": ["203.0.113.0/24", "198.51.100.0/24"]
    },
    "Bool": {
      "aws:ViaAWSService": "false"
    }
  }
}

Two things worth calling out. First, aws:SourceIp looks at the caller's public IP for calls made directly to AWS, but for calls made from inside a VPC through a VPC endpoint, you want aws:VpcSourceIp instead — it's the private IP inside the VPC, and it only populates when the call actually transits a VPC endpoint. Mixing the two up is a common reason a network-scoped policy silently fails for endpoint traffic: the condition key you checked simply isn't present on that request, so it evaluates as "condition not met" and the whole statement is skipped.

Second, that aws:ViaAWSService: false condition isn't decoration — without it, this Deny also blocks AWS services calling APIs on your behalf (e.g., CloudFormation invoking IAM during a stack operation), because those calls don't carry your IP at all and would otherwise fail the NotIpAddress check and get denied. This is the single most common way I've seen an IP-restriction policy break someone's CI pipeline the day after it ships.

Enforcing MFA: aws:MultiFactorAuthPresent and aws:MultiFactorAuthAge

For your most sensitive actions — deleting a CloudTrail trail, deactivating GuardDuty, changing another user's credentials — require not just "authenticated" but "authenticated with MFA, recently":

{
  "Effect": "Deny",
  "Action": [
    "iam:DeleteUser",
    "iam:DeleteRole",
    "iam:UpdateAccessKey",
    "cloudtrail:StopLogging",
    "guardduty:DeleteDetector"
  ],
  "Resource": "*",
  "Condition": {
    "BoolIfExists": {
      "aws:MultiFactorAuthPresent": "false"
    }
  }
}

Use BoolIfExists rather than a plain Bool here. aws:MultiFactorAuthPresent is simply absent from the request context for some call types (notably calls made with temporary credentials from certain federation flows, or service-to-service calls), and a plain Bool condition against a missing key evaluates as false — which, counterintuitively, means the Deny condition ("aws:MultiFactorAuthPresent": "false") matches and the action gets denied even for legitimate non-interactive callers. BoolIfExists only evaluates the condition when the key is present, so it doesn't misfire on requests where MFA presence genuinely isn't reportable.

Add aws:MultiFactorAuthAge (in seconds) if you want to force re-authentication for stale sessions rather than trusting an MFA check from ten hours ago:

"Condition": {
  "NumericGreaterThan": {
    "aws:MultiFactorAuthAge": "3600"
  }
}

Combining conditions safely: the Deny/NotAction trap

The pattern above — Effect: Deny plus NotAction — is powerful but easy to get backwards. NotAction in a Deny statement means "deny everything except these actions," so the actions you list are the ones exempted from the deny, not the ones targeted by it. I've seen policies where someone wanted to restrict a set of dangerous actions and reached for NotAction, accidentally exempting exactly the actions they meant to lock down while denying everything else in the account. If your intent is "deny these specific actions unless X," use a plain Action list. Reserve NotAction for "deny everything except these few things" — the safe subset you're carving out, like the two IAM self-service calls in the aws:SourceIp example above, which a locked-out user still needs to fix their own password.

The other non-obvious trap: explicit Deny always wins, but a missing condition key doesn't always mean "deny." Whether an absent key trips the condition depends entirely on which condition operator you used (Bool vs BoolIfExists, StringEquals vs StringEqualsIfExists). IAM doesn't warn you when a policy's condition silently never matches because the key isn't in the request context — it just evaluates false and moves on, and the action proceeds under whatever your next-most-permissive statement allows. Test conditional Deny policies with the IAM Policy Simulator against the actual principal and a realistic set of request parameters before you rely on them, not just against a policy you're reading and assuming is correct.

Rolling this out

Don't attach a new condition-scoped Deny directly to a broad group and walk away. Roll it out the way I roll out any access-tightening change: attach it to a single test principal first, run the actual workflows that principal needs (including CI and automation, not just interactive console use), check CloudTrail for errorCode: AccessDenied events you didn't expect, and only then widen the attachment to the group or account-wide SCP. Tag-based and MFA-based conditions are cheap to write and expensive to debug in production if you get the IfExists variant wrong — the fifteen minutes in the policy simulator is worth it.

Terraform S3 native locking: kill your DynamoDB table

Mon, 10 Aug 2026 00:00:00 GMT

For as long as I've been writing Terraform, "S3 backend" meant "S3 backend plus a DynamoDB table for locking," full stop. You'd provision a tiny pay-per-request table, grant your CI role dynamodb:GetItem/PutItem/ DeleteItem on it, and never think about it again except when someone asked "wait, why do we have a DynamoDB table for a static site's infra?" As of Terraform 1.10, that table is no longer required — S3 itself can do the locking, using conditional writes. This site's own terraform/backend.hcl runs on exactly this setup, no DynamoDB table anywhere in the account. Here's the history, the config, and the migration path if you're still running the old way.

Why DynamoDB was ever in the picture

State locking exists to stop two terraform apply runs from racing each other and corrupting state — classic "last writer wins" data loss. S3 never had a native compare-and-swap primitive, so Terraform's S3 backend piggybacked on DynamoDB's conditional PutItem (attribute_not_exists) to implement a lock: acquire a lock by writing an item keyed on the state path, release it by deleting the item. It worked well, but it meant every S3 backend needed a second AWS service, a second IAM policy, and a second thing to provision correctly before your very first terraform init — plus a manual dynamodb:DeleteItem to clean up a stuck lock after a killed CI job, which everyone running Terraform in CI has done at least once.

Amazon added conditional writes to S3 itself in August 2024 (If-None-Match / If-Match support on PutObject), and HashiCorp shipped support for using that directly as a locking mechanism in Terraform 1.10 (November 2024) via a new backend argument: use_lockfile. No DynamoDB, no second service, no separate IAM policy — the same S3 permissions your state already needed now cover locking too.

The config

This is the actual backend block from this repo, split the way Terraform backends normally are: the non-secret pieces in a partial config file (terraform/backend.hcl), and the empty backend "s3" {} stanza in versions.tf so nothing hardcoded ends up needing per-environment variables.

# terraform/versions.tf
terraform {
  required_version = ">= 1.10"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.60, < 8.0"
    }
  }

  # Remote state in S3 with native locking (Terraform >= 1.10, no DynamoDB).
  # Non-secret values are supplied at init time via backend.hcl.
  backend "s3" {}
}

# terraform/backend.hcl
bucket       = "jstgtech-web-tfstate"
key          = "site/terraform.tfstate"
region       = "us-east-1"
encrypt      = true
use_lockfile = true

terraform init -backend-config=backend.hcl

That's the whole thing. use_lockfile = true tells the S3 backend to write a <key>.tflock object next to your state object during plan/apply, using a conditional PutObject so a second concurrent run gets a hard failure instead of a silent overwrite, and deletes the lock object when the run finishes (or on terraform force-unlock if a run got killed before it could clean up). No dynamodb_table argument anywhere — the S3 backend has supported that argument for years for the old locking path, and you can technically still set both during a transition (more on that below), but a fresh setup like this one just doesn't need it.

Gotchas that'll actually bite you

Versioning on the state bucket is still mandatory, and separate from ** locking.** Locking stops concurrent writes; versioning is what saves you when someone runs apply against a bad plan or you need to roll back a corrupted state file. use_lockfile doesn't touch this at all — you still need aws_s3_bucket_versioning with status = "Enabled" on the bucket, the same as the DynamoDB-locking days. This repo's bootstrap layer (terraform/bootstrap/main.tf) sets it explicitly:

 resource "aws_s3_bucket_versioning" "tfstate" {
    bucket = aws_s3_bucket.tfstate.id
    versioning_configuration {
      status = "Enabled"
    }
  }

Your IAM role needs write permissions it might not already have. The lock file is a real S3 object, so whatever role runs Terraform needs s3:PutObject and s3:DeleteObject on the state bucket — most roles that could already read/write state already have this, but if you'd scoped a role down to s3:GetObject + s3:PutObject on just the state key, tighten it to cover the <key>.tflock path too (or just the whole prefix, which is simpler and what most setups do — s3:* scoped to the bucket ARN, not wildcarded across accounts).
The backend block cannot reference variables, locals, or anything ** computed — this predates native locking but people relearn it every time.** You cannot do backend "s3" { bucket = var.state_bucket }. That's why backend "s3" {} is empty in versions.tf and everything lives in backend.hcl, passed at init time with -backend-config. If you have multiple environments, you'll have multiple .hcl files (or a templated one generated by CI before init), not variables inside the block.
use_lockfile needs Terraform ≥ 1.10, checked hard. If someone on your team is still on 1.9 (or an old pinned CI container), init will just error on the unrecognized argument. Bump required_version in the same PR that adds use_lockfile so the mismatch fails loud at init instead of someone silently running an older binary against a backend config it doesn't understand.
Stuck locks are unlocked differently now. With DynamoDB you'd aws dynamodb delete-item the lock row by hand, or use terraform force-unlock <LOCK_ID> which did the same thing under the hood. With native locking, force-unlock still works — Terraform reads the lock ID from the .tflock object's contents — but if you're ever debugging by hand, you're looking for an S3 object, not a DynamoDB item, and it's named <key>.tflock next to your actual state object in the same bucket.

Migrating an existing DynamoDB-locked backend

If you've got a working dynamodb_table setup today, don't just delete the table and flip use_lockfile on in the same change — do it in two steps so you're never in a state where in-flight infra changes could hit a locking config mismatch:

**Add use_lockfile = true to backend.hcl alongside the existing dynamodb_table entry**, then run:

   terraform init -reconfigure

-reconfigure (not -migrate-state) is correct here — you're not moving state to a new bucket/key, you're just changing backend configuration on the same backend type. Terraform will pick up native locking going forward. At this point both mechanisms are technically configured, but Terraform only uses one lock mechanism per backend version — as of the 1.10–1.13 line, setting both dynamodb_table and use_lockfile together is explicitly supported as a transition state precisely for this migration.

Run a real plan and apply to confirm the new lock path works — watch for the .tflock object appearing in the state bucket during the run (aws s3 ls s3://your-tfstate-bucket/ --recursive | grep tflock while it's running, or just check after a plan that holds the lock long enough).
Once you're confident, remove dynamodb_table from backend.hcl and run terraform init -reconfigure again. Nothing about your state file changes in this step — you're purely dropping a backend config key.
Delete the DynamoDB table — but not immediately. Leave it a week or two after step 3 in case you need to roll a teammate's stale local config back, then terraform state rm/destroy it via whatever provisioned it originally (if the table itself was Terraform-managed in a bootstrap layer, remove the resource block and apply; if it was created by hand, just delete it from the console or CLI). For a low-traffic project this is maybe $1–2/month in the pay-per-request billing mode most people use for lock tables, so the savings are more about **one fewer resource to provision, IAM-scope, and explain to the next person reading your Terraform** than meaningful dollars — the real win is operational, not financial.

When DynamoDB locking still earns its keep

I wouldn't rip it out reflexively everywhere:

You're pinned below Terraform 1.10 for reasons outside your control (an older provider requiring an old core version, an internal policy that hasn't approved the upgrade yet). use_lockfile simply isn't available.
Multiple tools lock against the same state outside Terraform's own CLI — e.g., a custom automation script or a different IaC tool cooperating on the same DynamoDB lock table by convention. S3 native locking is Terraform-backend-specific; nothing else in that ecosystem understands .tflock objects the way tools built around the DynamoDB API might.
You're on OpenTofu and haven't checked whether your pinned version has the equivalent support yet — OpenTofu forked before this landed and added it on its own timeline, so don't assume version-number parity with Terraform means feature parity here.

For a single-account, single-tool setup like this site's — GitHub Actions running terraform plan/apply via OIDC, nothing else touching state — none of those apply, which is exactly why terraform/backend.hcl here has no dynamodb_table line at all.

Service spotlight: Amazon Bedrock and its guardrails feature

Mon, 10 Aug 2026 00:00:00 GMT

Every team building on top of a foundation model eventually asks the same question: call the provider's API directly, or go through a managed layer. Bedrock's answer is "stay inside your AWS account boundary and get one API across multiple model providers" — worth understanding precisely, since "it's just an API wrapper" undersells what it actually replaces, and overselling it as "AWS's AI service" undersells how much is still on you.

What it actually is

Bedrock is a managed access layer to foundation models from multiple providers — Anthropic's Claude family, Meta's Llama, Amazon's own Nova models, Mistral, and others — through one consistent API and SDK, with no GPU infrastructure for you to provision or manage. You're not running a model; you're calling one that AWS hosts, billed per request. The value proposition over calling a provider directly is staying inside your existing AWS security boundary: IAM for auth instead of a separate API key to rotate and store, VPC endpoints so traffic never leaves AWS's network, CloudTrail logging every invocation, and data that (per AWS's terms) isn't used to train underlying models — which matters if your compliance posture already assumes AWS's shared responsibility model and adding a new third-party vendor relationship is its own review cycle.

Guardrails

Guardrails for Amazon Bedrock is a configurable content-filtering layer you attach to any model call, independent of which underlying model you're using. It can block or mask categories of content (hate speech, violence, prompt-injection attempts), redact detected PII in both the prompt and the response (SSNs, credit card numbers, emails — configurable per field), enforce topic denial (block responses about specific configured topics entirely — "don't discuss competitor products," for instance), and run a contextual grounding check that scores a RAG response against its source documents to catch hallucination before it reaches the user. It runs as a policy applied at invocation time, which means you can update filtering rules without redeploying application code or retraining anything.

The honest caveat: guardrails are pattern- and classifier-based, not a guarantee. They meaningfully reduce the surface area for bad outputs and give you an audit trail of what was blocked, but they're a mitigation layer, not a substitute for output validation in anything security- or compliance-sensitive downstream.

Where it earns its keep

Multi-model flexibility without rewriting integration code. Swapping which model handles a workload — testing Claude against Llama for a given task, or moving to a newer model version — is a parameter change in the same API call, not a new SDK integration.
Regulated environments where "customer data leaves AWS's network boundary" is itself the compliance blocker, independent of which model is actually good enough for the task.
Fine-tuning and RAG without standing up your own vector infrastructure. Bedrock Knowledge Bases handles embedding, chunking, and retrieval against a vector store (OpenSearch Serverless, Aurora, Pinecone) with a managed ingestion pipeline, if you'd rather not build that yourself.

The pricing gotcha

Bedrock bills per token, in and out, and pricing varies by model — a larger, more capable model can be an order of magnitude more expensive per token than a smaller one for the same request. The trap is defaulting every call in an application to the largest/newest model "because it's the best," when a cheaper model handles the bulk of routine requests (simple classification, short summarization) just as well. Provisioned Throughput (reserved capacity, billed hourly regardless of usage) is worth it only at sustained high volume; below that threshold, on-demand per-token pricing is cheaper even though the per-unit rate looks higher, because you're not paying for idle reserved capacity between requests.

A practical tip

Route by task complexity instead of hardcoding one model for an entire application — a cheap, fast model for classification/extraction/short responses, a larger model reserved for requests that actually need deeper reasoning. Bedrock's consistent API across model families makes that routing a config decision rather than a rewrite, which is most of the point of using a multi-model layer instead of committing to one provider's SDK directly.

Service spotlight: Amazon EventBridge beyond Lambda glue

Mon, 10 Aug 2026 00:00:00 GMT

Most teams meet EventBridge as "the thing that triggers a Lambda on a cron schedule" and stop there. That's a legitimate use, but it undersells the service — EventBridge is an event bus, and the pattern-matching/routing layer underneath it is worth understanding on its own, separate from the scheduler feature that happens to share the console.

What it actually is

An event bus is a named channel that receives JSON events and evaluates them against a set of rules. Each rule has an event pattern — a partial JSON match against the event's fields — and a list of targets (Lambda, SQS, Step Functions, Kinesis, another event bus, over 20 AWS services directly). When an event matches, EventBridge fans it out to every matching rule's targets, in parallel, with built-in retry and an optional dead-letter queue per target. You publish an event once; the bus decides who cares.

Every account gets a default bus that also receives events natively from ~200 AWS services (an S3 upload, an ECS task state change, a CodePipeline stage transition) with zero integration code — that's the part people miss. You can also create custom buses for your own application events, and partner buses for SaaS integrations (Datadog, PagerDuty, Zendesk) that push events directly into your account.

Where it beats point-to-point wiring

The alternative to EventBridge is usually "Lambda A calls Lambda B directly" or "publish to an SNS topic with N subscriptions." Both work, but they couple the publisher to knowing who's listening. EventBridge inverts that: the publisher emits one event describing what happened, and consumers declare what they care about via pattern matching, without the publisher's code ever changing when a fourth consumer shows up. Adding "also send a Slack notification when an order ships" is a new rule, not a code change to the order service.

Schema discovery is the other underused feature: point EventBridge at a bus and it infers a schema registry from the events flowing through it, including versioning as the shape evolves, and can generate strongly-typed bindings (Java, Python, TypeScript) for consumers. For anything beyond a handful of hand-maintained event types, that beats a wiki page describing "here's what fields this event has, trust me."

Archive and replay

Every bus can have an archive attached — EventBridge stores a copy of matching events (with a retention period you set, including indefinite) and can replay them into the bus later, re-triggering the same rules. That's the feature that turns "we had a bug in the order-confirmation-email Lambda for six hours" from a data-backfill script into a console button: fix the Lambda, replay the archived window, done. It's also genuinely useful for testing — replay production events into a rule pointed at a dev target instead of hand-crafting test payloads.

The rule-limit gotcha

Each event pattern rule can have at most 5 targets, and a bus can have up to 300 rules by default (a soft limit, raisable). Teams that treat EventBridge as a single giant bus with dozens of broad-matching rules and long target lists eventually hit both limits and end up debugging which of 15 similar-looking rules actually matched a given event — the console's "test pattern" tool helps, but the real fix is scoping rules narrowly by detail-type and source from the start rather than one catch-all rule per consumer with a growing target list.

Pricing

Custom events published to a custom bus are billed per million events; events matched from AWS service integrations on the default bus are free — only your own published events cost anything. Archive storage is billed separately by GB, and replays re-invoke targets, so replaying a large archived window against a Lambda target bills exactly like the original traffic did. It's cheap at normal volumes, but a botched pattern that accidentally matches everything (an empty or overly broad source filter) and fans out to an expensive target is the way people get a surprise line item, same as any fan-out system.

A practical tip

Give every custom event a detail-type and source from day one, even for a single-consumer event — retrofitting rule scoping after five teams are publishing to the same bus with inconsistent field names is far more painful than establishing the convention up front. If you're publishing from multiple services, agree on a naming scheme (source: "orders.checkout", detail-type: "OrderPlaced") before the first event ships, not after.

Service spotlight: Amazon SQS queue depth, DLQs, and visibility

Mon, 10 Aug 2026 00:00:00 GMT

SQS is old, boring, and one of the most reliable services AWS runs — which is exactly why it's easy to wire up carelessly and not notice until a queue backs up in production. The service itself rarely fails; the mistakes are almost always in how visibility timeouts, DLQs, and consumer scaling are configured around it.

The core model, briefly

A producer sends a message; a consumer polls, receives it, processes it, and explicitly deletes it. Between receive and delete, the message is invisible to other consumers for the visibility timeout duration — not deleted, just hidden, so a second consumer doesn't pick up the same message while the first is still working it. If the consumer crashes, times out, or never calls delete, the message reappears after the timeout expires and gets redelivered. That reappear-on-failure behavior is the entire reliability model, and most SQS production issues trace back to a mismatch somewhere in that loop.

Mistake 1: visibility timeout shorter than processing time

If your consumer takes 90 seconds to process a message but the queue's visibility timeout is set to the default 30 seconds, the message becomes visible again while it's still being processed — a second consumer picks it up, and now you're processing the same message twice concurrently. For anything non-idempotent (charging a card, sending an email, decrementing inventory), that's a correctness bug, not just wasted compute. Set the visibility timeout to comfortably exceed your maximum expected processing time, not the average — or use ChangeMessageVisibility to extend it dynamically from inside a long-running handler if processing time varies widely.

Mistake 2: no DLQ, or a DLQ nobody watches

A dead-letter queue catches messages that fail processing repeatedly (by maxReceiveCount) instead of retrying forever and blocking the queue behind one poison message. Skipping a DLQ means a single malformed message can loop indefinitely, burning consumer capacity on retries that will never succeed. But the more common failure is having a DLQ and never alarming on it — messages quietly pile up, nobody notices for weeks, and by the time someone checks, the redrive window (or business relevance of the messages) has passed. A DLQ without a CloudWatch alarm on ApproximateNumberOfMessages Visible is a DLQ that isn't doing its job.

Mistake 3: treating queue depth as the only signal

ApproximateNumberOfMessagesVisible tells you how many messages are waiting, but on its own it doesn't tell you whether that's a processing outage or just a traffic spike your consumers will burn down in ten minutes. Pair it with ApproximateAgeOfOldestMessage — a queue that's deep but where the oldest message is only 90 seconds old is healthy and scaling; a queue where the oldest message is 40 minutes old means something downstream is actually stuck, regardless of current depth. Age-of-oldest is usually the better alarm trigger for "something is broken" versus "we're busy."

Standard vs FIFO

Standard queues are at-least-once delivery with best-effort ordering and effectively unlimited throughput — the right default for most workloads, and your consumer logic needs to be idempotent regardless (duplicate delivery is a normal, expected occurrence, not an edge case). FIFO queues add exactly-once processing and strict ordering **within a message group**, at the cost of a throughput ceiling (3,000 messages/sec with batching, per API action) and higher per-request cost. Reach for FIFO only when ordering is a genuine correctness requirement (e.g., applying account balance changes in sequence) — defaulting to FIFO "to be safe" trades away throughput headroom for a guarantee most workloads don't actually need.

A practical tip

Set a redrive policy that moves messages back from the DLQ to the source queue (SQS supports this natively now, no custom script needed) as part of your incident-response runbook, not as something you improvise during an outage. Deciding in advance how many retries a message deserves before landing in the DLQ, and what "redrive after the bug is fixed" actually looks like operationally, is a five-minute conversation before go-live and a much worse one at 2am with a few thousand stuck messages.

Service spotlight: Aurora Serverless v2 without the v1 cold starts

Mon, 10 Aug 2026 00:00:00 GMT

Aurora Serverless v1 had a reputation problem, and it was earned: scale-to- zero meant a cold start that could take tens of seconds on the next connection, and scaling itself worked in coarse steps that didn't handle sudden bursts well. Enough people got burned that "Aurora Serverless" became shorthand for "don't." v2 is a different architecture, not a patched v1, and it's worth revisiting that reputation now that the two are easy to conflate.

What changed

v1 scaled by swapping the entire database to a differently-sized instance behind the scenes — a real cutover with a brief connection drop, done in discrete capacity steps. v2 scales capacity in place, in increments of 0.5 ACUs (Aurora Capacity Units, each roughly 2 GiB of memory plus proportional CPU and networking), typically in under a second, with no connection interruption. You set a min and max ACU range per cluster (as low as 0, up to 256 per instance) and Aurora adjusts within it continuously based on actual load — CPU, memory, and active connections — rather than stepping through a small number of predefined sizes.

Crucially, v2 instances can participate in a Global Database, use Multi-AZ with the same failover mechanics as provisioned Aurora, and even mix with provisioned instances in the same cluster (serverless reader, provisioned writer, or vice versa) — none of which v1 supported. That mixed- instance-class capability is the feature that makes it viable for real production topologies instead of just dev/test databases.

Where it earns its keep

Variable, hard-to-predict load — a multi-tenant SaaS database where tenant activity spikes unpredictably, or a workload with a strong daily/ weekly cycle (busy on weekday mornings, near-idle overnight) where sizing for peak means paying for idle capacity most of the time.
Dev/test/staging environments that see bursts of activity during work hours and near-zero the rest of the time — set a low min ACU and let it ride down without anyone managing an instance-stop/start schedule.
New workloads where you genuinely don't know the right instance size ** yet.** Instead of guessing an r6g.xlarge and resizing later (a disruptive operation on provisioned Aurora), set a wide ACU range and let the actual traffic tell you where it settles — then, if it settles at a steady high number, consider whether provisioned pricing would now be cheaper for that stable load.

Where provisioned Aurora still wins

For a steady, predictable, high-utilization workload, provisioned Aurora with Reserved Instance pricing is cheaper than v2 at the equivalent capacity — the per-ACU-hour rate carries a premium over the equivalent provisioned instance-hour, same trade as Fargate versus EC2. If your database sits at a consistent 8 ACUs of load 24/7 with no meaningful variance, you're paying for elasticity you're not using.

There's also a real floor: 0.5 ACU minimum for an "always-on" cluster is roughly comparable to a small t4g instance's baseline cost — v2 doesn't scale to true zero like v1 could (v2 got a scale-to-zero option later, but it reintroduces a cold-start pause on the next connection, the exact trade-off v2 was built to avoid at higher tiers, so it's really only sane for genuinely idle dev environments).

The scaling-lag gotcha

Scaling is fast but not instant, and it reacts to load — it doesn't predict it. A workload that goes from idle to a hard spike in under a second (a flash-sale-style traffic burst) can briefly get throttled or see elevated latency while Aurora scales up to meet it, because there's a real control loop with a reaction time, not a pre-provisioned buffer sitting ready. Setting a higher minimum ACU than "what average load needs" gives the scaling logic headroom to absorb bursts before it has to react, at the cost of paying for that headroom continuously — a direct latency-versus-cost knob, not a free win.

A practical tip

Watch the ServerlessDatabaseCapacity CloudWatch metric against your min/ max bounds for a couple of weeks under real traffic before treating the range as tuned. A cluster that's pinned at its max ACU most of the day isn't "elastic," it's a provisioned database paying the serverless premium — that's the signal to either raise the max or move to provisioned capacity for the baseline and reserve v2 for genuinely variable secondary workloads.

Service spotlight: AWS AppSync and its resolver-cost model

Mon, 10 Aug 2026 00:00:00 GMT

GraphQL servers usually mean running (and scaling, and patching) a Node process that resolves fields by calling out to your actual data sources. AppSync's pitch is skipping that process entirely — you define a schema and per-field resolvers that AppSync itself executes, with no server of yours in the request path unless a resolver specifically needs one. That's a meaningfully different operational model, and it comes with a cost model that's easy to underestimate if you're used to thinking in request-per- second server pricing.

What it actually is

You upload a GraphQL schema and attach a resolver to each field that needs one — a small piece of logic that maps the incoming GraphQL selection to a call against a data source: DynamoDB, Aurora (via RDS Data API), OpenSearch, EventBridge, HTTP endpoints, or a Lambda function for anything custom. Resolvers used to be written in VTL (Apache Velocity Template Language, borrowed from API Gateway's mapping templates) — verbose, unfamiliar syntax most teams didn't already know. AppSync now supports JavaScript resolvers as the default recommended option, which is a meaningfully better developer experience if you're starting fresh; VTL still works for existing resolvers and some advanced batching patterns.

The other core piece is built-in real-time subscriptions over WebSockets — a client subscribes to a mutation, and AppSync pushes the result to every subscribed client when it fires, with the pub/sub infrastructure entirely managed. That's the feature that's genuinely hard to replicate cheaply with a hand-rolled server; it's Socket.io-style infrastructure you don't have to run.

Where it earns its keep

DynamoDB-backed APIs, especially. A direct AppSync-to-DynamoDB resolver (no Lambda in between) is close to the cheapest, lowest-latency path from a GraphQL query to a NoSQL read — no cold start, no intermediate compute to size or scale.
Apps that need live data — collaborative tools, dashboards, chat — where subscriptions replace a polling loop or a separately-built WebSocket service.
Fine-grained, field-level auth. AppSync resolvers can enforce authorization per field (via Cognito groups, IAM, Lambda authorizers, or OIDC), so a mobile client and an admin client can query the same schema and see different fields without maintaining two APIs or two sets of REST endpoints.

Where it's the wrong tool

If most of your resolvers end up calling a Lambda anyway (because the logic is too custom for a direct data-source resolver), you've mostly rebuilt "API Gateway plus Lambda" with GraphQL schema validation on top — worth it if GraphQL's client-side benefits (one round trip, client-specified shape) matter to you, not worth it if you just wanted a managed API layer. And teams unfamiliar with GraphQL's N+1 problem will hit it here exactly like anywhere else: a nested field resolver that fires once per parent item needs batching (AppSync's batch invoke for Lambda resolvers, or DynamoDB BatchGetItem) or it'll quietly turn one query into hundreds of downstream calls.

The pricing gotcha

AppSync bills per query/mutation request and, separately, per resolver invocation within that request — a query resolving five nested fields is one request but potentially five (or more, with N+1) billed resolver executions, plus real-time subscription connection-minutes and message counts on top if you're using them. A single GraphQL query that looks simple from the client can fan out into a surprising number of billed resolver calls server-side; the AWS docs' pricing page undersells how fast that adds up on a deeply nested schema, and it's worth actually tracing a representative query's resolver count before assuming the bill will track request count linearly.

A practical tip

Use pipeline resolvers to chain multiple resolution steps (an auth check, then a DynamoDB read, then a transform) into a single resolver attached to one field, rather than pushing that logic into nested field resolvers on the schema — it collapses what would be several billed resolver invocations per query into fewer, more predictable ones, and keeps the N+1 pattern from creeping in as the schema grows.

Service spotlight: AWS Backup replaces per-service snapshot scripts

Mon, 10 Aug 2026 00:00:00 GMT

Every AWS account I've inherited that predates AWS Backup has the same shape: a Lambda that snapshots EBS volumes on a cron, a separate RDS automated backup window nobody's checked in years, and DynamoDB point-in-time recovery toggled on for some tables and not others, with no single place to see whether any of it is actually working. AWS Backup exists to replace that patchwork with one policy.

What it actually is

AWS Backup is a managed service that centralizes backup scheduling, retention, and recovery across most stateful AWS resource types — EBS, RDS/Aurora, DynamoDB, EFS, FSx, Storage Gateway volumes, and EC2 instances as a whole (not just their volumes) — under one backup plan. A plan defines a schedule (cron expression), a retention period, and a lifecycle that can transition backups to cold storage and eventually expire them, then you assign resources to the plan by tag or by explicit resource ID. Everything the plan touches shows up in one backup vault with a consistent job history, so "did last night's backup actually succeed" is one console view instead of checking five different services' own backup mechanisms.

Where it earns its keep

One policy instead of five. Tag every resource that needs a daily backup with backup: daily, assign that tag to a plan, and every new EBS volume or RDS instance with that tag is covered automatically going forward — no per-resource setup step anyone can forget.
Cross-service consistency. Retention and lifecycle rules are defined once in the plan, not reimplemented per service with each service's own quirks (RDS automated backups cap at 35 days retention on their own; AWS Backup's retention isn't bound by that).
Backup vault lock. A vault can be locked into compliance mode, which makes backups genuinely immutable and undeletable — including by the account root user — for the configured retention period. That's the feature that actually matters for ransomware resilience: an attacker with full account access still can't delete backups a compliance-locked vault is holding.
Cross-account and cross-region copy. A plan can copy backups to a separate backup account and a separate region in the same job, which is the AWS-native way to satisfy "backups must survive the loss of the source account" without building custom replication.

What it still leaves you to configure

AWS Backup schedules and retains backups; it doesn't validate that a restore actually works. Restore testing is a separate feature (Restore Testing plans) that you have to opt into explicitly and point at real resources — untested backups are a common source of "we had backups but the restore didn't work" incidents, and AWS Backup existing doesn't change that unless you actually run the restore tests.

It also doesn't replace application-consistent backup logic for databases that need it. RDS/Aurora snapshots through AWS Backup use the same underlying mechanism as native RDS snapshots (so they're crash-consistent, and for most engines that's sufficient), but if you're backing up something like a self-managed database on EC2 that needs quiesce-before-snapshot logic, AWS Backup's EC2/EBS support won't give you that for free — you still need your own pre-snapshot hooks.

The pricing model

Costs are backup storage (billed per GB-month, warm vs cold tier priced differently) plus, for some resource types, restore costs. The lifecycle transition to cold storage (available for EBS, RDS, DynamoDB, and a few others) meaningfully cuts long-retention costs, but cold-tier backups typically have a minimum retention period before transition and can carry early-deletion or restore-time costs, so a plan built for "retain 7 years, transition to cold at 90 days" needs those constraints modeled, not just the headline GB-month price.

A practical tip

Turn on AWS Backup Audit Manager (a separate but related feature) once you have plans running — it continuously evaluates whether resources are actually covered by a backup plan matching your organization's policy and flags drift, which catches the "someone launched a new RDS instance without the required tag" gap that a plan alone won't.

Service spotlight: AWS Config for compliance-as-code and drift detection

Mon, 10 Aug 2026 00:00:00 GMT

Terraform tells you what infrastructure should look like at apply time. It doesn't tell you what changed at 3am when someone with console access manually flipped a security group rule to unblock themselves and forgot to revert it. That gap — configuration drift between what you declared and what's actually running — is what AWS Config is built to close.

What it actually is

AWS Config continuously records the configuration state of supported resources in your account (over 300 resource types) as a timestamped history, and evaluates that state against Config rules — either AWS managed rules (s3-bucket-public-read-prohibited, restricted-ssh, iam-password-policy) or custom rules backed by your own Lambda function or Guard/CloudFormation Guard policy. Every recorded change produces a configuration item, and Config keeps a full timeline, so you can ask "what did this security group's rules look like at 2:14pm last Tuesday" and get an actual answer, not a guess reconstructed from CloudTrail events.

Rules run either on a schedule (periodic) or triggered by a configuration change (change-triggered), and each evaluation produces a compliant/non-compliant verdict per resource that shows up in one dashboard across the account — or, aggregated, across an entire Organization via Config aggregators.

Where it earns its keep

Drift detection that isn't tied to your IaC tool. Config doesn't know or care that a resource was created by Terraform — it evaluates whatever's actually running against your rules, which catches manual console changes, break-glass fixes that weren't reverted, and changes made by tools outside your Terraform state entirely.
Point-in-time configuration history for incident response. "What changed on this resource in the hour before the incident" is a Config timeline query, which is a much faster starting point than paging through raw CloudTrail events trying to reconstruct resource state by hand.
Automated remediation. A non-compliant finding can trigger an SSM Automation document via remediation actions — auto-revoking a security group rule that opens 0.0.0.0/0 on port 22, for example — so drift doesn't just get flagged, it gets fixed without a human in the loop for well-understood violations.
Organization-wide compliance posture in one place. Conformance packs bundle related rules (CIS benchmark, PCI-DSS-aligned checks) and deploy across every account in an Organization via an aggregator, giving one compliance score instead of per-account spot checks.

Where it doesn't replace Terraform (or vice versa)

Config is observational and reactive — it tells you what's true now and evaluates it against policy, but it doesn't prevent a bad change from happening in the first place the way a Terraform plan/apply gate or an SCP does. And Terraform's state file, while it also describes resource configuration, only reflects what Terraform itself last applied — it goes stale the moment something changes outside Terraform, which is exactly the blind spot Config is designed to catch. The two are complementary: Terraform (plus policy-as-code gates like Checkov) prevents drift at apply time; Config catches whatever gets through anyway, including changes that never went through Terraform at all.

The pricing model

Config bills per configuration item recorded and per **rule evaluation**, both of which scale with account activity and resource count — a large, churny account with many resource types and frequent changes will rack up config-item costs faster than a small, stable one. Turning on recording for every supported resource type across every account in an Organization without checking projected volume first is the most common way a Config bill surprises someone; scope the recorder to resource types you actually intend to have rules for if cost is a concern, rather than defaulting to "record everything."

A practical tip

Start with a conformance pack for a benchmark you already care about (the AWS-provided CIS or Foundational Security Best Practices packs are good defaults) instead of hand-picking individual managed rules — it gets you broad, curated coverage immediately, and you can prune or add rules from there once you see what's actually flagging in your account.

Service spotlight: AWS Fargate and its per-task pricing surprises

Mon, 10 Aug 2026 00:00:00 GMT

"Just use Fargate" is the default answer to "how do I run this container" often enough that it's worth being precise about what it trades away, not just what it saves. It's not free serverless magic — it's EC2 capacity management moved from your team to AWS, billed per task instead of per instance, and that shift changes both your ops burden and your cost curve.

What it actually is

Fargate is a launch type for ECS (and EKS) that removes the EC2 instance layer entirely. With the EC2 launch type, you manage a cluster of instances, size them, patch them, and bin-pack tasks onto them yourself (or via Cluster Autoscaler-style capacity providers). With Fargate, you specify vCPU and memory per task definition, and AWS runs each task on its own right-sized, isolated compute — no instances to see, patch, or pack. You still define everything else about the container (image, environment, networking, IAM task role) exactly like EC2-backed ECS; only the compute layer underneath changes.

Where it earns its keep

Spiky or unpredictable workloads. A batch job that runs for ten minutes an hour doesn't need a warm EC2 fleet sized for peak sitting idle the other fifty minutes. Fargate tasks start in roughly 30-60 seconds and you pay only for the vCPU/memory-seconds actually consumed.
Teams without dedicated infra headcount. No AMI patching pipeline, no instance-type selection exercise, no capacity provider tuning. That's a real operational cost removed, not just a marketing line.
Per-task network isolation. Every Fargate task gets its own elastic network interface, so security groups apply at the task level naturally — useful for multi-tenant workloads where you don't want tasks sharing a host's network namespace.

Where EC2-backed ECS still wins

Fargate's per-vCPU-hour and per-GB-hour pricing is meaningfully higher than the equivalent on-demand EC2 price for the same resources — AWS is charging for not having to manage the instance, and that premium is real. For steady-state, high-utilization workloads (a fleet of API servers running 24/7 at 60-80% CPU), a well-managed EC2-backed ECS cluster with Reserved Instances or Savings Plans underneath is usually meaningfully cheaper for the same compute — you're just paying someone (your own team) in ops time instead of paying AWS in margin. The crossover point depends on utilization and how much that ops time actually costs, but "always Fargate" and "always EC2" are both wrong defaults; it's a per-workload call.

Fargate also has hard ceilings EC2 doesn't: max 4 vCPU / 30 GB memory per task on the standard configuration tier (higher limits exist but require opt-in and aren't universally available), no GPU support, and no control over the underlying kernel or host-level tuning (sysctls, huge pages) that some latency-sensitive workloads need.

Fargate Spot

Fargate Spot runs tasks on spare capacity at up to a 70% discount versus on-demand Fargate pricing, with the same two-minute interruption warning model as EC2 Spot. It's a good fit layered into an ECS capacity provider strategy — e.g., a base count of on-demand Fargate tasks to guarantee minimum capacity, with everything above that scaled on Fargate Spot. That works well for stateless, horizontally-scaled services behind a load balancer where losing one task briefly just means the ALB stops routing to it and ECS replaces it; it's a poor fit for long-running batch jobs that don't checkpoint, since an interruption mid-job means starting over.

The pricing gotcha

Fargate bills per-task at 1-second granularity with a 1-minute minimum, and — this is the part people miss — vCPU and memory are billed independently at their configured amounts, not at actual usage. A task defined with 2 vCPU / 4 GB but that only ever uses 0.5 vCPU still bills for the full 2 vCPU the whole time it's running. Over-provisioning task definitions "to be safe" is the single most common way Fargate bills come in higher than expected — right-size against actual CloudWatch Container Insights utilization, not a guess, and revisit it after the workload has run for a few weeks under real traffic.

A practical tip

If you're running dozens of small, short-lived tasks, check whether ECS Service Connect or batching multiple containers into one task definition (as sidecars sharing the task's vCPU/memory allocation) reduces total billed task-time versus one task per container — the per-task minimum billing granularity means many tiny tasks can cost more in aggregate than fewer, right-sized ones doing the same work.

Service spotlight: AWS Step Functions for real workflows

Mon, 10 Aug 2026 00:00:00 GMT

Every few months I see a team reinvent Step Functions badly: a Lambda that invokes another Lambda, wrapped in a try/except that retries three times and then writes a row to DynamoDB so a cron job can poll for stuck items. That's a state machine with extra steps — literally. Step Functions is AWS's managed orchestrator for exactly this shape of problem, and it's worth knowing precisely when it earns its keep and when it's overkill.

What it actually is

A Step Function is a JSON (or YAML, via the newer workflow studio) state machine defined in Amazon States Language. Each state does one thing — invoke a Lambda, call another AWS service directly via an "SDK integration," branch on a condition, wait, fan out over a list, or hand off to a human approval step — and the service itself handles the transitions, retries, timeouts, and error handling between them. Execution history is retained and visualized automatically, so when something fails at 2am you get a diagram with a red X on the exact state that broke, not a pile of CloudWatch Logs you have to stitch together by request ID.

There are two flavors, and picking the right one matters:

Standard workflows are built for long-running, auditable processes. Exactly-once execution, up to a year of runtime, full execution history retained in the console. Priced per state transition.
Express workflows are built for high-volume, short-duration work (under 5 minutes) — think per-request orchestration behind an API. At-least-once execution, no persistent execution history in the console (it goes to CloudWatch Logs instead, which you pay for separately), priced per invocation duration and memory, closer to Lambda's pricing model.

Using Standard for a workflow that fires 50,000 times a day doing simple API-to-API glue work is the single most common way people get an unpleasantly large Step Functions bill.

When to reach for it

Multi-step processes with real failure modes — order fulfillment, video transcoding pipelines, ML training/inference chains, anything with a "do A, then B, and if B fails, do C instead of D" shape. Encoding that in nested Lambda try/except blocks gets unreadable fast; a state machine makes the actual business logic visible as a diagram.
Fan-out/fan-in work. The Map state runs a step over every item in an array — in Distributed Map mode, up to 10,000 concurrent child executions reading directly from S3 or a JSON array, without you writing a single line of concurrency-control code.
Long-running processes that need to survive restarts. A Standard workflow waiting on a human approval, an external webhook, or a batch job that takes six hours doesn't cost you anything while it's waiting — Step Functions isn't polling, it's holding state and will resume the instant a callback token comes back.
Direct AWS SDK integrations. Step Functions can call over 200 AWS services' APIs directly from a state definition — start a Glue job, put an item in DynamoDB, publish to SNS — with no Lambda in between. That's one less function to deploy, monitor, and patch for what's really just a passthrough API call.

When NOT to reach for it

If your "workflow" is two steps with no meaningful failure branching — call API A, then call API B with A's result — you don't need a state machine, you need a Lambda function or even just synchronous code in your existing service. The overhead of authoring, deploying, and versioning a state machine definition isn't worth it for something a single function handles in ten lines. And for very high-throughput, sub-second, simple orchestration (think: per-request routing in a hot path), Express workflows can work, but you're often better served by keeping that logic in application code and reserving Step Functions for where the auditability and visual execution history actually pay off.

The pricing gotcha

Standard workflows charge per state transition, not per execution — and transitions add up faster than people expect. A Map state iterating over 1,000 items, each running three sequential states, is 3,000 transitions in one execution, not one. At $0.025 per 1,000 transitions, that's cheap in isolation, but a workflow that fans out over large datasets on a frequent schedule can quietly become one of the more expensive things in an account. The fix isn't to avoid Map — it's to use Distributed Map, which counts child workflow executions differently and is built for exactly this high-fan-out case, plus to actually look at the "state transitions" line item in Cost Explorer before a workflow goes from a proof of concept to a production schedule running every five minutes.

A practical tip

Use ResultSelector and OutputPath inside state definitions to trim what gets passed downstream instead of piping entire upstream payloads (including that giant DynamoDB item or Lambda response) through every subsequent state. Standard workflow execution history and each state's input/output are capped at 256 KB — passing bloated payloads through unfiltered is the most common way people hit that limit and get a cryptic States.DataLimitExceeded error on a workflow that's otherwise working fine.

Service spotlight: CloudFront Functions vs Lambda@Edge

Mon, 10 Aug 2026 00:00:00 GMT

CloudFront gives you two different ways to run code at the edge, and they're not tiers of the same thing — they're built for genuinely different jobs, with different runtime models, different latency profiles, and a pricing gap wide enough that picking the wrong one for a simple task is a real cost mistake, not just a style preference.

The two options

CloudFront Functions runs a restricted subset of JavaScript in a lightweight, purpose-built runtime embedded directly in CloudFront's edge locations — sub-millisecond execution, no cold starts, and it can run on viewer request/response events only (the events closest to the end user, before/after CloudFront's cache). It has hard constraints: no network access, no filesystem access, a 10 KB code size limit at the "basic" compute tier (2 MB on the newer "advanced" tier, which raised limits and added things like key-value store access), and a 1 MB max HTTP request/response size to operate on.

Lambda@Edge runs actual Lambda functions (Node.js or Python, your choice of runtime and package) at CloudFront edge locations, and can run on all four CloudFront event types — viewer request, origin request, origin response, and viewer response — with network access, larger code packages, and access to other AWS services from within the function. It has real cold starts (worse than standard Lambda, since it's replicating across edge regions) and materially higher per-invocation and per-GB-second pricing than CloudFront Functions.

Picking based on the job, not the label

The deciding question isn't "which is more powerful" — Lambda@Edge always wins that comparison — it's **does this task need origin access or first-party AWS service calls**. If the answer is no, CloudFront Functions is almost always the right choice:

URL rewrites and redirects (the trailing-slash/index.html rewrite pattern static sites need for pretty URLs) — pure string manipulation on the request, no origin needed. CloudFront Functions territory, and meaningfully cheaper and faster than Lambda@Edge for exactly this job.
Header manipulation — adding security headers, normalizing a header for cache-key purposes, stripping a header before it hits the origin. Same story: no origin call needed, CloudFront Functions handles it at a fraction of the cost.
A/B testing via cookie-based routing, viewer-side auth token ** validation** (checking a JWT's signature without calling out to a service) — still viewer-request/response only, still fits.

Reach for Lambda@Edge only when the task genuinely needs something CloudFront Functions structurally can't do:

Calling another AWS service or an external API from inside the function (looking up a value in DynamoDB to decide which origin to route to) — CloudFront Functions has no network access at all, full stop.
Modifying the origin request/response, not just the viewer-facing side — e.g., rewriting the request CloudFront sends to a custom origin based on logic too complex for a viewer-request rewrite alone.
Larger dependencies or non-JS runtimes — image manipulation libraries, anything needing Python or a sizable npm dependency tree that won't fit CloudFront Functions' size ceiling.

The pricing and latency gap

CloudFront Functions is priced per invocation, at a rate roughly two orders of magnitude cheaper than Lambda@Edge's per-invocation-plus-duration pricing, and it runs with no cold start because it's not a full Lambda execution environment being spun up — it's closer to a purpose-built string- processing VM than a general compute runtime. For a function firing on every single request to a high-traffic distribution (which URL-rewrite and header functions typically do), that pricing gap is not academic — running a basic redirect rule as Lambda@Edge instead of a CloudFront Function on a busy site can be the difference between a rounding-error cost and a noticeable line item.

A practical tip

Default to CloudFront Functions and only reach for Lambda@Edge when you hit one of its actual constraints (no network access, code size, event type) — not the other way around. It's easy to reach for Lambda@Edge out of familiarity with Lambda generally, but for the viewer-request rewrite and header-manipulation jobs that make up most CloudFront edge-compute use cases, it's strictly worse on both cost and latency for no functional gain.

Service spotlight: querying years of audit history with CloudTrail Lake

Mon, 10 Aug 2026 00:00:00 GMT

The standard CloudTrail setup — trail to S3, Athena table on top, manual partition projection — works, but it's a pipeline you own: someone has to get the Glue table schema right, keep partitions current, and remember it exists when an incident finally needs it at 2am. CloudTrail Lake exists to delete that pipeline.

What it actually is

CloudTrail Lake is a managed, queryable event store for CloudTrail data. You create an event data store, point it at management events, data events, or both, across one account or an entire AWS Organization, and AWS ingests, indexes, and retains that data for up to seven years without you touching S3 or Glue. Querying is plain SQL against the event data store through the CloudTrail console or the cloudtrail-data API — no Athena table definitions, no partition maintenance, no worrying that a schema drift in raw JSON broke your queries.

Under the hood it's still built on the same event structure as classic CloudTrail-to-S3, so anything you already know about CloudTrail event fields (eventName, sourceIPAddress, userIdentity, requestParameters) transfers directly — the difference is entirely in ingestion and query ergonomics, not in what's being captured.

Where it earns its keep

Incident response and forensics. "Who called DeleteBucket in the last six months" is a SELECT with a WHERE clause instead of an Athena table setup exercise done under pressure during an active incident. That difference matters most exactly when you have the least patience for infrastructure work.
Organization-wide queries. An event data store can aggregate events from every account in an AWS Organization into one queryable store, which is a real improvement over stitching together per-account S3 buckets and cross-account Athena access.
Long retention without a lifecycle policy to babysit. Seven years of retention is configured once at store creation, not maintained via an S3 lifecycle rule someone has to remember not to break.
Federated queries via generative AI tools. CloudTrail Lake events can be queried through Amazon Q and Bedrock Agents' natural-language interfaces for teams that want "show me anomalous IAM activity this week" without writing SQL by hand — useful for on-call engineers who aren't CloudTrail SQL experts.

Where it doesn't replace the classic setup

CloudTrail Lake is not a replacement for a trail delivering to S3 if you need that data available to other tooling — a SIEM ingesting raw CloudTrail JSON from S3, a security data lake feeding a different query engine, or a compliance requirement for immutable S3 object storage with Object Lock. Lake's event data store is queryable through its own API and console, not a general-purpose object store other systems can read from directly. Most mature setups run both: a trail to S3 for downstream tooling and archival, and an event data store for ad hoc SQL investigation.

The pricing model is the actual decision

CloudTrail Lake bills per GB ingested, with a choice between two pricing options at event-data-store creation: one-year extendable retention (higher ingestion price, retention extendable up to 7 years, data can be exported) or seven-year retention (lower ingestion price, fixed at 7 years, no export). That choice is set per store and isn't something you casually change later, so decide upfront whether you'll ever need to export the underlying data — if there's any chance you will, pay for the extendable tier even though it costs more per GB, because re-ingesting historical data into a differently-configured store isn't an option.

For accounts with high API call volume (a busy CI/CD pipeline hitting AWS APIs constantly, or verbose data events on S3/Lambda), ingestion costs can add up fast — model this against your actual CloudTrail event volume before turning on data events across every account in an Organization, rather than discovering the bill after the fact.

A practical tip

Start with management events only in the event data store, and add data events (S3 object-level, Lambda invocations) selectively for the handful of buckets or functions where object-level audit trail actually matters — data events are dramatically higher volume than management events, and are the single most common reason a CloudTrail Lake bill comes in far higher than expected.

Service spotlight: Kinesis Data Streams, shards, and when SQS was simpler

Mon, 10 Aug 2026 00:00:00 GMT

I've seen more than one team reach for Kinesis Data Streams because "streaming" sounded like the right word for their use case, then spend weeks fighting shard math for a workload that one SQS queue and a Lambda would have handled with a fraction of the operational overhead. Kinesis is the right tool for a specific shape of problem — it's worth being precise about what that shape is before picking it.

What it actually is

Kinesis Data Streams is an append-only, partitioned log: producers write records to a stream, the stream is divided into shards, and each shard is an ordered sequence that multiple independent consumers can read from the same position without deleting records for each other. That last part is the core difference from SQS: an SQS message is deleted once a consumer processes it, so only one logical consumer group gets each message, whereas a Kinesis record stays in the stream (default 24 hours, extendable to 365 days) and any number of consumer applications can read the full history independently, at their own pace, from their own checkpoint.

Each shard supports up to 1 MB/sec or 1,000 records/sec of writes and 2 MB/sec of reads (5 reads/sec) in standard consumer mode, or up to 2 MB/sec per shard per consumer with enhanced fan-out, which gives each registered consumer a dedicated read throughput instead of sharing the base 2 MB/sec across all of them.

Where it earns its keep

Multiple independent consumers of the same event stream. A clickstream that needs to feed a real-time dashboard, a fraud-detection pipeline, and a data lake ingestion job simultaneously — each reading the full stream independently — is exactly what Kinesis's replay-without- deletion model is for. SQS would need a fan-out pattern (SNS to multiple queues) to approximate this, and even then each queue consumer still deletes its own copy rather than sharing an ordered log.
Strict ordering within a partition key. Records with the same partition key always land in the same shard and are delivered in the order written. SQS FIFO queues offer ordering too, but Kinesis's partition-key-to-shard model scales that ordering guarantee across much higher throughput than a single FIFO message group.
Replay and reprocessing. Because records persist in the stream for the configured retention window, a consumer that falls behind or a new consumer version that needs to reprocess history from an earlier point can do so — you can't rewind an SQS queue.

Where SQS is genuinely simpler

If you have one logical consumer (or a consumer group where each message should be handled exactly once by exactly one worker), SQS is less to operate: no shard count to manage, no manual scaling decision when throughput grows, and its default at-least-once, auto-scaling queue model just works without capacity planning. Kinesis in provisioned mode requires you to explicitly resize (split or merge) shards as throughput changes — under-provisioned shards throttle producers with ProvisionedThroughputExceededException, and over-provisioned shards are paying for capacity you don't use. On-demand mode removes manual shard management (Kinesis scales automatically based on observed throughput, within limits), trading that operational burden for a meaningfully higher per-stream price than provisioned mode at low-to-moderate volume.

Consumer-side, Kinesis Client Library (KCL) applications carry more operational surface than an SQS consumer loop — checkpointing, lease management via a DynamoDB table KCL creates for you, and shard rebalancing on scale-out are all things that can silently misbehave (a consumer falling behind and not alarming on IteratorAge is the classic failure mode) in ways a plain SQS ReceiveMessage loop doesn't have to think about at all.

A practical tip

Before reaching for Kinesis, ask whether you actually need **multiple independent consumers replaying the same ordered stream**. If the honest answer is "no, I have one processing pipeline," SQS (or SQS FIFO, if you need ordering within a single consumer group) is very likely the simpler, cheaper, lower-maintenance choice — save Kinesis for when the fan-out and replay semantics are the actual requirement, not just the more impressive-sounding service name.

Service spotlight: cheap DNS-level failover with Route 53 health checks

Mon, 10 Aug 2026 00:00:00 GMT

Multi-region failover has a reputation for requiring a global load balancer, cross-region routing infrastructure, and a meaningful engineering lift. For a lot of workloads — a static site with a backup origin, an API with a standby region, anything where "route traffic away from the broken endpoint" is the whole requirement — Route 53 health checks and failover routing get you most of the way there with nothing more than DNS records.

What it actually is

A Route 53 health check polls an endpoint (HTTP, HTTPS, or TCP) on an interval (default 30 seconds, or 10 seconds for the "fast" option) from a distributed set of AWS health checker locations worldwide, and marks the endpoint healthy or unhealthy based on a configurable failure threshold — requiring multiple consecutive failures across multiple checker locations before flipping status, which avoids a single flaky network path in one region triggering an unnecessary failover.

Failover routing policy then ties DNS answers to that health check status: you create a primary record pointing at your main endpoint and associate it with the health check, and a secondary record pointing at a backup endpoint. While the primary's health check passes, Route 53 answers DNS queries with the primary record. The moment the health check fails, Route 53 stops returning the primary record and starts answering with the secondary — no load balancer, no application-level failover logic, just DNS resolving to a different answer once the health check trips.

Where it earns its keep

Static or largely static sites with a backup origin. A CloudFront distribution normally in front of an S3 bucket, with a health check on the S3 origin and a failover record pointing at a backup S3 bucket in another region (or even a "we're down" static page hosted elsewhere) — cheap insurance against a regional S3 outage with no compute involved.
API failover across regions without a global load balancer. For workloads where eventual consistency of DNS propagation is acceptable (see the TTL caveat below), pointing a primary record at a region-A API Gateway/ALB and a secondary at region B, health-checked on each, is meaningfully simpler to set up and reason about than Global Accelerator or a multi-region Application Load Balancer setup, if anycast-level failover speed isn't required.
Health checks that inspect more than "is it up." A health check can match on a specific string in the response body, not just a 200 status — useful for catching an endpoint that's returning 200 with a degraded/error payload, which a naive load balancer health check sometimes misses.
Composable with other routing policies. Failover pairs with weighted, latency-based, and geolocation routing at the record level (e.g., latency-based routing across regions, each of which has its own failover pair underneath), so you're not limited to a single global primary/secondary if the topology is more complex.

The catch: DNS TTL and caching

DNS failover is only as fast as clients actually re-resolve the record. A record with a 300-second TTL means some fraction of clients — and, more unpredictably, any resolver or client that doesn't strictly honor TTL — keep hitting the now-dead primary for up to that TTL after failover triggers, sometimes longer with misbehaving caching resolvers. Set a low TTL (30-60 seconds) on records used for failover well before you need it — TTL changes themselves take time to propagate, since resolvers that already cached the old TTL keep using it until their current cache entry expires, so this isn't something you can fix in the moment of an actual outage.

Health check evaluation itself also isn't instant: with the default 30-second interval and default failure threshold, detecting an outage and flipping DNS can take on the order of a minute or two end to end, which is well short of what many RTO targets actually need for a fully automated failover — model that latency into your DR plan rather than assuming "we have failover configured" means "we have sub-minute RTO."

A practical tip

Create a calculated health check (one that aggregates the status of several child health checks with AND/OR/NOT logic) when "healthy" means more than one endpoint being reachable — e.g., requiring both the API and its database dependency's health check to pass before Route 53 considers the primary healthy, instead of failing over on a symptom while the actual root cause goes undetected.

Service spotlight: when S3 Intelligent-Tiering beats hand-rolled lifecycle rules

Mon, 10 Aug 2026 00:00:00 GMT

S3 lifecycle rules that move objects to Infrequent Access after 30 days and Glacier after 90 are a reasonable default when access patterns are predictable. They're a bad fit when they're not — and "predictable access pattern" is an assumption worth questioning for a lot of buckets that actually hold a mix of hot and cold data with no clean age-based line between them.

What it actually is

S3 Intelligent-Tiering is a storage class that automatically moves objects between access tiers based on observed access patterns instead of a fixed age threshold you define upfront. Objects start in the Frequent Access tier; after 30 consecutive days with no access, they move to Infrequent Access automatically; after 90 days with no access, optionally to Archive Instant Access; and, if you opt in to the deeper tiers, after 180 days to Archive Access and Deep Archive Access — with the key property that if an object in a colder tier is accessed again, it moves back to Frequent Access automatically, and there's no retrieval fee for the Frequent, Infrequent, or Archive Instant Access tiers (retrieval from the two deepest archive tiers works like Glacier and does require a restore request with retrieval time).

This is the core difference from a manual lifecycle policy: a lifecycle rule moves objects on a **fixed schedule regardless of whether they're still being accessed**, so an object accessed on day 89 still transitions to Glacier on day 90 if that's what the rule says, and pulling it back out means eating a retrieval cost and (for Glacier-class tiers) a wait. Intelligent-Tiering only moves objects that have genuinely gone cold, and un-cools them automatically the moment access resumes.

Where it earns its keep

Unpredictable or mixed access patterns. A bucket holding user uploads where some files get accessed constantly and others never again, with no way to know in advance which is which per object, is exactly the case a fixed-age lifecycle rule handles badly and Intelligent-Tiering handles by design.
No operational tuning required. There's no age threshold to pick, monitor, and revisit as access patterns change over the life of the bucket — Intelligent-Tiering adapts continuously, so a bucket's usage pattern shifting six months from now doesn't require anyone to notice and adjust a lifecycle rule.
No retrieval fees on tier transitions within the standard tiers. Pulling an infrequently-accessed object back out doesn't carry the same per-GB retrieval charge a Standard-IA or Glacier object would, which matters for buckets where "infrequent" doesn't mean "never."

Where a plain lifecycle rule is still cheaper

Intelligent-Tiering charges a small per-object monthly monitoring and automation fee on top of storage costs, for every object over 128 KB (objects smaller than that are charged at Frequent Access rates and never monitored or transitioned, since the monitoring fee would exceed any storage savings). For a bucket with millions of small objects, that per-object fee adds up fast and can exceed what you'd pay just leaving everything in Standard, let alone what a manual lifecycle-to-IA rule would cost. And if access patterns genuinely are predictable — logs that are always hot for 30 days and never touched again, backups that are written once and only ever read during a disaster recovery event — a fixed lifecycle rule with no monitoring fee at all is strictly cheaper, because you already know the answer Intelligent-Tiering would spend money figuring out.

A practical tip

Run S3 Storage Lens or check the bucket's access patterns via S3 Inventory before choosing between the two — if you can already describe the access pattern in one sentence ("logs go cold after 30 days, always"), write a lifecycle rule. If the honest answer is "it varies by object and I don't know the split," Intelligent-Tiering is worth the monitoring fee to stop guessing.

Service spotlight: wiring automatic rotation into Secrets Manager

Mon, 10 Aug 2026 00:00:00 GMT

Storing a database password in Secrets Manager instead of a .env file is the easy part. The part that actually earns the "we rotate credentials" line in a security questionnaire is automatic rotation — and that's where most setups I've reviewed stop halfway, with a secret stored but never actually rotated because nobody wired up the rotation Lambda.

What it actually is

Secrets Manager stores secrets encrypted with KMS and, for the services that matter most, can rotate them on a schedule automatically via a rotation Lambda function that Secrets Manager invokes. For RDS, Aurora, DocumentDB, and Redshift, AWS provides pre-built rotation Lambda templates (deployed via a SAM app from the Secrets Manager console or CLI) that implement the standard four-step rotation process without you writing the logic yourself:

createSecret — generate a new password and stage it as AWSPENDING
setSecret — set that new password on the actual database
testSecret — verify the new credential actually authenticates
finishSecret — promote AWSPENDING to AWSCURRENT, completing the rotation

That staged, four-step model exists specifically so a failure partway through doesn't lock you out: AWSCURRENT only moves to the new password after testSecret confirms it works, so a broken setSecret step leaves the database still accepting the old, still-AWSCURRENT password rather than stranding the app with a password nothing accepts.

Where it earns its keep

No credential ever needs to be manually rotated by a human again once it's wired up — the whole point of rotation is removing "someone remembers to change the password quarterly" from the list of things that depend on human follow-through, which is also the thing that reliably doesn't happen.
Application code stays credential-agnostic. Apps fetch the current secret value via GetSecretValue at connection time (ideally through the Secrets Manager RDS/JDBC connector libraries, which handle the AWSCURRENT/AWSPENDING transition transparently) instead of having a password baked into config, so a rotation doesn't require an app deploy — as long as the app re-fetches rather than caching the credential for its entire process lifetime.
Multi-user rotation strategy for zero-downtime cutover. For workloads that can't tolerate any connection using a stale credential during rotation, Secrets Manager supports an alternating-user rotation strategy — two database users, rotation alternates which one is AWSCURRENT, so old connections using the previous user keep working until they naturally cycle rather than being cut off mid-rotation.

Where it goes wrong in practice

The most common failure isn't rotation itself — it's **rotation never running successfully because the Lambda can't reach the database**. The rotation Lambda needs network access to the database (correct VPC subnets, security group rules allowing it in) and the Secrets Manager VPC endpoint if the Lambda runs without internet egress; get either wrong and rotation fails silently on schedule, over and over, until someone notices the secret's LastRotatedDate hasn't moved in months. Alarm on rotation failures explicitly (CloudWatch metric filter on the rotation Lambda's error logs, or EventBridge on RotationFailed) rather than assuming "I set a rotation schedule" means it's actually rotating.

The other gap: rotation changes the secret in Secrets Manager and on the database, but doesn't retroactively fix every place a credential might be cached — a long-lived connection pool that doesn't recycle connections, or a sidecar that read the secret once at container start and never again, keeps using the old credential until it happens to reconnect. Rotation strategy has to account for how long-lived your actual connections are, not just how often the schedule fires.

A practical tip

Set the rotation schedule's testing in a non-production secret first, and deliberately break setSecret (point it at a nonexistent user, say) to confirm the four-step staging really does leave AWSCURRENT untouched on failure before you trust it against a production database credential — verifying the failure mode is safe is worth the ten minutes it takes.

Service spotlight: SSH-less, bastion-less EC2 access with SSM

Mon, 10 Aug 2026 00:00:00 GMT

I still see security groups with port 22 open to a bastion host's IP range, a bastion host someone has to patch, and an SSH key rotation process nobody actually follows. Session Manager has been GA for years and solves this completely, and yet the bastion pattern persists mostly out of habit — worth a clear-eyed look at what actually changes when you switch.

What it actually is

Session Manager is a feature of AWS Systems Manager that opens an interactive shell session to an EC2 instance (or on-prem/other-cloud server running the SSM agent) entirely over the agent's outbound HTTPS connection to the SSM service — no inbound port needs to be open on the instance at all, not even 22. Authentication and authorization happen through IAM: if your IAM identity has ssm:StartSession permission (typically scoped by resource tag) and the instance has SSM's managed policy on its instance role, you get a shell. No SSH key to distribute, no bastion host to maintain, no security group rule to remember to remove later.

Every session is logged: session start/end, the identity that connected, and — if you enable it — the full session transcript to CloudWatch Logs or S3, plus KMS encryption of session data in transit. That's a meaningfully better audit story than SSH access, where "who ran what command on this box" usually means reconstructing it from shell history files that a user can edit or delete.

Where it earns its keep

Removing SSH entirely from your security posture. No open port 22 means one less thing for a port scanner to find and one less credential (an SSH key) to leak, rotate, or worry about ending up in a public repo.
Instances with no public IP. A fully private-subnet EC2 instance with no NAT and no bastion is normally unreachable for interactive access; Session Manager works over a VPC endpoint (com.amazonaws.region.ssm plus the two related endpoints), so instances with zero internet egress are still reachable for administration.
Temporary, auditable access grants. Because access is IAM policy, not a distributed key, revoking someone's shell access to production is an IAM policy change that takes effect immediately — no key rotation across a fleet, no bastion account cleanup.
Port forwarding without a bastion. aws ssm start-session --document-name AWS-StartPortForwardingSession tunnels a local port to a remote one (handy for reaching an RDS instance in a private subnet from a laptop) without opening the database to the internet or standing up a jump host at all.

What still trips people up

The SSM agent has to be installed, running, and able to reach the SSM service endpoints — most current AMIs (Amazon Linux 2023, recent Ubuntu/ Windows AMIs) ship with it preinstalled, but older custom AMIs and minimal/hardened base images often don't, and "Session Manager isn't working" on those turns out to be a missing or outdated agent, not an IAM problem. The instance also needs outbound HTTPS reachability to the SSM endpoints — either a NAT gateway/instance, or the three SSM-related VPC interface endpoints for fully private subnets — and forgetting the VPC endpoints on a private-subnet instance is the most common "it works in one VPC and not another" support ticket.

The other gap: Session Manager solves interactive shell access, but it's not a substitute for scoped, least-privilege instance role permissions. Someone with ssm:StartSession on an instance whose instance role has broad S3 or IAM permissions still inherits that instance's blast radius once they're in the shell — Session Manager changes how you get access, it doesn't change what that access can do once you're there.

A practical tip

Turn on session logging to CloudWatch Logs and set a retention/alerting policy on it from day one — the audit trail is the single biggest advantage over SSH, and it's only useful if someone's actually watching or alarming on it, not just accumulating in a log group nobody queries until an incident forces the question.

Service spotlight: running an AWS Well-Architected Tool review

Mon, 10 Aug 2026 00:00:00 GMT

"We should do a Well-Architected review" tends to get said in the same tone as "we should write more tests" — a good idea everyone agrees with and nobody schedules. Having actually run several of these against real production workloads, the tool is more useful than its reputation as a compliance checkbox exercise suggests, provided you go in knowing what it will and won't tell you.

What it actually is

The AWS Well-Architected Tool is a free console tool that walks a defined workload (you register one per application or system you want reviewed) through a structured questionnaire organized by the **six pillars**: Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimization, and Sustainability. Each pillar has a set of questions ("How do you manage identity and permissions for people and machines?"), each question has a list of best-practice choices you select as implemented or not, and unchosen best practices become flagged risk items — categorized as high or medium risk — in a summary report per pillar.

The output is a milestone: a snapshot of your answers and risk items at a point in time, which you can re-run later and diff against to show whether risk actually went down after remediation work, not just that a review happened once.

Where it earns its keep

Structured coverage across dimensions people forget under deadline ** pressure.** Reliability and cost questions in particular tend to get skipped in day-to-day feature work; walking through all six pillars on a schedule forces a periodic look at "do we have alarms on this," "is this single-AZ," and "why is this instance still on-demand" even when no one's actively fighting a fire in those areas.
A shared vocabulary for architecture discussions. Once a team has gone through a review together, "that's a Reliability risk" or "that's not really Well-Architected on the Security pillar" becomes shorthand everyone in the room understands the same way, which speeds up later design reviews.
Lenses for workload-specific guidance. Beyond the six generic pillars, AWS publishes lenses (Serverless, SaaS, Machine Learning, Container Build, and others) with more specific best practices for that architecture style — the Serverless lens asks meaningfully different questions than the generic Operational Excellence pillar does, and is worth applying if your workload fits one.
AWS credits for remediation, sometimes. Depending on your account's relationship with AWS (particularly through a Partner or an active Enterprise Support engagement), completing a review and acting on findings can be tied to funding programs — worth checking with your account team if cost is a factor in prioritizing the work.

Where it falls short

The questionnaire is generic by design, which means its questions are necessarily abstracted away from your actual architecture — it will ask whether you have a documented incident response runbook, but it can't tell you whether your specific runbook is any good, or whether your DR RTO target is realistic for your actual failure modes. It's a structured prompt for a conversation your team needs to have, not a substitute for that conversation, and treating it as a checkbox exercise where you tick "implemented" without actually verifying the practice is in place defeats the entire point — self-reported answers are only as honest as the person answering them.

It also doesn't automatically inspect your account — answering the questionnaire accurately still requires someone to actually go look at what's configured, which is real work. (AWS's separate Trusted Advisor and Well-Architected Tool's own automated checks integration can pull some findings in automatically for a subset of best practices, narrowing but not eliminating that manual-verification gap.)

A practical tip

Don't try to review an entire application's full six-pillar surface in one sitting — split the review across a couple of working sessions per pillar with the people who actually own that area (security engineer for the Security pillar, whoever owns the on-call rotation for Reliability), and treat the output high-risk items as backlog tickets with owners, not a report that goes in a drawer once the review is done.

CVE-2026-16812: a CVSS 10 hole in your SD-WAN control plane

Mon, 10 Aug 2026 00:00:00 GMT

Most weeks the scariest CVE is scary because of what it's attached to — a database, a CI/CD server, a pile of customer data. This week it's scarier because of what it's in front of: the single management plane for an entire SD-WAN fleet. CVE-2026-16812, an unauthenticated OS command injection in Arista's VeloCloud Orchestrator (VCO) On-Prem, scores a perfect CVSS 10.0, is under active exploitation, and CISA added it to the Known Exploited Vulnerabilities catalog with a federal remediation deadline of July 30 (CISA). If you run any branch offices, retail sites, or hybrid-cloud links over VeloCloud, this is the one to stop and read carefully.

Root cause

VCO is the web console that operators use to configure and monitor every VeloCloud Edge device in a deployment — think of it as the "cloud console" for your own private SD-WAN. According to Arista's advisory, the flaw lets "remote attackers access privileged functionality that was intended only for internal use and should not be remotely accessible" (BleepingComputer). In plain terms: some internal-only code path that shells out to the underlying OS is reachable from the external web interface without authentication, and attacker-controlled input reaches that shell call unsanitized. No credentials, no prior access, no user interaction — just network reachability to the VCO web UI is enough to get arbitrary command execution as whatever user the orchestrator process runs as.

The uncomfortable detail from The Register's coverage is that the exposure is structural: the orchestrator is "exposed by default, with no configuration capable of removing that exposure entirely" (The Register). This isn't a case of someone forgetting to put the admin panel behind a VPN — On-Prem VCO's own design assumes some surface has to be internet- reachable for edge devices to phone home to it, and that's the same surface the vulnerable code path sits on. Arista's Hosted and Dedicated (SaaS) deployments were already patched before the advisory went public, which narrows this specifically to organizations running their own VCO instance rather than using Arista's managed service.

Blast radius

This is where CVE-2026-16812 earns the CVSS 10. Compromising VCO doesn't just hand over one server — it hands over the control plane for every VeloCloud Edge device the orchestrator manages: branch routers, site configurations, routing policy, and the tunnels those edges use to reach each other and the cloud. Arista's own guidance says exploitation "may compromise the confidentiality, integrity, and availability of the orchestrator and data managed by the orchestrator," and researchers have flagged the realistic follow-on as attackers using orchestrator access to push malicious configuration to managed edges, effectively turning a single web app bug into a foothold across every site in the WAN. If your VCO manages edges that terminate into a VPC or a colo where your workloads live, that's the pivot path from "someone popped our SD-WAN console" to "someone is on our network."

CISA has already observed active exploitation, and researchers have published three IP addresses seen scanning and delivering payloads: 8.19.75.217, 206.72.242.124, and 206.72.242.162. Worth feeding those into your firewall/IDS blocklists today regardless of your patch status — they're a known-bad signal, not a mitigation on their own.

Remediation

Arista has shipped fixed builds for every affected On-Prem branch:

5.2.x → 5.2.3.14
6.1.x → 6.1.3.4
6.4.x → 6.4.2.4
7.0.x → 7.0.0.1

Patch first, but don't stop there. Because the flaw grants unauthenticated RCE and has apparently been exploited since before public disclosure, Arista's own advisory goes further than "update and move on" — it recommends rotating credentials, validating that managed edge devices haven't had unauthorized configuration changes pushed to them, and treating a confirmed-compromised instance as a candidate for restore-from-clean rather than trusting an in-place patch. That's a strong signal from the vendor that patching alone may not be sufficient remediation if you can't rule out prior compromise — check orchestrator and edge audit logs for activity around the known-bad IPs above before you consider this closed.

If patching isn't immediate, restrict the VCO web interface to trusted management networks now. It won't fully close the hole given how the product's exposure is architected, but cutting off casual internet scanning buys time, and it's the same "assume it's reachable, so fence it" posture you'd want for any management plane with this much downstream reach.

The bigger lesson

The recurring theme this year is that orchestration and management planes — CI/CD servers, RMM consoles, and now SD-WAN orchestrators — are increasingly the preferred target, precisely because compromising one system fans out into control over everything it manages. If you run on-prem management software for infrastructure with real blast radius, the question worth asking isn't just "is it patched" but "what does an attacker get if this one box falls" — and whether that answer is something you're comfortable with by design, not just by patch level.

CVE-2025-30066: The GitHub Action Tag You Trusted Lied

Mon, 10 Aug 2026 00:00:00 GMT

Every workflow file with a line like uses: tj-actions/changed-files@v45 is making a promise to itself that a tag won't change out from under it. In March 2025 that promise broke for real: attackers rewrote the tags on tj-actions/changed-files, a GitHub Action used in over 23,000 repositories, to point at a malicious commit that dumped CI/CD secrets straight into public workflow logs — tracked as CVE-2025-30066 (CISA). It's one of the cleanest real-world illustrations of why "pin a version" and "pin a commit" are not the same security control, and it's directly relevant if your CI — like this site's — runs on GitHub Actions.

Root cause

tj-actions/changed-files is a popular action that reports which files changed in a PR or push, and plenty of pipelines gate steps on its output. On March 14, 2025, its maintainers discovered that a large number of the project's version tags had been silently repointed to a commit that never went through review: it injected a Node.js payload that scanned GitHub Runner memory for credentials — cloud access keys, PATs, npm tokens, private RSA keys — and printed them, base64-encoded, into the workflow run log (Wiz).

The entry point traces back further, and it's the more interesting part of the story. Wiz's investigation found the compromise was likely cascading: three days earlier, on March 11, attackers had already hijacked the v1 tag of a different action, reviewdog/action-setup, and pointed it at malicious code of their own (CVE-2025-30154) (StepSecurity). tj-actions/eslint-changed-files happened to depend on reviewdog/action-setup, and the maintainers' own bot ran with a GitHub personal access token — a token that Wiz believes was harvested through that same compromised dependency and then used to push the malicious commit and rewrite the changed-files tags (Wiz). One stolen credential in one small action rippled into a second, much more widely used one within days.

The structural flaw both incidents share is that a tag like @v1 or @v45 is just a mutable pointer, not a content hash. GitHub lets any tag be force- pushed to a new commit at any time by anyone with write access — there's no cryptographic binding between the ref your workflow trusts and the code that actually runs when the runner checks it out. Reviewing an action once and pinning @v4 doesn't protect you from a maintainer account (or its automation token) getting popped six months later.

Blast radius

The initial estimate — every one of the 23,000+ repos using tj-actions/changed-files — made headlines fast, but the real exposure window was narrower and still meaningfully bad. The malicious code was live between roughly March 12, 00:00 UTC and March 15, 12:00 UTC; anyone who ran the action against a public repo during that window had whatever secrets were in scope for the job printed straight into a log anyone could read, before GitHub could pull them down (GitHub Advisory Database). On the upstream reviewdog side, deeper analysis found the confirmed secret-leak count was smaller than the initial panic suggested — around 218 repositories actually had secrets exposed in logs, not tens of thousands (BleepingComputer) — but "smaller than feared" still means real cloud credentials and tokens sitting in public logs for anyone who scraped them in time. GitHub temporarily pulled the tj-actions/changed-files repository entirely while the malicious commit was reverted and tags restored, which is its own signal of how seriously the platform treated it.

Remediation

The fix that actually closes this hole is boring and mechanical: pin every third-party Action to a full 40-character commit SHA, not a tag — uses: tj-actions/changed-files@a1b2c3... instead of @v45. A SHA can't be silently repointed the way a tag can, so a compromised maintainer token can rewrite tags all day and your pinned workflow still runs the commit you reviewed. Pinning by SHA does mean you lose automatic updates, which is why you pair it with Dependabot or Renovate — both can open PRs that bump the pinned SHA (with the new tag as a comment for readability) so upgrades stay a reviewed diff instead of an invisible tag move. On the org side, GitHub lets you restrict which Actions are allowed to run at all (allow-list specific actions or require them to be from verified creators), which is worth turning on for anything beyond a personal repo. If you ran an affected action during the exposure window, don't stop at patching: pull the workflow run logs, search them for anything that looks like a leaked credential, and rotate every secret that was in scope for those jobs — assume exposure rather than hoping the log got pulled in time.

The bigger lesson

A third-party GitHub Action is a dependency with write access to your CI environment and, by extension, whatever your CI can reach — cloud credentials, deploy keys, package registry tokens. Marketplace popularity and a green checkmark on the README aren't a security review; they're a popularity contest. This site's own CI runs entirely on GitHub Actions, and the practical takeaway is the same one that applies to any npm install: pin to something immutable, let a bot handle the update diffs, and treat every action in your workflow file as code you're choosing to run with your secrets — because that's exactly what it is.

CVE-2025-0282: the Ivanti VPN zero-day, and its sequel

Mon, 10 Aug 2026 00:00:00 GMT

If you manage remote access for a living, Ivanti Connect Secure has probably cost you a weekend at some point in the last three years. The latest entry is CVE-2025-0282, an unauthenticated stack-based buffer overflow in Connect Secure, Policy Secure, and Neurons for ZTA gateways that Mandiant caught being exploited as a zero-day starting in mid-December 2024, weeks before Ivanti shipped a fix (Google Cloud / Mandiant). CISA added it to the Known Exploited Vulnerabilities catalog and gave federal agencies until January 15 to patch (CISA KEV). It's the third major Ivanti edge-device RCE chain in two years, and that pattern is the actual story here.

Root cause

CVE-2025-0282 is a classic memory-safety bug: a stack-based buffer overflow (CWE-121) in a web-facing component of Connect Secure that a remote, unauthenticated attacker can trigger with a crafted request, landing arbitrary code execution on the appliance with no credentials and no user interaction (Rapid7). It carries a CVSS base score of 9.0 — full compromise, network attack vector, no privileges required.

This isn't Ivanti's first appliance-RCE rodeo. A year earlier it was CVE-2023-46805 (an auth bypass) chained with CVE-2024-21887 (a command injection) to get the same outcome on the same product line (CISA). The underlying story repeats: Connect Secure is a Linux-based appliance running a web application stack that's directly exposed to the internet by design — that's the whole point of a VPN gateway — and a memory-corruption or auth-logic bug in that stack turns "reachable on 443" straight into root on the box. There's no perimeter behind the perimeter device.

Blast radius

Mandiant's incident response found a genuinely elaborate malware ecosystem dropped on compromised appliances, tracked as the SPAWN family: SPAWNANT (the installer/persistence mechanism), SPAWNMOLE (a tunneler for pivoting into the internal network), SPAWNSNAIL (an SSH backdoor), and SPAWNSLOTH (a log-tampering tool to blind forensic analysis) — plus two newly observed tools, the PHASEJAM dropper and DRYHOOK credential harvester (Google Cloud / Mandiant). Most notably, SPAWNANT was built specifically to tamper with Ivanti's own Integrity Checker Tool manifest, so the appliance would report clean to the exact tool defenders were told to trust.

Mandiant attributes the activity, with medium confidence, to UNC5337 — believed to be part of UNC5221, the same China-nexus espionage cluster behind the 2023/2024 Ivanti chain — going after VPN gateways specifically because they sit at the network edge with credentials and session state flowing through them and, historically, no EDR agent watching what runs on them (The Record). At the time of disclosure, tens of thousands of Connect Secure instances were sitting exposed to the internet, and exploitation had already been underway for roughly a month before the public advisory (CyberSecurityNews). And the pattern didn't stop there: three months later, Ivanti disclosed CVE-2025-22457, another unauthenticated stack-based buffer overflow in the same product line — initially misjudged as a low-severity denial-of-service bug until Mandiant showed it was remotely exploitable — and again observed exploited in the wild by a suspected China-nexus actor before agencies could patch (Google Cloud / Mandiant).

Remediation

Patched builds for CVE-2025-0282 have been available since January 8, 2025: Connect Secure 22.7R2.5, Policy Secure 22.7R1.2, and Neurons for ZTA gateways 22.7R2.3 (Ivanti advisory). If you're still behind those versions, that's the first move, no exceptions.

But given that SPAWNANT was purpose-built to falsify the Integrity Checker Tool's output, a clean ICT scan on an unpatched or recently patched device isn't proof of a clean appliance. Run both the internal and external ICT, but treat it as one signal among several, not a verdict — cross-reference against Ivanti's published indicators of compromise, check for unexpected outbound tunnels or SSH listeners, and if you have any reason to believe a device was exposed during the exploitation window, the safer path is a factory reset and clean rebuild from a patched image rather than trusting an in-place upgrade to have removed a persistence mechanism designed to survive exactly that. Rotate every credential and certificate that ever transited the appliance — VPN gateways see user passwords, session tokens, and often service-account secrets, all of which should be considered burned if compromise can't be ruled out.

The bigger lesson

Three major RCE chains on the same Ivanti product line in under two years, each exploited as a zero-day before a patch existed, each attributed to state-nexus actors going straight for the edge — this isn't bad luck, it's a target selection. Edge appliances are internet-facing by requirement, run vendor firmware most security teams can't instrument the way they'd instrument a server, and sit exactly where credentials and network access converge. If you operate any VPN gateway, firewall, or SSL-VPN appliance — Ivanti or otherwise — the operational question isn't "are we patched" as a one-time checkbox, it's whether you have a standing plan for zero-day exploitation on a box you can't put an agent on: network segmentation around the appliance, egress monitoring for exactly the kind of tunneling SPAWNMOLE does, and a rebuild-from-clean playbook you've actually rehearsed before you need it at 2am.

Java deserialization after Log4Shell: CVE-2023-46604

Mon, 10 Aug 2026 00:00:00 GMT

Log4Shell (CVE-2021-44228) made "Java deserialization" a household phrase for a few terrifying weeks in December 2021, but it wasn't a one-off. Less than two years later, CVE-2023-46604, an unauthenticated remote code execution bug in Apache ActiveMQ's OpenWire protocol, gave attackers the same outcome — arbitrary command execution with zero credentials — and it wasn't a lookup-and-log JNDI trick this time, it was textbook insecure deserialization in a message broker that sits on the network path of a huge number of Java shops. CISA added it to the Known Exploited Vulnerabilities catalog within days, and ransomware crews had it weaponized before most defenders had patched (Rapid7). If you assumed Log4Shell was the last time a Java serialization bug would take down a fleet of production systems overnight, this one is worth sitting with.

Root cause

ActiveMQ's OpenWire is the binary wire protocol brokers and clients use to talk to each other, and by default it's listening on port 61616 with no authentication required to establish a connection. Inside that protocol, one command type — ExceptionResponse — carries a class name and a message string so a broker can tell a client "here's the exception that happened." The marshalling code that unpacks it, BaseDataStreamMarshaller.createThrowable, takes that attacker-supplied class name off the wire and instantiates it directly, passing the attacker-supplied message string into the constructor, without first checking that the class is actually a Throwable (Rapid7).

That's the whole bug. Anyone who can open a TCP connection to the OpenWire port can send a crafted EXCEPTION_RESPONSE packet naming any class on the server's classpath — including Spring's ClassPathXmlApplicationContext, which happily fetches and executes an XML bean definition from an attacker-controlled URL. Point that at a remote XML file that defines a ProcessBuilder-backed bean and the broker runs your shell command. It's the same family as the Commons Collections gadget chains that made Java deserialization famous a decade ago — take a class that legitimately does something dangerous when instantiated or invoked, and abuse a deserializer that doesn't discriminate about what it's allowed to construct. The fix Apache shipped adds exactly the type check that should have been there from the start: reject any "exception" class name that isn't actually a Throwable before instantiating it (Rapid7).

Blast radius

CVE-2023-46604 scores CVSS 9.8, and researchers rated exploitation complexity as trivial — no auth, no user interaction, one crafted packet (Huntress). The result is arbitrary command execution as whatever OS user runs the broker process, and in the wild that turned into exactly what you'd expect once a reliable pre-auth RCE with a public PoC exists: a race between opportunistic and targeted actors. Rapid7 tracked exploitation attributed to HelloKitty ransomware starting within days of disclosure (Rapid7), Trend Micro and Sekoia both documented the Kinsing cryptomining botnet using it to drop miners and rootkits on Linux hosts (Trend Micro, Sekoia), and SOC Prime and others tracked TellYouThePass ransomware riding the same bug (SOCRadar). Message brokers tend to sit deep in the architecture — application servers, integration layers, IoT backends all talk to them — so a broker compromise isn't an edge-of-network incident, it's a foothold with a direct line to whatever internal services trust that broker's traffic.

Remediation

Apache patched the marshalling logic and shipped fixed releases across every supported branch: 5.15.16, 5.16.7, 5.17.6, and 5.18.3, with 6.0.0 also carrying the fix (Apache ActiveMQ). If you're still running an unpatched broker, upgrading is non-negotiable — this isn't a "add authentication in front of it" situation, because the flaw is in how the broker parses its own wire protocol before any application-level auth applies. Restricting network access to port 61616 to only trusted broker and client hosts is a reasonable stopgap while you schedule the upgrade, but given how quickly this was weaponized, treat any internet-facing or broadly-reachable OpenWire port as already assumed-compromised and hunt for follow-on cryptominer or webshell activity, not just confirm the patch applied.

The bigger lesson

What's actually changed since Log4Shell is the plumbing around detection: WAF and IDS vendors now ship signatures for OpenWire-style deserialization payloads within hours of a PoC landing, CISA's KEV catalog gives defenders a authoritative "this is being exploited right now" signal instead of relying on vendor advisories alone, and software composition analysis tools flag vulnerable broker and library versions in CI before they ever ship. On the JDK side, JEP 415's context-specific deserialization filters (Java 17+) let you scope an allowlist to a specific ObjectInputStream instead of one JVM-wide filter that's either too loose or breaks half your app (Baeldung) — a real improvement over JEP 290's blunt instrument.

What hasn't changed is the underlying pattern: a component trusts a class name or a byte stream from the network more than it should, and something reachable on the classpath turns that trust into code execution. A 2024 NDSS study found over 3,600 GitHub Java projects still carrying known deserialization vulnerabilities (Java Code Geeks) — the detection tooling is faster, but it's still catching the same class of bug, not preventing it from being written. If your Java services still do implicit ObjectInputStream deserialization or unauthenticated binary protocol parsing anywhere on the network path, Log4Shell and ActiveMQ are both telling you the same thing: the fix belongs in the code that trusts the bytes, not in the WAF rule that flags them after the fact.

Exposed Kubernetes API Servers Are an RBAC Time Bomb

Mon, 10 Aug 2026 00:00:00 GMT

You don't need a zero-day to take over a Kubernetes cluster. You need an API server that answers on the public internet and an anonymous-auth flag nobody turned off. Aqua Nautilus researchers spent three months scanning for exactly that combination and found it everywhere: hundreds of organizations with clusters already under active attack, most of it opportunistic cryptomining riding in through anonymous access that had somehow been granted real privileges (CSO Online, Aqua Nautilus). This isn't a rare edge case — Shadowserver's internet-wide scans have repeatedly found north of 380,000 Kubernetes API servers responding publicly, the large majority in the US and Western Europe (Shadowserver, The Register). Exposure alone doesn't mean compromise, but it's the precondition every campaign in this space depends on, and the misconfigurations that turn "reachable" into "owned" are boringly common.

Root cause

Kubernetes has two places where "no credentials" quietly becomes "some credentials." The first is the API server and kubelet's anonymous-auth setting, which historically defaulted to enabled and maps unauthenticated requests to the system:anonymous user (or the system:unauthenticated group). By itself that's supposed to be harmless — anonymous requests should have no RBAC permissions bound to them. In practice, Aqua's kubelet research found operators binding that anonymous identity to real roles, sometimes admin-level ones, usually as a shortcut to get some internal tool or health check working without wiring up proper auth (Aqua Security). Once that binding exists, anyone who can reach the API server or the kubelet's HTTPS port (10250) has exactly the access that role grants — no login required. Aqua's kube-hunter tool has a dedicated check for this pattern precisely because it recurs so often across audited clusters (Aqua Security / kube-hunter).

The second gap is the default service account. Every pod gets one mounted automatically unless a manifest explicitly sets automountServiceAccountToken: false, and that token grants whatever RBAC role is bound to the account — which in a lot of real clusters is broader than anyone intended, because default service accounts accumulate permissions over time as people bind roles the fast way instead of the correct way. Layer that onto an internet-reachable control plane — sometimes because of a misconfigured cloud load balancer, sometimes because someone ran kubectl proxy --address=0.0.0.0 --accept-hosts='.*' on a bastion and forgot about it, a specific misconfiguration Aqua called out by name in its research — and a single unauthenticated HTTP request is enough to start probing.

Blast radius

The API server isn't just another service — it's the thing that can schedule code onto every node in the cluster. Anonymous or default-account access that includes pod-create permission lets an attacker launch a pod with a hostPath mount or privileged: true, which is a direct path off the container and onto the underlying node. From the node, the attacker can read the kubelet's credentials and every service account token scheduled there, pivoting sideways into other namespaces the original access point never touched. In cloud-hosted clusters, pods often carry workload-identity credentials (IRSA on EKS, Workload Identity on GKE) or can reach the instance metadata service directly, so cluster compromise routinely becomes cloud-account compromise. This is also the exact class of risk that made CVE-2018-1002105 so severe a few years back — a bug that let unauthenticated requests reach the API server's proxy layer and come out the other side with cluster-admin, a reminder that "you're talking to the API server" and "you have privileged access" are one and the same threat model even without a misconfiguration involved (Tenable). In practice, most of what Aqua and others observe on these exposed clusters is quieter than that: TeamTNT- and Kinsing-style campaigns that drop XMRig miners across every node they can reach, plus at least one documented "RBAC buster" campaign that used the access specifically to plant a persistent backdoor role rather than just mine coins (Tigera, CSO Online).

Remediation

Disable anonymous auth. Run kubelets with --anonymous-auth=false and route real authentication through --client-ca-file or a token webhook; never bind system:anonymous or system:unauthenticated to any ClusterRole, including "harmless-looking" ones.
Stop exposing the control plane. Put the API server behind a private endpoint and security-group/authorized-network restrictions (EKS private endpoint access, GKE authorized networks, or equivalent) instead of a public load balancer with no source restriction.
Audit RBAC for least privilege, especially bindings on default service accounts — Kubernetes' own docs have a good-practices guide for this and it's worth running against every namespace, not just the obviously sensitive ones (Kubernetes docs).
Set automountServiceAccountToken: false on any pod that doesn't actually call the Kubernetes API, so a compromised pod doesn't automatically hand over a credential.
Lock down the kubelet port (10250) with network policies and firewall rules — it shouldn't be reachable from outside the cluster network, full stop.
Turn on audit logging and alert on anonymous or unexpected-identity requests hitting the API server; that signal is cheap to collect and catches this exact pattern early.

The bigger lesson

None of this requires a novel exploit — every campaign cited here rode in on defaults nobody explicitly locked down. Kubernetes was built to be flexible for cluster operators first and secure-by-default second, and that tradeoff means the burden of hardening anonymous access, RBAC bindings, and network exposure sits entirely on whoever stood the cluster up. Attackers aren't finding your cluster through cleverness; they're running the same internet-wide scans Shadowserver publishes for free and checking whether you did the hardening step or skipped it. Assume they've already scanned you, and go verify which answer they got.

Leaked credentials in public repos get used in minutes

Mon, 10 Aug 2026 00:00:00 GMT

GitGuardian's 2026 State of Secrets Sprawl report counted 28.65 million new hardcoded secrets pushed to public GitHub commits in 2025 alone — a 34% jump year over year and the largest single-year increase they've recorded (GitGuardian). That's not a backlog of old mistakes; it's this year's output. And the same report found that 64% of secrets that leaked back in 2022 are *still valid* today (GitGuardian). Meanwhile Unit 42's tracking of the EleKtra-Leak campaign found attackers detecting and using exposed AWS IAM credentials within roughly five minutes of them landing on GitHub (Unit 42). Committing a secret to a public repo isn't a "someone might find this eventually" risk. It's closer to broadcasting it live.

Root cause

Almost none of this is malicious. It's .env files added to a repo before anyone wrote a .gitignore entry for them, database URLs hardcoded into a config "just for local testing," service-account JSON dropped into a test fixture, or a CI job that echoes an environment variable into build logs that are themselves public. GitGuardian's report also flags AI-assisted coding as a growing contributor — commits with AI-generated code leak secrets at roughly double the baseline rate, and secrets tied to AI services specifically were up 81% year over year (GitGuardian), which tracks: an assistant that's never seen your .gitignore will happily paste a working API key straight into a code sample.

The part engineers consistently underestimate is that deleting the secret in a follow-up commit doesn't delete it from the repo. Git preserves history — the credential is still sitting in an earlier commit object, reachable by anyone who clones the repo or even just browses the commit log on GitHub. Force-pushing a rewritten history to your own branch doesn't help either if the repo was ever public even briefly: forks, local clones, and GitHub's own caches can retain the blob indefinitely. The only thing that actually neutralizes a leaked secret is invalidating the credential itself — history rewrites are cleanup, not remediation.

Blast radius

This is the part that should change how you triage. Automated scanners watch GitHub's public event stream continuously, and multiple independent research efforts have clocked exploitation in the single-digit minutes: Unit 42 measured attackers weaponizing exposed IAM keys from the EleKtra-Leak campaign in about five minutes (Unit 42), and independent write-ups of live incidents describe leaked AWS keys being grabbed and turned into GPU cryptomining fleets within 4 to 11 minutes of the push, run across every region the credential could reach (TechRadar). The economics are simple: spinning up GPU instances for Monero mining costs the attacker nothing (it's your bill) and the scanning infrastructure is fully automated, so there's no human in the loop slowing things down. There is no "we'll rotate it during the next sprint" window here — by the time a human notices the commit, the credential has often already been used.

Remediation

The controls that matter, roughly in order of leverage:

Push protection. GitHub's secret scanning can block a push before a known-format secret ever lands in the repo, and its partner program automatically notifies the issuing provider (AWS, Stripe, and ~150 others) when a valid secret is detected in a public repo, sometimes triggering revocation faster than the committer even sees the alert (GitHub Docs). Turn this on org-wide, not per-repo.
Pre-commit scanning. Tools like gitleaks or trufflehog run locally before a commit is even made, catching secrets in formats GitHub doesn't recognize (internal API keys, custom tokens) and catching them before they touch history at all, which is strictly better than catching them after.
Treat any committed secret as compromised, full stop. Don't reach for git filter-branch or a force-push as the first response — rotate the credential at the source first. A scrubbed history with a still-valid key behind it protects nobody; an invalidated key with a messy history is a Tuesday.
Shrink the blast radius with short-lived credentials. The strongest structural fix is having fewer long-lived secrets to leak in the first place. This site's own CI/CD (see the GitHub Actions OIDC to AWS tutorial) uses OpenID Connect federation instead of storing AWS access keys as repo secrets — GitHub Actions requests a short-lived, auto-expiring token scoped to a specific IAM role for each run, so there's no long-lived AWS key sitting in secrets storage to leak in the first place. Where OIDC isn't an option, aggressive credential expiry and scoped-down IAM policies get you most of the same benefit.

The bigger lesson

Prevention controls — push protection, pre-commit hooks, .gitignore discipline — are worth having, but GitGuardian's own numbers show they aren't closing the gap: secrets leaked have grown 152% since 2021 against a 98% growth in GitHub's developer base (GitGuardian), meaning the leak rate is outpacing the population producing the leaks. Some secret will eventually get past whatever scanner you've configured, because prevention is a filter and filters have gaps by definition. What actually determines whether that leak becomes an incident report or a non-event is how fast you can rotate the credential and how little damage it can do in the window before you notice. That's why the real investment isn't just "stop secrets from leaking" — it's "assume one will leak, and make sure it's short-lived, narrowly scoped, and rotated before a bot ever gets a chance to use it."

MOVEit and the MFT zero-day exploitation playbook

Mon, 10 Aug 2026 00:00:00 GMT

In May 2023 the Clop ransomware group quietly started exploiting an unpatched SQL injection in Progress Software's MOVEit Transfer — days before anyone outside the attackers knew it existed. By the time the dust settled, more than 2,700 organizations and upward of 90 million individuals had data stolen through CVE-2023-34362, making it one of the largest single-vulnerability breaches on record (CISA, TechCrunch). It wasn't an isolated incident — it was the clearest example yet of a pattern that keeps repeating against managed file transfer (MFT) software: find one flaw in an internet-facing appliance built to move sensitive data, and turn it into hundreds of breaches in a matter of days.

Root cause

MOVEit Transfer's vulnerability was a textbook SQL injection: an unauthenticated attacker could send crafted input to the web application and manipulate backend SQL queries against MySQL, Microsoft SQL Server, or Azure SQL, escalating from unauthorized database access to remote code execution (Rapid7). Clop used it to drop a custom ASP.NET web shell — dubbed LEMURLOOT, typically written to disk as human2.aspx to blend in with MOVEit's legitimate human.aspx — giving them a durable, authenticated-looking foothold for pulling files straight out of the transfer database (Mandiant). Forensic teams found evidence attackers had mapped the database schema in advance, meaning this wasn't smash-and-grab improvisation — it was a prepared exploit chain, sat on for weeks, then fired against every internet-reachable MOVEit instance Clop could find.

The deeper problem is what MFT software is. Products like MOVEit, GoAnywhere, and Cleo exist specifically to sit at the network edge, accept authenticated (and sometimes unauthenticated) file uploads from external partners, and hold the resulting files — often full of PII, financial records, or healthcare data — until someone downstream picks them up. That's three attractive properties stacked on one box: it has to be internet-facing by design, it processes untrusted external input as its core function, and the data sitting on it is exactly what a data-extortion crew wants. A vulnerability class that would be a moderate finding on an internal app becomes catastrophic on an MFT server because the "why would an attacker target this" question answers itself.

Blast radius

Clop's MOVEit campaign wasn't a slow-burn intrusion — CISA and the FBI described mass, opportunistic exploitation across a few days before switching to the extortion phase (CISA). Confirmed victims spanned government agencies (multiple U.S. federal agencies among them), airlines (British Airways, Aer Lingus), payroll provider Zellis and its downstream customers, the BBC, and the government of Nova Scotia — the breadth reflecting how MOVEit sits inside countless unrelated organizations' back-office data flows (BankInfoSecurity). By late October 2023, tracking firm Emsisoft put the confirmed count at 2,559 organizations and over 66 million individuals, with later tallies climbing past 2,700 organizations and roughly 90+ million people as more downstream disclosures rolled in through 2024 (Cloudskope).

Notably, Clop didn't encrypt anything. This was pure data-theft-and-extort: steal the files, then list victims on a leak site with a payment deadline. That's become the default MFT playbook — encryption is optional, exposure of the stolen files is the leverage, and it's fast to scale because the same exploit chain works against every unpatched instance simultaneously. It repeated almost exactly with the earlier GoAnywhere MFT SQLi/RCE (CVE-2023-0669) and again in December 2024 against Cleo's Harmony, VLTrader, and LexiCom products, where Clop chained CVE-2024-50623 and CVE-2024-55956 — the second flaw shipped because the first patch was incomplete — to deploy a Java backdoor and hit organizations that thought they'd already remediated (Rapid7, BleepingComputer).

Remediation

Patch immediately and don't stop there — Progress shipped fixes for CVE-2023-34362 within days, but two related SQLi flaws (CVE-2023-35036, CVE-2023-35708) surfaced shortly after in the same code paths, the same "one patch wasn't the whole story" pattern seen later with Cleo. Treat a single advisory as the start of a remediation window, not the end.

For detection, CISA's advisory (AA23-158A) published concrete IOCs worth hunting for regardless of which MFT product you run: unexpected .aspx files in the web root (human2.aspx for MOVEit specifically), new/unknown admin or service accounts created around the exploitation window, SQL injection patterns and anomalous query volume in application logs, and outbound connections to unfamiliar IPs shortly after suspicious file activity (CISA). CISA also published YARA and Sigma rules for LEMURLOOT specifically — run them even after patching, since a compromise that predates your patch won't be undone by it.

Architecturally, stop treating MFT servers like ordinary web apps. Segment them into their own network zone with tightly scoped egress — a file transfer server has no legitimate reason to be initiating arbitrary outbound connections, so alerting on unexpected outbound data flows is one of the highest-signal detections available. Put a WAF in front of the web interface and use it to block known exploit patterns while you patch. Minimize what's actually exposed to the internet: if partners can reach you over a VPN or IP allowlist instead of the open web, do that. And treat the underlying files as sensitive at rest — encrypt them, and don't let the transfer server itself be the only thing standing between "in transit" and "exfiltrated."

The bigger lesson

MOVEit, GoAnywhere, and Cleo aren't unrelated incidents — they're the same shape of failure recurring because MFT software occupies a structural sweet spot for attackers: internet-facing by requirement, processing untrusted input by design, and holding exactly the data a data-extortion crew wants to steal. That combination doesn't exist for MFT alone — it applies to any edge appliance that ingests external data and holds something valuable afterward. If you run one, the operative question isn't "has this been patched recently," it's "what happens to every file that's touched this box if it's compromised tomorrow" — and whether your network segmentation, egress monitoring, and patch cadence would actually catch it before it becomes the next mass-breach headline.

The 198M-voter S3 leak that still explains most breaches

Mon, 10 Aug 2026 00:00:00 GMT

Most S3 exposure incidents don't require a zero-day, a phishing email, or even much skill — just someone who knows how to type an S3 URL into a browser. In June 2017, UpGuard security researcher Chris Vickery found exactly that: a bucket named dra-dw — "Deep Root Analytics Data Warehouse" — sitting fully public on the open internet, no password and no authentication anywhere in front of it. Inside was 1.1 terabytes of data on nearly every registered voter in the United States: 198 million records with names, dates of birth, home addresses, phone numbers, and modeled ethnicity and religion scores, compiled by contractors working for the Republican National Committee (Gizmodo, UpGuard). Deep Root Analytics had, in UpGuard's account of the incident, simply set the bucket's permissions to public instead of private (CyberScoop). That's a 2017 incident — old enough that AWS has since shipped several generations of guardrails specifically built to prevent it — and yet the identical failure mode, a bucket flipped to public and found by a researcher scanning the internet rather than by the company that owns it, keeps showing up in breach reports nearly every year since. That's the real story here: not one incident, but a misconfiguration class that refuses to die.

Root cause

S3 buckets are private by default. Getting one exposed takes a deliberate (if often thoughtless) action, and over a decade of these incidents the paths to that action have stayed remarkably consistent. Legacy ACLs are the classic one: S3's original access-control-list model lets you grant read or write access to predefined groups, and two of those groups are routinely confused for "my organization" when they actually mean "the entire internet" or "any authenticated AWS account on the planet" — not your customers, not your team, literally anyone with an AWS account. Bucket policies cause the same damage a different way: a policy meant to scope access to a specific role or CloudFront distribution gets written with a wildcard principal ("Principal": "*") or an overly broad condition that accidentally satisfies for anonymous requests too. And a large share of real-world exposures trace back to third-party tooling — backup utilities, data pipelines, static site generators, BI exports — that defaults to a public bucket because it's the path of least resistance for the vendor's quickstart guide, and nobody revisits the setting once it works. None of these require malice. They require a permission model with more than one way to say "public" and no default nudge toward noticing you've done it.

Blast radius

The Deep Root leak wasn't found by Deep Root. It was found by a third-party researcher running broad scans for open cloud storage, who stumbled onto a predictable Amazon subdomain, dra-dw.s3.amazonaws.com (UpGuard). That's the pattern across nearly every S3 exposure story of the last decade — Verizon's exposed customer call records via a third-party vendor, Dow Jones's 2.2 million subscriber records, Accenture's exposed API keys and credentials — the company almost never discovers its own leak. It's found by a security researcher, a journalist, or occasionally a criminal, all of whom have the same tooling: internet-wide bucket-name scanners that turn "is this bucket public" into a solved, automatable question. Once a bucket is enumerable, the blast radius is everything in it, immediately, with no exploit chain required — the exposure is the vulnerability. For Deep Root that meant PII on 198 million Americans sitting downloadable for at least several days before Vickery's report reached the company and the bucket was locked down (CyberScoop).

Remediation

AWS's answer to this exact failure mode is S3 Block Public Access, launched in November 2018 specifically in response to years of incidents like this one — it's a set of four switches, settable at the bucket or the whole-account level, that override any ACL or bucket policy trying to grant public access, so a misconfigured policy simply can't take effect (AWS). As of April 2023, AWS made this the default for every new bucket and disabled ACLs by default account-wide, which closes off the legacy-ACL path that caused Deep Root's leak unless someone deliberately opts back in. Layer on top of that: IAM Access Analyzer for S3, which continuously flags buckets reachable from outside your account or organization, including via cross-account bucket policies that Block Public Access alone won't catch; an AWS Config rule (s3-bucket-public-read-prohibited / s3-bucket-public-write-prohibited) that continuously evaluates every bucket against the policy you've set, rather than only at creation time; and an SCP at the AWS Organizations level denying any principal from disabling Block Public Access or attaching a public bucket policy in the first place, so an individual account or a rushed engineer can't quietly opt back into exposure. Finally, turn on CloudTrail S3 data events (off by default, since they're high-volume) for buckets holding anything sensitive — object-level GetObject logging is what tells you whether an exposure window was ever actually accessed, versus merely theoretically reachable, which matters enormously for incident scoping and breach-notification decisions.

The bigger lesson

This misconfiguration has been headline news since at least 2017 — Verizon, Dow Jones, Accenture, WWE, Pentagon contractor INSCOM, and Deep Root Analytics all made the same mistake within about a twelve-month stretch — and it's still a recurring line item in breach reports today, years after AWS built free, one-click guardrails specifically to stop it. The reason isn't that the fix is hard; it's that S3's original design let "public" be an opt-in property of individual objects and policies, which means it only takes one wrong click, one stale ACL, or one default-public third-party tool to undo. Block Public Access flips that assumption: it makes private the property you have to fight to escape, at the account level, instead of the property you have to remember to defend at the object level. Every org still running without Block Public Access enforced by SCP, without Access Analyzer alerting, and without Config rules gating drift is one misclick away from being next month's version of this story.

CVE-2024-21626 and runc's shared-kernel blast radius

Mon, 10 Aug 2026 00:00:00 GMT

Every container on a multi-tenant host is a guest sharing one kernel with every other tenant. Namespaces and cgroups make that kernel look partitioned, but they don't partition the kernel's code — a bug reachable from inside a container is reachable from the same privilege level the kernel runtime itself trusts. CVE-2024-21626, the "Leaky Vessels" flaw in runc, is the clearest recent proof: a leaked file descriptor let a malicious image escape its container and land as root on the host, no exploit chain required beyond a crafted Dockerfile (Snyk, The Hacker News). And it wasn't a one-off — a fresh batch of runc container-escape CVEs landed in late 2025, hitting the exact same trust boundary from a different angle.

Root cause

runc is the low-level OCI runtime underneath Docker, containerd, and most of Kubernetes' container execution path — it's the thing that actually calls pivot_root, sets up namespaces, and execs the container process. CVE-2024-21626 came from an order-of-operations bug: runc could leak an internal file descriptor referencing the host's working directory before it finished walling the process off with pivot_root. A malicious image that set its WORKDIR to something like /proc/self/fd/7 could ride that leaked fd straight into a directory on the host filesystem, escaping the container rootfs entirely (GitHub PoC, Red Hat).

The 2025 trio — CVE-2025-31133, CVE-2025-52565, and CVE-2025-52881, disclosed by a SUSE researcher in November — is the same story with a different mechanism: race conditions in how runc mounts /dev and enforces "masked paths" under /proc. An attacker who controls container startup config can swap /dev/null or /dev/pts/$n for a symlink pointing at a sensitive procfs file milliseconds before runc bind-mounts it, tricking the runtime into mounting host /proc paths read-write inside the container, or redirecting writes meant for a scoped procfs entry to arbitrary host paths like /proc/sysrq-trigger (Sysdig, OCI advisory). Different bug class, same underlying truth: the isolation boundary is enforced entirely by kernel code that every container process can call into, and a single logic slip anywhere in that code path breaks the boundary for everyone sharing the kernel.

Blast radius

An attacker who lands one of these isn't stealing data from a container — they're getting arbitrary code execution on the host, at whatever privilege the container runtime holds, which is typically root. On a shared build farm, CI runner, or multi-tenant Kubernetes node, that one escape gives an attacker every other container's filesystem, secrets mounted into other pods, the kubelet's credentials, and a jumping-off point to the rest of the cluster. Leaky Vessels was especially nasty for CI/CD because it triggers just by building a malicious image — you don't need to run untrusted code in production, you just need your pipeline to pull an attacker-controlled base image or Dockerfile and build it (Palo Alto Networks).

Working proof-of-concept exploit code for CVE-2024-21626 has been public on GitHub since shortly after disclosure, which is exactly the scenario that turns a CVSS score into an operational problem — no 0-day skill required, just an unpatched runtime and a way to get a container built or started. The 2025 trio requires a bit more precision (winning a mount race), but the outcome is the same: root on the node, and from there, lateral movement across whatever else that node was trusted to isolate.

Remediation

Patch the runtime, not just the orchestrator sitting on top of it — Docker and Kubernetes ship runc bundled, so "update Kubernetes" doesn't automatically mean "update runc" if you're on a vendored or older build:

CVE-2024-21626: fixed in runc 1.1.12, containerd 1.6.28 / 1.7.13, and Docker Engine 25.0.2 (Docker Scout)
CVE-2025-31133 / CVE-2025-52565 / CVE-2025-52881: fixed in runc ** 1.2.8, 1.3.3, and 1.4.0-rc.3** (SecurityWeek)

Beyond patching, this bug class is exactly why "container" and "security boundary" shouldn't be treated as synonyms for anything you don't trust. Defense in depth that actually helps here: run rootless containers so a kernel-level escape lands as an unprivileged host user instead of root; apply seccomp and drop capabilities aggressively so even a successful escape has less to work with; and for genuinely untrusted workloads — public CI runners, multi-tenant SaaS sandboxes, anything running code you didn't write — put a real boundary under the shared kernel with gVisor (a userspace syscall shim the exploit has to get through first) or Kata Containers (a real VM boundary per workload, so a kernel exploit only pops that one micro-VM).

The bigger lesson

runc, containerd, and every OCI-compliant runtime built on the shared-kernel model give you process isolation, not a security boundary against a sufficiently novel kernel bug — and this class of vulnerability proves that gap isn't hypothetical, it's recurred across at least two unrelated bug classes in two years. That's fine for internal services running your own trusted images. It's a real risk for anything that runs code you don't control: public CI, contributor-submitted builds, multi-tenant platforms. For those, the honest question isn't "did we patch runc" — it's "why are we relying on a shared kernel to isolate an adversary in the first place," and whether a VM-backed runtime like Kata or a syscall-filtering sandbox like gVisor should be the default instead of the exception.

Cloud roundup: unauthenticated TeamCity RCE now in KEV

Sun, 09 Aug 2026 00:00:00 GMT

If you run a self-hosted CI/CD server, stop reading and go patch it. That's the headline today — the rest is a mix of AWS platform news and more actively-exploited flaws in tools a lot of us have sitting in our stacks.

TeamCity RCE is now under active exploitation

CISA added CVE-2026-63077, a critical (CVSS 9.8) unauthenticated remote code execution flaw in JetBrains TeamCity On-Premises, to the Known Exploited Vulnerabilities catalog on August 5 (CISA, The Hacker News). An attacker can abuse the agent polling protocol to skip authentication entirely and run arbitrary OS commands as the TeamCity server process — which means your build agents, artifacts, and any credentials TeamCity holds for deploying into AWS are all in play. JetBrains has patches out (upgrade to 2025.11.7 or 2026.1.3, per JetBrains' advisory); federal agencies were given until August 8 to remediate under BOD 26-04. If your TeamCity server has any path to the public internet, treat this like the SolarWinds-style CI/CD compromise scenario it is and patch today, then audit what credentials that server has been handing out.

Two more KEV additions worth knowing about

The same CISA update cycle also added CVE-2026-9198, a code injection bug in Langflow (the visual builder a lot of teams use to prototype LLM pipelines) that gives unauthenticated RCE on default installs — telemetry shows 650+ exploitation attempts from 244 unique IPs since early July (The Hacker News). And CVE-2026-34486, a missing-encryption flaw in Apache Tomcat's EncryptInterceptor, is reportedly being hit by an AI-orchestrated, China-nexus automated attack campaign. If you've stood up Langflow for an internal AI experiment and forgotten about it, that's exactly the kind of exposed dev tool this campaign is scanning for — take: audit anything you spun up "just to try AI stuff" six months ago and never locked down.

AWS Lambda gets a real bandwidth bump

Less urgent, more useful: AWS Lambda functions outside a VPC with 2 GB+ of memory now scale network bandwidth proportionally, from 625 Mbps at 2 GB up to 3,000 Mbps at 10 GB, at no extra charge (AWS). That's a meaningful jump for anything doing large payload transfers, S3 streaming, or bursty data-heavy work in Lambda — previously you were capped at 625 Mbps regardless of memory. Catch: it's not automatic. You have to request the "Network bandwidth per execution environment" quota bump via Service Quotas before your functions see it, so if you have latency-sensitive Lambdas moving a lot of data, it's worth requesting now even if you don't need it yet — quota changes aren't instant.

Also worth a look

CloudWatch now offers managed Prometheus collectors that auto-provision and scale to pull OpenTelemetry/Prometheus metrics from EKS, EC2, ECS, MSK, and OpenSearch without you running your own collector fleet — worth a look if you're maintaining a hand-rolled Prometheus scraping setup on EKS. And separately, N-able confirmed active exploitation of an N-central authentication bypass (CVE-2026-18577) that let an attacker pivot from a compromised N-central server into managed endpoints — a reminder that RMM/MSP tooling is a high-value target precisely because of the blast radius one compromised instance gives an attacker.

Bottom line

Today's theme is "your build and automation tooling is the target, not just your app." Patch TeamCity if you run it, sweep for forgotten Langflow instances, and grab the free Lambda bandwidth quota bump while it's on your mind.

JSTGTECH

Cloud roundup: macOS Screen Sharing bug now under attack

macOS Screen Sharing flaw is being exploited to plant Monero miners

ShieldBreak: an unpatched Windows Defender bypass with a public PoC

EC2 gets built-in application health checks

Also worth a look

Bottom line

Cutting NAT gateway costs with VPC endpoints that actually help

Two different mechanisms, and it matters which one you reach for

Adding a gateway endpoint

Adding an interface endpoint

Where NAT still has to stay

Finding out what's actually costing you, before you guess

The gotcha: adding the endpoint doesn't guarantee it gets used

What to actually do

Cloud roundup: S3 finally names the policy that denied you

S3 access-denied errors now name the exact policy ARN

AWS Client VPN ships a scriptable CLI and centralized policy controls

Bedrock adds OpenAI's Daybreak Red and Blue for authorized security work

Bottom line

Service spotlight: Amazon Cognito user pools vs identity pools

What it actually is

Where it earns its keep

Where it goes wrong in practice

A practical tip

The Capital One breach: an SSRF bug into 100M records

Root cause

Blast radius

Remediation

The bigger lesson

Cloud roundup: 153GB LiteLLM breach exposes cloud secrets

153GB of stolen credentials surface from the March LiteLLM attack

Amazon EKS now lets you tune scheduler, controller manager, and API server parameters

OpenSearch/Elasticsearch Extended Support surcharge is about to double

Bottom line

Cloud roundup: Metabase CVSS 10 SQLi confirmed exploited

Metabase's CVSS 10 SQL injection now has a CVE and is in CISA KEV

Cisco ASA and FTD hit with an actively exploited VPN DoS flaw

Microsoft's August Patch Tuesday: ~400 fixes, one already exploited

Also worth a look

Bottom line

Cloud roundup: New ransomware exploits N-central bug

StormEncryptor ransomware ties back to the N-central auth bypass

EC2 gets application-level health checks, not just instance-level

AWS DRS now preserves UEFI boot mode on Linux failover

Also worth a look

Bottom line

Cloud roundup: LoadMaster RCE hits its KEV deadline today

Progress LoadMaster command injection hits its remediation deadline today

A CVSS 10 zero-day in Metabase, no CVE yet

N-able N-central: the first patch wasn't enough

AWS Security Hub Extended adds supply chain security

Bottom line

CI/CD for a static site: build, sync, and invalidate

The three stages, and why order matters

Sync: cache-control headers aren't optional

The --delete gotcha: it respects filters, both ways

Invalidate: scoped paths vs. wildcard, and the pricing surprise

Rolling this out

CloudFront invalidations without nuking your AWS bill

How invalidation pricing actually works

The real fix: stop invalidating, start fingerprinting

Where you still need surgical invalidation

The gotcha: invalidations aren't instant, and they queue

What to actually do

Ditch IAM access keys: GitHub Actions OIDC to AWS

How the trust actually works

Step 1: create the OIDC provider

Step 2: write a trust policy scoped tighter than you think you need

Step 3: use it in the workflow

Trade-offs worth knowing before you migrate

Rolling it out without a big-bang cutover

IAM Identity Center: kill per-account IAM users for good

What Identity Center actually replaces

Setting it up

1. Choose your identity source

2. Create permission sets

How permission sets map to account assignments

Walkthrough: one group, three accounts

CLI usage: aws sso login

The `--delete` gotcha: it respects filters, both ways

CLI usage: `aws sso login`