A lot of people are complaining about bot traffic. I was curious as to the impact on this site, but for the past six years it’s been deployed to Netlify, and I didn’t have a way to see raw request volume without paying for a limited Analytics feature, and I couldn’t see disaggregated referer and user-agent without paying a lot more money for enterprise log drains.
AWS has powerful observability features that provide exactly what I want, but moving a Netlify site to S3 + CloudFront isn’t that simple. AWS S3 websites are limited in a few ways, and a CloudFront + S3 REST deployment is more powerful but more barebones out of the box: by default it doesn’t support directory redirects, index file rewrites, or custom response headers. Those features are there by default on Netlify, but on AWS you have to write custom CloudFront functions to handle them.
So I wrote Hedgerules to deploy those CloudFront functions, enabling me to switch my site from Netlify to AWS without losing any functionality, and gaining the observability I wanted.
The Hedgerules documentation includes a detailed comparison table for all its features. The most critical ones are:
- Directory index redirects —
/pathredirects to/path/ - Index file rewriting —
/path/serves/path/index.htmlfrom S3 - Hugo alias redirects — redirects from Hugo’s
aliasesfrontmatter - Custom redirects — user-defined
_hedge_redirects.txtfile - Custom response headers — global defaults and per-page overrides via
_hedge_headers.json
Hedgerules also has a niche feature that I believe is totally unique to it:
it understands a special {/path}
request path token
when parsing custom response header definitions,
which is replaced by the request path
(and guaranteed to be prefixed with a /).
I use this to emit
Onion-Location headers
for every page with just one entry.
Hedgerules includes a Hugo theme
that provides templates for its _hedge_redirects.txt and _hedge_headers.json files.
(It doesn’t affect your site’s style or generated HTML pages at all.)
As part of this, it creates a map of all
redirects
and headers.
This is similar to
Hugo redirects and partial functions
from all the way back in 2021,
and uses the more modern Hugo
return
statement rather than a clunky partial that expects you to pass it a scratch var.
AWS observability
Now that Hedgerules has handled those issues for me, I can pick up a bunch of useful AWS features for approximately free, like raw logs with CloudWatch Logs Insights, and visuals with CloudWatch Dashboards.
Here are some example screenshots from the dashboard for my main website:
And here is a live tail of my access logs:
Finally, I can use AWS Athena to query historical logs in a SQL dialect:
On Netlify
My experience with Netlify has been wonderful, and I still have maybe a dozen sites there. I think the best endorsement comes from the reason I made Hedgerules in the first place: there’s a bunch of extra bot traffic out there, and I just had no idea because Netlify is handling it all for me.