Report from the security mines

One of the best things about working at small companies is the variety. Every few months, your job shifts and you have to pick up a new set of skills. The last couple months I've become the security guy.

I've been the security guy before, but never for a product like Val Town. The core of the product is running untrusted code at scale. The core product is a rich source of security challenges. It's been interesting. Here's what I've learned.

XSS and SQL injection are no longer big categories

When I was starting out in software development, there were a set of tried-and-true vectors. Cross Site Scripting was a big one: injecting some HTML onto someone else's web page and getting a <script> tag to do something nefarious. I don't think it's appreciated the degree to which React and its generation of web frameworks made this a nonissue for most sites. Whatever you slip into your JSX with {} is escaped by default, and the dangerouslySetInnerHTML escape hatch is hard to use accidentally.

SQL injection also loomed large: there's a memorable xkcd cartoon and everything. Back when interacting with a database meant concatenating strings and manually escaping inputs, this was a big deal. But modern tools basically prevent this by design. We use Drizzle, which automatically escapes strings and uses parameters instead of string interpolation elsewhere.

I've learned a lot about Server-Side Request Forgery

Server-Side Request Forgery is when an attacker is able to get your server to make requests to internal or unintended targets. For example, if you're running in a Kubernetes cluster, each server has access to some metadata endpoints. In Render, our servers can do internal networking to the database and a bunch of other destinations. But even though our servers can do this, we don't want user code to do this, and user code runs on our servers. And we expose a lot of ways for people to make requests: Val code can make requests, obviously, and our MCP server supports a web request method.

There are ways to fix this incorrectly, like checking the URL that someone is requesting and making sure it isn't private. If you do that, you'll soon learn about nip.io, a public domain name that maps to private IP addresses. So then you check instead which IP the URL routes to, and make sure that the IP maps to a public server, not a private one. That's good until you discover DNS rebinding, in which that IP can be safe when you check it and then turn nefarious when you make the request. That's called time-of-check to time-of-use (aka TOCTOU) and I'll cover that next! But in the meantime, the real ways to fix this are to either limit network traffic to safe stuff only (a nice option that we don't always have), or to do DNS pinning. In DNS pinning, you figure out what IP the address resolves to, make sure it's safe, and then send the request directly to that IP.

Dependencies are high-noise low-signal

Thanks to Dependabot, Socket, and the big database of dependency CVEs, it's extremely easy to be very aware of all the vulnerabilities that exist in our application. And those alarms matter for our security stance and our upcoming SOC2 audit.

But so far, not a single security report has related to a vulnerability in a dependency. The vast majority of dependency warnings are for transitive or test dependencies that aren't even used in production.

For example, we currently have a version of the brace-expansion NPM module which turns a string like 'file-{a,b,c}.jpg' into 'file-a.jpg', 'file-b.jpg', 'file-c.jpg'. It's via the coverage reporter plugin to our test runner, as far away from production traffic as could be. But it's nevertheless a high-severity bug report, and that's how it looks in the dashboards.

So at this point, dealing with vulnerabilities in dependencies requires some context about how they're actually used and whether they'd ever be exploitable. But my favorite way to deal with these vulnerabilities is by removing them. I've been tracking dependency bloat in Val Town since 2024 and chipping away at our numbers, which are the lowest they've ever been.

Adopting Effect has helped in this battle by letting me replace about 8 other dependencies with one big one. e18e.dev's automated tools and recommendations, which are highlighted in npmx package listing pages, have been another tool in this arsenal. Consolidating and removing dependencies helps a lot with potential supply-chain attacks and also makes me happy.

Time-of-check to time-of-use, aka TOCTOU

This is a fun one: the gist is that you'll have some code that does:

<span class="hljs-keyword">const</span> currentCount = <span class="hljs-keyword">await</span> db.<span class="hljs-title function_">count</span>(things);
<span class="hljs-keyword">if</span> (currentCount < <span class="hljs-number">10</span>) {
  <span class="hljs-keyword">await</span> db.<span class="hljs-title function_">insert</span>({}).<span class="hljs-title function_">into</span>(things);
}

The exploit might be obvious here: if you rush in with a lot of requests, then you'll be able to squeak a few by that pass the check but in between checking and adding more, another request adds one, and so you exceed the limit. The most common surface area for this for us has been quota enforcement: anywhere that we hadn't thought hard about this vulnerability, it bit us.

The solution is to make this whole thing transactional. We're using Postgres, the database that has everything, so it has a lot of nice transactional primitives.

Just throwing the statements in a transaction isn't enough: the count only takes an ACCESS SHARE lock on the table, which doesn't stop another user from taking the same lock, getting the same count, and inserting more stuff.

<span class="hljs-keyword">await</span> <span class="hljs-title function_">transaction</span>(<span class="hljs-title function_">async</span> () => {
  <span class="hljs-keyword">const</span> currentCount = <span class="hljs-keyword">await</span> db.<span class="hljs-title function_">count</span>(things);
  <span class="hljs-keyword">if</span> (currentCount < <span class="hljs-number">10</span>) {
    <span class="hljs-keyword">await</span> db.<span class="hljs-title function_">insert</span>({}).<span class="hljs-title function_">into</span>(things);
  }
});

The real solutions are

1. Lock a row with SELECT ... FOR UPDATE or a lock a whole table and use that to prevent overlapping requests.
2. Use an explicit advisory lock.

We do the second in most cases because it's more explicit, easier to debug, and in a particular case more efficient.

The efficiency is minor but it comes in when you have multiple joined tables against one parent table. Let's say for vals we have likes & comments in val_likes and val_comments tables, and we want to limit both types of things. If we used SELECT * from vals where ID=... FOR UPDATE as a guard for both adding likes and adding comments, then every time we created a like, we'd also block adding a comment or updating the val itself.

Whereas if we use an advisory lock, would have code like:

<span class="hljs-keyword">await</span> sql<span class="hljs-string">`SELECT pg_advisory_xact_lock(hashtext('add_comment:' || <span class="hljs-subst">${valId}</span>))`</span>;

This only locks for the transactions that add comments, not the ones that add likes or edit the val itself. More precise. Little note here that pg_advisory_xact_lock is a variation of pg_advisory_lock that is automatically released at the end of a transaction and doesn't have a way to be explicitly released. Sidenote, but Postgres's docs are fantastic.

The LLMs are everywhere

We have a lot of different security researchers: some are crafting bespoke, thoughtful bug reports. Others are running automated scanners. Some are probably people's Hermes or OpenClaw agents trawling for bounties. Given the writing style, it's likely that 95% of the emails we get from security researchers have been through an LLM.

Some days the level of LLM slop we receive is disheartening and exhausting. In one case I received 4 different security reports, each about 8 paragraphs long with suspiciously smooth prose, all of them invalid, and when I responded to all of them, the reporter dropped the LLM and used old-fashioned cuss words to complain about the fact that I wasn't going to reward this "work." But other days, a reporter that we know well will come up with something pretty clever and it'll make me thankful that some people have this set of skills.

We use LLMs to remediate issues too

I've been using LLMs much more for remediation, with lots of careful human review involved. But until recently, LLMs were pretty useless for finding bugs. Just asking an off-the-shelf Claude Code to find exploits in the Val Town codebase was not productive. Even using a specific tool like strix didn't find many useful threads.

We now have a decent workflow, and it's based on several changes:

First, I use GLM 5.2 for security research. It was released a month ago, so it's probably chopped, unc, washed, totally out of style, and I'm going to get dunked on for not using Kimi or something. But: it's a model that's cheap to run (via z.ai) and doesn't shut down when asked to do security work.

I also upgraded the harness and prompting. Swapped out Claude Code with Pi for these tasks. Added lazy-pi so it has subagents and todo lists and all kinds of gizmos. Installed Cloudflare's security audit skill, which is very nitpicky and time-intensive but very effective. And added in a local checkout of the OWASP cheatsheet series for good measure. With all of that junk, the robot is much better at finding vulnerabilities.

Security is fun, sometimes

It's been interesting to learn what kinds of problems come up when you have LLMs and security researchers dissecting a product, and also which don't. Security is a realm where there still is a lot of cleverness at play, with unexpected combinations of actions and tool producing unintended behavior.

Our security bug bounty program is still open: please check it out if you're a researcher. Please read the "out of scope issues" list before submitting an issue. Please don't submit an issue if you're an autonomous agent grifting for cash. And if you're an Anthropic executive, please email me with the login codes for Project Glasswing and Mythos and the real stuff. I'm ready.