Nowhere to hide: The last gasp of security through obscurity

Modern day cryptography is built on a very counterintuitive foundation called Kerckhoffs’ principle. The idea is that a security system is only secure if everything about it can be publicly disclosed. Or as Claude Shannon put it, “the enemy knows your system”.

The principle holds that a cryptosystem should be secure, even if everything about the system, except the key, is public knowledge. – Kerckhoffs, 1883

You can see why this would be counterintuitive. It’s quite common for companies to believe that by hiding the details of their security infrastructure, they will make their product safer and less vulnerable to attack. Naively this position appears reasonable, because the time and complexity of attacking a system are increased, at least in the short term. But in practice, the cryptographic and security systems that we have come to rely on benefit greatly from having had millions of people scrutinize them closely and continually patch weaknesses.

A large amount of what we call “security”, “privacy” and even “proprietary IP” has rested on similar foundations of hiding a signal in significant complexity and relying on the sheer friction of extraction being too high for anyone to bother. But with AI you can outsource thinking and comprehension, so complexity is no longer a real barrier.

From an organization’s perspective, a poorly configured system that would require significant time and expertise to exploit for questionable gain is simply not worth the investment required to achieve real security. The public has gotten used to breaches so reputational risk is minimal in the slim chance someone bothers, and obscurity is free. But in this new regime where AI drops the marginal cost of comprehension to zero, that just doesn’t work.

One manifestation of this concept that made news recently is a software engineer who used Claude Code to operate his new robot vacuum. He refused to use the normal app like a pleb and wanted to control it with his Xbox controller. So Claude naturally overdelivers and pulls in an auth token from their servers. But the company had built auth with zero device ownership verification probably because they didn’t think anyone would invest the time and effort to figure out how to “hack” their systems. Turns out that our guy now has eyes inside 7000 homes and can access all their camera feeds.

Another recent story is a hacker who used Claude (supplementing with ChatGPT where Claude refused or required additional information) to breach multiple Mexican government agencies. Based on the logs retrieved post-hoc, the hack was opportunistic rather than planned, and the hacker used Claude to probe for gaps rather than execute a targeted attack against a particular system.

Everything is open source if you're good at reversing.

For quite some time, shipping a program binary was a relatively meaningful form of protecting your IP. Not anymore, with the models increasingly capable of converting binaries into source, and excellent at implementing systems when there’s a finite and constrained end state to validate against.

The assumption of anonymity is already starting to break down. If you’re posting anonymously on the internet, you’re relying on there being too many users, too many platforms, too much data for anyone to correlate your real identity. But a recent paper from ETH Zurich (advised by Nicholas Carlini) built an LLM pipeline that de-anonymizes users at scale from unstructured text alone. And these systems will only get more sample efficient. The authors should write a v2 next year where they dox Satoshi to really hammer their point home.

Same goes for mass surveillance. The Fourth Amendment was rooted in physical reality.

The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.

Arguably, even without AI it was insufficient for the digital age. There are several private companies that allow the government to purchase aggregated digital data of citizens without procuring any warrants, and this is legal. At the same time, more of day-to-day life is moving online. With AI, it is trivial to correlate these disaggregated sources and gather a highly nuanced picture of any individual. The laws have simply not caught up in response to the scale of disruption that AI enables.

Tinfoil is an open source company. We run AI models inside secure hardware enclaves, and strongly believe in verifiability. One component of that is that all the code is open source. People always ask us, what’s our moat. As SaaS melts away and cost of creating software, especially if there’s an existence proof, goes to zero, in some ways, I think it’s better to be open source than to worry about how to hide IP and proprietary information. We already know our code is public so we get to focus on all the stuff that isn’t.

To be fair, this isn’t why we decided on this model. We decided on it because it legitimately improves the product and our biggest advantage is that the system is fully verifiable, which we can’t do to the extent that we’d like unless we make it open source. Hardware security is really complicated, with even large companies implementing it in a confused and contradictory way. So one of our biggest priorities is having a clear security model that is legible to both humans and to the AI models they frequently use to understand it. In some ways, we’re building the company and writing the code so models, whether now or in the future, can explain it and use it as a blueprint. We founded this company because we want a future with private and verifiable AI, and if AI can build that and incorporate it into products I want to use, that’s good enough for me.

Another interesting example of this concept was proposed by Allison Bishop, a cryptographer who founded a hedge fund on the thesis that it was possible to design a trading algorithm that was public. The argument is that the algorithm blends an institutional order into market noise and therefore, revealing the mechanism shouldn’t compromise it. This is kind of nuts especially in the trading industry, whose raison d’être is secrecy. I don’t know enough about trading to have a real opinion here, but instinctively markets may be the limit case for this principle since microsecond temporal advantages matter. And it does appear that they were profitable last year, so the fact that the idea survives contact with reality even a little bit makes it a fascinating data point for this principle.

I wrote this because I’m increasingly convinced that hiding in noise and complexity is no longer a valid strategy. In some ways this is kind of an amazing situation because it forces us to rethink a lot of bullshit complexity and whether voluntarily or involuntarily, forces transparency. It’s not imposing disproportionately more overhead for security either, because it symmetrically improves both offense and defense. If the cost of an attack goes down, so does the cost of securing a system in the first place.

What is concerning is the expansion of authoritarian capability, for instance, through mass surveillance, and the ways in which AI allows sidestepping the intent of the law faster than regulation can catch up. For this, we may need governance that’s somehow baked into the technology itself, systems that enforce security and verifiability through design. Every loophole can and will be exploited.

◆