The Glass House Fallacy
Part II of my challenge that open source (AI) transparency in itself does not lead to better, more ethical development outcomes
I’m very grateful to a machine learning engineer I really respect, as I got a beautiful and detailed response from him to my previous text. He makes a few points worth stressing at the outset:
I made an error in platform identification (Vertex AI is proprietary, not open source), and that is my mistake indeed. Nevertheless, I feel like the broader argument stands.
Verification capability almost always equals security advantage.
Organizations have the capacity to perform proper vetting.
The ability to verify is more valuable than controlled development.
Market competition should take precedence over mandatory security measures.
The last four points are assumptions that reveal a classic technologist perspective where technical capability almost always equals implementation, and where market forces and individual responsibility are seen as superior to institutional oversight. It is a perfectly valid philosophical position, but one that often seems to overlook (without malice, I would assume) the practical realities of how most organizations actually operate in the real world.
Verification Fallacy
Let’s start with the verification fallacy. The ability to verify code directly is no doubt valuable, but it presents us with a fallacy. The argument that open source is more secure because you can verify the code assumes two things:
That verification is actually happening at scale,
That verification automatically leads to security.
I argue that neither assumption holds up under scrutiny. While the potential for verification exists, the reality is that most deployments of open source AI models occur without meaningful verification. Rather, users of these models rather assume that some Good Samaritan did it for them. It literally is the digital equivalent of having a transparent house where anyone could theoretically check for intruders, but who actually does?
False Dichomoty
You can make an argument that I was implying that closed source is better. However, that isn’t the case at all – it’s that openness alone doesn’t solve our fundamental security and ethical challenges. The real issue isn’t about open versus closed; it’s about our current inability to effectively govern AI development in either paradigm.
Consider this: when Thomas Wolf (a link my respected colleague shared) advocates for open source security advantages, he’s speaking from the perspective of a highly sophisticated organization with deep technical expertise. But what about the thousands of organizations deploying these models without such capabilities? What about those who just don’t care at all? What about those who do have ill intents? There’s a reason why Open AI (not an epitome of open source, but a valid illustration for the argument nevertheless) cut off China’s access to their solutions.
The ability to verify doesn't equal actual verification.
My colleague also mentions that with closed source, you can only trust, not verify. But this presents a false dichotomy. In practice, most organizations using open source AI are also operating on trust - they're just trusting a different set of actors. They trust:
The community to catch vulnerabilities
Package maintainers to act ethically
Their own ability to detect malicious code
Rather than arguing for or against open source, we need to acknowledge that both open and closed source models have security challenges that mere transparency doesn't solve. We need:
Better governance frameworks that work across both paradigms
Practical security measures that don't rely solely on theoretical capacity to verify
Recognition that security through transparency only works with active, capable oversight
Conclusion
The core issue isn't about choosing between open and closed source - it's about acknowledging that neither paradigm inherently solves our security and ethical challenges. We need to move beyond this dichotomy and focus on building practical solutions that work in the real world, where most organizations lack the resources for comprehensive code verification.
The question isn't whether we can verify the code - it's whether we will, and what happens when we don't.
November 2024