Beyond Secure by Design- Where is the Quality Software Bible?
By Matt Abrams (TFX Venture Partner, Advisor, Board Member, & Principal of the Abrams Group), Karl Hightower (TFX Venture Partner, Vice President, Chief Data & Analytics Officer, Stanford University), and Brandon Shelton (Founder & Managing Partner, TFX Capital)
INTRODUCTION
In the summer of July 2024, a few errant lines of code deployed to production without sufficient testing caused over $5 billion in damages, impacting 300 of the Fortune 500 companies. While this error stranded travelers and made Crowdstrike a household name, the incident raises fundamental questions for investors, users, and stewards of the digital environment: Why does it take critical, costly, or nearly life-or-death events to institute high-quality software?
What are the incentives for developers to institute software assurance, and for customers to insist on quality standards that go beyond the low bar of “it’s passed basic testing”?
Is this an opportunity for a Quality Software Bible that isn’t only Secure by Design, but also a set of best practices and governing principles to emerge in this dynamic, AI-fueled environment?
STANDARDS VERSUS SPEED
There are internationally recognized standards with respect to software quality (ISO/IEC 5055:2021). However, when you’ve only got 12-24 months of runway before you’re out of business- Speed is everything for startups selling to the enterprises. Investor pressure to land customers, scale rapidly, and raise follow-on capital is real.
Oversight of software development at this early stage varies, and may include a fractional CTO, an off-shore or near-shore development shop, or networks of friends.
While the fundamentals of software engineering (robustness, security, performance, maintainability) are well known, startups nevertheless incur sizable technical debt in these early years as they battle to establish product market fit. For high-reliability organizations dealing with sensitive and regulated patient data (HIPAA) the problems are immense.
US healthcare providers typically operate with limited budgets for additional CIO/CTO personnel.
“The cost to create a sandbox to test new software or new products may be in the low millions of dollars for mid to large size healthcare companies, but the opportunity cost of missing out on differentiating services that can be gained from these new software implementations can be exponentially larger.
The other side of this is the mistakes of putting things in without isolated testing can create HIPAA violations, massive financial damage, brand reputation degradation, and ultimately the loss of patient trust.” -Karl Hightower, Chief Data & Analytics Officer, Stanford Healthcare
QA or DIE
At TFX Capital, we have portfolio companies selling to high-reliability organizations in finance and healthcare- All of which face enormous risks from software development. To address this, HRO’s can develop software in-house or purchase it from vendors, both options come with tradeoffs. As the Crowdstrike incident proved, enterprise-level vendors make mistakes too!
However, the demands of HROs can be overwhelming for startups. We have seen these institutions squeeze already lean startup technology teams with long procurement cycles and budgets that do not meet their needs. Enterprise wants everything yesterday and at the highest standards possible, which is ideal for startups! But is the risk profile of a startup’s “bootstrapped” software development too great for an enterprise customer to tolerate?
Thankfully, several of TFX Capital’s portfolio companies have punched through these enterprise sales barriers, such as Tidal Cyber and Xona.
What sets them apart?
These companies have full-time CTOs who are actively upholding quality assurance processes in partnership with their large enterprise customers despite being startups themselves.
They employ a “work smarter, not harder” mindset and leverage automation where possible.
However, some of our companies have punched through the challenging enterprise sales barrier such as Tidal Cyber and Xona. These companies have full-time CTOs who are actively upholding quality assurance processes in partnership with their large, enterprise customers despite being startups themselves. They employ a “work smarter, not harder” mindset and leverage automation where possible.
“Across the development and technology processes, we require developers, engineers, etc, to be the infrastructure janitors across all the complex services that are now required in modern applications. There are early solutions (Codezero Technologies Inc.) that help do the same for infrastructure and reduce the need for every development team to be infrastructure plumbers and janitors.” – Matt Abrams, TFX Capital Venture Partner
HOLD THE LINE
There will be more and more startups trying to create solutions for enterprise customers, and for now, venture capital will be there to propel them. Whether in the microservice array or fully embedded in a major Operating System (i.e. Crowdstrike and Microsoft Windows), software creators/developers must hold the line when it comes to quality management controls.
In recent years we have seen faster and more expansive code development at large firms, offshore development groups, startups and everywhere in between. As more code is created and connected, and as more developers bring different knowledge (and gaps) to engineering, the opportunity for a bug or bad code to be written/released continues to grow. Companies and engineering leaders must continually adapt quality processes and adequately resist mounting speed pressures. And if a startup has leveraged offshore software development for cost purposes, they will inevitably have to refactor this code as they scale, and bring development in-house. The ability to test the current state can be limited without proper documentation.
THE X-FACTOR: (G)AI
Artificial General Intelligence (AGI) and General Artificial Intelligence (GAI) are becoming mainstream definitions as thousands of researchers and billions of dollars of venture capital build the next and best solutions for a wide array of opportunities.
“The resources used to train the model(s) can be repurposed to run millions of instances of it (this matches projected cluster sizes by ~2027), and the model can absorb information and generate actions at roughly 10x-100x human speed.” -Dario Amodei, CEO of Anthropic (Oct’24)
Future software development may come exclusively from trained models, and we are seeing ‘nudges’ and other intelligence tools for developers to complete the code they are working on.
While this is exciting to help expand the already limited bandwidth of human software engineers, one doesn’t have to dream too big to foresee risks with autonomous software development tools at scale.
We can see AI being used to deliver robust QA/QC outcomes, sophisticated ‘embedding models’ to help companies leverage their data security efforts, and other efforts to expand, expedite, and scale protections.
Humans must remain in the decision loop of software actions and decisions at some level to ensure proper oversight and control. Remember, the adversary gets a vote in future cybersecurity battles.
SOFTWARE QUALITY RECOMMENDATIONS
Here are our recommendations for Founders and technologists alike:
- See something, say something. This adage must be part of the operational fabric for every startup, engineering team, and leader. Engineers, no matter their level or role, should be empowered to point out issues when they see or anticipate them. Build a safe process for reporting issues.
- Ownership. While ownership of quality software design and development should be ubiquitous, someone must fully own quality. Clear roles and responsibilities must be established, no matter the organization’s size.
- AI matters. As greater and greater tools become available, teams need to leverage AI for code improvements and QA testing. Humans need to remain in the loop, but cost-effective tools are becoming increasingly available and need to be adapted.
- Dawn of the architect.Properly architecting products and platforms from the very beginning is essential to avoid legacy technical debt as you scale.
While software tools and QA techniques have been expanding at organizations for decades, there are three converging trends calling for a new way ahead.
First, the cybersecurity threat environment has never been greater with nation-state adversaries and other bad actors persistently searching for code entry points.
Second, venture funding for enterprise software companies has been growing since the early 2000s. There are more and more startups with venture funding banging on the doors of enterprise customers—some have adopted our recommendations, but many have not.
Third, the ascent of GAI and AI is extraordinary, as leaps are being made monthly. These techniques and resources will dramatically expand the toolkits for engineers (for good and bad), requiring an adaptable framework or “Quality Software Bible” to ensure quality code is shipped.
As Marietje Schaake writes in Tech Coup, “In our final report we concluded that, in practice, it is impossible to make fully secure software.” But it’s imperative that we try, and discipline and ownership are paramount for everyone involved in the digital ecosystem as we embark on 2025 and beyond.
