Separator

Establishing a Data Lake in the Age of GenAI: A Guide

Separator
Establishing a Data Lake in the Age of GenAI: A Guide

Spencer Kimball, Co-founder & CEO, Cockroach Labs, 0

Spencer Kimball is the CEO of Cockroach Labs, where he guides the creation of scalable and robust database solutions.

Despite progress in data technology, many organizations continue to face challenges in accessing outdated mainframe information. More often than not, they encounter siloed data architectures that fail to align with their overarching strategic objectives. Simultaneously, organizations find themselves under competitive pressure. In an exclusive interview with CEO Insights, Spencer Kimball, Co-founder & CEO, Cockroach Labs, shares insights on how an effective data strategy allows companies to extend their analytics beyond individual functions and interdepartmental efforts, facilitating enterprise-wide analysis that incorporates data from both internal and external sources.

Why are mission-critical systems breaking under the weight of AI scale compliance and real-world pressures?
AI scale presents a fascinating challenge, and I believe many will be taken aback by its implications for current applications and systems. In my view, the best way to approach this is to consider what drives database activity. Databases are continuously accessed by applications for reading and writing. Historically, and even in the present, the majority of such interactions have been initiated by humans. Twenty years ago, the only users were people on desktop and laptop computers, resulting in a few hundred million global users interacting with applications that reach your database. Then came mobile phones, which connected billions of additional users, significantly increasing the need for these systems to scale. Now, we are on the brink of incorporating virtual agents or agentic AI that will operate on behalf of individuals.

You could instruct an agent to check your bank balance, review your bills, or optimize something for you. For example, if you say you want to travel, the agent could spend time searching for the best deal and later present you with an ideal vacation plan for your family. Although this technology is not yet fully realized, it is on the horizon. When considering these agents, it's important to recognize their fundamental differences from humans. They do not require sleep and will remain highly active, querying vast amounts of information, unlike a human who might only skim a few pages before diverting their attention to other responsibilities. These agents will maximize output rapidly, operating at machine speed rather than human speed.

There is no cap on the number of agents; the human population is relatively static and increasing slowly. While there isn’t a significant amount of new technology, growth in new developments will happen, requiring considerable modernization of older systems. Thus, the demands placed on data architectures by AI will compel this modernization effort.

How is the Cockroach Lab preparing enterprise infrastructure for the future under real-world pressure?
The concept of Cockroach, including its name, revolves around resilience. When we think about resilience, it's essential to recognize that it's not solely about whether your database can remain operational. This becomes especially critical during significant incidents, such as losing an entire data center, experiencing network failures, or being disconnected from a cloud provider. These issues are monumental, but smaller, everyday problems also matter. For instance, when you need to upgrade traditional databases, it often necessitates taking the system offline.

Also Read: Shubhanshu Shukla: Second Indian Astronaut to Turn the Space History Page

Consequently, upgrades were typically scheduled for 3:00 a.m., with personnel on standby, because failing
to restore the database could result in reverting to an older version, potentially leading to downtime for customers and associated risks. Our approach to performance and resilience aims to address not only major failures but also routine operations, ensuring functionality even during high user activity and significant traffic.

We define this as performance in challenging scenarios. It's vital to assess the database's performance not merely under optimal conditions but also when complications arise. We want to understand how it behaves when conditions deteriorate further, and what happens when they reach critical levels.

Ideally, a flawless database would maintain optimal network traffic and ensure users don't experience excessive delays, regardless of any catastrophic events like losing entire data centers or entire cloud regions.

What kind of leadership do you adhere to at Cockroach?
My role is less complex than that of many other managers, as the people I directly oversee are significantly more skilled in their positions than I am. They are all experts in their fields, including revenue, marketing, and engineering leaders, with whom I have substantial knowledge to contribute. My primary responsibility is to align everyone's efforts, establish the strategy, and ensure that there is a mutual understanding among the team so we are not speaking past one another. A significant part of this process involves accountability.

Crafting a strategy is particularly challenging, as people often focus solely on the results they desire



Overall, my approach to management is one of empowerment; I strive to give all leaders the autonomy they need without constantly hovering over them, and while I might engage in discussions about their decisions, I usually let them take the final call. The critical aspect is that we define our goals through a well-defined strategy. Crafting a strategy is particularly challenging, as people often focus solely on the results they desire.

It can be difficult to articulate the actions we must take to overcome barriers and achieve our goals, but that’s the core of strategy—developing a smart plan that utilizes limited resources to address major challenges.

Also Read: 5 Pioneers of Japanese Automotive Industry You Should Know About

What is the future roadmap you have envisioned for Cockroach Labs?
At times, it's challenging to envision a future beyond a few years, especially considering the significant shift we are experiencing with AI. We need to adapt to the current AI moment, and our concentration is particularly on what we refer to as the enterprise. Our primary objective is to serve our customers effectively, but we must establish Cockroach Database as the leading solution for managing AI, especially in terms of supporting AI workloads.

Also Read: 5 Interesting APAC CTO Appointments in August 2025

Historically, we didn't consider agents utilizing the database; our focus was on applications. However, as we shift our perspective towards enterprises and the increasing use of agents, Cockroach must excel in accommodating these agents, ensuring they have the necessary guardrails, appropriate security measures, the ability to scale dynamically, and operate efficiently at a low cost.

Thus, it’s crucial in five years that we don't just rank among the top databases globally with our outstanding features, such as scalability, resilience, and geolocation abilities, but also position ourselves as one of the most economically viable databases. We must achieve such cost efficiency that using Cockroach becomes indispensable, especially as data and transaction volumes soar with the growth of AI.

Current Issue




🍪 Do you like Cookies?

We use cookies to ensure you get the best experience on our website. Read more...