[…] Stonebraker says that the spark for the idea for DBOS, which is short for database operating system, came when he was listening to a talk by Zacharia, who among other things was the creator of the Spark in-memory database while at the AMPLab at the University of California Berkeley and the co-founder and chief technology officer of Databricks, which has commercialized Spark.
“This talk was at Stanford three and a half years ago,” Stonebraker tells The Next Platform. “And Matei said that Databricks was routinely orchestrating a million Spark subtasks on sizeable clouds and that Databricks had to keep track of scheduling a million things. He said that this can’t be done with traditional operating system scheduling, and so this was done out of a Postgres database. And then he started to whine that Postgres was too slow, and I told him we can do better than that.”
[…]
And rather than fight about it, Stonebraker and Zaharia teamed up to create an operating system based on a database rather than a database bolt on for an operating system.
[…]
tonebraker and Zaharia played around with ideas, and built a prototype operating system on VoltDB to prove it would work; then they founded a company to commercialize the idea in April 2023 and secured $8.5 million initial seed funding to start building the real DBOS. Engine Ventures and Construct Capital led the funding, along with Sinewave and GutBrain Ventures.
[…]
“The state that the operating system has to keep track of – memory, files, messages, and so on – is approximately linear to the resources you have got,” says Stonebraker. “So without me saying another word, keeping track of operating system state is a database problem not addressed by current operating system schedulers. Moreover, OLTP database performance has gone up dramatically, and that is why we thought instead of running the database system in user space on top of the operating system, why don’t we invert our thinking 180 degrees and run the operating system on top of the database, with all of the operating services are coded in SQL?”
[…]
FoundationDB is a blazingly fast NoSQL database, which means that it does support the ACID properties of a relational database but which does not offer full SQL compliance. (Stonebraker tells us that DBOS eventually will do that, which seems to imply the underlying database engine will change.) Right now, DBOS has been tested running across 1,000 cores running applications coded in TypeScript, but Stonebraker says there is no reason to believe that DBOS can’t scale across 1 million cores or more and support Java, Python, and other application languages as they are needed by customers.
[…]
thanks to the distributed database underpinnings of its kernel, it can do things that a Linux kernel just cannot do. And it can do all of these things without a full Linux OS and without Kubernetes containing things, and without having to bolt Postgres onto the side of the database middleware.
One is provide reliable execution, which means that is a program running atop DBOS is ever interrupted, it starts where it left off and does not have to redo its work from some arbitrary earlier point and does not crash and have to start from the beginning. And because every little bit of the state of the operating system – and therefore the applications that run atop it – is preserved, you can go backwards in time in the system and restart the operating system if it experiences some sort of anomaly, such as a bad piece of application software running or a hack attack. You can use this “time travel” feature, as Stonebraker calls it, to reproduce what are called heisenbugs – ones that are very hard to reproduce precisely because there is no shared state in the distributed Linux and Kubernetes environment and that are increasingly prevalent in a world of microservices.
[…]
One last thing. We know of operating systems that had an intimate relationship with a database, but this twist is actually a new one in that the operating system kernel/schedular is itself largely a database and services are created in database languages.
For example. IBM’s System/38 and AS/400 minicomputers had a relational database at the heart of the operating system and in fact the database was the only file system allowed on these machines from 1978 through 1996, at which time IBM pulled the database out of the operating system and added in the OS/2 Parallel File System to give a POSIX-compliant, ASCII formatted file system for the AS/400. (Which is known today as the IBM i proprietary operating system.) The Pick operating system similarly had an integrated database, too. And of course, the “Longhorn” version of Windows Server 2008 was supposed to have the WinFS file system, which was based on a relational database, embedded in it, but that effort was spiked a decade and a half ago.
Which brings us to that one last thing: There is no reason why DBOS cannot complete the circle and not only have a database as an operating system kernel, but also have a relational database as the file system for applications.
Source: The Cloud Outgrows Linux, And Sparks A New Operating System
Robin Edgar
Organisational Structures | Technology and Science | Military, IT and Lifestyle consultancy | Social, Broadcast & Cross Media | Flying aircraft