User:Iamamz3/okvs.dev
Ordered Key-Value Store Development[edit | edit source]
Introduction[edit | edit source]
While there is no shortage of database systems, there is none that meets all those requirements:
- Can represent, and query poly-structured data:
- relational / graphical
- recursive / hierarchical
- time / spatial / geometric
- textual
- Support efficient:
- pagination
- versioning
- horizontal scaling
- down scaling
- Support ACID transactions across objects
- Full control from a high level general purpose programming language
We could argue indefinitely that one or more of those requirements are unnecessary, overkill, and that YAGNI. We could argue that by relaxing a few of the requirements, a particular software or more likely a set of software can come close. We could argue endlessly that building yet-another-database is NIH syndrome, wheel re-invention that curse the software industry with fragmentation and fatigue. We could invoke UNIX philosophy, enterprise software architectures, experiences, know-how, failed patterns, thousands of errors fixed, production use, decades of good services, and big communities, we could mention mainstream legacy.
Standing on the shoulders of giants, this book is merely trying to bring the attention of the community to a growing trend in the software engineering field, and demonstrate what you can achieve with an Ordered Key-Value Store (OKVS). Most of what is described in this document already exists in the wild hidden in Software as a Service (SaaS) products, but also sometime readily accessible in software programmed with C++, Java, JavaScript, Scala, C#, Go, Rust, Python, Clojure, and Scheme.
You have already used an Ordered Key-Value Store. And certainly, you know about the components upon which it is built. As matter of fact, if you are a software engineer, you will not learn new algorithms, or new data-structures, instead you will discover how to use your existing knowledge differently to mimic what you were doing before with better alternatives that are more economical.
Ordered Key Value Stores are useful, usable and used.
A question holds: Will Ordered Key-Value Store remain a tool for the experts? Maybe it will become more widely accepted in the tool-belt of software developers? One way to find out the response to that question is to answer another question: Are all needs of all domains covered by the abstractions provided in this document, or there is still some specific needs that arise in some domains that require surgical intervention?
Continuation[edit | edit source]
If LISP family of language are programmable programming languages thanks to runtime compilation, and syntax transformers (macros), then Ordered Key-Value Stores are programmable storage systems. OKVS can cover much representation grounds, thanks to extensions, higher level abstractions, you can represent whatever you want or whatever you need. Hence like LISP, you need to think in terms of Domain Specific Language, and answer questions such as: What problem do I need to solve? In this research, you need to figure not only the most handy representation, a representation that is more specific than a graph. Graph or more likely property graph are handy, after all everything is graph, we will see that sometimes more performance alternatives exists. To sort this out, you also need to plan read and write workloads, and in particular have some rough sketch of what can happen online, in sub-second latency, and at the other side of the continuum things that do not require short latency such as background tasks. Another aspect that you need to take into account is space, both in-memory space and on-disk space. OKVS will help you with that, while making yourself the economy of learning yet-another-programming-paradigm every time you need to solve a new problem. OKVS builds upon what you know, and you build the tools that fit your problem almost (?) perfectly, and scale to large number of problems.
This book will try to give you practical experience building systems that use OKVS.
In the first chapter, a language called Fantastic Kernel, that is a LISP forked from Scheme dialect, will be introduced. Fear not. Kernel was chosen because of its small while powerful surface, as long as you are not entitled against parentheses, you will be able to learn that language in little time. Fantastic will be present through the book to illustrate the concepts. To supplement Fantastic, and with the same goal, to ease the reader in their apprentice path to grasp seemingly new concepts, the essential programming interface of an OKVS, dubbed OKDB, will be annotated with formal notation. The first chapter comes with some discussion, and illustrate each interface element with small recipes. It wants to cover as much grounds of the essential aspects of OKVS so that you can think on your own, so that you can build a castle in the sky without going through the rest of the book. The goal of the first chapter is to present all the pieces of the puzzle, so that you build your own things, your own way, letting your imagination truly flow.
The rest of the book is goal-driven and project-driven. Each problem will be solved with a specific extension or set of extensions. The chapters are named after the project problem, not the extensions, so that you can yourself make the exercise to solve (even as a mind play) the problem with or without an OKVS. It is encouraged to think about the solutions before reading the book.
Chapters[edit | edit source]
| # | Title | Description | % |
|---|---|---|---|
| 1 | Fantastic Kernel | Introduction to Fantastic Kernel programming language | |
| 2 | OKDB | The interface to program an OKVS Database called OKDB | |
| short.it | Give an URL, take a short URL | ||
| FollowBlogBlogGood | A database for a simple blog | ||
| Oh My Palette | A collection of color palettes | ||
| Linkify | Fuzzy entity name resolution | ||
| Last Hope | An audit-trail for web navigation | ||
| Passtech | A code snippet database with a search engine | ||
| Around | A crowd sourced time and geographical knowledge base | ||
| Jungle | A realtime database for cooperation | ||
| Word Dog | A dictionary for friends of words | ||
| TootDB | A database of fediverse toots | ||
| Razlebol | An structured log aggregator |
OKDB & Extensions[edit | edit source]
| # | Title | Description | % |
|---|---|---|---|
| okdb | OKVS Database | ||
| nstore2 | Generic Tuple Store | ||
| vnstore2 | Versioned Generic Tuple Store | ||
| bitstore | Bi-Temporal Store | ||
| ftstore | Full-Text Store | ||
| fzstore | Fuzzy Store | ||
| xzstore | XZ-ordered curve for objects with spatial extent | ||
| eavstore | Entity-Attribute-Value Store | ||
| rstore | Record Store | ||
| gstore | Property Graph Store |
