r/rust • u/pixelooz • 1d ago
🛠️ project Finally finished building a relational database from scratch.
Hey everyone,
I'm a 4th year CS undergrad, I've been learning about databases since the last year as I eventually wanted to build one. After giving the build several attempts by sometimes building just the storage engine, or the front-end, etc., I finally did complete the build last week. I felt like its ready to share, so I'm making this post here.
Repo: https://github.com/pixelooz/hinder-db
Its called HinderDb. It's a cli only (for now) database with a custom slotted page B+Tree storage engine, buffer pool, secondary indexes, wal, volcano style execution pipeline, etc.
I wrote it in rust as I like writing rust and this would be another way to get better at it, plus the domain just seemed like a good fit.
It was mostly a learning project for me, but ended up better than I had orginally planned for, so I'm quite happy with that. I originally got interested in databases and stuff through Designing Data Intensive Applications book, and since then always wanted to build one. I had already given the storage engine a couple of tries, by reading some already existing material (Writing your own database type of books) and codebases, but this was the first time writing it complete, and its works quite nice.
I am keeping this as my main portfolio project as well since I'll be applying for jobs/intenships. So, if any working professionals can offer any feedback - something I should improve, any apparent flaws, or the next thing I should look into. I'll really appreciate it.
P.S: I have solid knowledge in backend as well and I'll be applying for on-site (in India) as well as global remote positions, so if you can offer any advice regarding navigating the job market rn, it'd be much appreciated.
25
u/Solus161 1d ago
This would put my pet sqlite clone project to shame. Not everyone dare to go for this kind of challenge, just for the sake of learning.
12
u/pixelooz 1d ago
Thanks man 😭. Actually I was already studying databases for about an year, so went all out as I eventually want to get into the same field. Mind sharing your work?
5
u/Solus161 1d ago
It’s just a Codecrafters challenge, nothing fancy, just to learn the concepts. I did it 4+ months into Rust. Building this kind of Redis clone is a strong point on CV. But yours is kinda serious, like 10x. I just invest my time into system programming (webdev is so boring lol) and surely follow your github.
2
u/pixelooz 1d ago
Lol, same sentiments, I also delved into systems engineering cause I got bored of web dev. Hopefully I can find a job in this market 🥹. You should check out the editor I made (quire), it's not as good architecturally but u might find it interesting.
8
u/Developer5702 1d ago
Databases are hard, kudos man!
Going to be using this repository as a reference when I build my own relational DB (kinda new to Rust and systems programming, so it will take me quite a while).
Congratulations for the project.
1
u/pixelooz 1d ago
Thanks a lot 😁. By all means, use it. I hope it provides enough educational value.
3
u/Doctor--STORM 1d ago
Once done with relational modelling - for analytics try using dbt or sqlmesh
1
9
u/jykke 1d ago
Good start. It has 31 tests. As a comparison, sqlite3 has 1194 tests (487k lines).
16
u/pixelooz 1d ago
Ik Ik, this is obviously a toy compared to real DBs. I just wanted to cover as much core surface area as I could and have a good foundation to build on as I learn more. Plus for other people who are interested in DBs to have a digestible reference to start from.
6
u/wintrmt3 1d ago
Sqlite has way more tests that are not open source, that's part of why it's very hard to port it.
3
2
u/Aggravating-Sign-136 1d ago
Omg!!! I just started getting into databases. 😭Don’t mind if i take inspiration.
1
2
u/cl3dson 1d ago
Congrats, inspiring see this willingness to learn stuff in this day and age of vibe coding, do you mind sharing the learning material you used ? any books or articles you used as reference?
2
u/pixelooz 1d ago
Hey! Thanks man 😄. For learning material the primary sources were cmu lectures, database internals, mkdb (both the go and rust one) - my page architecture was inspired from the go mkdb only. Designing Data Intensive Applications (even though not needed for the implementation) is good for the right mental model and much more. The sqlite documentations and related articles on that (my secondary indexes and wals are based on that model only). And last but not least, the plethora of write your own small database tutorials out there that I did on and off throughout the year for the familiarity of writing the database, like the csatck one, although they are very simple in their implementation, they are rather helpful.
2
2
1
u/Prestigious_Swift 1d ago
Wow, cool project. I started similar project about the same time, but I took a little different path doing a lot of tests, benchmarks, optimizations and other stuff. That's why it has much less functionality than yours, also it has much less comments. Check it out. And I’ve also just started my first year of a bachelor’s.
1
u/pixelooz 1d ago
Hey! Nice work man, you have good test coverage as well, I started with tests as well, but had to remove them as I had to adjust scope a little, and that broke most of the tests. Would love to see yours completed. May I ask which architecture are you following currently as reference?
1
u/Prestigious_Swift 1d ago
To be honest I dont follow any existing architecture (maybe on accident idk). I mean I take some concepts from others, but mostly I try to think of everything by myself.
1
u/pixelooz 1d ago
Damn! Do you have a lot of experience in database or systems engineering or some related field? Cause I mean, otherwise it would take so much time coming up with ideas without some existing reference.
1
u/Prestigious_Swift 20h ago
I mean I dont have WAL, joins, transactions, subqueries, planner. IMO they are like 80% of the difficulty of project.
1
u/NeedleworkerPale8110 1d ago
Big kudos man, that's an impressive. I'm curious about 2 things: 1. How much time did you invested weekly on it 2. If you'd had to do it again, what would you do differently?
1
u/pixelooz 1d ago edited 20h ago
Thanks man! 😁
weekly I did about 5-7hrs per day when I was writing the code. When I was just studying the theory and architecture, I would give about 3-4 hrs per day on it. Rarely I did both at the same time unless I was just referring to it.
I would study the query binder+planner+optimizer much more in detail than I did this time, it was much more intense than I thought it would be, and I would also define the scope of the implementation much more in detail, as the vagueness and - let's start will change later - cost me a lot of refactor; for a new impl I would maintain a balance.
1
u/No-Height-8011 1d ago
Hell yeah, bptree! I’m working on my own relational database right now, and I had to refactor my entire pager and tree architecture cause my types were so messed up. I ended up making pages a trait so that I could write universal methods for serializing and deserializing pages. Another thing, since a pageid is basically just an offset, my pageid actually wraps a nonzerousize. This way, I can serialize optional pageids (like the leaf node linked list pointers) as either a usize or None as 0.
Anyways I’m about to start working on the executor, I’ve basically got a minimal sql ast. Any tips? Our architectures are super similar (I copied postgresql)
2
u/pixelooz 1d ago edited 1d ago
Damn, it's peculiar how similar the architecture is😂. I did debate whether to make optional page IDs an option, but ended up settling with booleans only cause I wanted to avoid the repeated ceremony of pattern matching, thinking booleans will be just one line, but I think option would've been the same in hindsight.
Regarding the executor, honestly volcano iterator model was the easiest part in my implementation, the aggregates gave me a tough time but other than that they were easy, however query planner was the toughest to figure out for me. I would say study it more than anything for the execution pipeline, cmu lectures are enough for the mental model of executors I think. Best of luck!
Btw, mind sharing your repo? Would love to take a look.
1
u/No-Height-8011 12h ago
Absolutely!
https://github.com/kawiggles/database
The main branch is basically the 1.0/proof of concept. Check out the pager-refactor for the refactor I mentioned. The implementation is a mess in the first version. It’s definitely nowhere near as far along as your’s, but I’m getting there. Planning on doing a volcano iterator for execution.
2
u/pixelooz 11h ago
This is nice, also yeah, our impls are not that far. Also I can see that you are attempting btrees merges as well...dayum! That shits hard, I chose the easy path of deferring it to vaccum. If you get it right let me know, kinda wanna implement for mine.
The iterators will be fun.1
u/No-Height-8011 11h ago
Merges were a fucking pain. I think I got it right, I’m writing the test suite right now. This is actually my second implementation of merges; you can see the first in the main branch. It’s ridiculously ugly though
1
u/Ok_Plastic_3224 1d ago edited 1d ago
I love when people go for project like these, I was also building a database from scratch myself and I want to know how long does it take you to complete this project?
1
u/pixelooz 1d ago
Thanks! Btw, I don't understand the question, can you be a little more specific?
1
u/Ok_Plastic_3224 1d ago
I am asking from when were you working on this project
1
u/pixelooz 1d ago
Took me about a year on and off. Reading books, writing demos/small impls, tried different types of storage engines as well. There's a bitcask one in my repos. If I sat consistently it would take about 6 months I guess from starting.
1
u/Ok_Plastic_3224 1d ago
Greate since I am also working on my project and based on what you describe its definitely would take a year or two to even finish the half of it for me.
1
1
u/misplaced_my_pants 1d ago
You should add pipe syntax!
https://www.databricks.com/blog/sql-gets-easier-announcing-new-pipe-syntax
1
u/pixelooz 1d ago
Oh, this looks interesting, I wonder how the parser would be written for this. Do you by any chance have a reference I can look into for the parser?
2
1
u/IAMPowaaaaa 1d ago
what resources are you using to implement this stuff?
1
u/pixelooz 1d ago
For learning material the primary sources were cmu lectures, database internals, mkdb (both the go and rust one), The sqlite documentations and related articles on that (my secondary indexes and wals are based on that model only). And write your own small database tutorials like the csatck one, although they are very simple in their implementation, they are still helpful.
1
u/Grand-Challenge-558 17h ago
Rust developers are wild 😊 I love that people actually build stuff like this. Huge congrats!
1
0
u/GreenOrg 4h ago
Please mention what AI tools you used for that project? What is the ratio between your coding and AI vibe coding?
Note: I just checked this project with Copilot, and it said 90% of the project was generated by AI (code, commits, modules, and architecture).
1
53
u/SnooCalculations7417 1d ago
great work. you could take it a small step further and have it serve as the backend for a small portfolio website. pyo3 would be your friend here as you could make an adapter seemlessly for python, make a fastapi service to serve it, and have a simple SPA up and running in no time i think