New CEPH Cluster

[u/Frequent-Wear-3948]

Hi guys,

We're considering a new cluster for CEPH, nothing too fancy - erasure coded setup 8+2, with 10 disks and 2x NVMe, 6 nodes for good resiliency. Now we're considering splitting the available nvme space as Bluestore space and rest for the metadata caching when we pull S3 objects from Amazon.

How do we decide the size needed for Bluestore ? Is there any rule of thumb there ?

Here is the h/w config we're planning on using

https://gpumachines.com/shared/gigabyte-20r264-s30-aav1-98b877

Any suggestions are most welcome ! Appreciate all your help people from trenches :)

[u/mattk404]

Unless you're doing very not recommended things with failure domains you're 6 nodes can't do 8+2 EC. You need 10 nodes.

[u/Frequent-Wear-3948 → u/mattk404]

That's a fair point ! We were under assumption 10 disks, 10 OSDs, 8+2 but didn't think about main principle of failure domains. Guess 4+2 it is then.

[u/gaidzak → u/Frequent-Wear-3948]

11 nodes; K+M+1 unless changing min_size to K

[u/Smooth_Flounder_9854 → u/mattk404]

You can‘t do 8+2, but you can do other aggressive ECs if you want and understand the trade offs. What’s your goal on performance? Performance is one of the major trade offs you have when you start to become more aggressive on the ec.

[u/i_love_hotsauce → u/Smooth_Flounder_9854]

I mean, he can, but should never do this - by setting the crush failure domain to osd instead of host, he can do 8+2

[u/edijo → u/i_love_hotsauce]

he can, but should never do this - by setting the crush failure domain to osd

Why "never do this"? Disks fail much more often than (server) nodes. Also, you may have unequal nodes.

I understand that you can minimize the risk of having many PG on a failed node by using MSR with crush-osds-per-failure-domain=2

[u/joeyx22lm → u/edijo]

Not an acceptable risk for any production I've ever been involved with.

[u/edijo → u/joeyx22lm]

Not every storage needs 99.999% uptime, some are fine at 95's. Especially the "economy" ones, using EC. You can move disks to another box(es) in the very rare case when it is necessary. Even cheap refurbished servers work for decades and have hot-swap disk bays and hot-swap power supplies.

Sure, if you buy servers specifically for the cluster instead of "using what you get", you go for identical machines and node redundancy.

[u/joeyx22lm → u/edijo]

I am very well versed in using older hardware, using what you've got.

Also I wouldn't rely on "HDDs die more frequently than nodes do", I know the statistics and I do not run a data center but I have had nodes go down due to all kinds of unexpected issues.

OSD-level fault domain is still not an acceptable risk for any production I've ever been involved with. A single node going down, taking out multiple storage pools, would keep me up at night.

[u/edijo → u/joeyx22lm]

But if you have EC n+2 (or +3) and allow only 2 osd on a box, it doesn't go "down" much more than with node redundancy.

[u/Smooth_Flounder_9854 → u/i_love_hotsauce]

It depends on the usecase. With 6 hosts they can have a different failure domain than host with a higher EC. For example, with a 12+3 they can have a failure domain of 1 host + 1 OSD as the expense of higher IOPS for writes/reads. This would also require custom crush rules.

[u/Micketeer]

6 nodes for resiliancy but you want 8+2? That does not add up. 

General recommendation for DB size is 4% of disk. 

[u/SystEng]

"erasure coded setup 8+2, with 10 disks [...] 6 nodes for good resiliency"

[...] "Guess 4+2 it is then."

The failure domain could be the OSD instead of the server, but k=8,m=2 with 10 OSDs (and especially 10 HDD OSDs) seems to me to have some huge disadvantages.

In an ideal case the number of (failure domains and) servers and OSDs should be rather larger than the width of the EC set/replica set.

10 OSDs in 6 servers with k=4,m=2 is definitely better but consider this complication:

• With 10 OSDs and 6 servers you can have 4 servers with 2 OSDs and 2 servers with 1 OSDs (or worse).

• k=4,m=2 with 6 servers requires that all objects be chunked across 6 servers.

• What happens when either OSD in the 2 servers that only have 1 OSD become full?

I would also guess (the configuration link does not work for me) the HDDs have large capacity too to "optimize" cost "effectiveness" and I reckon that Ceph requires a lot of IOPS-per-TB but many people know better.

[u/SystEng → u/SystEng]

10 OSDs in 6 servers with k=4,m=2 is definitely better

If that is not clear enough: that is a massive euphemism. But from an earlier discussion it seems that configurations are members of the Ceph community and can easily take offense. :-) I liked the previous massive euphemism "tight during recovery windows". :-)

[u/SystEng → u/SystEng]

"I would also guess (the configuration link does not work for me)"

That link now works for me and it turns out that each of the six servers has 10x 24TB HDD OSDs, not 10 among the 6 servers.

"Ceph requires a lot of IOPS-per-TB but many people know better." "I liked the previous massive euphemism "tight during recovery windows". :-)"

6 servers with 32GiB each with 10x 24TB SATA HDDs each with EC k=4,m=2 and 2x 4TB SSDs shared among those 10 HDDs for DB+WAL and metadata is a bravely optimistic configuration. A pattern that is quite popular.

[u/TheSov]

i see you addressed the EC stuff, so ill get right to the crux of the issue, you dont have enough ram. this cluster will suck.

[u/Frequent-Wear-3948 → u/TheSov]

oh that's interesting point. So you think 128GB of RAM per node is not enough ? Does this have any relation to HDD capacity per node ? Thanks for the tip so far, much appreciated

[u/TheSov → u/Frequent-Wear-3948]

spinning disks require big caches for good speed. unless you are fine with large latencies. i assume u are using the 2 ssd's mirrored for the WALs/ROCKDBs? that covers your write. you are dealing with objects in EC that uses more memory than you think, 128GB maybe ok for your setup, i dont know your actual use case. but from my experience i would lean toward more ram and over engineer than to be insufficient, also ceph is scale out not scale up, if i was also you i wouldnt fill every disk on the chassis, and do half and get more chassis'. so if u ever have to expand u just buy disks instead of having to source chassis as well.

[u/Frequent-Wear-3948 → u/TheSov]

tbh it's all about the budgets these days. Ram prices are an extortion so while I totally agree with you having more, not sure if we can make it this financial year lol :) Appreciate the feedback though, what I can do is to get i.e. 2x 64GB sticks each, so i have big room for expansion

[u/ihavenospeed]

coming from someone who dealt with rocksdb spillover on a multi-petabyte S3 cluster with billions of objects, I'd suggest you first assess what workload you want to optimize for.

As for the pulling from Amazon S3, you have to make an idea about the caching mechanism, I personally go for a separate nvme backed pool, but there are also other options.

As someone else already said, 6 nodes for an 8+2 won't do the trick unfortunately, you could do it tough by choosing the osd failure domain but I highly suggest you not to do so; even with a 4+2 that would suit your 6 host layout you'd be tight during recovery windows.

Personal take? Claude and GPT is your best friend when making such ideas.

[u/Frequent-Wear-3948 → u/ihavenospeed]

Much appreciated ! Thank you for your suggestions here. So possibly add 2 more NVMe via m.2 would do the trick then change to 4+2, but as you said it would be tight.

I know I can ask all these questions to claude or gpt, but that's the thing - sometimes it's better to verify with ppl who actually have experience with larger deployments rather than listening to GPT later saying - Opps sorry you're correct ;)

Thanks again

[u/ihavenospeed → u/Frequent-Wear-3948]

fair point on the gpts thing, I use it for brainstorming but I've fed it my infra over time so it's tuned to my context.

on the nvme, adding more gives you more room for rocksdb and less blast radius when one dies, but it won't help the EC math in any form. thats purely host count.

My rule of thumb is a fixed ratio between block devices and db devicse, then you split each nvme into fixed LVs so every osd gets it's own dedicated LV for rocksdb. proportional to your layout rather then a fixed percentage.

And on the EC, if you can stretch the budget get more hosts before you buy anything else, with exactly 6 and 4+2 a host going down means the pgs sit degraded with nowhere to rebuild onto, since the other 5 already hold a shard so I say 7 gets you self healing, 8 gives you margin.

ps. if you can avoid m.2 use u.2/3

[u/_--James--_ → u/Frequent-Wear-3948]

If you are relying on AI for answers, while you got the right answers in this sub by all rights. I have to suggest reaching out to a SI partner that handles Ceph specifically. I would suggest Croit personally. Reach out to them, get a SE engagement going and run through your needs and wants, and let them guide you.

You do not have enough hardware to support your build, that is the bottom line. You also probably do not have enough bottom end IO to support your throughput load, but that would need to be properly benchmarked, stated, and mathed to really know. You need to start at your current state IO + sizing requirements and then build down into the hardware, not bottom up.

[u/ihavenospeed → u/_--James--_]

I think your reply was meant for OP; but if that was meant for me, I rely on AI that I specifically trained to speed-up troubleshooting or to brainstorm ideas when needed, but i've been running this stuff for some time so I can catch when it invents stuff. For someone with no hands-on experience, you can try your luck on a lab, not on production.

[u/_--James--_ → u/ihavenospeed]

Yes, i was talking to the OP. Looking at the opening Post and the OP's reply's, AI is not enough for them to build this right. This is why SI's exist and I stand by what i said. AI is not bad, its just not strong enough for the OP in their current state.

Such as if the OP went to Claud/GPT and said "I have 6 servers I want EC8+2" and AI did not kick back "This is not possible with out extreme concern" the AI failed that model, which it sounds like what happened.