Post-SNO BOM: Scaling to Three Nodes

The validation BOM covered one node and the network infrastructure — enough to prove the design works. This post covers what happened next: scaling to three nodes for Phase 1, every price change and hardware swap along the way, and the total cost of the cluster so far.

Validation exists to catch surprises before they get expensive. It caught several.

What changed from the validation BOM

NICs: three purchases, two returns

The NIC selection was the most expensive lesson. Covered in detail in the hardware validation post, but the cost impact belongs here:

No money lost (all returned), but about a month of shipping and waiting. The CX4121C ended up being the cheapest of the three — and the only one that works.

For Nodes 5-6, bought two more CX4121C at the same price. No surprises the second time.

RAM: buy now or pay more later

Node 4 came from the seller with 64 GB (2 x 32 GB DDR4-3200). Enough for validation. But DDR4 prices were climbing — global supply situation pushing modules from ~400 PLN to 850 PLN per 32 GB stick in a few months.

Decision: buy all RAM now. Upgraded Node 4 to 128 GB (added 2 x 32 GB at 850 PLN each), and ordered Nodes 5-6 with 128 GB from the start. Total RAM cost is higher than planned, but waiting would have been worse.

Boot SSD: Intel D3-S3610 price doubled

The D3-S3610 400 GB was 220 PLN for Node 4. By the time Nodes 5-6 were ordered, the same drive was going for ~440 PLN on Allegro. Not worth it at that price.

Replacement: Toshiba THNSF8400CCSE 400 GB — enterprise SATA SSD, 199 PLN each. Same enterprise endurance class, lower price than the now-inflated D3-S3610.

Both are enterprise MLC/eMLC SSDs with high write endurance — what matters for etcd.

NVMe: PNY CS1030 stays (for now)

The PNY CS1030 is DRAM-less and consumer-grade. Not ideal for a Ceph OSD. But after validation, the decision is: it works, Ceph isn’t deployed yet, and enterprise NVMe can wait. No point spending on enterprise drives before Rook-Ceph is running and I can measure actual I/O patterns.

All three nodes use the seller-provided NVMe for now. Enterprise replacement comes when it matters.

HDD: still deferred

HDDs are the most expensive per-node component and not needed until Rook-Ceph is deployed with a slow pool. Buying 3 x 20 TB drives to sit on a shelf while I’m still bootstrapping the cluster doesn’t make sense. Purchased after the platform is running.

Nodes 5-6: the Phase 1 purchase

Same seller on Allegro, same Dell OptiPlex 5090 SFF chassis. Different config — 128 GB RAM from the start and a 480 GB SSD included (used as the seller-provided NVMe, same role as Node 4’s PNY CS1030).

One new DAC cable — the validation setup already had enough for Node 4 and the trunk, but Nodes 5-6 need one more connection to the CRS317.

Node 4 upgrade

Node 4 goes from 64 GB to 128 GB. All three nodes now identical at 128 GB.

Complete Phase 1 BOM

Everything purchased across validation and Phase 1, in one table:

What’s not in the total

• HDDs — purchased after Rook-Ceph is deployed. Estimated 3 x ~1,500-2,500 PLN depending on capacity (16-24 TB). Adds ~4,500-7,500 PLN.
• Enterprise NVMe — replaces the seller-provided drives when Ceph performance data justifies it. Estimated 3 x ~200-400 PLN.
• Returned NICs — X710 (299 PLN) and CX312A (289.90 PLN) were refunded. No net cost, just time.

Cost breakdown

Let’s be honest — a homelab isn’t a cheap hobby. ~19,600 PLN (~4,560 EUR) for three SFF desktops, networking gear, and enterprise boot SSDs. And that’s before HDDs and enterprise NVMe — the full build will land somewhere around 25,000-28,000 PLN (~5,800-6,500 EUR). That’s real money, and it’s worth being transparent about it.

But as I wrote in the first post of this series: this isn’t Plex in a container. It’s infrastructure that mirrors what I work with professionally — from physical cabling to storage class definitions. The skills transfer directly. A training course covering the same material costs more and teaches less.

72% on compute is expected — the Dell chassis with i7-11700 and 128 GB RAM is the most expensive component. The network infrastructure (router + switch + modules + cables) is 20% but it’s a one-time cost that serves Phase 1 and Phase 2 without changes.

Lessons about buying used hardware

1. Prices change fast. The D3-S3610 doubled in a few months. DDR4 32 GB modules went from ~400 to 850 PLN. If you know you need something, buy it when the price is right, not when you need it.

2. Same seller, identical units. All three chassis from the same Allegro seller. Identical BIOS versions, identical NVMe slots, identical PCIe behavior. Mixing sellers means mixing hardware revisions and potential compatibility surprises.

3. Enterprise SSDs are commodity. The Toshiba THNSF8400CCSE was cheaper than the Intel D3-S3610 and serves the same purpose. Don’t get attached to a specific model — check what’s available at the right price.

4. NIC validation saves real money. Three NICs tested, two returned. If I’d bought five X710s upfront, that’s ~1,500 PLN in returns and weeks of delay. Buy one, test, then bulk order.

5. Defer what you can. HDDs and enterprise NVMe can wait. The cluster runs without them — Rook-Ceph isn’t deployed until Phase 1 is stable. No point tying up capital in components that sit idle.

What’s still coming

The cluster as it stands — three nodes with 128 GB RAM, 10G networking, enterprise boot SSDs, and validated NICs — is ready for Phase 1 deployment. Total investment so far: ~19,600 PLN (~4,560 EUR). With HDDs and enterprise NVMe still ahead, the full Phase 1 build will likely land around ~25,000-28,000 PLN (~5,800-6,500 EUR).

Not cheap. But cheaper than the knowledge it builds.