Sharding a Database Q: How frequently would we need to add machines to our pool ?
What is the benefit of having storage utilization less than 80%?
How did you arrive with the number 8?
Typically in production systems, if resource utilization goes beyond a certain level, it is hard to recover. Eg. if CPU goes to 80% plus, the system will slow down and will become unusable. Same with storage, remember there could be other actors in a distributed system including OS. So, it is always safe to assume that not every resource can be used upto it full capacity or 100%. 80% is a good safe bet but will really depend on growth rate and other factors.
Doesn't this depend on the number of machines we are using at that instant ? If I am using X number of machines at time t1, then the data on each machine would grow only by 1/X TB daily. This is assuming resharding happens every time a new machine is added to distribute the keys equally. So as we keep adding more machines over time the number X grows while as specified the data growth rate is 1 TB per day hence 1/X would keep getting smaller and the time period after which we would need to add a new machine should keep increasing.
Shouldn’t we also consider the requirement of data replication and add an additional machine (to replicate data) to the new machine which gets added every 8 days ?