What Is Sharding in MongoDB And How it Works?

Sharing is an algorithm for dispersing information across multiple computers, allowing horizontal scaling (instead of vertical scaling).

Vertical scaling is the process of improving the performance of a single device or single server with an enhanced CPU, increased RAM, or an increase in storage capacity.

If physical limitations weren’t an issue, vertical scaling is preferred because of its ease of use. In the real world, no single machine can handle the demands of modern workloads.

Read Also: Read our detailed guide on Stateful vs. Stateless firewalls: which one is best for you.

Horizontal scaling is the solution. Also called scale-out, it’s the act of adding nodes that divide the data set and load. Horizontal scaling allows near-infinite scaling to handle vast volumes of data and heavy workloads.

How Does Sharding Work?

Shards, Routers, and Config Servers

Within MongoDB it is a Sharded-Cluster is composed of many:

  • Shards
  • routers
  • Replica sets for the config server

The term “shard” refers to a replica set that is one subset of a collection (also called chunks). A replica set is made up of one or more mongod nodes that share the same data set.

Read Also: Read our detailed guide on best Headless CMS for publishers.

Replica sets are highly available. Sharding lets us partition the data over multiple replica sets and consequently increase the resources of databases.

router(or mongos)acts as a query router for applications that use clients that handle writes and read operations. The router stores the metadata of the config server to ensure that it can guide requests to the correct Shard or Shards.

The config servers keep the entire metadata of a sharded group. The metadata includes the state and organization of every component and data in the sharded cluster.

Read Also: Read our detailed guide on Nessus vs OpenVAS: which one is best for you?

In its most basic configuration, using one shard, a cluster that is sharded could be like this:sharded-mongodb-cluster-testingSource: https://www.mongodb.com/basics/sharding

This configuration isn’t yet ready for production, but it’s suitable in a test environment. For production environments, we strongly suggest using multiple shards/mongos routers:

sharded-mongodb-cluster-production-environmentSource: https://www.mongodb.com/basics/sharding

What are the Benefits of Sharding?

Sharding lets you scale the database you use to manage increasing demands to an almost infinite extent. It is accomplished by increasing the amount of data that can be read and writtenstorage capacity, and your database’s availability. Let’s examine each of them in more in detail:

More efficient read and write speed: by distributing the data set over several shards, both capacities for write and read operations can be increased if these operations are limited to one shard.

It is contingent on the way you define the key for shards in each of the sharded collections.

Storage capacity increased: Additionally, in growing the number of shards, you can increase the total capacity of storage. It will allow for near-infinite scaling.

 High availability: Because every shard is a replica set, each piece that is stored is replicated. Because all data are distributed when one shard fails, the entire database is still functional to reading and writing from all the remaining shards.

Shard Keys and Chunks

Shard Keys

Sharing a collection is a process that requires the user to choose the method of distribution of the collection’s contents among different shards. MongoDB makes use of its shard keys to spread the collection’s content across shards. It is done by assigning a set of values to the shard.

Read Also: Read our detailed guide on rust building server. 

Fields within every document determine shard keys. The fields’ values will pick the shard on which the document will be placed according to the shard ranges and the number of chunks. The information is then stored in the replica of the config server set.

mongodb-shard-keys-and-chunks

https://www.mongodb.com/basics/sharding

Every shard key must have an index or auto-build as part of the sharding commands if the collection is incomplete and the necessary index isn’t present.

Read Also: Read our detailed guide on Kubernetes on premise. 

The key that shards are stored in can directly affect the cluster’s performance and can cause problems with your application’s access patterns. Utilize the following information to determine the correct key for your shard.

Chunks

Chunks are the logical categories of documents that are moved across shards according to the sharding range mappings at a particular moment in time. Each chunk comprises an inclusive lower range and an exclusive upper range based on the key for shards.

When a chunk reaches an exact size, or when the workload calls for it, the chunk is cut into two. The balanced component will automatically distribute the chunks between shards in order so that they remain as evenly as they can be.

Read Also: Read our detailed guide on best space engineers server hosting platform for you. 

How to Implement Sharding

The most cost-effective,  most convenient, and  easiest way to manage and deploy a sharded cluster is through MongoDB Atlas, the Database-as-a-Service that clarifies sharded cluster implementation.

All you have to be able to do is to turn on:

implementing-sharding-mongodb

https://www.mongodb.com/basics/sharding

Why Use MongoDB Atlas?

  • Install your sharded cluster within minutes using just a few clicks. It lets you split your collection across different regions and cloud providers and create one cluster to cover the global network.
  • Scale in and out with one command using unified monitoring tools and performance advisory.
  • It’s affordable since it costs only on your Shards infrastructure, not for configuration routers or servers.
  • It comes with built-in security for enterprise and encryption for access to your sharded cluster and auditing.
  • Updates to application that are no downtime and OS platform patching is done for you.

Read Also: Read our detailed guide on what is content filtering? definition, types, and best practices. 

If you are ready to built it by your own follow these steps. It would help if you replaced everything between < and > with the appropriate values for your configuration. We’ll set up the config server and shard(s) using replica sets with three nodes and a mongos server.

Let’s start.

Set Up the Config Server

Each replica set of config servers can include the number of mongod processes (up to 50) but with the following restrictions: no arbiters and no members with zero priority. In each case the above, you’ll need to begin the process using an option called the option –configsvr option. For instance:

mongod –configsvr –replSet <configReplSetName> –dbpath <path> –port 27019 –bind_ip localhost,<hostname(s)|ip address(es)>

Then, integrate with  solely one  process of the replica set.

mongo –host <hostname> –port 27019

Then execute rs.initiate() on only one of the members of the replica set:

rs.initiate(
{
_id: “<configReplSetName>”,
configsvr: true,
members: [
{ _id : 0, host : “<cfg1.example.net:27019>” },
{ _id : 1, host : “<cfg2.example.net:27019>” },
{ _id : 2, host : “<cfg3.example.net:27019>” }
]
}
)

Once the replica of the config server is in place and running, it is possible to create shards.

Read Also: Read our detailed guide on what is deskTop virtualization and how it works?

Set Up Shards

As previously mentioned, each shard can be described as a copy on its own. This procedure is identical to the config servers; however, using this –shardsvr option. Be sure to select an alternate name for  each replica sets.

mongod –shardsvr –replSet <shardReplicaSetNamereplSetname> –dbpath <path> –port 27018 –bind_ip <cluster hostname(s)|ip address(es)>

Then, integrate through solely one process of the replica set:

mongo –host <hostname> –port 27018

Then, run rs.initiate() on solely one  process of the replica set. Be sure to remove the — configsvr option:

rs.initiate(
{
_id: “<shardReplicaSetNamereplSetnamereplSetName>”,
members: [
{ _id : 0, host : “<shard-host1.example.net:2701827019>” },
{ _id : 1, host : “<shard-host2.example.net:2701827019>” },
{ _id : 2, host : “<shard-host3.example.net:2701827019>” }
]
}
)

Begin with mongos

Lastly, configure mongos and direct them to the replica servers of your config server:

mongos –configdb
<configReplSetName>/<cfg1.example.net:27019>,<cfg2.example.net:27019>,<cfg3.example.net:27019> –bind_ip localhost,<cluster hostname(s)|ip address(es)>

In a production environment, more than one mongo is required to prevent congestion and ensure availability. Typically, beginning with at least three mongo instances is a good idea.

Configure and Turn On Sharding for the Database

Connect your Mongos to:

mongo –host <hostname> –port 27017

Add your shards to the cluster. Make this one time per shard.

sh.addShard( “<shardReplicaSetName>/<shard-host1.example.net:27018>,<shard-host2.example.net:27018>,<shard-host3.example.net:27018>”)

Sharding can be enabled on your database:

sh.enableSharding(“<database>”)

Last but not least, make sure you shard your collection with your sh.shardCollection() technique. It can be done by the hashed sharding method, which spreads your data equally across shreds.

sh.shardCollection(“<database>.<collection>”, { <shard key field> : “hashed” , … } )

Or through range-based sharding, which permits the optimal distribution of data between shards according to key values of the shard. For specific sets of information, it may allow queries to be made across a variety of information more effective. The command is like this:

sh.shardCollection(“<database>.<collection>”, { <shard key field> : 1, … } )

That’s it! Now you’ve set up your first cluster that is sharded. Every interaction with your application from now on must be handled via routers (mongos instances) only.

Read Also: Read our detailed guide on best AngularJS Frameworks for rapid application development

Considerations Before Sharding

Here are some things to take into consideration before creating a sharded cluster:

There is no way of (easily) returning: The first and the least essential point issue is once the collection is shraded, there is no simple solution to “unshared” it. It is necessary to consider whether other limitations are required to the application’s present and future demands.

Broadcast operations: If the mongos router cannot determine whether a query is handled by one particular shard, it will broadcast that query to each shard within the cluster.

After each shard has replied to the question, it mongos can combine the responses of all shards and send an answer for the application that is running it. It can be extremely costly. It is essential to reduce this risk by ensuring proper setup of the key for sharding.

Administration complexity: The complexity of administration is that with the many servers and components to handle comes more complexity for the infrastructure aspect.

It can be lowered entirely with the help of MongoDB Atlas or, in part, orchestration tools like Our Kubernetes Operator or Ops Manager automation.

Multiple routers: Multiple routers can reduce the latency of client applications. However, it can also add load on the configuration servers.

Managed by chunks: The bit of data located in a shard gets placed into the form of a block. In time, chunks expand and are split at various rates, and this needs rebalancing. In the majority of cases it is done in automated ways by the balancer that is run by the config server.

However, in some instances, it may necessitate manual maintenance. Poor management of chunks could result in performance degradation and data skews across the fragments.

The MongoDB documentation includes a comprehensive list of limitations to be reviewed before deploying your first cluster sharded.

Final Thoughts

We’ve discussed the fundamentals of sharding and its advantages and drawbacks, and how to begin your first cluster that is sharded. For more details, look up the Sharding Manual or the methods for sharding in MongoDB’s documentation.

Read Also: Read our detailed guide on what is firewall? and what are advantages of host-based firewall and network-base firewall?

Frequently Asked Questions

What exactly is MongoDB Sharding?
Sharding is the act of partitioning data into many replicas (hosts) that allows horizontal scaling.

What is the process behind the MongoDB Sharding function?
The MongoDB database, a sharded-based cluster, is comprised of routers, shards, and configuration servers.

The data is distributed among the shards, and routers manage client requests while the configuration servers manage the shard state overall.

What are shard keys?
Fields within every document determine shard keys. The contents of those fields will evaluate the shard in which the document will depend on the shard ranges and chunks size.

The information is then stored in the replica of the config server set.

Every shard key requires an index or auto-build one in the sharding command if the collection is incomplete and the required index is not present.

The shard key can have an immediate impact on the cluster’s performance and may cause bottlenecks in your applications’ access patterns. 

Leave a comment