Replica and Shard setup in mongodb

Understanding replication and sharding in the context of databases (especially with MongoDB) is crucial for designing scalable and high-availability systems.

Replication

Replication involves creating copies of your data across multiple servers to ensure redundancy and availability. MongoDB achieves this using a replica set, which is a group of mongod instances that maintain the same data set. Replica sets provide high availability and data redundancy.

Primary: The primary node receives all write operations.
Secondary: Secondary nodes replicate the data from the primary. They can be used for read operations (depending on the read preference).
Arbiter: An arbiter node does not store data but participates in the election process to choose a new primary.

Sharding

Sharding is a method for distributing data across multiple machines. It is used to support deployments with large data sets and high throughput operations.

Shard: Each shard holds a subset of the sharded data. Shards can be replica sets or single mongod instances.
Config Servers: Store metadata and configuration settings for the cluster.
Mongos: Acts as a query router, providing an interface between client applications and the sharded cluster.

Command Prompt Implementation

Setting Up a Replica Set

Start MongoDB Instances: Start three MongoDB instances on different ports. For simplicity, we'll run them on a single machine.

bash

D:\Software\MongoDb\mongodb\bin>

mkdir \data\rs11 \data\rs12 \data\rs13 \data\rs21 \data\rs22 \data\rs23 \data\rs31 \data\rs32 \data\rs33 \data\cfgsvr1 \data\cfgsvr2 \data\cfgsvr3 \data\cfg1 \data\cfg2 \data\cfg3

mongod --configsvr --logpath D:\data\cfgsvr1\1.log --dbpath D:\data\cfgsvr1 --port 27018

mongod --replSet envest --logpath D:\data\rs11\1.log --dbpath D:\data\rs11 --port 28017 --smallfiles --oplogSize 64

mongod --replSet envest --logpath D:\data\rs12\1.log --dbpath D:\data\rs12 --port 28018 --smallfiles --oplogSize 64

mongod --replSet envest --logpath D:\data\rs13\1.log --dbpath D:\data\rs13 --port 28019 --smallfiles --oplogSize 64

Initialize the Replica Set: Connect to one of the instances and initialize the replica set.

bash

mongo --port 28017

In the MongoDB shell:

bash

rs.initiate()

//enter key button press

rs.add("IN-L1155:28018")

rs.add("IN-L1155:28019")

rs.config()

Check Replica Set Status:

bash

rs.status()

5. Create database name:

bash

use KcmStreamService

6. Give user with grand privileges on database

bash

db.createUser({user:"app",pwd:"app@pwd",roles:[{role:"readWrite",db:"KcmStreamService"}]});

7. Create collection or table

bash

>db.createCollection("KCM_COMPANY_PERMISSION_DATA")

>db.createCollection("KCM_USER_PERMISSION_DATA")

>db.createCollection("KCM_PERMISSION_DATA_USAGE")

>db.createCollection("KCM_PERMISSION_DATA_AUDIT")

>db.createCollection("KCM_COMPANY_STATS")

>db.createCollection("KCM_COMPANY_STATS_HIST")

>db.createCollection("KCM_USER_STATS")

>db.createCollection("KCM_USER_STATS_HIST")

>db.createCollection("KCM_COMPANY_CONFIG")

>db.createCollection("KCM_COMPANY_CONFIG_AUDIT")

>db.createCollection("KCM_ADMIN_AUTH_CONFIG")

>db.createCollection("KCM_COMPANY_LOGIN_STATS")

>db.createCollection("KCM_COMPANY_EVENT_STATS")

8. Execute the operations:

>show collections

> use admin

switched to db admin

> db.system.users.find()

>use KcmStreamService

>db.KCM_COMPANY_EVENT_STATS.find()

Setting Up Sharding

Start MongoDB Instances for config server: Start three MongoDB instances on different ports. For simplicity, we'll run them on a single machine.

bash

D:\Software\MongoDb\mongodb\bin>

mkdir \data\rs11 \data\rs12 \data\rs13 \data\rs21 \data\rs22 \data\rs23 \data\rs31 \data\rs32 \data\rs33 \data\cfgsvr1 \data\cfgsvr2 \data\cfgsvr3 \data\cfg1 \data\cfg2 \data\cfg3

mongod --configsvr --logpath D:\data\cfgsvr1\1.log --dbpath D:\data\cfgsvr1 --port 27018

mongod --configsvr --logpath D:\data\cfgsvr2\1.log --dbpath D:\data\cfgsvr2 --port 27019

mongod --configsvr --logpath D:\data\cfgsvr3\1.log --dbpath D:\data\cfgsvr3 --port 27020

2. Start Shard Servers: Start the shard servers, which can be either standalone instances or replica sets.

bash

mongod --shardsvr --replSet envest --logpath \data\rs11\1.log --dbpath \data\rs11 --port 28017 --smallfiles --oplogSize 64

mongod --shardsvr --replSet envest --logpath \data\rs12\1.log --dbpath \data\rs12 --port 28018 --smallfiles --oplogSize 64

mongod --shardsvr --replSet envest --logpath \data\rs13\1.log --dbpath \data\rs13 --port 28019 --smallfiles --oplogSize 64

mongod --shardsvr --replSet net --logpath \data\rs21\1.log --dbpath \data\rs21 --port 28020 --smallfiles --oplogSize 64

mongod --shardsvr --replSet net --logpath \data\rs22\1.log --dbpath \data\rs22 --port 28021 --smallfiles --oplogSize 64

mongod --shardsvr --replSet net --logpath \data\rs23\1.log --dbpath \data\rs23 --port 28022 --smallfiles --oplogSize 64

mongod --shardsvr --replSet aniket --logpath \data\rs31\1.log --dbpath \data\rs31 --port 28023 --smallfiles --oplogSize 64

mongod --shardsvr --replSet aniket --logpath \data\rs32\1.log --dbpath \data\rs32 --port 28024 --smallfiles --oplogSize 64

mongod --shardsvr --replSet aniket --logpath \data\rs33\1.log --dbpath \data\rs33 --port 28025 --smallfiles --oplogSize 64

Initialize the Replica Set for first node: Connect to one of the instances and initialize the replica set.

bash

mongo --port 28017

In the MongoDB shell for first node:

bash

rs.initiate()

//enter key button press

rs.add("IN-L1155:28018")

rs.add("IN-L1155:28019")

rs.config()

rs.status()

5. Initialize the Replica Set for second node: Connect to one of the instances and initialize the replica set.

bash

mongo --port 28020

In the MongoDB shell for second node:

bash

rs.initiate()

//enter key button press

rs.add("IN-L1155:28021")

rs.add("IN-L1155:28022")

rs.config()

rs.status()

7. Initialize the Replica Set for third node: Connect to one of the instances and initialize the replica set.

bash

mongo --port 28023

In the MongoDB shell for third node:

bash

rs.initiate()

//enter key button press

rs.add("IN-L1155:28024")

rs.add("IN-L1155:28025")

rs.config()

rs.status()

9. Start the Mongos Router:

Start the mongos instance and connect it to the config servers.

bash

mongos --configdb IN-L1155:27018,IN-L1155:27019,IN-L1155:27020 --chunkSize 10

10. Add Shard to Cluster:

Connect to the mongos instance and add the shard.

bash

mongo

sh.addShard("mandal/IN-L1155:28017")

sh.addShard("kartik/IN-L1155:28020")

sh.addShard("aniket/IN-L1155:28023")

Check Shard Set Status:

bash

rs.status()

12. Create database name:

bash

use KcmStreamService

13. Give user with grand privileges on database

bash

db.createUser({user:"app",pwd:"app@pwd",roles:[{role:"readWrite",db:"KcmStreamService"}]});

14. Create collection or table

bash

>db.createCollection("KCM_COMPANY_PERMISSION_DATA")

>db.createCollection("KCM_USER_PERMISSION_DATA")

>db.createCollection("KCM_PERMISSION_DATA_USAGE")

>db.createCollection("KCM_PERMISSION_DATA_AUDIT")

>db.createCollection("KCM_COMPANY_STATS")

>db.createCollection("KCM_COMPANY_STATS_HIST")

>db.createCollection("KCM_USER_STATS")

>db.createCollection("KCM_USER_STATS_HIST")

>db.createCollection("KCM_COMPANY_CONFIG")

>db.createCollection("KCM_COMPANY_CONFIG_AUDIT")

>db.createCollection("KCM_ADMIN_AUTH_CONFIG")

>db.createCollection("KCM_COMPANY_LOGIN_STATS")

>db.createCollection("KCM_COMPANY_EVENT_STATS")

15. Execute the operations:

>show collections

> use admin

switched to db admin

> db.system.users.find()

>use KcmStreamService

>db.KCM_COMPANY_EVENT_STATS.find()

Summary

Replication ensures high availability and redundancy by copying data across multiple nodes.
Sharding allows horizontal scaling by distributing data across multiple shards.
Commands:

Replication: Use rs.initiate(), rs.add(), and rs.status().
Sharding: Use mongod for config servers and shards, mongos for the query router, and sh.addShard(), sh.enableSharding(), and sh.shardCollection() for managing shards.

By following these steps, you can set up a robust MongoDB environment that ensures high availability, redundancy, and scalability.

For More DSA Related information, visit

Ø Bench mark of compiler using Ackerman function

Ø Find the Missing Number

Ø To check if the rows of a matrix are circularly identical in Java

Ø how to check loop in array

Ø 100 door puzzle programs

Ø Frequency Weaving Logic & Spiral Printing of a Rectangle

Ø Zig Zag Matrix print multiple way

Ø Gready Algorithm’s or knapsack algorithms

Ø understanding recursive method for binary tree

Ø Dynamic Programming: Max Square Sub-matrix of 1s in a Matrix