From c7d61cf0d2947234a724f1f74783fd2282f84d57 Mon Sep 17 00:00:00 2001 From: Bob Grabar Date: Fri, 16 Aug 2013 16:12:13 -0400 Subject: [PATCH] DOCS-94 DOCS-95 migrate sharded cluster to new hardware --- bin/builddata/htaccess.yaml | 7 + source/includes/tip-hostnames.rst | 5 + .../toc-sharded-cluster-maintenance.yaml | 5 + source/tutorial.txt | 1 + .../tutorial/add-shards-to-shard-cluster.txt | 10 +- source/tutorial/deploy-shard-cluster.txt | 2 +- .../manage-sharded-cluster-balancer.txt | 85 +++---- ...onfig-servers-with-different-hostnames.txt | 38 ++- ...rate-config-servers-with-same-hostname.txt | 31 ++- ...igrate-sharded-cluster-to-new-hardware.txt | 232 ++++++++++++++++++ 10 files changed, 341 insertions(+), 75 deletions(-) create mode 100644 source/includes/tip-hostnames.rst create mode 100644 source/tutorial/migrate-sharded-cluster-to-new-hardware.txt diff --git a/bin/builddata/htaccess.yaml b/bin/builddata/htaccess.yaml index fff3b23a96c..d31d8ab2936 100644 --- a/bin/builddata/htaccess.yaml +++ b/bin/builddata/htaccess.yaml @@ -2865,6 +2865,13 @@ redirect-path: '/MongoDB-manual-v2.2.tar.gz' url-base: '/mongodb-manual-v2.2.tar.gz' type: 'redirect' code: 301 +outputs: + - 'v2.2' +--- +redirect-path: '/tutorial/migrate-sharded-cluster-to-new-hardware' +url-base: '/administration/sharded-clusters' +type: 'redirect' +code: 303 outputs: - 'v2.2' ... \ No newline at end of file diff --git a/source/includes/tip-hostnames.rst b/source/includes/tip-hostnames.rst new file mode 100644 index 00000000000..07a640d11e4 --- /dev/null +++ b/source/includes/tip-hostnames.rst @@ -0,0 +1,5 @@ +.. tip:: To avoid downtime, give each config server a logical DNS name + (unrelated to the server's physical or virtual hostname). Without + logical DNS names, moving or renaming a config server requires + shutting down every :program:`mongod` and :program:`mongos` instance + in the sharded cluster. \ No newline at end of file diff --git a/source/includes/toc-sharded-cluster-maintenance.yaml b/source/includes/toc-sharded-cluster-maintenance.yaml index 56c9654033c..1d400f223d2 100644 --- a/source/includes/toc-sharded-cluster-maintenance.yaml +++ b/source/includes/toc-sharded-cluster-maintenance.yaml @@ -30,6 +30,11 @@ description: | Replaces a config server that has become inoperable. This procedure assumes that the hostname does not change. --- +file: /tutorial/migrate-sharded-cluster-to-new-hardware +description: | + Migrate a sharded cluster to a different hardware system, for example, + when moving a pre-production environment to production. +--- file: /tutorial/backup-sharded-cluster-metadata description: | Create a backup of a sharded cluster's metadata while keeping the diff --git a/source/tutorial.txt b/source/tutorial.txt index c866cfcd168..bd34573e208 100644 --- a/source/tutorial.txt +++ b/source/tutorial.txt @@ -71,6 +71,7 @@ Sharding - :doc:`/tutorial/migrate-config-servers-with-same-hostname` - :doc:`/tutorial/migrate-config-servers-with-different-hostnames` - :doc:`/tutorial/replace-config-server` +- :doc:`/tutorial/migrate-sharded-cluster-to-new-hardware` - :doc:`/tutorial/backup-sharded-cluster-metadata` - :doc:`/tutorial/backup-small-sharded-cluster-with-mongodump` - :doc:`/tutorial/backup-sharded-cluster-with-filesystem-snapshots` diff --git a/source/tutorial/add-shards-to-shard-cluster.txt b/source/tutorial/add-shards-to-shard-cluster.txt index 553eec2cda6..333b26ffc02 100644 --- a/source/tutorial/add-shards-to-shard-cluster.txt +++ b/source/tutorial/add-shards-to-shard-cluster.txt @@ -24,13 +24,7 @@ You interact with a sharded cluster by connecting to a :program:`mongos` instance. 1. From a :program:`mongo` shell, connect to the :program:`mongos` - instance. Issue a command using the following syntax: - - .. code-block:: sh - - mongo --host --port - - For example, if a :program:`mongos` is accessible at + instance. For example, if a :program:`mongos` is accessible at ``mongos0.example.net`` on port ``27017``, issue the following command: @@ -38,7 +32,7 @@ instance. mongo --host mongos0.example.net --port 27017 -#. Add each shard to the cluster using the :method:`sh.addShard()` +#. Add a shard to the cluster using the :method:`sh.addShard()` method, as shown in the examples below. Issue :method:`sh.addShard()` separately for each shard. If the shard is a replica set, specify the name of the replica set and specify a member of the set. In diff --git a/source/tutorial/deploy-shard-cluster.txt b/source/tutorial/deploy-shard-cluster.txt index 1e98bfcde87..d07791ef1d2 100644 --- a/source/tutorial/deploy-shard-cluster.txt +++ b/source/tutorial/deploy-shard-cluster.txt @@ -83,7 +83,7 @@ When you start the :program:`mongos` instance, specify the hostnames of the three config servers, either in the configuration file or as command line parameters. -.. tip:: .. include:: /includes/fact-use-cnames-for-config-servers.rst +.. include:: /includes/tip-hostnames.rst To start a :program:`mongos` instance, issue a command using the following syntax: diff --git a/source/tutorial/manage-sharded-cluster-balancer.txt b/source/tutorial/manage-sharded-cluster-balancer.txt index fcdea2ed6f8..b82b068fc74 100644 --- a/source/tutorial/manage-sharded-cluster-balancer.txt +++ b/source/tutorial/manage-sharded-cluster-balancer.txt @@ -6,13 +6,27 @@ Manage Sharded Cluster Balancer .. default-domain:: mongodb -This page describes provides common administrative procedures related +This page describes common administrative procedures related to balancing. For an introduction to balancing, see :ref:`sharding-balancing`. For lower level information on balancing, see :ref:`sharding-balancing-internals`. .. seealso:: :doc:`/tutorial/configure-sharded-cluster-balancer` +Check the Balancer State +------------------------ + +The following command checks if the balancer is enabled (i.e. that the +balancer is allowed to run). The command does not check if the balancer +is active (i.e. if it is actively balancing chunks). + +To see if the balancer is enabled in your :term:`cluster `, issue the following command, which returns a boolean: + +.. code-block:: javascript + + sh.getBalancerState() + .. _sharding-balancing-check-lock: Check the Balancer Lock @@ -57,15 +71,6 @@ This output confirms that: has the lock. For version 2.0 and later, the value of an active lock is ``2``; for earlier versions the value is ``1``. -.. optional:: - - You can also use the following shell helper, which returns a - boolean to report if the balancer is active: - - .. code-block:: javascript - - sh.getBalancerState() - .. _sharding-schedule-balancing-window: Schedule the Balancing Window @@ -153,46 +158,36 @@ all migration, use the following procedure: sh.setBalancerState(false) - .. note:: - - If a migration is in progress, the system will complete the - in-progress migration before stopping. - -#. To later re-enable the balancer, see :ref:`sharding-balancing-re-enable`. + If a migration is in progress, the system will complete the + in-progress migration before stopping. -After disabling, you can use the following -operation in the :program:`mongo` shell to determine if there are -no migrations in progress: +#. To verify that the balancer has stopped, issue the following command, + which returns ``false`` if the balancer is stopped: -.. code-block:: javascript - - use config - while( sh.isBalancerRunning() ) { - print("waiting..."); - sleep(1000); - } - -The :method:`sh.setBalancerState()` helper provides -a wrapper on the following process, which may be useful if you -need to run this operation from a driver that does not have helper -functions: + .. code-block:: javascript -#. Connect to any :program:`mongos` in the cluster using the - :program:`mongo` shell. + sh.getBalancerState() -#. Issue the following command to switch to the :ref:`config-database`: + Optionally, to verify no migrations are in progress after disabling, + issue the following operation in the :program:`mongo` shell: .. code-block:: javascript use config + while( sh.isBalancerRunning() ) { + print("waiting..."); + sleep(1000); + } -#. Issue the following update to disable the balancer: +.. note:: - .. code-block:: javascript + To disable the balancer from a driver that does not have the + :method:`sh.startBalancer()` helper, issue the following command from + the ``config`` database: - db.settings.update( { _id: "balancer" }, { $set : { stopped: true } } , true ); + .. code-block:: javascript -#. To later re-enable the balancer, see :ref:`sharding-balancing-re-enable`. + db.settings.update( { _id: "balancer" }, { $set : { stopped: true } } , true ) .. _sharding-balancing-re-enable: .. _sharding-balancing-enable: @@ -208,18 +203,18 @@ re-enable it: #. Issue one of the following operations to enable the balancer: - - From the :program:`mongo` shell, issue: + From the :program:`mongo` shell, issue: - .. code-block:: javascript + .. code-block:: javascript - sh.setBalancerState(true) + sh.setBalancerState(true) - - From a driver that does not have the :method:`sh.startBalancer()` helper, - issue the following from the ``config`` database: + From a driver that does not have the :method:`sh.startBalancer()` helper, + issue the following from the ``config`` database: - .. code-block:: javascript + .. code-block:: javascript - db.settings.update( { _id: "balancer" }, { $set : { stopped: false } } , true ) + db.settings.update( { _id: "balancer" }, { $set : { stopped: false } } , true ) Disable Balancing During Backups -------------------------------- diff --git a/source/tutorial/migrate-config-servers-with-different-hostnames.txt b/source/tutorial/migrate-config-servers-with-different-hostnames.txt index c4685469e35..42951c6d8c9 100644 --- a/source/tutorial/migrate-config-servers-with-different-hostnames.txt +++ b/source/tutorial/migrate-config-servers-with-different-hostnames.txt @@ -8,23 +8,40 @@ This procedure migrates a :ref:`config server ` in a :doc:`sharded cluster ` to a new server that uses a different hostname. Use this procedure only if the config server *will not* be accessible via the same hostname. + +Changing a :ref:`config server's ` hostname +**requires downtime** and requires restarting every process in the +sharded cluster. If possible, avoid changing the hostname so that you can instead use the procedure to :doc:`migrate a config server and use the same hostname `. -#. Disable the cluster balancer process temporarily. See +To migrate all the config servers in a cluster, perform this procedure +for each config server separately and migrate the config servers in +reverse order from how they are listed in the :program:`mongos` +instances' :setting:`configdb` string. Start with the last config server +listed in the :setting:`configdb` string. + +1. Disable the cluster balancer process temporarily. See :ref:`sharding-balancing-disable-temporarily` for more information. -#. Shut down the :ref:`config server ` you are moving. +#. Shut down the config server. + + This renders all config data for the sharded cluster "read only." + +#. Copy the contents of :setting:`dbpath` from the old config server to + the new config server. - This will render all config data for your cluster "read only:" + For example, to copy the contents of :setting:`dbpath` to a machine + named ``mongodb.config2.example.net``, you might issue a command + similar to the following: .. code-block:: sh rsync -az /data/configdb mongodb.config2.example.net:/data/configdb -#. Start the config instance on the new system. The default invocation - is: +#. Start the config server instance on the new system. The default + invocation is: .. code-block:: sh @@ -32,22 +49,21 @@ procedure to :doc:`migrate a config server and use the same hostname #. Shut down all existing MongoDB processes. This includes: - - all :program:`mongod` instances or :term:`replica sets ` + - the :program:`mongod` instances or :term:`replica sets ` that provide your shards. - the :program:`mongod` instances that provide your existing :ref:`config databases `. - - all :program:`mongos` instances in your cluster. + - the :program:`mongos` instances. #. Restart all :program:`mongod` processes that provide the shard servers. -#. Restart the config instances that have *not* changed hostnames. +#. Update the :setting:`configdb` setting for each :program:`mongos` + instances. -#. Update the :option:`--configdb ` parameter (or - :setting:`configdb`) for all :program:`mongos` instances and - restart all :program:`mongos` instances. +#. Restart the :program:`mongos` instances. #. Re-enable the balancer to allow the cluster to resume normal balancing operations. See the diff --git a/source/tutorial/migrate-config-servers-with-same-hostname.txt b/source/tutorial/migrate-config-servers-with-same-hostname.txt index b1e8cd3ed71..8a0f00a7348 100644 --- a/source/tutorial/migrate-config-servers-with-same-hostname.txt +++ b/source/tutorial/migrate-config-servers-with-same-hostname.txt @@ -8,33 +8,44 @@ This procedure migrates a :ref:`config server ` in a :doc:`sharded cluster ` to a new system that uses *the same* hostname. -#. Shut down the config server that you are moving. +To migrate all the config servers in a cluster, perform this procedure +for each config server separately and migrate the config servers in +reverse order from how they are listed in the :program:`mongos` +instances' :setting:`configdb` string. Start with the last config server +listed in the :setting:`configdb` string. - This will render all config data for your cluster :ref:`read only - `. +.. start-migrate-config-server-with-same-hostname + +#. Shut down the config server. + + This renders all config data for the sharded cluster "read only." #. Change the DNS entry that points to the system that provided the old config server, so that the *same* hostname points to the new system. - How you do this depends on how you organize your DNS and hostname resolution services. -#. Move the entire :setting:`dbpath` file system tree from the old - config server to the new config server. This command, issued on the - old config server system, may resemble the following: +#. Copy the contents of :setting:`dbpath` from the old config server to + the new config server. + + For example, to copy the contents of :setting:`dbpath` to a machine + named ``mongodb.config2.example.net``, you might issue a command + similar to the following: .. code-block:: sh - rsync -az /data/configdb mongo-config0.example.net:/data/configdb + rsync -az /data/configdb mongodb.config2.example.net:/data/configdb -#. Start the config instance on the new system. The default invocation - is: +#. Start the config server instance on the new system. The default + invocation is: .. code-block:: sh mongod --configsvr +.. end-migrate-config-server-with-same-hostname + When you start the third config server, your cluster will become writable and it will be able to create new splits and migrate chunks as needed. diff --git a/source/tutorial/migrate-sharded-cluster-to-new-hardware.txt b/source/tutorial/migrate-sharded-cluster-to-new-hardware.txt new file mode 100644 index 00000000000..6feb2b41b03 --- /dev/null +++ b/source/tutorial/migrate-sharded-cluster-to-new-hardware.txt @@ -0,0 +1,232 @@ +=============================================== +Migrate a Sharded Cluster to Different Hardware +=============================================== + +.. default-domain:: mongodb + +.. contents:: Migrate Sharded Cluster: + :backlinks: none + :local: + +This procedure moves the components of the :term:`sharded cluster` to a +new hardware system without downtime for reads and writes. + +.. TODO Discuss whether a bullet list might be a better way to give + a skimmable picture of the macro procedure. + +.. To move to new hardware: + +.. - :ref:`migrate-to-new-hardware-disable-balancer` + - :ref:`migrate-to-new-hardware-config-servers` + - :ref:`migrate-to-new-hardware-restart-mongos` + - :ref:`migrate-to-new-hardware-shards` + - :ref:`migrate-to-new-hardware-enable-balancer` + +.. important:: While the migration is in progress, do not attempt to change to the + :doc:`cluster metadata `. Do not use + any operation that modifies the cluster metadata *in any way*. For + example, do not create or drop databases, create or drop collections, + or use any sharding commands. + +If your cluster includes a shard that comprises a :term:`standalone` +:program:`mongod` instance instead of a :term:`replica set`, consider +:doc:`converting the standalone to a replica set +` to simplify migration and +to let you keep the cluster online during future maintenance. Migrating +a shard as standalone is a multi-step process and may require downtime. + +To migrate a cluster to new hardware, perform the following tasks. + +.. _migrate-to-new-hardware-disable-balancer: + +Disable the Balancer +-------------------- + +Disable to the balancer to stop :doc:`chunk migration +` and do not perform any metadata write +operations until the process finishes. If a migration is in progress, the +balancer will complete the in-progress migration before stopping. + +To disable the balancer, connect to one of the cluster's +:program:`mongos` instances and issue the following method: + +.. code-block:: javascript + + sh.stopBalancer() + +To check the balancer state, issue the :method:`sh.getBalancerState()` +method. + +For more information, see :ref:`sharding-balancing-disable-temporarily`. + +.. _migrate-to-new-hardware-config-servers: + +Migrate Each Config Server Separately +------------------------------------- + +Migrate each :ref:`config server ` by starting +with the *last* config server listed in the :setting:`configdb` string. +Proceed in reverse order of the :setting:`configdb` string. Migrate and +restart a config server before proceeding to the next. +Do not rename a config server during this process. + +.. note:: Renaming a config server requires extra steps and cluster + downtime. If you *must* rename a config server, use this procedure + instead: + :doc:`/tutorial/migrate-config-servers-with-different-hostnames`. + +.. important:: Start with the *last* config server listed in :setting:`configdb`. + +.. include:: /tutorial/migrate-config-servers-with-same-hostname.txt + :start-after: start-migrate-config-server-with-same-hostname + :end-before: end-migrate-config-server-with-same-hostname + +.. _migrate-to-new-hardware-restart-mongos: + +Restart the ``mongos`` Instances +-------------------------------- + +If the :setting:`configdb` string is changing, you must shut down *all* +:program:`mongos` instances before changing the :setting:`configdb` +string. This avoids errors in the sharded cluster over +:setting:`configdb` string conflicts. + +If the :setting:`configdb` string will remain the same, you can migrate +the :program:`mongos` instances sequentially or all at once. + +1. Shut down the :program:`mongos` instances using the + :dbcommand:`shutdown` command. If the :setting:`configdb` string is + changing, shut down *all* :program:`mongos` instances. + +#. If the hostname has changed for any of the config servers, update the + :setting:`configdb` string for each :program:`mongos` instance. The + :program:`mongos` instances must all use the same :setting:`configdb` + string. The strings must list identical host names in identical order. + + .. include:: /includes/tip-hostnames.rst + +#. Restart the :program:`mongos` instances being sure to use the + updated :setting:`configdb` string if hostnames have changed. + +For more information, see :ref:`sharding-setup-start-mongos`. + +.. _migrate-to-new-hardware-shards: + +Migrate the Shards +------------------ + +Migrate the shards one at a time. For each shard, follow the appropriate +procedure in this section. + +.. _migrate-replica-set-shard: + +Migrate a Replica Set Shard +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To migrate a sharded cluster, migrate each member separately. First +migrate the non-primary members, and then migrate the :term:`primary` +last. + +If the replica set has two voting members, first add an +:doc:`arbiter ` to the replica set to ensure +the set keeps a majority of its votes available as you migrate the +members. You can remove the arbiter after you migrate the primary. + +.. _migrate-replica-set-shard-member: + +Migrate a Member of a Replica Set Shard +```````````````````````````````````````` + +1. Shut down the :program:`mongod` process. To ensure a + clean shutdown, use the :dbcommand:`shutdown` command. + +#. Move the data directory (i.e., the :setting:`dbpath`) + to the new machine. + +#. Restart the :program:`mongod` process at the new + location. + +#. Connect to the replica set's current primary. + +#. If the hostname of the member has changed, use + :method:`rs.reconfig()` to update the :doc:`replica set configuration + document ` with the new hostname. + + For example, the following sequence of commands updates the hostname + for the instance at position ``2`` in the ``members`` array: + + .. code-block:: javascript + + cfg = rs.conf() + cfg.members[2].host = "pocatello.example.net:27017" + rs.reconfig(cfg) + + For more information on updating the configuration document, see + :ref:`replica-set-reconfiguration-usage`. + +#. To confirm the new configuration, issue :method:`rs.conf()`. + +#. Wait for the member to recover. To check the member's state, issue + :method:`rs.status()`. + +Migrate the Primary in a Replica Set Shard +````````````````````````````````````````````` + +The procedure to migrate the primary includes the failover process, +which renders the replica set unavailable to perform reads or accept +writes until the process completes. Typically this takes 10 seconds or +more. You may want to plan the migration during a maintenance window. + +1. Step down the primary to allow the normal :ref:`failover + ` process. + To step down the primary, connect to the primary and issue the either + the :dbcommand:`replSetStepDown` command or the + :method:`rs.stepDown()` method. The following example shows the + :method:`rs.stepDown()` method: + + .. code-block:: javascript + + rs.stepDown() + +#. Once the primary has stepped down and another member has assumed the + ``PRIMARY`` state, as observed in the output of + :method:`rs.status()`, migrate the stepped-down primary. Use the + procedure to :ref:`migrate-replica-set-shard-member`. + +Migrate a Standalone Shard +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The suggested best practice for migrating a standalone shard is to +:doc:`convert the standalone to a replica set +` and then use the +procedure for :ref:`migrating a replica set shard +`. The best +practice in a sharded cluster is to use replica sets for all shards, +which provides continued availability during maintenance windows. + +Migrating a shard as standalone is a multi-step process and may require +downtime. If the shard is the :term:`primary shard` for a database, the +process includes the :dbcommand:`movePrimary` command, which can result +in missing data if run on a live system where data is changing in the +database. To migrate the standalone shard, use the procedure to +:doc:`/tutorial/remove-shards-from-cluster`. + +.. _migrate-to-new-hardware-enable-balancer: + +Re-Enable the Balancer +---------------------- + +Re-enable the balancer to resume :doc:`chunk migrations +`. + +Connect to one of the cluster's :program:`mongos` instances and pass +``true`` to the :method:`sh.setBalancerState()` method: + +.. code-block:: javascript + + sh.setBalancerState(true) + +To check the balancer state, issue the :method:`sh.getBalancerState()` +method. + +For more information, see :ref:`sharding-balancing-enable`.