2.5.3 Removing Servers
======================

Sometimes you need to remove a server from a cluster. Either because the hardware
or the operation system has become defect or simply because you are re-provisioning
the data center.

There are two methods of removing a server from a cluster, called `soft leave`
and `hard leave`. The difference between these two methods lies in a trade-off
between how quickly the server can actually be removed, not creating excessive
network/IO load and not violating any redundancy guarantees.

### Soft Leave

When `soft-leaving` a server, a flag is added to that server's configuration that
tells the cluster not to put any new data onto the server and to start rebalancing
the data owned by the leaving server to other nodes.

However, this rebalancing algorithm is designed not to create excessive network
or IO/CPU load by rebalancing all the data at once and to never violate the
redundancy guarantees.

This means that the data will be moved from the leaving server at a fairly slow
pace and you should not actually disconnect the leaving server until all data was
moved away from there.

Depending on the amount of data stored on the leaving server, cluster size and
available network bandwidth a slow leave operation might take many hours to days
to complete.

The slow leave operation is therefore applicable, if you are reprovisioning and
should not be used to remove defect hosts.

The general command to perform a soft leave is (assuming you want to remove `nodeX`)

    $ evqlctl cluster-remove-server --server_name "nodeX" --soft

However, you might have to pass configuration options. Assuming you are running
zookeeper as the coordination service on `localhost:2181` and the cluster name
is `mycluster`, the full command line could look like this:

    $ evqlctl cluster-remove-server \
        -C cluster.name=mycluster \
        -C cluster.coordinator=zookeeper \
        -C cluster.zookeeper_hosts=nue01.prod.fnrd.net:2181 \
        --server_name "nodeX" \
        --soft


### Hard Leave

The `hard leave` operation will immediately remove the server from the cluster
and mark all data previously stored on the node as "dead". After this happens,
the other (live) servers will detect that some partitions have fewer replicas
than they should have and will immediately start to assign new servers to those
partitions and start replicating.

This means a hard leave is instant: you can immediately disconnect the server
after the hard-leave operation has completed. A hard leave is also applicable
for defect hardware (where the machine/server is already dead and is removed from
  the cluster after-the-fact).

However, a hard leave results in an instant rebalance that might create some
network load and will temporarily reduce the number of replicas
(i.e. the redundancy level) for some data. So you have to be careful not to
execute too many hard leaves at once as this might lead to data loss.

Do not use the hard leave to reprovision the cluster, but only for defect nodes.

The command to perform a hard leave is (assuming you want to remove `nodeX`)

    $ evqlctl cluster-remove-server --server_name "nodeX" --hard
