[[ch-configure]]
== Configuring DRBD

[[s-prepare-storage]]
=== Preparing your lower-level storage

After you have installed DRBD, you must set aside a roughly
identically sized storage area on both cluster nodes. This will
become the _lower-level device_ for your DRBD
resource. You may use any type of block device found on your
system for this purpose. Typical examples include:

* A hard drive partition (or a full physical hard drive),

* a software RAID device,

* an LVM Logical Volume or any other block device configured by the
  Linux device-mapper infrastructure,

* any other block device type found on your system.

You may also use _resource stacking_, meaning you can use one DRBD
device as a lower-level device for another. Some specific
considerations apply to stacked resources; their configuration is
covered in detail in <<s-three-nodes>>.

NOTE: While it is possible to use loop devices as lower-level devices
for DRBD, doing so is not recommended due to deadlock issues.

It is _not_ necessary for this storage area to be empty before you
create a DRBD resource from it. In fact it is a common use case to
create a two-node cluster from a previously non-redundant
single-server system using DRBD (some caveats apply -- please refer to
<<s-metadata>> if you are planning to do this).

For the purposes of this guide, we assume a very simple setup:

* Both hosts have a free (currently unused) partition named
  +/dev/sda7+.

* We are using <<s-internal-meta-data,internal meta data>>.

[[s-prepare-network]]
=== Preparing your network configuration

It is recommended, though not strictly required, that you run your
DRBD replication over a dedicated connection. At the time of this
writing, the most reasonable choice for this is a direct,
back-to-back, Gigabit Ethernet connection. When DRBD is run
over switches, use of redundant components and the +bonding+ driver
(in +active-backup+ mode) is recommended.

It is generally not recommended to run DRBD replication via routers,
for reasons of fairly obvious performance drawbacks (adversely
affecting both throughput and latency).

In terms of local firewall considerations, it is important to
understand that DRBD (by convention) uses TCP ports from 7788 upwards,
with every resource listening on a separate port. DRBD uses _two_
TCP connections for every resource configured. For proper DRBD
functionality, it is required that these connections are allowed by
your firewall configuration.

Security considerations other than firewalling may also apply if a
Mandatory Access Control (MAC) scheme such as SELinux or AppArmor is
enabled. You may have to adjust your local security policy so it does
not keep DRBD from functioning properly.

You must, of course, also ensure that the TCP ports
for DRBD are not already used by another application.

It is not possible to configure a DRBD resource to support more than
one TCP connection. If you want to provide for DRBD connection
load-balancing or redundancy, you can easily do so at the Ethernet
level (again, using the +bonding+ driver).

For the purposes of this guide, we assume a
very simple setup:

* Our two DRBD hosts each have a currently unused network interface,
  +eth1+, with IP addresses +10.1.1.31 and 10.1.1.32+ assigned to it,
  respectively.

* No other services are using TCP ports 7788 through 7799 on either
  host.

* The local firewall configuration allows both inbound and outbound
  TCP connections between the hosts over these ports.


[[s-configure-resource]]
=== Configuring your resource

All aspects of DRBD are controlled in its configuration file,
+/etc/drbd.conf+. Normally, this configuration file is just a skeleton
with the following contents:

-------------------------------------
include "/etc/drbd.d/global_common.conf";
include "/etc/drbd.d/*.res";
-------------------------------------

By convention, +/etc/drbd.d/global_common.conf+ contains the
<<s-drbdconf-global,+global+>> and <<s-drbdconf-common,+common+>>
sections of the DRBD configuration, whereas the +.res+ files contain
one <<s-drbdconf-resource,+resource+>> section each.

It is also possible to use +drbd.conf+ as a flat configuration file
without any +include+ statements at all. Such a configuration,
however, quickly becomes cluttered and hard to manage, which is why
the multiple-file approach is the preferred one.

Regardless of which approach you employ, you should always make sure
that +drbd.conf+, and any other files it includes, are _exactly
identical_ on all participating cluster nodes.

The DRBD source tarball contains an example configuration file in the
+scripts+ subdirectory. Binary installation packages will either
install this example configuration directly in +/etc+, or in a
package-specific documentation directory such as
+/usr/share/doc/packages/drbd+.

This section describes only those few aspects of the configuration
file which are absolutely necessary to understand in order to get DRBD
up and running. The configuration file's syntax and contents are
documented in great detail in <<re-drbdconf>>.


[[s-drbdconf-example]]
==== Example configuration

For the purposes of this guide, we assume a
minimal setup in line with the examples given in the
previous sections:

.Simple DRBD configuration (+/etc/drbd.d/global_common.conf+)
-------------------------------------
global {
  usage-count yes;
}
common {
  net {
    protocol C;
  }
}
-------------------------------------

.Simple DRBD resource configuration (+/etc/drbd.d/r0.res+)
-------------------------------------
resource r0 {
  on alice {
    device    /dev/drbd1;
    disk      /dev/sda7;
    address   10.1.1.31:7789;
    meta-disk internal;
  }
  on bob {
    device    /dev/drbd1;
    disk      /dev/sda7;
    address   10.1.1.32:7789;
    meta-disk internal;
  }
}
-------------------------------------

This example configures DRBD in the following fashion:

* You "opt in" to be included in DRBD's usage statistics (see
  <<fp-usage-count>>).

* Resources are configured to use fully synchronous replication
  (<<s-replication-protocols,Protocol C>>) unless explicitly specified
  otherwise.

* Our cluster consists of two nodes, +alice and bob+.

* We have a resource arbitrarily named +r0+ which uses +/dev/sda7+ as
  the lower-level device, and is configured with
  <<s-internal-meta-data,internal meta data>>.

* The resource uses TCP port 7789 for its network connections, and
  binds to the IP addresses 10.1.1.31 and 10.1.1.32, respectively.

The configuration above implicitly creates one volume in the
resource, numbered zero (+0+). For multiple volumes in one resource,
modify the syntax as follows:

.Multi-volume DRBD resource configuration (+/etc/drbd.d/r0.res+)
-------------------------------------
resource r0 {
  volume 0 {
    device    /dev/drbd1;
    disk      /dev/sda7;
    meta-disk internal;
  }
  volume 1 {
    device    /dev/drbd2;
    disk      /dev/sda8;
    meta-disk internal;
  }
  on alice {
    address   10.1.1.31:7789;
  }
  on bob {
    address   10.1.1.32:7789;
  }
}
-------------------------------------

NOTE: Volumes may also be added to existing resources on the fly. For
an example see <<s-lvm-add-pv>>.

[[s-drbdconf-global]]
==== The +global+ section

This section is allowed only once in the configuration. It is normally
in the +/etc/drbd.d/global_common.conf+ file. In a single-file
configuration, it should go to the very top of the configuration
file. Of the few options available in this section, only one is of
relevance to most users:

[[fp-usage-count]]
.+usage-count+
The DRBD project keeps statistics about the usage of various DRBD
versions. This is done by contacting an HTTP server every time a new
DRBD version is installed on a system. This can be disabled by setting
+usage-count no;+.  The default is +usage-count ask;+ which will
prompt you every time you upgrade DRBD.

DRBD's usage statistics are, of course, publicly available: see
http://usage.drbd.org.


[[s-drbdconf-common]]
==== The +common+ section

This section provides a shorthand method to define configuration
settings inherited by every resource. It is normally found in
+/etc/drbd.d/global_common.conf+. You may define any option you can
also define on a per-resource basis.

Including a +common+ section is not strictly required, but strongly
recommended if you are using more than one resource. Otherwise, the
configuration quickly becomes convoluted by repeatedly-used options.

In the example above, we included +net { protocol C; }+ in the
+common+ section, so every resource configured (including +r0+)
inherits this option unless it has another +protocol+ option
configured explicitly. For other synchronization protocols available,
see <<s-replication-protocols>>.

[[s-drbdconf-resource]]
==== The +resource+ sections

A per-resource configuration file is usually named
+/etc/drbd.d/<resource>.res+.  Any DRBD resource you define must be
named by specifying resource name in the configuration. You may use
any arbitrary identifier, however the name must not contain characters
other than those found in the US-ASCII character set, and must also
not include whitespace.

Every resource configuration must also have two +on <host>+ sub-sections
(one for every cluster node). All other configuration settings are
either inherited from the +common+ section (if it exists), or derived
from DRBD's default settings.

In addition, options with equal values on both hosts
can be specified directly in the +resource+ section. Thus, we can
further condense our example configuration as follows:

-------------------------------------
resource r0 {
  device    /dev/drbd1;
  disk      /dev/sda7;
  meta-disk internal;
  on alice {
    address   10.1.1.31:7789;
  }
  on bob {
    address   10.1.1.32:7789;
  }
}
-------------------------------------


[[s-first-time-up]]
=== Enabling your resource for the first time

After you have completed initial resource configuration as outlined in
the previous sections, you can bring up your resource.

Each of the following steps must be completed on both nodes.

.Create device metadata
This step must be completed only on initial device
creation. It initializes DRBD's metadata:
-------------------------------------
drbdadm create-md resource
v08 Magic number not found
Writing meta data...
initialising activity log
NOT initializing bitmap
New drbd meta data block sucessfully created.
-------------------------------------

.Enable the resource
This step associates the resource with its backing device (or devices,
in case of a multi-volume resource), sets replication parameters, and
connects the resource to its peer:
-------------------------------------
drbdadm up resource
-------------------------------------

.Observe +/proc/drbd+
DRBD's virtual status file in the +/proc+ filesystem, +/proc/drbd+,
should now contain information similar to the following:

-------------------------------------
cat /proc/drbd
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by buildsystem@linbit, 2011-12-20 12:58:48
 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:524236
-------------------------------------

NOTE: The +Inconsistent/Inconsistent+ disk state is expected at this
point.

By now, DRBD has successfully allocated both disk and network
resources and is ready for operation. What it does not know yet is
which of your nodes should be used as the source of the initial device
synchronization.

[[s-initial-full-sync]]
=== The initial device synchronization

There are two more steps required for DRBD to become fully
operational:

.Select an initial sync source
If you are dealing with newly-initialized, empty disk, this choice is
entirely arbitrary. If one of your nodes already has valuable data
that you need to preserve, however, _it is of crucial importance_ that
you select that node as your synchronization source.  If you do
initial device synchronization in the wrong direction, you will lose
that data. Exercise caution.


.Start the initial full synchronization
This step must be performed on only one node, only on initial resource
configuration, and only on the node you selected as the
synchronization source. To perform this step, issue this command:

-------------------------------------
drbdadm primary --force resource
-------------------------------------

After issuing this command, the initial full synchronization will
commence. You will be able to monitor its progress via
+/proc/drbd+. It may take some time depending on the size of the
device.

By now, your DRBD device is fully operational, even before the initial
synchronization has completed (albeit with slightly reduced
performance). You may now create a filesystem on the device, use it as
a raw block device, mount it, and perform any other operation you
would with an accessible block device.

You will now probably want to continue with <<ch-admin>>, which
describes common administrative tasks to perform on your resource.

[[s-using-truck-based-replication]]
=== Using truck based replication

In order to preseed a remote node with data which is then to be kept
synchronized, and to skip the initial device synchronization, follow
these steps.

This assumes that your local node has a configured, but disconnected
DRBD resource in the Primary role.  That is to say, device
configuration is completed, identical +drbd.conf+ copies exist on both
nodes, and you have issued the commands for
<<s-initial-full-sync,initial resource promotion>> on your local node
-- but the remote node is not connected yet.


* On the local node, issue the following command:
-------------------------------------
drbdadm new-current-uuid --clear-bitmap <resource>
-------------------------------------

* Create a consistent, verbatim copy of the resource's data _and its
  metadata_. You may do so, for example, by removing a hot-swappable
  drive from a RAID-1 mirror.  You would, of course, replace it with a
  fresh drive, and rebuild the RAID set, to ensure continued
  redundancy. But the removed drive is a verbatim copy that can now be
  shipped off site.  If your local block device supports snapshot
  copies (such as when using DRBD on top of LVM), you may also create
  a bitwise copy of that snapshot using +dd+.


* On the local node, issue:
-------------------------------------
drbdadm new-current-uuid <resource>
-------------------------------------

Note the absence of the +--clear-bitmap+ option in this second
invocation.

* Physically transport the copies to the remote peer location.

* Add the copies to the remote node. This may again be a matter of
  plugging a physical disk, or grafting a bitwise copy of your shipped
  data onto existing storage on the remote node.  Be sure to restore
  or copy not only your replicated data, but also the associated DRBD
  metadata. If you fail to do so, the disk shipping process is moot.

* Bring up the resource on the remote node:
-------------------------------------
drbdadm up resource
-------------------------------------

After the two peers connect, they will not initiate a full device
synchronization. Instead, the automatic synchronization that now
commences only covers those blocks that changed since the invocation
of +drbdadm{nbsp}--clear-bitmap{nbsp}new-current-uuid+.

Even if there were _no_ changes whatsoever since then, there may still
be a brief synchronization period due to areas covered by the
<<s-activity-log,Activity Log>> being rolled back on the new
Secondary. This may be mitigated by the use of
<<p-checksum-sync,checksum-based synchronization>>.

You may use this same procedure regardless of whether the resource is
a regular DRBD resource, or a stacked resource. For stacked resources,
simply add the +-S+ or +--stacked+ option to +drbdadm+.
