Home

Awesome

Chef-BCS

Join the chat at https://gitter.im/bloomberg/chef-bcs

DESCRIPTION

Installs and configures Ceph, a distributed network storage and filesystem designed to provide excellent performance, reliability, and scalability.

The current version is focused on installing and configuring Ceph for CentOS and RHEL.

Prerequisites

  1. Vagrant - https://www.vagrantup.com/downloads.html (for development or just spinning up VM version of cluster - not needed for bare metal cluster)
  2. VirtualBox - https://www.virtualbox.org/wiki/Downloads
  3. Git

Instructions

  1. Fork/clone repo
  2. Navigate to [whatever path]/ceph-bcs/bootstrap/vms/vagrant directory
  3. Launch Vagrant version to see how it works and to do development and testing by issuing ./CEPH_UP command (in /bootstrap/vms/vagrant directory)

Process (Vagrant)

Assuming you're in the path mentioned in #2 above.

To start a normal build simply do the following (no proxy):

./CEPH_UP

NB: If you want to test the upstream ceph-chef cookbook then clone that repo, make your changes, copy your cloned repo into the cookbooks section of the is cloned repo and then run the following command to start the build and test:

./CEPH_UP -d 0 <-- Run in debug mode

NB: Behind firewall:

./CEPH_UP -p [whatever your http(s) proxy url]

OR

./CEPH_UP -d 0 -p [whatever your http(s) proxy url] <-- Run in debug mode

What happens...

  1. Download CentOS 7.1 box version from Chef Bento upstream (7.2 and 7.3 versions of the bento/centos have sshd issues)
  2. Download required cookbooks including ceph-chef which is the most important
  3. Issue vagrant up that creates 4 VMs (dynamic and part of yaml file in /bootstrap/vms directory)
  4. Spins down VMs and adds network adapters and interfaces, sets up folder sharing and start VMs again
  5. Mounts shared folders (makes it easy to move cookbooks etc to VMs) and sets network and then setups up the bootstrap node ceph-bootstrap as a Chef Server
  6. Sets up chef-client on all other VMs
  7. Adds roles for specific Ceph types such as ceph-mon and ceph-osd etc for the given VM
  8. Updates the environment json file (contains all of your override values of the defaults - different one for vagrant.json, staging.json and/or production.json) [Only vagrant.json is used in this repo. You will need to create the specific environment json file for your targeted environment]
  9. Creates the Ceph Monitors first (ceph-mon role)
  10. Creates the Ceph OSD nodes (ceph-osd role)
  11. Creates the Ceph RGW node (ceph-radosgw role)
  12. Creates the Ceph restapi node (ceph-restapi role)
  13. Finishes the cluster simply by enabling the services

Nodes (Vagrant) - Creates an S3 Ceph Object Store Example Cluster

These are the default names. You can can call them anything you want. The main thing is to keep them numbered and not named like a pet but instead, named like cattle :)

ceph-bootstrap - Bootstrap node that acts as the Chef Server, Repo Mirror (in some cases) and Cobbler Server

ceph-vm1 - VM that has the ceph-mon, ceph-osd and ceph-radosgw roles applied

ceph-vm2 - VM that has the ceph-mon and ceph-osd roles applied

ceph-vm3 - VM that has the ceph-mon and ceph-osd roles applied

NOTE: ceph-bootstrap does NOT contain any ceph functionality

RADOS Gateway (RGW) uses civetweb as the embedded web server. You can login to any VM and issue a simple curl command (i.e., curl localhost or curl ceph-vm1.ceph.example.com or curl ceph-vm1). The hosts file is updated on all three VMs to support FQDN and short names.

Login to VMs (Vagrant)

Must be located in the [wherever root dir]/bootstrap/vms/vagrant directory (vagrant keeps a .vagrant directory with node information in it)

Command(s):

vagrant ssh ceph-bootstrap

vagrant ssh ceph-vm1

vagrant ssh ceph-vm2

vagrant ssh ceph-vm3

NOTE: These names can be changed in the [wherever root dir]/bootstrap/vms/servers_config.yaml file.

Sidebar: Vagrant uses port forwarding on the first network adapter of a given VM it manages. It then uses ssh port on the localhost to make it simple on itself.

Helper Scripts (used in development to break tasks into smaller units of work)

<wherever repo>/bootstrap/common

<wherever repo>/bootstrap/vms

<wherever repo>/bootstrap/vms/vagrant

Note: The only one you must call is CEPH_UP which starts the whole process from creation of VMs to running Ceph cluster

For documentation on how to use this cookbook, refer to the USAGE section.

Note: The documentation is a WIP along with a few other features. This repo is actively managed.

If there are issues then please go to the ISSUES section in this repo.

REQUIREMENTS

Chef

>= 12.8+

Platform

Tested as working:

Cookbooks

[IMPORTANT - Cookbook that everything else is based on]
https://github.com/ceph/ceph-chef

The ceph cookbook requires the following cookbooks from Chef:

https://supermarket.chef.io/

GEMS

The following two GEMS will need to be pulled down and loaded onto the production nodes for envrionments that can't reach the outside. The bootstrap_prereqs.sh does this automatically.

TEMPLATES

The following templates are Jinja2 based templates. The jinja_render.py found in bootstrap/templates reads the production yaml data files and runs through these files and builds the production.json, kickstart, linux grub and operations key files. The erb are Chef templates but the jinja_render script builds and puts those erb files in the template/default area of the cookbook as part of the preprocess.

USAGE

Ceph cluster design is beyond the scope of this README, please turn to the public wiki, mailing lists, visit our IRC channel, or contact Red Hat:

http://ceph.com/docs/master http://ceph.com/resources/mailing-list-irc/

This cookbook can be used to implement a chosen cluster design. Most of the configuration is retrieved from node attributes, which can be set by an environment or by a wrapper cookbook. A basic cluster configuration will need most of the following attributes:

Most notably, the configuration does NOT need to set the mon initial members, because the cookbook does a node search to find other mons in the same environment.

The other set of attributes that this recipe needs is node['ceph']['osd_devices'], which is an array of OSD definitions, similar to the following:

Using a Policy Wrapper Cookbook

To automate setting several of these node attributes, it is recommended to use a policy wrapper cookbook. This allows the ability to use Chef Server cookbook versions along with environment version restrictions to roll out configuration changes in an ordered fashion.

It also can help with automating some settings. For example, a wrapper cookbook could peek at the list of harddrives that ohai has found and populate node['ceph']['osd_devices'] accordingly, instead of manually typing them all in:

node.override['ceph']['osd_devices'] = node['block_device'].each.reject{ |name, data| name !~ /^sd[b-z]/}.sort.map { |name, data| {'journal' => "/dev/#{name}"} }

For best results, the wrapper cookbook's recipe should be placed before the Ceph cookbook in the node's runlist. This will ensure that any attributes are in place before the Ceph cookbook runs and consumes those attributes.

Ceph Monitor

Ceph monitor nodes should use the ceph-mon role.

Includes:

Ceph Metadata Server

Ceph metadata server nodes should use the ceph-mds role.

Includes:

Ceph OSD

Ceph OSD nodes should use the ceph-osd role

Includes:

Ceph RADOS Gateway

Ceph RADOS Gateway nodes should use the ceph-radosgw role

ATTRIBUTES

General

Ceph MON

Ceph OSD

Ceph MDS

Ceph RADOS Gateway (RGW)

Note: Only supports the newer 'civetweb' version of RGW (not Apache)

Resources/Providers

ceph_client

The ceph_client LWRP provides an easy way to construct a Ceph client key. These keys are needed by anything that needs to talk to the Ceph cluster, including RGW, CephFS, and RBD access.

Actions

Parameters

ceph_cephfs

The ceph_cephfs LWRP provides an easy way to mount CephFS. It will automatically create a Ceph client key for the machine and mount CephFS to the specified location. If the kernel client is used, instead of the fuse client, a pre-existing subdirectory of CephFS can be mounted instead of the root.

Actions

Parameters

ceph_pool

The ceph_pool LWRP provides an easy way to create and delete Ceph pools.

It assumes that connectivity to the cluster is setup and that admin credentials are available from default locations, e.g. /etc/ceph/ceph.client.admin.keyring.

Actions

Parameters

DEVELOPING

Style Guide

This cookbook requires a style guide for all contributions. Travis will automatically verify that every Pull Request follows the style guide.

  1. Install ChefDK
  2. Activate ChefDK's copy of ruby: eval "$(chef shell-init bash)"
  3. bundle install
  4. bundle exec rake style

Testing

This cookbook uses Test Kitchen to verify functionality. A Pull Request can't be merged if it causes any of the test configurations to fail.

  1. Install ChefDK
  2. Activate ChefDK's copy of ruby: eval "$(chef shell-init bash)"
  3. bundle install
  4. bundle exec kitchen test aio-debian-74
  5. bundle exec kitchen test aio-ubuntu-1204
  6. bundle exec kitchen test aio-ubuntu-1404

LICENSE

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.