Home

Awesome

Ceph-Chef Cookbook

Join the chat at https://gitter.im/ceph/ceph-chef License

DESCRIPTION

Installs and configures Ceph, a distributed network storage and filesystem designed to provide excellent performance, reliability, and scalability. Supports Hammer and higher releases (nothing below Hammer is supported in this repo).

Once Hammer support has stopped in Ceph then it will be removed from this cookbook as an option.

The current version is focused on installing and configuring Ceph for Ubuntu, CentOS and RHEL.

For documentation on how to use this cookbook, refer to the USAGE section.

Ceph-Chef works with ALL of the latest versions of Ceph and includes Ceph-Mgr recipe.

Recommendation

There are many Enterprises that use the cookbook to install and manage Ceph. Below are a few reference Chef Wrapper Repos and projects that use Ceph-Chef.

https://github.com/bloomberg/chef-bcs. That Chef App (repo) uses this repo for Bloomberg's large clusters. The chef-bcs repo is an S3 Object Store Cluster used in multiple data centers.

https://github.com/cepheus-io/cepheus or http://cepheus.io. This is a powerful and flexible example of use Ceph-Chef. Everything is data driven so you can simply change the data and use it to build your own Ceph cluster or as a reference. It also provides a full management stack for Ceph.

Note: The documentation is a WIP along with a few other features. This repo is actively managed.

For help, use Gitter chat, mailing-list or issues here in this repo.

NOTE: Users of ceph-cookbook

The original ceph-cookbook will remain and may continue to be updated (see that repo for specifics). We tried to use some of the interesting features of ceph-cookbook but we added a lot of enhancements and simplifications. Simply replacing ceph-cookbook with ceph-chef will not work without a few modifications. Also, ceph-chef only works with Chef 12.8+ and Hammer and higher. Nothing below the Hammer release of Ceph is supported in this repo. In addition, only civitweb is used going forward (not Apache).

NOTE: The current LWRP are using the style prior to Chef version 12.5. There will be a new release shortly that will support the now recommended way of handling custom resources. To make that change easier we will be using a helper cookbook called Poise. Using Poise makes creating custom resources and common services very simple.

Chef

>= 12.8+

Platform

Tested as working:

Cookbooks

The ceph cookbook requires the following cookbooks from Chef:

https://supermarket.chef.io/

USAGE

Ceph cluster design and Ceph support are beyond the scope of this README, please turn to the:

public wiki, mailing lists, visit our IRC channel, or contact Red Hat:

http://ceph.com/docs/master http://ceph.com/resources/mailing-list-irc/

This cookbook can be used to implement a chosen cluster design. Most of the configuration is retrieved from node attributes, which can be set by an environment json file or by a wrapper cookbook that updates the attributes directly. A basic cluster configuration will need most of the following attributes:

Most notably, the configuration does NOT need to set the mon initial members, because the cookbook does a node search based on TAGS or ENVIRONMENTS to find other mons in the same environment. However, you can add them to node['ceph']['config']['global']['mon initial members'] = <whatever mon ip list you want>

The other set of attributes that this recipe needs is node['ceph']['osd']['devices'], which is an array of OSD definitions, similar to the following:

Ceph Admin Commands

An example of finding a mon socket in a generic like environment. ceph-conf --name mon.$(hostname -s) --show-config-value admin_socket

Trouble Shooting

Pools - After creating it appears that some of the PGs are stuck 'creating+peering'. This can be caused by a number of things. Most likely an OSD is not blocking the creation. Do something like: ceph pg ls-by-pool <whatever the pool name> | grep creating

Something like: ceph pg ls-by-pool .rgw | grep creating

It should show you the PG number as the first column. Use that to query it to see if something is blocking it. ceph pg <pg num> query

Something like: ceph pg 175.f7 query

This will return a lot of json data. You can grep or look for 'blocking'. If found then restart the given OSD. You can find the host for the OSD with: ceph osd find <OSD number>

Once you're on the host simply restart the specific OSD with: sudo service ceph restart osd.<whatever the number>

Using a Policy Wrapper Cookbook

To automate setting several of these node attributes, it is recommended to use a policy wrapper cookbook. This allows the ability to use Chef Server cookbook versions along with environment version restrictions to roll out configuration changes in an ordered fashion.

It also can help with automating some settings. For example, a wrapper cookbook could peek at the list of hard drives that ohai has found and populate node['ceph']['osd_devices'] accordingly, instead of manually typing them all in:

node.override['ceph']['osd_devices'] = node['block_device'].each.reject{ |name, data| name !~ /^sd[b-z]/}.sort.map { |name, data| {'journal' => "/dev/#{name}"} }

For best results, the wrapper cookbook's recipe should be placed before the Ceph cookbook in the node's runlist. This will ensure that any attributes are in place before the Ceph cookbook runs and consumes those attributes.

Ceph Monitor

Ceph monitor nodes should use the ceph-mon role.

Includes:

Ceph Metadata Server

Ceph metadata server nodes should use the ceph-mds role. NB: Only required for Ceph-FS

Includes:

Ceph OSD

Ceph OSD nodes should use the ceph-osd role

Includes:

Ceph RADOS Gateway (RGW)

Ceph RGW nodes should use the ceph-radosgw role

ATTRIBUTES

General

Ceph MON

Ceph OSD

Ceph MDS

Ceph RADOS Gateway (RGW)

Note: Only supports the newer 'civetweb' version of RGW (not Apache)

Resources/Providers

ceph_client

The ceph_client LWRP provides an easy way to construct a Ceph client key. These keys are needed by anything that needs to talk to the Ceph cluster, including RGW, CephFS, and RBD access.

Actions

Parameters

ceph_cephfs

The ceph_cephfs LWRP provides an easy way to mount CephFS. It will automatically create a Ceph client key for the machine and mount CephFS to the specified location. If the kernel client is used, instead of the fuse client, a pre-existing subdirectory of CephFS can be mounted instead of the root.

Actions

Parameters

ceph_pool

The ceph_pool LWRP provides an easy way to create and delete Ceph pools.

It assumes that connectivity to the cluster is setup and that admin credentials are available from default locations, e.g. /etc/ceph/ceph.client.admin.keyring.

Actions

Parameters

DEVELOPING

Style Guide

This cookbook requires a style guide for all contributions. Travis will automatically verify that every Pull Request follows the style guide.

  1. Install ChefDK
  2. Activate ChefDK's copy of ruby: eval "$(chef shell-init bash)"
  3. bundle install
  4. bundle exec rake style

Testing

This cookbook uses Test Kitchen to verify functionality. A Pull Request can't be merged if it causes any of the test configurations to fail.

  1. Install ChefDK
  2. Activate ChefDK's copy of ruby: eval "$(chef shell-init bash)"
  3. bundle install
  4. bundle exec kitchen test aio-debian-74
  5. bundle exec kitchen test aio-ubuntu-1204
  6. bundle exec kitchen test aio-ubuntu-1404

AUTHORS

CONTRIBUTIONS

We welcome ALL contributions:

  1. Fork
  2. Create a feature branch
  3. Make changes
  4. Test
  5. Make Pull Request

Your changes will be tested and if all goes well it will be merged - Thanks!!