Awesome
OpenCompany Storage Service
Background
A lack of transparency results in distrust and a deep sense of insecurity.
-- Dalai Lama
Teams struggle to keep everyone on the same page. People are hyper-connected in the moment with chat and email, but it gets noisy as teams grow, and people miss key information. Everyone needs clear and consistent leadership, and the solution is surprisingly simple and effective - great leadership updates that build transparency and alignment.
With that in mind we designed Carrot, a software-as-a-service application powered by the open source OpenCompany platform and a source-available web UI.
With Carrot, important company updates, announcements, stories, and strategic plans create focused conversations that keep everyone aligned without interruptions. When information is shared transparently, it inspires trust, new ideas and new levels of stakeholder engagement. Carrot makes it easy for leaders to engage with employees, investors, and customers, creating alignment for everyone.
Transparency expectations are changing. Organizations need to change as well if they are going to attract and retain savvy teams, investors and customers. Just as open source changed the way we build software, transparency changes how we build successful companies with information that is open, interactive, and always accessible. Carrot turns transparency into a competitive advantage.
To get started, head to: Carrot
Overview
The OpenCompany Storage Service handles data access and data management of open company content and data. It supports other OpenCompany services, such as the Web application and the Digest Service, as well as open API access.
Local Setup
Prospective users of Carrot should get started by going to Carrot.io. The following local setup is for developers wanting to work on the OpenCompany Storage Service.
Most of the dependencies are internal, meaning Leiningen will handle getting them for you. There are a few exceptions:
- Java - a Java 8+ JRE is needed to run Clojure
- Leiningen 2.9.1+ - Clojure's build and dependency management tool
- RethinkDB v2.3.6+ - a multi-modal (document, key/value, relational) open source NoSQL database
Java
Your system may already have Java 8+ installed. You can verify this with:
java -version
If you do not have Java 8+ download it and follow the installation instructions.
An option we recommend is OpenJDK. There are instructions for Linux and Homebrew can be used to install OpenJDK on a Mac with:
brew update && brew cask install adoptopenjdk8
Leiningen
Leiningen is easy to install:
- Download the latest lein script from the stable branch.
- Place it somewhere that's on your $PATH (
env | grep PATH
)./usr/local/bin
is a good choice if it is on your PATH. - Set it to be executable.
chmod 755 /usr/local/bin/lein
- Run it:
lein
This will finish the installation.
Then let Leiningen install the rest of the dependencies:
git clone https://github.com/open-company/open-company-storage.git
cd open-company-storage
lein deps
RethinkDB
RethinkDB is easy to install with official and community supported packages for most operating systems.
RethinkDB for Mac OS X via Brew
Assuming you are running Mac OS X and are a Homebrew user, use brew to install RethinkDB:
brew update && brew install rethinkdb
If you already have RethinkDB installed via brew, check the version:
rethinkdb -v
If it's older, then upgrade it with:
brew update && brew upgrade rethinkdb && brew services restart rethinkdb
Follow the instructions provided by brew to run RethinkDB every time at login:
ln -sfv /usr/local/opt/rethinkdb/*.plist ~/Library/LaunchAgents
And to run RethinkDB now:
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.rethinkdb.plist
Verify you can access the RethinkDB admin console:
open http://localhost:8080/
After installing with brew:
- Your RethinkDB binary will be at
/usr/local/bin/rethinkdb
- Your RethinkDB data directory will be at
/usr/local/var/rethinkdb
- Your RethinkDB log will be at
/usr/local/var/log/rethinkdb/rethinkdb.log
- Your RethinkDB launchd file will be at
~/Library/LaunchAgents/homebrew.mxcl.rethinkdb.plist
RethinkDB for Mac OS X (Binary Package)
If you don't use brew, there is a binary installer package available for Mac OS X from the Mac download page.
After downloading the disk image, mounting it (double click) and running the rethinkdb.pkg installer, you need to manually create the data directory:
sudo mkdir -p /Library/RethinkDB
sudo chown <your-own-user-id> /Library/RethinkDB
mkdir /Library/RethinkDB/data
And you will need to manually create the launchd config file to run RethinkDB every time at login. From within this repo run:
cp ./opt/com.rethinkdb.server.plist ~/Library/LaunchAgents/com.rethinkdb.server.plist
And to run RethinkDB now:
launchctl load ~/Library/LaunchAgents/com.rethinkdb.server.plist
Verify you can access the RethinkDB admin console:
open http://localhost:8080/
After installing with the binary package:
- Your RethinkDB binary will be at
/usr/local/bin/rethinkdb
- Your RethinkDB data directory will be at
/Library/RethinkDB/data
- Your RethinkDB log will be at
/var/log/rethinkdb.log
- Your RethinkDB launchd file will be at
~/Library/LaunchAgents/com.rethinkdb.server.plist
RethinkDB for Linux
If you run Linux on your development environment (good for you, hardcore!) you can get a package for you distribution or compile from source. Details are on the installation page.
RethinkDB for Windows
RethinkDB isn't supported on Windows directly. If you are stuck on Windows, you can run Linux in a virtualized environment to host RethinkDB.
Configuration
An AWS SQS queue is used to pass messages to the OpenCompany Storage service from the OpenCompany Authentication service. Setup SQS Queues and key/secret access to the queue using the AWS Web Console or API.
You will also need to subscribe the aws-sqs-auth-queue
SQS queue to the Authentication service SNS topic:
Go to the AWS SQS Console and select the aws-sqs-auth-queue
SQS queue configured above. From the 'Queue Actions' dropdown, select 'Subscribe Queue to SNS Topic'. Select the SNS topic you've configured your Authentication service instance to publish to, and click the 'Subscribe' button.
Make sure you update the section in project.clj
that looks like this to contain your specific configuration:
:dev [:qa {
:env ^:replace {
:db-name "open_company_storage_dev"
:liberator-trace "false" ; liberator debug data in HTTP response headers
:hot-reload "true" ; reload code when changed on the file system
:open-company-auth-passphrase "this_is_a_dev_secret" ; JWT secret
:aws-access-key-id "CHANGE-ME"
:aws-secret-access-key "CHANGE-ME"
:aws-sqs-bot-queue "CHANGE-ME" ; SQS queue to pass on requests to the Slack Bot
:aws-sqs-email-queue "CHANGE-ME" ; SQS queue to pass on requests to the Email service
:aws-sqs-auth-queue "CHANGE-ME" ; SQS queue to read notifications from the Auth service
:aws-sqs-storage-queue "CHANGE-ME" ; SQS queue to read requests from the Bot service
:aws-sns-storage-topic-arn "" ; SNS topic to publish notifications (optional)
:log-level "debug"
}
You can also override these settings with environmental variables in the form of OPEN_COMPANY_AUTH_PASSPHRASE
and
AWS_ACCESS_KEY_ID
, etc. Use environmental variables to provide production secrets when running in production.
Shared Secrets
A secret, open-company-auth-passphrase
, is shared between the OpenCompany services for creating and validating JSON Web Tokens.
HTTP Behavior
There are two options controlling HTTP behavior:
liberator-trace
will include detailed debug info in HTTP response headers that correspond to each step in the Liberator decision graph for that request.
The hot-reload
configuration controls if each request checks for updated code to load, useful for development, but not for production.
Both of these settings take the string true
or false
.
AWS
AWS SQS queues are used to pass messages from the Storage Service to other OpenCompany services. Setup an SQS Queue and key/secret access to the queue using the AWS Web Console or API and update the corresponding aws-
configuration properties with the key, secret and queue names.
An optional AWS SNS pub/sub topic is used to push notifications of content changes to interested listeners. If you want to take advantage of this capability, configure the aws-sns-storage-topic-arn
with the ARN (Amazon Resource Name) of the SNS topic you setup in AWS.
Technical Design
The Storage service is composed of these main responsibilities:
- CRUD of orgs, boards, and entries
- Access control to orgs, boards, and entries
- Notifying the Bot service of new orgs via SQS
- Notifying the Bot service and Email service of share requests via SQS
- Notifying the Bot service and Email service of new invites via SQS
- Notifying the Email service of password reset and email validation requests via SQS
- Publishing org, board and entry change notifications to interested subscribers via SNS
The Storage service provides a HATEOAS REST API:
The Storage service shares a RethinkDB database instance with the Interaction service.
Usage
Prospective users of Carrot should get started by going to Carrot.io. The following usage is for developers wanting to work on the OpenCompany Storage Service.
You can verify all is well with your RethinkDB instance and get familiar with RethinkDB ReQL query language by using the Data Explorer:
open http://localhost:8080/
REPL
Next, you can try some things with Clojure by running the REPL from within this project:
lein migrate-db
lein repl
Server
Start a production instance:
lein start!
Or start a development instance:
lein start
API Requests
You'll need a JWToken to use the REST API via cURL as an authenticated user. The token is passed in the Authorization
header with each request. You can either extract your own token from the headers of requests from your web browser,
or you can generate a token using the Authentication service.
You can see a list of common API requests and responses by loading the oc-storage.paw
file from this repository
into the Paw API Development tool.
Import sample data
To import company sample data from an edn file run:
lein run -m oc.storage.util.import -- ./opt/samples/18f.edn
use -d
to erase the company while importing like this:
lein run -m oc.storage.util.import -- -d ./opt/samples/green-labs.edn
To add all the company sample data in a directory (each file with a .edn
extension), run:
lein run -m oc.storage.util.import -- ./opt/samples/
use -d
to erase companies while importing like this:
lein run -m oc.storage.util.import -- -d ./opt/samples/
To add sample data on a production environment, specify the production database name:
DB_NAME="open_company_storage" lein run -m oc.storage.util.import -- -d ./opt/samples/18F.edn
or
DB_NAME="open_company_storage" lein run -m oc.storage.util.import -- -d ./opt/samples/
Generate sample data
To generate sample data into an existing org, run:
lein run -m oc.storage.util.generate -- <org-slug> <config-file> <start-date> <end-data>
e.g.
lein run -m oc.storage.util.generate -- 18f ./opt/generate.edn 2017-01-01 2017-06-30
See the sample generation config file ./opt/generate.edn
for how the sample data generation can be customized.
Force initial indexing or re-indexing
You can initiate an initial indexing of all the content stored in the storage service, or a re-indexing of the content from the commandline. Ensure you have AWS_SQS_SEARCH_INDEX_QUEUE
set to the search service SQS queue you are using:
export AWS_SQS_SEARCH_INDEX_QUEUE=change-me
Then run:
lein full-index
Or you can do it from the REPL. Ensure you have the aws-sqs-search-index-queue
configuration set to the search service SQS queue you are using:
config/aws-sqs-search-index-queue
then, from the REPL, run:
(send-data-to-search-index)
Testing
Tests are run in continuous integration of the master
and mainline
branches on Travis CI:
To run the tests locally:
lein test!
Participation
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.
License
Distributed under the GNU Affero General Public License Version 3.
Copyright © 2015-2022 Stoat Labs, Inc.
This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.