Awesome

Nabu Catalog

Setting up your dev environment

Use direnv to add bin to your path

PATH_add bin

Bring up the environment

# Build the base image
docker compose build

# Install the gems
nabu_run bundle

# Bring up all the containers
docker compose up

This brings up the following containers

app - the rails app
search - Solr instance for search (dev + test)
db - mysql data base (dev + test)
s3 - s3 mock

You can then easily run all the standard commands by prefixing with nabu

nabu_run bundle install
nabu_run bundle exec rake db:prepare
nabu_run bundle exec guard # Test runner

Production

The application is designed to be deployed with containers into an AWS account using CDK

To bootstrap a new account

# Setup an AWS account and credentials as per your preferred method and set the environment to use it
AWS_PROFILE=nabu
REGION=ap-southeast-2
ACCOUNT=$(aws sts get-caller-identity | jq -r .Account)
cdk bootstrap aws://$ACCOUNT/$REGION

If ECR complains about access

ACCOUNT=$(AWS_PROFILE=nabu-stage aws sts get-caller-identity | jq -r .Account)
AWS_PROFILE=nabu-stage aws ecr get-login-password --region ap-southeast-2 | docker login --username AWS --password-stdin $ACCOUNT.dkr.ecr.ap-southeast-2.amazonaws.com

Deployment

Use CDK to deploy new code via docker as well as any infrastructure changes

bin/release stage
bin/release prod

If necessary:

bin/aws/ecs_rake deploy:migrate
bin/aws/ecs_rake searchkick:reindex

Importing a production database into your development environment

AWS_PROFILE=nabu-prod bin/aws/db_backup
mysql -h 127.0.0.1 -u root nabu_devel < ../schema.sql
pv ../data.sql | mysql -h 127.0.0.1 -u root nabu_devel
nabu_run bin/rails db:environment:set RAILS_ENV=development
nabu_run bin/rake db:migrate
nabu_run bin/rake searchkick:reindex:all

New Ethnologue data

We use the following source locations

Run the following rake task to import everything

bundle exec rake import:ethnologue

This will

Add new countries and update names
Update country names
Add new languages and update names (Only Living languages)
Add mappings of language to countries
All existing languages that are retired are marked as such, incl name change.
Where name changes occurred items in CollectionLanguage, ItemContentLanguage, ItemSubjectLanguage are updated with the replacement language code.
Where splits happened, a message is printed.

OAI-PMH

OLAC available at:

http://catalog.paradisec.org.au/oai/item

The feeds that OLAC harvests:

Individual item:

http://catalog.paradisec.org.au/oai/item?verb=GetRecord&identifier=oai:paradisec.org.au:AA1-002&metadataPrefix=olac

RIF-CS available at:

http://catalog.paradisec.org.au/oai/collection

use resulting server on an OAI repository explorer:

http://www.language-archives.org/register/register.php (OLAC)
http://re.cs.uct.ac.za/
http://oval.base-search.net/ (OAI-PMH validator)
http://validator.oaipmh.com/ (OAI-PMH validator)
http://repox.gulbenkian.pt/repox/jsp/testOAI-PMH.jsp (test protocol)

URLs to test:

The feed that ANDS harvests:

http://catalog.paradisec.org.au/oai/collection?verb=ListRecords&metadataPrefix=rif

Test at ANDS:

https://demo.ands.org.au/registry/orca/admin/data_source_view.php?data_source_key=paradisec.org.au

Feed for a single collection:

http://catalog.paradisec.org.au/oai/collection?verb=GetRecord&metadataPrefix=rif&identifier=oai:paradisec.org.au:AA2

Setup Secrets

aws secretsmanager list-secrets

aws secretsmanager put-secret-value --secret-id ARN --secret-string "{\"site_key\":\"***\", \"secret_key\":\"***\"}"

Upgrades

We should regularly make sure we are running the latest versions of third-party packages

# Ruby gems
nabu_run bundle outdated
nabu_run bundle update

# node modules
nabu_run yarn upgrade-interactive

# New rails version
rails new nabu --database=mysql --javascript=esbuild --css=sass --skip-action-cable