Home

Awesome

Domo

Mix.install([:domo], force: true)

About

Elixir CIMethod TDDhex.pm version
<!-- Documentation0 -->

A library to validate values of nested structs with their type spec t() and associated precondition functions.

<!-- Documentation0 -->

Example apps

🔗 JSON parsing and validation example

🔗 Commanded + Domo combo used in Event Sourcing and CQRS app

🔗 Ecto + Domo combo in example_avialia app

🔗 TypedStruct + Domo combo in example_typed_integrations app

<!-- Documentation1 -->

Description

The library adds constructor, validation, and reflection functions to a struct module. When called, constructor and validation functions guarantee the following:

If the conditions described above are not met, the constructor and validation functions return an error.

The business rule expressed via the precondition function is type-associated, which affects all structs referencing the appropriate type and using Domo.

In terms of Domain Driven Design, types and associated precondition functions define the invariants relating structs to each other.

use Domo {: .info}

When you use Domo, the Domo module will define the following functions:

See the Callbacks section for more details.

<!-- Documentation1 -->

Tour

<p align="center" class="hidden"> <a href="https://livebook.dev/run?url=https%3A%2F%2Fgithub.com%2FIvanRublev%2FDomo%2Fblob%2Fmaster%2FREADME.md"> <img src="https://livebook.dev/badge/v1/blue.svg" alt="Run in Livebook" /> </a> </p>

Let's say that we have a LineItem and PurchaseOrder structs with relating invariant that is the sum of line item amounts should be less then order's approved limit. That can be expressed like the following:

defmodule LineItem do
  use Domo

  defstruct amount: 0

  @type t :: %__MODULE__{amount: non_neg_integer()}
end

defmodule PurchaseOrder do
  use Domo

  defstruct id: 1000, approved_limit: 200, items: []

  @type id :: non_neg_integer()
  precond(id: &(1000 <= &1 and &1 <= 5000))

  @type t :: %__MODULE__{
          id: id(),
          approved_limit: pos_integer(),
          items: [LineItem.t()]
        }
  precond(t: &validate_invariants/1)

  defp validate_invariants(po) do
    amounts = po.items |> Enum.map(& &1.amount) |> Enum.sum()

    if amounts <= po.approved_limit do
      :ok
    else
      {:error, "Sum of line item amounts (#{amounts}) should be <= to approved limit (#{po.approved_limit})."}
    end
  end
end

Then PurchaseOrder struct can be constructed consistently with functions generated by Domo like the following:

PurchaseOrder.new()
{:ok, %PurchaseOrder{approved_limit: 200, id: 1000, items: []}}

The constructor function takes any Enumerable as the input value:

{:ok, po} = PurchaseOrder.new(%{approved_limit: 250})
{:ok, %PurchaseOrder{approved_limit: 250, id: 1000, items: []}}

It returns the descriptive keyword list if there is an error in input arguments. And it validates nested structs automatically:

PurchaseOrder.new(id: 500, items: [%LineItem{amount: -5}])
{:error,
 [
   items: "Invalid value [%LineItem{amount: -5}] for field :items of %PurchaseOrder{}.
    Expected the value matching the [%LineItem{}] type.
    Underlying errors:
       - The element at index 0 has value %LineItem{amount: -5} that is invalid.
       - Value of field :amount is invalid due to Invalid value -5 for field :amount
         of %LineItem{}. Expected the value matching the non_neg_integer() type.",
   id: "Invalid value 500 for field :id of %PurchaseOrder{}. Expected the
    value matching the non_neg_integer() type. And a true value from
    the precondition function \"&(1000 <= &1 and &1 <= 5000)\"
    defined for PurchaseOrder.id() type."
 ]}

The returned errors are verbose and are intended for debugging purposes. See the Error messages for a user section below for more options.

The manually updated struct can be validated like the following:

po
|> Map.put(:items, [LineItem.new!(amount: 150)])
|> PurchaseOrder.ensure_type()
{:ok, %PurchaseOrder{approved_limit: 200, id: 1000, items: [%LineItem{amount: 150}]}}

Domo returns the error if the precondition function validating the t() type as a whole fails:

updated_po = %{po | items: [LineItem.new!(amount: 180), LineItem.new!(amount: 100)]}
PurchaseOrder.ensure_type(updated_po)
{:error, [t: "Sum of line item amounts should be <= to approved limit"]}

Domo supports sum types for struct fields, for example:

defmodule FruitBasket do
  use Domo
  
  defstruct fruits: []
  
  @type t() :: %__MODULE__{fruits: [String.t() | :banana]}
end
FruitBasket.new(fruits: [:banana, "Maracuja"])
{:ok, %FruitBasket{fruits: [:banana, "Maracuja"]}}
{:error, [fruits: message]} = FruitBasket.new(fruits: [:banana, "Maracuja", nil])
IO.puts(message)
Invalid value [:banana, "Maracuja", nil] for field :fruits of %FruitBasket{}. Expected the value matching the [<<_::_*8>> | :banana] type.
Underlying errors:
   - The element at index 2 has value nil that is invalid.
   - Expected the value matching the <<_::_*8>> type.
   - Expected the value matching the :banana type.

Getting the list of the required fields of the struct that have type other then nil or any is like that:

PurchaseOrder.required_fields()
[:approved_limit, :id, :items]

For debug purposes you can get the original type spec t() for the struct. That might be useful when the struct's type spec is generated by some other library such as TypedEctoSchema. Do it like the following:

PurchaseOrder.__t__()
%PurchaseOrder{
  id: integer() | nil
  ...
}

See the Callbacks section for more details about functions added to the struct.

Error messages for a user

It's possible to attach error messages to types with the precond macro to display them later to the user. To filter such kinds of messages, pass the maybe_filter_precond_errors: true option to Domo generated functions like that:

defmodule Book do
  use Domo

  defstruct [:title, :pages]

  @type title :: String.t()
  precond title: &(if String.length(&1) > 1, do: :ok, else: {:error, "Book title is required."})

  @type pages :: pos_integer()
  precond pages: &(if &1 > 2, do: :ok, else: {:error, "Book should have more then 3 pages. Given (#{&1})."})

  @type t :: %__MODULE__{title: nil | title(), pages: nil | pages()}
end

defmodule Shelf do
  use Domo

  defstruct books: []

  @type t :: %__MODULE__{books: [Book.t()]}
end

defmodule PublicLibrary do
  use Domo

  defstruct shelves: []

  @type t :: %__MODULE__{shelves: [Shelf.t()]}
end

library = struct!(PublicLibrary, %{shelves: [struct!(Shelf, %{books: [struct!(Book, %{title: "", pages: 1})]})]})

PublicLibrary.ensure_type(library, maybe_filter_precond_errors: true)
{:error,
 [
   shelves: [
     "Book title is required.",
     "Book should have more then 3 pages. Given (1)."
   ]
]}

That output contains only a flattened list of precondition error messages from the deeply nested structure.

<!-- Documentation2 -->

Custom constructor function

Sometimes a default value for the struct's field must be generated during the construction. To reuse the new(!)/1 function's name and keep the generated value validated, instruct Domo to use another name for its constructor with gen_constructor_name option, like the following:

defmodule Foo do
  use Domo, skip_defaults: true, gen_constructor_name: :_new

  defstruct [:id, :token]

  @type id :: non_neg_integer()
  @type token :: String.t()
  precond token: &byte_size(&1) == 8

  @type t :: %__MODULE__{id: id(), token: token()}

  def new(id) do
    _new(id: id, token: random_string(8))
  end

  def new!(id) do
    _new!(id: id, token: random_string(8))
  end

  defp random_string(length),
    do: :crypto.strong_rand_bytes(length) |> Base.encode64() |> binary_part(0, length)
end
Foo.new!(15245)
%Foo{id: 15245, token: "e8K9wP0e"}

Compile-time and Run-time validations

At the project's compile-time, Domo performs the following checks:

At run-time, Domo validates structs matching their t() types.

Domo compiles TypeEnsurer module from struct's t() type to do all kinds of validations. There is a generated function with pattern matchings and guards for each struct's field. Constructor and validation functions of the struct delegate the work to the appropriate TypeEnsurer module.

After the compilation, the flow of control of the nested StructA validation can look like the following:

+--------------------+
| PurchaseOrder      |     +---------------------------+
|                    |     | PurchaseOrder.TypeEnsurer |
| new(!)/1 ----------|--_  |                           |
| ensure_type(!)/1 --|-----|-> ensure_field_type/1     |
+--------------------+     |   private functions       |
                           +----|----------------------+
                                |
+-----------------------+       |
| LineItem              |       |  +-------------------------+
|                       |       |  | LineItem.TypeEnsurer    |
| new(!)/1              |       |  |                         |
| ensure_type(!)/1      |       +--|-> ensure_field_type/1   |
+-----------------------+          |   private functions     |
                                   +-------------------------+

In interactive mode (iex / livebook) Domo generates TypeEnsurer module dynamically as the last step of struct's module definition.

In mix compile mode Domo generates all TypeEnsurer modules after elixir compiler finishes its job. The generated code can be found in _build/MIX_ENV/domo_generated_code folder. However, that is for information purposes only. The following compilation will overwrite all changes there.

Depending types tracking

Let's suppose a structure field's type depends on a type defined in another module. When the latter type or its precondition changes, Domo recompiles the former module automatically to update its TypeEnsurer to keep the type validation up to date.

Domo tracks type-depending modules and touches appropriate files during compilation.

That works for any number of intermediate modules between the module defining the struct's field and the module defining the field's final type.

Integration with Ecto

Ecto schema changeset can be automatically validated to conform to t() type and associated preconditions. Then the changeset function can be shortened like the following:

<!-- livebook:{"force_markdown":true} -->
defmodule Customer do
  use Ecto.Schema
  use Domo, skip_defaults: true

  import Ecto.Changeset
  import Domo.Changeset

  schema "customers" do
    field :first_name, :string
    field :last_name, :string
    field :birth_date, :date

    timestamps()
  end

  @type t :: %__MODULE__{
          first_name: String.t(),
          last_name: String.t(),
          birth_date: Date.t()
        }

  def changeset(changeset, attrs) do
    changeset
    |> cast(attrs, __schema__(:fields))
    # Domo.Changeset defines validate_type/1 function.
    |> validate_type()
  end 
end

The Domo validation comes with the price of about 1.5x times slower than the equivalent Ecto's validate_N functions.

See detailed example is in the example_avialia project.

Integration with libraries generating t() type for a struct

Domo is compatible with most libraries that generate t() type for a struct and an Ecto schema, f.e. typed_struct and typed_ecto_schema respectfully. Just use Domo in the module, and that's it.

An example is in the example_typed_integrations project.

TypedStruct's submodule generation with :module option currently is not supported.

Installation

To use Domo in a project, add the following line to mix.exs dependencies:

<!-- livebook:{"force_markdown":true} -->
{:domo, "~> 1.5"}

And the following line to the project's mix.exs file:

<!-- livebook:{"force_markdown":true} -->
compilers: [:domo_compiler] ++ Mix.compilers()

To exclude the generated TypeEnsurer modules from mix test --coverage add the following line, that works since Elixir v1.13, to the project's mix.exs:

<!-- livebook:{"force_markdown":true} -->
test_coverage: [ignore_modules: [~r/\.TypeEnsurer$/]]

To avoid mix format putting extra parentheses around precond/1 macro call, add the following import to the .formatter.exs:

<!-- livebook:{"force_markdown":true} -->
[
  import_deps: [:domo]
]

Phoenix hot-reload

To enable Phoenix hot-reload for struct's type ensurers built by Domo, update the compilers in the mix.exs file like the following:

<!-- livebook:{"force_markdown":true} -->
compilers: [:domo_compiler] ++ Mix.compilers() ++ [:domo_phoenix_hot_reload]

And add the following line to the endpoint's configuration in the config.exs file:

<!-- livebook:{"force_markdown":true} -->
config :my_app, MyApp.Endpoint,
  reloadable_compilers: [:phoenix, :domo_compiler] ++ Mix.compilers() ++ [:domo_phoenix_hot_reload]

Umbrella application

Add the Domo dependency and compilers config as mentioned in the section above to the mix.exs file for each app using Domo.

You may add the same domo compiler config line to the app itself and to the root umbrella's mix.exs to enable recompile command to work correctly for iex -S mix run in the root.

If you have the Phoenix hot-reload configured for one of the web apps in umbrella then the :domo_phoenix_hot_reload compiler should be added to all dependency apps used by the given web one.

Configuration

<!-- using_options -->

The options listed below can be set globally in the configuration with config :domo, option: value. The value given with use Domo, option: value overrides the global setting.

Run the Application.put_env(:domo, :verbose_in_iex, true) to enable verbose messages from domo in Interactive Elixir console.

<!-- using_options -->

Limitations

Parametrized types are not supported because it adds lots of complexity. Library returns {:type_not_found, :key} error for @type dict(key, value) :: [{key, value}] definition. Library returns error for type referencing parametrized type like @type field :: container(integer()).

Primitive types referencing themselves are not supported. Library returns an error for @type leaf :: leaf | nil definition. On the other hand, structs referencing themselves are supported. The library will build TypeEnsurer for the following definition @type t :: %__MODULE__{leaf: t | nil} and validate.

Performance 🐢

Compilation-time

Library affects the project's full recompilation time almost insignificantly.

The compilation times for the business application with 38 structs (8 fields each on average) having 158 modules in total are the following:

Mode       Average (by 3 measurements)  Deviation
No Domo    14.826s                      11.92
With Domo  15.711s                      7.34

Comparison: 
No Domo    14.826s 
With Domo  15.711s - 1.06x slower

Run-time

The library ensures the correctness of data types at run-time, which comes with the computation price.

One of the standard tasks is translating the map with string values received as a form into a validated nested struct.

For benchmark, we use Album struct having many Track structs.

When comparing Domo generated new!/1 constructor function and Ecto changeset using a set of validate_.../2 functions internally, both have equivalent execution times. However, in the former case, the memory consumption is about 25% bigger.

In the case of Ecto changeset that uses Domo.Changeset.validate_type/1 function the computation takes about 1.5x times longer.

One of the benchmark results is shown below. The benchmark project is /benchmark_ecto_domo folder. You can run it with mix benchmark command.

Operating System: macOS
CPU Information: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.11.0
Erlang 24.3.3

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 8 s
memory time: 2 s
parallel: 1
inputs: none specified
Estimated total run time: 36 s

Benchmarking Domo Album.new!/1...
Benchmarking Domo.Changeset validate_type/1...
Benchmarking Ecto.Changeset validate_.../1...

Name                                     ips        average  deviation         median         99th %
Domo Album.new!/1                       5.31      188.25 ms     ±1.79%      188.28 ms      196.57 ms
Ecto.Changeset validate_.../1           5.21      191.85 ms     ±1.94%      191.00 ms      202.22 ms
Domo.Changeset validate_type/1          3.76      266.19 ms     ±1.20%      266.58 ms      271.01 ms

Comparison: 
Domo Album.new!/1                       5.31
Ecto.Changeset validate_.../1           5.21 - 1.02x slower +3.59 ms
Domo.Changeset validate_type/1          3.76 - 1.41x slower +77.93 ms

Memory usage statistics:

Name                              Memory usage
Domo Album.new!/1                    245.73 MB
Ecto.Changeset validate_.../1        186.59 MB - 0.76x memory usage -59.13956 MB
Domo.Changeset validate_type/1       238.69 MB - 0.97x memory usage -7.04444 MB

**All measurements for memory usage were the same**

Migration

To migrate to a new version of Domo, please, clean and recompile the project with mix clean --deps && mix compile command.

Adoption

It's possible to adopt Domo library in the project having user-defined constructor functions named new/1 by refactoring to the Domo generated one. Here's how:

  1. Add :domo dependency to the project, configure compilers as described in the installation section
  2. Set the name of the Domo generated constructor function by adding config :domo, :gen_constructor_name, :_new option into the confix.exs file, to prevent conflict with user-defined constructor function name
  3. Add use Domo to existing struct, f.e. FirstStruct
  4. Change all struct building calls to be done with Domo generated function with the name set on step 3 f.e. FistStruct._new(%{...})
  5. Repeat for each struct in the project
  6. Remove original new/1 if it's not needed anymore and rename _new to new in the whole project
  7. Remove config :domo, :gen_constructor_name configuration because Domo generates constructor wiht new name by default.
<!-- Documentation2 -->

Callbacks

Constructor, validation, and reflection functions added to the struct module using Domo.

new!/1/0

<!-- new!/1 -->

Creates a struct validating type conformance and preconditions.

The argument is any Enumerable that emits two-element tuples (key-value pairs) during enumeration.

Returns the instance of the struct built from the given enumerable. Does so only if struct's field values conform to its t() type and all field's type and struct's type precondition functions return ok.

Raises an ArgumentError if conditions described above are not fulfilled.

This function will check if every given key-value belongs to the struct and raise KeyError otherwise.

<!-- new!/1 -->

new/2/1/0

<!-- new/2 -->

Creates a struct validating type conformance and preconditions.

The argument is any Enumerable that emits two-element tuples (key-value pairs) during enumeration.

Returns the instance of the struct built from the given enumerable in the shape of {:ok, struct_value}. Does so only if struct's field values conform to its t() type and all field's type and struct's type precondition functions return ok.

If conditions described above are not fulfilled, the function returns an appropriate error in the shape of {:error, message_by_field}. message_by_field is a keyword list where the key is the name of the field and value is the string with the error message.

Keys in the enumerable that don't exist in the struct are automatically discarded.

Options

<!-- new/2 -->

ensure_type!/1

<!-- ensure_type!/1 -->

Ensures that struct conforms to its t() type and all preconditions are fulfilled.

Returns struct when it's valid. Raises an ArgumentError otherwise.

Useful for struct validation when its fields changed with map syntax or with Map module functions.

<!-- ensure_type!/1 -->

ensure_type/2/1

<!-- ensure_type/2 -->

Ensures that struct conforms to its t() type and all preconditions are fulfilled.

Returns struct when it's valid in the shape of {:ok, struct}. Otherwise returns the error in the shape of {:error, message_by_field}.

Takes the same options as new/2.

Useful for struct validation when its fields changed with map syntax or with Map module functions.

<!-- ensure_type/2 -->

typed_fields/1/0

<!-- typed_fields/1 -->

Returns the list of struct's fields defined with explicit types in its t() type spec.

Does not return meta fields with __underscored__ names and fields having any() type by default.

Includes fields that have nil type into the return list.

Options

<!-- typed_fields/1 -->

required_fields/1/0

<!-- required_fields/1 -->

Returns the list of struct's fields having type others then nil or any().

Does not return meta fields with __underscored__ names.

Useful for validation of the required fields for emptiness. F.e. with validate_required/2 call in the Ecto changeset.

Options

<!-- required_fields/1 -->

__t__/0

<!-- __t__/0 -->

Returns the struct's type spec t() description for debugging purposes.

Domo compiled validation functions for the given struct based on the described type spec.

<!-- __t__/0 -->

Contributing

  1. Fork the repository and make a feature branch

  2. After implementing of the feature format the code with:

    mix format
    

    run linter and tests to ensure that all works as expected with:

    mix check || mix check --failed
    
  3. Make a PR to this repository

Changelog

v1.5.16

v1.5.15

v1.5.14

v1.5.13

v1.5.12

v1.5.11

v1.5.10

v1.5.9 (2022-11-11)

v1.5.8 (2022-09-11)

v1.5.7 (2022-08-06)

v1.5.6 (2022-06-12)

v1.5.5 (2022-06-12)

v1.5.4 (2022-05-23)

v1.5.3 (2022-04-10)

v1.5.2 (2022-01-02)

v1.5.1 (2021-12-12)

v1.5.0 (2021-12-05)

Breaking change:

v1.4.1 (2021-11-16)

v1.4.0 (2021-11-15)

Breaking changes:

v1.3.4 (2021-10-13)

v1.3.3 (2021-10-07)

v1.3.2 (2021-09-18)

v1.3.1 (2021-08-19)

v1.3.0 (2021-08-15)

v1.2.9 (2021-08-09)

v1.2.8 (2021-07-15)

v1.2.7 (2021-07-05)

v1.2.6 (2021-06-21)

v1.2.5 (2021-06-14)

v1.2.4 (2021-06-07)

v1.2.3 (2021-05-31)

v1.2.2 (2021-05-05)

v1.2.1 (2021-04-25)

v1.2.0 (2021-04-12)

v0.0.x - v1.0.x (2020-06-20)

Roadmap

License

Copyright © 2020-2022 Ivan Rublev

This project is licensed under the MIT license.