Awesome

Valix

Valix - Go package for validating requests

Contents

• Overview
• Installation
• Features
• Concepts
• Examples
  • Creating Validators
  • Using Validators
• Constraints
  • Common Constraints
  • Constraint Sets
  • Custom Constraints
  • Constraints Registry
  • Conditional Constraints
• Polymorphic Validation
• Validation Tags
• I18n Support

Overview

Validate JSON requests in the form of *http.Request, map[string]interface{} or []interface{}

Installation

To install Valix, use go get:

go get github.com/marrow16/valix

To update Valix to the latest version, run:

go get -u github.com/marrow16/valix

Features

• Deep validation (define validation for properties where those properties are objects or arrays of objects that also need validating)
• Create validators from structs or define them as code (see Creating Validators)
• Validate http.Request (into struct or map[string]interface{})
• Finds all validation violations - not just the first one! (see Using Validators) and provides information for each violation (property name, path and message)
• Rich set of pre-defined common constraints (see Common Constraints)
• Customisable constraints (see Custom Constraints)
• Conditional constraints to support partial polymorphic request models (see Conditional Constraints)
• Full Polymorphic Validation
• Support for i18n - enabling translation of validation messages (inc. http.Request language and region detection)
• Validators fully marshalable and unmarshalable (save/share validators as JSON)
• Highly extensible (add your own constraints, messages, presets etc. without a PR to this repository)
• 100% tested (see Codecov.io)
• Coming soon - generate validators from OpenApi/Swagger JSON

Concepts

Valix is based on the concept that incoming API requests (such as POST, PUT etc.) should be validated early - against a definition of what the request body should look like.

At validation (of a JSON object as example) the following steps are performed:

• Optionally check any constraints on the overall object (specified in Validator.Constraints)
• Check if there are any unknown properties (unless Validator.IgnoreUnknownProperties is set to true)
• For each defined property in Validator.Properties:
  • Check the property is present (if PropertyValidator.Mandatory is set to true)
  • Check the property value is non-null (if PropertyValidator.NotNull is set to true)
  • Check the property value is of the correct type (if PropertyValidator.Type is set)
  • Check the property value against constraints (specified in PropertyValidator.Constraints)
  • If the property value is an object or array (and PropertyValidator.ObjectValidator is specified) - check the value using the validator (see top of this process)

The validator does not stop on the first problem it finds - it finds all problems and returns them as a list (slice) of 'violations'. Each violation has a message along with the name and path of the property that failed. However, custom constraints can be defined that will either stop the entire validation or cease further validation constraints on the current property

Examples

Creating Validators

Validators can be created from existing structs - adding v8n tags (in conjunction with existing json tags), for example:

package main

import (
    "github.com/marrow16/valix"
)

type AddPersonRequest struct {
    Name string `json:"name" v8n:"notNull,mandatory,constraints:[StringNoControlCharacters{},StringLength{Minimum: 1, Maximum: 255}]"`
    Age int `json:"age" v8n:"type:Integer,notNull,mandatory,constraint:PositiveOrZero{}"`
}
var AddPersonRequestValidator = valix.MustCompileValidatorFor(AddPersonRequest{}, nil)

(see Validation Tags for documentation on v8n tags)

Or in slightly more abbreviated form (using & to denote constraint tokens):

package main

import (
    "github.com/marrow16/valix"
)

type AddPersonRequest struct {
    Name string `json:"name" v8n:"notNull,mandatory,&StringNoControlCharacters{},&StringLength{Minimum: 1, Maximum: 255}"`
    Age int `json:"age" v8n:"type:Integer,notNull,mandatory,&PositiveOrZero{}"`
}
var AddPersonRequestValidator = valix.MustCompileValidatorFor(AddPersonRequest{}, nil)

The valix.MustCompileValidatorFor() function panics if the validator cannot be compiled. If you do not want a panic but would rather see the compilation error instead then use the valix.ValidatorFor() function instead.

Alternatively, Validators can be expressed effectively without a struct, for example:

package main

import (
    "github.com/marrow16/valix"
)

var CreatePersonRequestValidator = &valix.Validator{
    IgnoreUnknownProperties: false,
    Properties: valix.Properties{
        "name": {
            Type:         valix.JsonString,
            NotNull:      true,
            Mandatory:    true,
            Constraints:  valix.Constraints{
                &valix.StringNoControlCharacters{},
                &valix.StringLength{Minimum: 1, Maximum: 255},
            },
        },
        "age": {
            Type:         valix.JsonInteger,
            NotNull:      true,
            Mandatory:    true,
            Constraints:  valix.Constraints{
                &valix.PositiveOrZero{},
            },
        },
    },
}

Additional validator options

Validators can have additional properties that control the overall validation behaviour. These properties are described as follows:

Using Validators

Once a validator has been created (using previous examples in Creating Validators), they can be used in several ways:

Validating a request into a struct

A request *http.Request can be validated into a struct:

package main

import (
    "encoding/json"
    "net/http"

    "github.com/marrow16/valix"
)

func AddPersonHandler(w http.ResponseWriter, r *http.Request) {
    addPersonReq := &AddPersonRequest{}
    ok, violations, _ := CreatePersonRequestValidator.RequestValidateInto(r, addPersonReq)
    if !ok {
        // write an error response with full info - using violations information
        valix.SortViolationsByPathAndProperty(violations)
        errResponse := map[string]interface{}{
            "$error": "Request invalid",
            "$details": violations,
        }
        w.WriteHeader(http.StatusUnprocessableEntity)
        w.Header().Set("Content-Type", "application/json")
        _ = json.NewEncoder(w).Encode(errResponse)
        return
    }
    // the addPersonReq will now be a validated struct
    w.Header().Set("Content-Type", "application/json")
    _ = json.NewEncoder(w).Encode(addPersonReq)
}

Validating a string or reader into a struct

A string, representing JSON, can be validated into a struct:

package main

import (
    "testing"

    "github.com/marrow16/valix"
    "github.com/stretchr/testify/require"
)

func TestValidateStringIntoStruct(t *testing.T) {
    str := `{
        "name": "",
        "age": -1
    }`
    req := &AddPersonRequest{}
    ok, violations, _ := AddPersonRequestValidator.ValidateStringInto(str, req)
    
    require.False(t, ok)
    require.Equal(t, 2, len(violations))
    valix.SortViolationsByPathAndProperty(violations)
    require.Equal(t, "Value must be positive or zero", violations[0].Message)
    require.Equal(t, "age", violations[0].Property)
    require.Equal(t, "", violations[0].Path)
    require.Equal(t, "String value length must be between 1 and 255 (inclusive)", violations[1].Message)
    require.Equal(t, "name", violations[1].Property)
    require.Equal(t, "", violations[1].Path)
    
    str = `{
        "name": "Bilbo Baggins",
        "age": 25
    }`
    ok, violations, _ = AddPersonRequestValidator.ValidateStringInto(str, req)
    
    require.True(t, ok)
    require.Equal(t, 0, len(violations))
    require.Equal(t, "Bilbo Baggins", req.Name)
    require.Equal(t, 25, req.Age)
}

Also, a reader io.Reader can be validated into a struct using the .ValidateReaderIntoStruct() method of the validator:

package main

import (
    "strings"
    "testing"

    "github.com/marrow16/valix"
    "github.com/stretchr/testify/require"
)

func TestValidateReaderIntoStruct(t *testing.T) {
	reader := strings.NewReader(`{
		"name": "",
		"age": -1
	}`)
	req := &AddPersonRequest{}

	ok, violations, _ := AddPersonRequestValidator.ValidateReaderInto(reader, req)

	require.False(t, ok)
	require.Equal(t, 2, len(violations))
	valix.SortViolationsByPathAndProperty(violations)
	require.Equal(t, "Value must be positive or zero", violations[0].Message)
	require.Equal(t, "age", violations[0].Property)
	require.Equal(t, "", violations[0].Path)
	require.Equal(t, "String value length must be between 1 and 255 (inclusive)", violations[1].Message)
	require.Equal(t, "name", violations[1].Property)
	require.Equal(t, "", violations[1].Path)

	reader = strings.NewReader(`{
		"name": "Bilbo Baggins",
		"age": 25
	}`)
	ok, violations, _ = AddPersonRequestValidator.ValidateReaderInto(reader, req)

	require.True(t, ok)
	require.Equal(t, 0, len(violations))
	require.Equal(t, "Bilbo Baggins", req.Name)
	require.Equal(t, 25, req.Age)
}

Validating a map

Validators can also validate a map[string]interface{} representation of a JSON object:

package main

import (
    "testing"

    "github.com/marrow16/valix"
    "github.com/stretchr/testify/require"
)

func TestValidateMap(t *testing.T) {
    req := map[string]interface{}{
        "name": "",
        "age": -1,
    }
  
    ok, violations := AddPersonRequestValidator.Validate(req)
    require.False(t, ok)
    require.Equal(t, 2, len(violations))
    valix.SortViolationsByPathAndProperty(violations)
    require.Equal(t, "Value must be positive or zero", violations[0].Message)
    require.Equal(t, "age", violations[0].Property)
    require.Equal(t, "", violations[0].Path)
    require.Equal(t, "String value length must be between 1 and 255 (inclusive)", violations[1].Message)
    require.Equal(t, "name", violations[1].Property)
    require.Equal(t, "", violations[1].Path)
  
    req = map[string]interface{}{
        "name": "Bilbo Baggins",
        "age": 25,
    }
    ok, _ = AddPersonRequestValidator.Validate(req)
    require.True(t, ok)
}

Validating a slice

Validators can also validate a slice []interface{} representation of a JSON object, where each object element in the slice is validated:

package main

import (
    "testing"

    "github.com/marrow16/valix"
    "github.com/stretchr/testify/require"
)

func TestValidateSlice(t *testing.T) {
    req := []interface{}{
        map[string]interface{}{
            "name": "",
            "age":  -1,
        },
        map[string]interface{}{
            "name": "Bilbo Baggins",
            "age":  25,
        },
    }
  
    ok, violations := AddPersonRequestValidator.ValidateArrayOf(req)
    require.False(t, ok)
    require.Equal(t, 2, len(violations))
    valix.SortViolationsByPathAndProperty(violations)
    require.Equal(t, "Value must be positive or zero", violations[0].Message)
    require.Equal(t, "age", violations[0].Property)
    require.Equal(t, "[0]", violations[0].Path)
    require.Equal(t, "String value length must be between 1 and 255 (inclusive)", violations[1].Message)
    require.Equal(t, "name", violations[1].Property)
    require.Equal(t, "[0]", violations[1].Path)
  
    req = []interface{}{
        map[string]interface{}{
            "name": "Frodo Baggins",
            "age":  20,
        },
        map[string]interface{}{
            "name": "Bilbo Baggins",
            "age":  25,
        },
    }
    ok, _ = AddPersonRequestValidator.ValidateArrayOf(req)
    require.True(t, ok)
}

Constraints

In Valix, a constraint is a particular validation rule that must be satisfied. For a constraint to be used by the validator it must implement the valix.Constraint interface.

Common Constraints

Valix provides a rich set of over 100 pre-defined common constraints (plus many common regex patterns) - see Constraints Reference wiki documentation for full reference with examples.

Constraint Sets

It is not uncommon in APIs for many properties in different requests to share a common set of constraints. For this reason, Valix provides a ConstraintSet - which is itself a Constraint but contains a list of sub-constraints.

type ConstraintSet struct {
    Constraints Constraints
    Message string
}

When checking a ConstraintSet, the contained constraints are checked sequentially but the overall set stops on the first failing constraint.

If a Message is provided (non-empty string) then that message is used for any of the failing constraints - otherwise the individual constraint fail messages are used.

The following is an example of a constraint set which imposes a complex constraint (although one that could probably be more easily achieved using valix.StringPattern)

package main

import (
    "unicode"
    "github.com/marrow16/valix"
)

var MySet = &valix.ConstraintSet{
    Constraints: valix.Constraints{
        &valix.StringTrim{},
        &valix.StringNotEmpty{},
        &valix.StringLength{Minimum: 16, Maximum: 64},
        valix.NewCustomConstraint(func(value interface{}, vcx *valix.ValidatorContext, this *valix.CustomConstraint) (bool, string) {
            if str, ok := value.(string); ok {
                if len(str) == 0 || str[0] < 'A' || str[0] > 'Z' {
                    return false, this.GetMessage(vcx)
                }
            }
            return true, ""
        }, ""),
        &valix.StringCharacters{
            AllowRanges: []unicode.RangeTable{
                {R16: []unicode.Range16{{'0', 'z', 1}}},
            },
            DisallowRanges: []unicode.RangeTable{
                {R16: []unicode.Range16{{0x003a, 0x0040, 1}}},
                {R16: []unicode.Range16{{0x005b, 0x005e, 1}}},
                {R16: []unicode.Range16{{0x0060, 0x0060, 1}}},
            },
        },
    },
    Message: "String value length must be between 16 and 64 chars; must be letters (upper or lower), digits or underscores; must start with an uppercase letter",
}

Constraint sets can also be registered, making them available in v8n struct tags.

Custom Constraints

If you need a constraint for a specific domain validation, there are two ways to do this...

Create a re-usable constraint (which implements the valix.Constraint interface), example:

package main

import (
    "strings"
    "github.com/marrow16/valix"
)

type NoFoo struct {
}

func (c *NoFoo) Check(value interface{}, vcx *valix.ValidatorContext) (bool, string) {
    if str, ok := value.(string); ok {
        return !strings.Contains(str, "foo"), c.GetMessage(vcx)
    }
    return true, ""
}

func (c *NoFoo) GetMessage(tcx I18nContext) string {
    return "Value must not contain \"foo\""
}

Or create a custom constraint on the fly with check function, example:<br> Note: Custom constraints using functions means that the validator cannot be marshalled/unmarshalled

package main

import (
    "strings"
    "github.com/marrow16/valix"
)

var myValidator = &valix.Validator{
    IgnoreUnknownProperties: false,
    Properties: valix.Properties{
        "foo": {
            Type:         valix.JsonString,
            NotNull:      true,
            Mandatory:    true,
            Constraints:  valix.Constraints{
                valix.NewCustomConstraint(func(value interface{}, vcx *valix.ValidatorContext, cc *valix.CustomConstraint) (bool, string) {
                    if str, ok := value.(string); ok {
                        return !strings.Contains(str, "foo"), cc.GetMessage(vcx)
                    }
                    return true, ""
                }, "Value must not contain \"foo\""),
            },
        },
    },
}

Constraints Registry

All of the Valix common constraints are loaded into a registry - the registry enables the v8n tags to reference these.

If you want to make your own custom constraint available for use in v8n tags, it must also be registered. For example:

package main

import (
    "strings"
    "github.com/marrow16/valix"
)

func init() {
    valix.RegisterConstraint(&NoFoo{})
}

// and the constraint can now be used in `v8n` tag...
type MyRequest struct {
    Name string `json:"name" v8n:"&NoFoo{}"`
}

Required/Unwanted Properties

When properties are required, or unwanted, according to the presence of other properties - use the v8n tag tokens required_with:<expr> or unwanted_with:<expr> (abbreviated forms +: and -: respectively).

The simplest expression is just the name of another property - the following example shows making two properties foo and bar mutually inclusive (i.e. if one is present then the other is required):

type ExampleMutuallyInclusive struct {
    Foo string `json:"foo" v8n:"+:bar, +msg:'foo required when bar present'"`
    Bar string `json:"bar" v8n:"+:foo, +msg:'bar required when foo present'"`
}

Or another example, using the unwanted_with: token, to make two properties mutually exclusive (i.e. if one is present then the other must not):

type ExampleMutuallyExclusive struct {
    Foo string `json:"foo" v8n:"-:bar, -msg:'foo and bar are mutually exclusive'"`
    Bar string `json:"bar" v8n:"-:foo, -msg:'foo and bar are mutually exclusive'"`
}

The required_with: and unwanted_with can also use more complex boolean expressions - the following example demonstrates making two out of three properties mutually inclusive but not all three:

type ExampleTwoOfThreeMutuallyInclusive struct {
    Foo string `json:"foo" v8n:"+:(bar || baz) && !(bar && baz), -:bar && baz"`
    Bar string `json:"bar" v8n:"+:(foo || baz) && !(foo && baz), -:foo && baz"`
    Baz string `json:"baz" v8n:"+:(foo || bar) && !(foo && bar), -:foo && bar"`
}

The boolean property expressions can also traverse up and down the object tree (using JSON . path style notation). The following demonstrates:

type ExampleUpAndDownRequired struct {
    Foo string `json:"foo" v8n:"+:sub.foo"`
    Bar string `json:"bar" v8n:"+:sub.bar"`
    Sub struct {
        SubFoo string `json:"foo" v8n:"+:..foo"`
        SubBar string `json:"bar" v8n:"+:..bar"`
    } `json:"sub"`
}

Additional expression functionality notes:

• Boolean operator ^^ (XOr) is also supported
• Path traversal also supports going up to the root object - e.g. /.foo.bar will go up to the root object and then descend down path foo.bar
• Prefixing property name with ~ (tilde) means check a context condition rather than property existence - e.g. ~METHOD_POST checks whether the METHOD_POST condition token has been set in the context
• The +: and -: validation tag tokens correspond to the valix.PropertyValidator.RequiredWith and valix.PropertyValidator.UnwantedWith fields respectively
• Use the valix.ParseExpression or valix.MustParseExpression functions to programmatically parse boolean property expressions
• Unfortunately, array index notation (e.g. sub[0]) is not currently supported (and neither is traversing array values)

Conditional Constraints

Sometimes, the model of JSON requests needs to vary according to some property condition. For example, the following is a beverage order for a tea and coffee shop:

type BeverageOrder struct {
    Type     string `json:"type" v8n:"notNull,required,&StringValidToken{Tokens:['tea','coffee']}"`
    Quantity int    `json:"quantity" v8n:"notNull,required,&Positive{}"`
    // only relevant to type="tea"...
    Blend    string `json:"blend" v8n:"notNull,required,&StringValidToken{Tokens:['Earl Grey','English Breakfast','Masala Chai']}"`
    // only relevant to type="coffee"...
    Roast    string `json:"roast" v8n:"notNull,required,&StringValidToken{Tokens:['light','medium','dark']}"`
}

The above validation will always expect the blend and roast properties to be present and their value to be valid. However, this is not the requirement of the model - we only want:

• the blend property to be present and valid when type="tea"
• the 'roast' property to be present and valid when type="coffee"

These validation requirements can be incorporated by using the 'when conditions':

type BeverageOrder struct {
    Type     string `json:"type" v8n:"notNull,required,order:-1,&StringValidToken{Tokens:['tea','coffee']},&SetConditionFrom{Parent:true}"`
    Quantity int    `json:"quantity" v8n:"notNull,required,&Positive{}"`
    // only relevant to type="tea"...
    Blend    string `json:"blend" v8n:"when:tea,notNull,required,&StringValidToken{Tokens:['Earl Grey','English Breakfast','Masala Chai']}"`
    // only relevant to type="coffee"...
    Roast    string `json:"roast" v8n:"when:coffee,notNull,required,&StringValidToken{Tokens:['light','medium','dark']}"`
}

Note in the above:

• on the Type field:
  • order:-1 means that this property is checked
  • &SetConditionFrom{} sets a validator context condition token from the incoming value of property type<br> therefore, either a validator condition token of tea or coffee will be set<br> Note: the Parent:true means properties at the same level as this will see the condition token
• on the Blend field the when:tea tag has been added - which means the blend property is only checked when there is a validator condition token of tea set
• on the Roast field the when:coffee tag has been added - which means the roast property is only checked when there is a validator condition token of coffee set

However, this second example may still not be strict enough - because it allows the blend property to be present when the type is "coffee" and the roast property to be present when the type is "tea"<br> This can be overcome by using the 'unwanted conditions':

type BeverageOrderStrict struct {
    Type     string `json:"type" v8n:"notNull,required,order:-1,&StringValidToken{Tokens:['tea','coffee']},&SetConditionFrom{Parent:true}"`
    Quantity int    `json:"quantity" v8n:"notNull,required,&Positive{}"`
    // Blend is only relevant when type="tea"...
    Blend    string `json:"blend" v8n:"when:tea,unwanted:!tea,notNull,required,&StringValidToken{Tokens:['Earl Grey','English Breakfast','Masala Chai']}"`
    // Roast is only relevant when type="coffee"...
    Roast    string `json:"roast" v8n:"when:coffee,unwanted:!coffee,notNull,required,&StringValidToken{Tokens:['light','medium','dark']}"`
}

Note in the above:

• the unwanted:!tea tag has been added to the Blend field - which means... "if condition token of tea has not been set then we do not want the blend property to be present"
• the unwanted:!coffee tag has been added to the Roast field - which means... "if condition token of coffee has not been set then we do not want the roast property to be present"

Polymorphic Validation

Sometimes, the model of JSON requests needs to vary completely according to some condition. The previous conditional constraints example can solve some of the conditional variance - but when different properties are required/not-required or same properties have different constraints under different conditions then polymorphic validation becomes necessary.

As an example, if the following requests all need to be validated using a single validator:

{
    "type": "tea",
    "quantity": 1,
    "blend": "Earl Grey|English Breakfast|Masala Chai"
}

{
    "type": "coffee",
    "quantity": 1,
    "roast": "light|medium|dark"
}

{
    "type": "soft",
    "quantity": 1,
    "brand": "Coca Cola",
    "flavor": "Regular|Diet|Zero|Cherry"
}

{
    "type": "soft",
    "quantity": 1,
    "brand": "Tango",
    "flavor": "Orange|Apple|Strawberry|Watermelon|Tropical"
}

A demonstrated solution to this can be see in the Polymorphic example code

Note: Polymorphic validators cannot be derived from struct tags (see Validation Tags) - because structs themselves cannot be polymorphic!

Validation Tags

Valix can read tags from struct fields when building validators. These are the v8n tags, in the format:

type example struct {
    Field string `v8n:"token[,token, ...]"`
}

Where the tokens correspond to various property validation options - as listed here:

Registering your own tag tokens

The v8n tag can also support custom tokens which can be registered using valix.RegisterCustomTagToken. Any registered custom tag tokens can be used in the v8n tag and will be processed when building a validator for a struct using valix.ValidatorFor

An example of how this is used can be found in examples/custom_tag_tokens_test.go

Tag token aliases

If you find that you're using the same v8n tag tokens repeatedly - you can create aliases for these and then just reference the alias using a $ prefix.

An example of how this is used can be found in examples/tag_aliases_test.go

Abbreviating constraints in tags

When specifying constraints in tags, especially with constraint args, the struct tags can become a little verbose. For example:

type MyStruct struct {
    Foo string `json:"foo" v8n:"&StringNoControlCharacters{},&StringUppercase{Message:'Upper only'},&StringLength{Minimum:10,Maximum:20,ExclusiveMin:true}"`
}

To overcome this, there are several things you can do:

1. Where there are no args for the constraint, the {} at the end can be dropped
2. Where the constraint struct has only one field or has a default field (tagged with <code>`v8n:"default"`</code>) then the arg name can be dropped
3. The constraints registry has pre-defined abbreviated forms
4. Constraint arg names can be abbreviated or shortened to closest matching name, e.g.
   1. Message can be abbreviated to Msg or msg (case-insensitive, remove vowels, replace double-characters with single)
   2. Minimum can be shortened to Min or min (or any other variation that matches only one target field name)
   3. ExclusiveMin can be shortened to excMin, exMin, eMin etc.
5. Where a constraint field is a bool and setting it to true - the value (:true) can be omitted

After those steps, the constraint tags would be:

type MyStruct struct {
    Foo string `json:"foo" v8n:"&strnocc,&strupper{'Upper only'},&strlen{stp,min:10,max:20,excMin}"`
}

Internationalisation Support

Valix has full I18n support for translating violation messages - which is both extensible and/or replaceable...

I18n support features:

• Support for both language and region (e.g. en, en-GB, en-US, fr and fr-CA etc.)
• Detection of request Accept-Language header<br>(when using Validator.RequestValidate or Validator.RequestValidateInto)
• Fallback language and region support
  • e.g. if fr-CA was requested but no Canadian specific translation then fr is used
  • e.g. if mt (Maltese) is an unsupported language but you want the fallback language to be it (Italian) then set this in the valix.DefaultFallbackLanguages variable, e.g. valix.DefaultFallbackLanguages["mt"] = "it"
• Default runtime language and region changeable<br>(set vars valix.DefaultLanguage and/or valix.DefaultRegion)
• Built-in valix.DefaultTranslator supports English, French, German, Italian and Spanish
  • more languages and regional variants can be added at runtime
  • replace translator with your own (implementing valix.Translator interface)
• Completely replaceable I18n support (replace variable valix.DefaultI18nProvider with your own)