Home

Awesome

Pagoda: Rapid, easy full-stack web development starter kit in Go

Go Report Card Test License: MIT Go Reference GoT Mentioned in Awesome Go Slack Widget

<p align="center"><img alt="Logo" src="https://user-images.githubusercontent.com/552328/147838644-0efac538-a97e-4a46-86a0-41e3abdf9f20.png" height="200px"/></p>

Table of Contents

Introduction

Overview

Pagoda is not a framework but rather a base starter-kit for rapid, easy full-stack web development in Go, aiming to provide much of the functionality you would expect from a complete web framework as well as establishing patterns, procedures and structure for your web application.

Built on a solid foundation of well-established frameworks and modules, Pagoda aims to be a starting point for any web application with the benefit over a mega-framework in that you have full control over all of the code, the ability to easily swap any frameworks or modules in or out, no strict patterns or interfaces to follow, and no fear of lock-in.

While separate JavaScript frontends have surged in popularity, many prefer the reliability, simplicity and speed of a full-stack approach with server-side rendered HTML. Even the popular JS frameworks all have SSR options. This project aims to highlight that Go templates can be powerful and easy to work with, and interesting frontend libraries can provide the same modern functionality and behavior without having to write any JS at all.

Foundation

While many great projects were used to build this, all of which are listed in the credits section, the following provide the foundation of the back and frontend. It's important to note that you are not required to use any of these. Swapping any of them out will be relatively easy.

Backend

Frontend

Go server-side rendered HTML combined with the projects below enable you to create slick, modern UIs without writing any JavaScript or CSS.

Storage

Originally, Postgres and Redis were chosen as defaults but since the aim of this project is rapid, simple development, it was changed to SQLite which now provides the primary data storage as well as persistent, background task queues. For caching, a simple in-memory solution is provided. If you need to use something like Postgres or Redis, swapping those in can be done quickly and easily. For reference, this branch contains the code that included those (but is no longer maintained).

Screenshots

Inline form validation

<img src="https://user-images.githubusercontent.com/552328/147838632-570a3116-1e74-428f-8bfc-523ed309ef06.png" alt="Inline validation"/>

Switch layout templates, user registration

<img src="https://user-images.githubusercontent.com/552328/147838633-c1b3e4f6-bbfd-44e1-b0be-884d1a83f8f4.png" alt="Registration"/>

Alpine.js modal, HTMX AJAX request

<img src="https://user-images.githubusercontent.com/552328/147838634-4b84c5d5-dc3b-4280-ac12-247ab22184a3.png" alt="Alpine and HTMX"/>

Getting started

Dependencies

Ensure that Go is installed on your system.

Start the application

After checking out the repository, from within the root, simply run make run:

git clone git@github.com:mikestefanello/pagoda.git
cd pagoda
make run

Since this repository is a template and not a Go library, you do not use go get.

By default, you should be able to access the application in your browser at localhost:8000. This can be changed via the configuration.

By default, your data will be stored within the dbs directory. If you ever want to quickly delete all data just remove this directory.

Running tests

To run all tests in the application, execute make test. This ensures that the tests from each package are not run in parallel. This is required since many packages contain tests that connect to the test database which is stored in memory and reset automatically for each package.

Service container

The container is located at pkg/services/container.go and is meant to house all of your application's services and/or dependencies. It is easily extensible and can be created and initialized in a single call. The services currently included in the container are:

A new container can be created and initialized via services.NewContainer(). It can be later shutdown via Shutdown().

Dependency injection

The container exists to faciliate easy dependency-injection both for services within the container as well as areas of your application that require any of these dependencies. For example, the container is automatically passed to the Init() method of your route handlers so that the handlers have full, easy access to all services.

Test dependencies

It is common that your tests will require access to dependencies, like the database, or any of the other services available within the container. Keeping all services in a container makes it especially easy to initialize everything within your tests. You can see an example pattern for doing this here.

Configuration

The config package provides a flexible, extensible way to store all configuration for the application. Configuration is added to the Container as a Service, making it accessible across most of the application.

Be sure to review and adjust all of the default configuration values provided in config/config.yaml.

Environment overrides

Leveraging the functionality of viper to manage configuration, all configuration values can be overridden by environment variables. The name of the variable is determined by the set prefix and the name of the configuration field in config/config.yaml.

In config/config.go, the prefix is set as pagoda via viper.SetEnvPrefix("pagoda"). Nested fields require an underscore between levels. For example:

http:
  port: 1234

can be overridden by setting an environment variable with the name PAGODA_HTTP_PORT.

Environments

The configuration value for the current environment (Config.App.Environment) is an important one as it can influence some behavior significantly (will be explained in later sections).

A helper function (config.SwitchEnvironment) is available to make switching the environment easy, but this must be executed prior to loading the configuration. The common use-case for this is to switch the environment to Test before tests are executed:

func TestMain(m *testing.M) {
    // Set the environment to test
    config.SwitchEnvironment(config.EnvTest)

    // Start a new container
    c = services.NewContainer()

    // Run tests
    exitVal := m.Run()

    // Shutdown the container
    if err := c.Shutdown(); err != nil {
        panic(err)
    }

    os.Exit(exitVal)
}

Database

The database currently used is SQLite but you are free to use whatever you prefer. If you plan to continue using Ent, the incredible ORM, you can check their supported databases here. The database driver is provided by go-sqlite3. A reference to the database is included in the Container if direct access is required.

Database configuration can be found and managed within the config package.

Auto-migrations

Ent provides automatic migrations which are executed on the database whenever the Container is created, which means they will run when the application starts.

Separate test database

Since many tests can require a database, this application supports a separate database specifically for tests. Within the config, the test database can be specified at Config.Database.TestConnection, which is the database connection string that will be used. By default, this will be an in-memory SQLite database.

When a Container is created, if the environment is set to config.EnvTest, the database client will connect to the test database instead and run migrations so your tests start with a clean, ready-to-go database.

When this project was using Postgres, it would automatically drop and recreate the test database. Since the current default is in-memory, that is no longer needed. If you decide to use a test database not in-memory, you can alter the Container initialization code to do this for you.

ORM

As previously mentioned, Ent is the supplied ORM. It can swapped out, but I highly recommend it. I don't think there is anything comparable for Go, at the current time. If you're not familiar with Ent, take a look through their top-notch documentation.

An Ent client is included in the Container to provide easy access to the ORM throughout the application.

Ent relies on code-generation for the entities you create to provide robust, type-safe data operations. Everything within the ent package in this repository is generated code for the two entity types listed below with the exception of the schema declaration.

Entity types

The two included entity types are:

New entity type

While you should refer to their documentation for detailed usage, it's helpful to understand how to create an entity type and generate code. To make this easier, the Makefile contains some helpers.

  1. Ensure all Ent code is downloaded by executing make ent-install.
  2. Create the new entity type by executing make ent-new name=User where User is the name of the entity type. This will generate a file like you can see in ent/schema/user.go though the Fields() and Edges() will be left empty.
  3. Populate the Fields() and optionally the Edges() (which are the relationships to other entity types).
  4. When done, generate all code by executing make ent-gen.

The generated code is extremely flexible and impressive. An example to highlight this is one used within this application:

entity, err := c.ORM.PasswordToken.
    Query().
    Where(passwordtoken.ID(tokenID)).
    Where(passwordtoken.HasUserWith(user.ID(userID))).
    Where(passwordtoken.CreatedAtGTE(expiration)).
    Only(ctx.Request().Context())

This executes a database query to return the password token entity with a given ID that belong to a user with a given ID and has a created at timestamp field that is greater than or equal to a given time.

Sessions

Sessions are provided and handled via Gorilla sessions and configured as middleware in the router located at pkg/handlers/router.go. Session data is currently stored in cookies but there are many options available if you wish to use something else.

Here's a simple example of loading data from a session and saving new values:

func SomeFunction(ctx echo.Context) error {
    sess, err := session.Get(ctx, "some-session-key")
    if err != nil {
        return err
    }
    sess.Values["hello"] = "world"
    sess.Values["isSomething"] = true
    return sess.Save(ctx.Request(), ctx.Response())
}

Encryption

Session data is encrypted for security purposes. The encryption key is stored in configuration at Config.App.EncryptionKey. While the default is fine for local development, it is imperative that you change this value for any live environment otherwise session data can be compromised.

Authentication

Included are standard authentication features you expect in any web application. Authentication functionality is bundled as a Service within services/AuthClient and added to the Container. If you wish to handle authentication in a different manner, you could swap this client out or modify it as needed.

Authentication currently requires sessions and the session middleware.

Login / Logout

The AuthClient has methods Login() and Logout() to log a user in or out. To track a user's authentication state, data is stored in the session including the user ID and authentication status.

Prior to logging a user in, the method CheckPassword() can be used to determine if a user's password matches the hash stored in the database and on the User entity.

Routes are provided for the user to login and logout at user/login and user/logout.

Forgot password

Users can reset their password in a secure manner by issuing a new password token via the method GeneratePasswordResetToken(). This creates a new PasswordToken entity in the database belonging to the user. The actual token itself, however, is not stored in the database for security purposes. It is only returned via the method so it can be used to build the reset URL for the email. Rather, a hash of the token is stored, using bcrypt the same package used to hash user passwords. The reason for doing this is the same as passwords. You do not want to store a plain-text value in the database that can be used to access an account.

Tokens have a configurable expiration. By default, they expire within 1 hour. This can be controlled in the config package. The expiration of the token is not stored in the database, but rather is used only when tokens are loaded for potential usage. This allows you to change the expiration duration and affect existing tokens.

Since the actual tokens are not stored in the database, the reset URL must contain the user and password token ID. Using that, GetValidPasswordToken() will load a matching, non-expired password token entity belonging to the user, and use bcrypt to determine if the token in the URL matches stored hash of the password token entity.

Once a user claims a valid password token, all tokens for that user should be deleted using DeletePasswordTokens().

Routes are provided to request a password reset email at user/password and to reset your password at user/password/reset/token/:user/:password_token/:token.

Registration

The actual registration of a user is not handled within the AuthClient but rather just by creating a User entity. When creating a user, use HashPassword() to create a hash of the user's password, which is what will be stored in the database.

A route is provided for the user to register at user/register.

Authenticated user

The AuthClient has two methods available to get either the User entity or the ID of the user currently logged in for a given request. Those methods are GetAuthenticatedUser() and GetAuthenticatedUserID().

Middleware

Registered for all routes is middleware that will load the currently logged in user entity and store it within the request context. The middleware is located at middleware.LoadAuthenticatedUser() and, if authenticated, the User entity is stored within the context using the key context.AuthenticatedUserKey.

If you wish to require either authentication or non-authentication for a given route, you can use either middleware.RequireAuthentication() or middleware.RequireNoAuthentication().

Email verification

Most web applications require the user to verify their email address (or other form of contact information). The User entity has a field Verified to indicate if they have verified themself. When a user successfully registers, an email is sent to them containing a link with a token that will verify their account when visited. This route is currently accessible at /email/verify/:token and handled by pkg/handlers/auth.go.

There is currently no enforcement that a User must be verified in order to access the application. If that is something you desire, it will have to be added in yourself. It was not included because you may want partial access of certain features until the user verifies; or no access at all.

Verification tokens are JSON Web Tokens generated and processed by the jwt module. The tokens are signed using the encryption key stored in configuration (Config.App.EncryptionKey). It is imperative that you override this value from the default in any live environments otherwise the data can be comprimised. JWT was chosen because they are secure tokens that do not have to be stored in the database, since the tokens contain all of the data required, including built-in expirations. These were not chosen for password reset tokens because JWT cannot be withdrawn once they are issued which poses a security risk. Since these tokens do not grant access to an account, the ability to withdraw the tokens is not needed.

By default, verification tokens expire 12 hours after they are issued. This can be changed in configuration at Config.App.EmailVerificationTokenExpiration. There is currently not a route or form provided to request a new link.

Be sure to review the email section since actual email sending is not fully implemented.

To generate a new verification token, the AuthClient has a method GenerateEmailVerificationToken() which creates a token for a given email address. To verify the token, pass it in to ValidateEmailVerificationToken() which will return the email address associated with the token and an error if the token is invalid.

Routes

The router functionality is provided by Echo and constructed within via the BuildRouter() function inside pkg/handlers/router.go. Since the Echo instance is a Service on the Container which is passed in to BuildRouter(), middleware and routes can be added directly to it.

Custom middleware

By default, a middleware stack is included in the router that makes sense for most web applications. Be sure to review what has been included and what else is available within Echo and the other projects mentioned.

A middleware package is included which you can easily add to along with the custom middleware provided.

Handlers

A Handler is a simple type that handles one or more of your routes and allows you to group related routes together (ie, authentication). All provided handlers are located in pkg/handlers. Handlers also handle self-registering their routes with the router.

Example

The provided patterns are not required, but were designed to make development as easy as possible.

For this example, we'll create a new handler which includes a GET and POST route and uses the ORM. Start by creating a file at pkg/handlers/example.go.

  1. Define the handler type:
type Example struct {
    orm *ent.Client
    *services.TemplateRenderer
}
  1. Register the handler so the router automatically includes it
func init() {
    Register(new(Example))
}
  1. Initialize the handler (and inject any required dependencies from the Container). This will be called automatically.
func (e *Example) Init(c *services.Container) error {
    e.TemplateRenderer = c.TemplateRenderer
    e.orm = c.ORM
    return nil
}
  1. Declare the routes

It is highly recommended that you provide a Name for your routes. Most methods on the back and frontend leverage the route name and parameters in order to generate URLs.

func (e *Example) Routes(g *echo.Group) {
    g.GET("/example", e.Page).Name = "example"
    g.POST("/example", c.PageSubmit).Name = "example.submit"
}
  1. Implement your routes
func (e *Example) Page(ctx echo.Context) error {
    // add your code here
}

func (e *Example) PageSubmit(ctx echo.Context) error {
    // add your code here
}

Errors

Routes can return errors to indicate that something wrong happened. Ideally, the error is of type *echo.HTTPError to indicate the intended HTTP response code. You can use return echo.NewHTTPError(http.StatusInternalServerError), for example. If an error of a different type is returned, an Internal Server Error is assumed.

The error handler is set to the provided Handler in pkg/handlers/error.go in the BuildRouter() function. That means that if any middleware or route return an error, the request gets routed there. This route conveniently constructs and renders a Page which uses the template templates/pages/error.gohtml. The status code is passed to the template so you can easily alter the markup depending on the error type.

Redirects

The pkg/redirect package makes it easy to perform redirects, especially if you provide names for your routes. The Redirect type provides the ability to chain redirect options and also supports automatically handling HTMX redirects for boosted requests.

For example, if your route name is user_profile with a URL pattern of /user/profile/:id, you can perform a redirect by doing:

return redirect.New(ctx).
    Route("user_profile").
    Params(userID).
    Query(queryParams).
    Go()

Testing

Since most of your web application logic will live in your routes, being able to easily test them is important. The following aims to help facilitate that.

The test setup and helpers reside in pkg/handlers/router_test.go.

Only a brief example of route tests were provided in order to highlight what is available. Adding full tests did not seem logical since these routes will most likely be changed or removed in your project.

HTTP server

When the route tests initialize, a new Container is created which provides full access to all of the Services that will be available during normal application execution. Also provided is a test HTTP server with the router added. This means your tests can make requests and expect responses exactly as the application would behave outside of tests. You do not need to mock the requests and responses.

Request / Response helpers

With the test HTTP server setup, test helpers for making HTTP requests and evaluating responses are made available to reduce the amount of code you need to write. See httpRequest and httpResponse within pkg/handlers/router_test.go.

Here is an example how to easily make a request and evaluate the response:

func TestAbout_Get(t *testing.T) {
    doc := request(t).
        setRoute("about").
        get().
        assertStatusCode(http.StatusOK).
        toDoc()
}

Goquery

A helpful, included package to test HTML markup from HTTP responses is goquery. This allows you to use jQuery-style selectors to parse and extract HTML values, attributes, and so on.

In the example above, toDoc() will return a *goquery.Document created from the HTML response of the test HTTP server.

Here is a simple example of how to use it, along with testify for making assertions:

h1 := doc.Find("h1.title")
assert.Len(t, h1.Nodes, 1)
assert.Equal(t, "About", h1.Text())

Pages

The Page is the major building block of your Handler responses. It is a struct type located at pkg/page/page.go. The concept of the Page is that it provides a consistent structure for building responses and transmitting data and functionality to the templates. Pages are rendered with the TemplateRenderer.

All example routes provided construct and render a Page. It's recommended that you review both the Page and the example routes as they try to illustrate all included functionality.

As you develop your application, the Page can be easily extended to include whatever data or functions you want to provide to your templates.

Initializing a new page is simple:

func (c *home) Get(ctx echo.Context) error {
    p := page.New(ctx)
}

Using the echo.Context, the Page will be initialized with the following fields populated:

Flash messaging

Flash messaging functionality is provided within the msg package. It is used to provide one-time status messages to users.

Flash messaging requires that sessions and the session middleware are in place since that is where the messages are stored.

Creating messages

There are four types of messages, and each can be created as follows:

The message string can contain HTML.

Rendering messages

When a flash message is retrieved from storage in order to be rendered, it is deleted from storage so that it cannot be rendered again.

The Page has a method that can be used to fetch messages for a given type from within the template: Page.GetMessages(typ msg.Type). This is used rather than the funcmap because the Page contains the request context which is required in order to access the session data. Since the Page is the data destined for the templates, you can use: {{.GetMessages "success"}} for example.

To make things easier, a template component is already provided, located at templates/components/messages.gohtml. This will render all messages of all types simply by using {{template "messages" .}} either within your page or layout template.

Pager

A very basic mechanism is provided to handle and facilitate paging located in pkg/page/pager.go. When a Page is initialized, so is a Pager at Page.Pager. If the requested URL contains a page query parameter with a numeric value, that will be set as the page number in the pager.

During initialization, the items per page amount will be set to the default, controlled via constant, which has a value of 20. It can be overridden by changing Pager.ItemsPerPage but should be done before other values are set in order to not provide incorrect calculations.

Methods include:

There is currently no template (yet) to easily render a pager.

CSRF

By default, all non GET requests will require a CSRF token be provided as a form value. This is provided by middleware and can be adjusted or removed in the router.

The Page will contain the CSRF token for the given request. There is a CSRF helper component template which can be used to easily render a hidden form element in your form which will contain the CSRF token and the proper element name. Simply include {{template "csrf" .}} within your form.

Automatic template parsing

Dealing with templates can be quite tedious and annoying so the Page aims to make it as simple as possible with the help of the template renderer. To start, templates for pages are grouped in the following directories within the templates directory:

Specifying which templates to render for a given Page is as easy as:

page.Name = "home"
page.Layout = "main"

That alone will result in the following templates being parsed and executed when the Page is rendered:

  1. layouts/main.gohtml as the base template
  2. pages/home.gohtml to provide the content template for the layout
  3. All template files located within the components directory
  4. The entire funcmap

The template renderer also provides caching and local hot-reloading.

Cached responses

A Page can have cached enabled just by setting Page.Cache.Enabled to true. The TemplateRenderer will automatically handle caching the HTML output, headers and status code. Cached pages are stored using a key that matches the full request URL and middleware is used to serve it on matching requests.

By default, the cache expiration time will be set according to the configuration value located at Config.Cache.Expiration.Page but it can be set per-page at Page.Cache.Expiration.

Cache tags

You can optionally specify cache tags for the Page by setting a slice of strings on Page.Cache.Tags. This provides the ability to build in cache invalidation logic in your application driven by events such as entity operations, for example.

You can use the cache client on the Container to easily flush cache tags, if needed.

Cache middleware

Cached pages are served via the middleware ServeCachedPage() in the middleware package.

The cache is bypassed if the requests meet any of the following criteria:

  1. Is not a GET request
  2. Is made by an authenticated user

Cached pages are looked up for a key that matches the exact, full URL of the given request.

Data

The Data field on the Page is of type any and is what allows your route to pass whatever it requires to the templates, alongside the Page itself.

Forms

The Form field on the Page is similar to the Data field, but it's meant to store a struct that represents a form being rendered on the page.

An example of this pattern is:

type ContactForm struct {
    Email      string `form:"email" validate:"required,email"`
    Message    string `form:"message" validate:"required"`
    form.Submission
}

Embedding form.Submission satisfies the form.Form interface and makes dealing with submissions and validation extremely easy.

Then in your page:

p := page.New(ctx)
p.Form = form.Get[ContactForm](ctx)

This will either initialize a new form to be rendered, or load one previously stored in the context (ie, if it was already submitted). How the form gets populated with values so that your template can render them is covered in the next section.

Submission processing

Form submission processing is made extremely simple by leveraging functionality provided by Echo binding, validator and the Submission struct located in pkg/form/submission.go.

Using the example form above, this is all you would have to do within the POST callback for your route:

Start by submitting the form along with the request context. This will:

  1. Store a pointer to the form so that your GET callback can access the form values (shown previously). That allows the form to easily be re-rendered with any validation errors it may have as well as the values that were provided.
  2. Parse the input in the POST data to map to the struct so the fields becomes populated. This uses the form struct tags to map form input values to the struct fields.
  3. Validate the values in the struct fields according to the rules provided in the optional validate struct tags.
var input ContactForm

err := form.Submit(ctx, &input)

Check the error returned, and act accordingly. For example:

switch err.(type) {
case nil:
    // All good!
case validator.ValidationErrors:
    // The form input was not valid, so re-render the form
    return c.Page(ctx)
default:
    // Request failed, show the error page
    return err
}

And finally, your template:

<form id="contact" method="post" hx-post="{{url "contact.post"}}">
    <input id="email" name="email" type="email" class="input" value="{{.Form.Email}}">
    <input id="message" name="message" type="text" class="input" value="{{.Form.Message}}">
</form

Inline validation

The Submission makes inline validation easier because it will store all validation errors in a map, keyed by the form struct field name. It also contains helper methods that your templates can use to provide classes and extract the error messages.

While validator is a great package that is used to validate based on struct tags, the downside is that the messaging, by default, is not very human-readable or easy to override. Within Submission.setErrorMessages() the validation errors are converted to more readable messages based on the tag that failed validation. Only a few tags are provided as an example, so be sure to expand on that as needed.

To provide the inline validation in your template, there are two things that need to be done.

First, include a status class on the element so it will highlight green or red based on the validation:

<input id="email" name="email" type="email" class="input {{.Form.GetFieldStatusClass "Email"}}" value="{{.Form.Email}}">

Second, render the error messages, if there are any for a given field:

{{template "field-errors" (.Form.GetFieldErrors "Email")}}

Headers

HTTP headers can be set either via the Page or the context:

p := page.New(ctx)
p.Headers["HeaderName"] = "header-value"

ctx.Response().Header().Set("HeaderName", "header-value")

Status code

The HTTP response status code can be set either via the Page or the context:

p := page.New(ctx)
p.StatusCode = http.StatusTooManyRequests

ctx.Response().Status = http.StatusTooManyRequests

Metatags

The Page provides the ability to set basic HTML metatags which can be especially useful if your web application is publicly accessible. Only fields for the description and keywords are provided but adding additional fields is very easy.

p := page.New(ctx)
p.Metatags.Description = "The page description."
p.Metatags.Keywords = []string{"Go", "Software"}

A component template is included to render metatags in core.gohtml which can be used by adding {{template "metatags" .}} to your layout.

URL and link generation

Generating URLs in the templates is made easy if you follow the routing patterns and provide names for your routes. Echo provides a Reverse function to generate a route URL with a given route name and optional parameters. This function is made accessible to the templates via funcmap function url.

As an example, if you have route such as:

e.GET("/user/profile/:user", handler.Get).Name = "user_profile"

And you want to generate a URL in the template, you can:

{{url "user_profile" 1}

Which will generate: /user/profile/1

There is also a helper function provided in the funcmap to generate links which has the benefit of adding an active class if the link URL matches the current path. This is especially useful for navigation menus.

{{link (url "user_profile" .AuthUser.ID) "Profile" .Path "extra-class"}}

Will generate:

<a href="/user/profile/1" class="is-active extra-class">Profile</a>

Assuming the current path is /user/profile/1; otherwise the is-active class will be excluded.

HTMX support

HTMX is an awesome JavaScript library allows you to access AJAX, CSS Transitions, WebSockets and Server Sent Events directly in HTML, using attributes, so you can build modern user interfaces with the simplicity and power of hypertext.

Many examples of its usage are available in the included examples:

All of this can be easily accomplished without writing any JavaScript at all.

Another benefit of HTMX is that it's completely backend-agnostic and does not require any special tools or integrations on the backend. But to make things easier, included is a small package to read and write HTTP headers that HTMX uses to communicate additional information and commands.

The htmx package contains the headers for the request and response. When a Page is initialized, Page.HTMX.Request will also be initialized and populated with the headers that HTMX provides, if HTMX made the request. This allows you to determine if HTMX is making the given request and what exactly it is doing, which could be useful both in your route as well as your templates.

If you need to set any HTMX headers in your Page response, this can be done by altering Page.HTMX.Response.

Layout template override

To facilitate easy partial rendering for HTMX requests, the Page will automatically change your Layout template to use htmx.gohtml, which currently only renders {{template "content" .}}. This allows you to use an HTMX request to only update the content portion of the page, rather than the entire HTML.

This override only happens if the HTMX request being made is not a boost request because boost requests replace the entire body element so there is no need to do a partial render.

Conditional processing / rendering

Since HTMX communicates what it is doing with the server, you can use the request headers to conditionally process in your route or render in your template, if needed. If your routes aren't doing multiple things, you may not need this, but it's worth knowing how flexible you can be.

A simple example of this:

if page.HTMX.Request.Target == "search" {
    // You know this request HTMX is fetching content just for the #search element
}

{{if eq .HTMX.Request.Target "search"}}
    // Render content for the #search element
{{end}}

CSRF token

If CSRF protection is enabled, the token value will automatically be passed to HTMX to be included in all non-GET requests. This is done in the footer template by leveraging HTMX events.

Rendering the page

Once your Page is fully built, rendering it via the embedded TemplateRenderer in your handler can be done simply by calling RenderPage():

func (c *home) Get(ctx echo.Context) error {
    p := page.New(ctx)
    p.Layout = templates.LayoutMain
    p.Name = templates.PageHome
    return c.RenderPage(ctx, p)
}

Template renderer

The template renderer is a Service on the Container that aims to make template parsing and rendering easy and flexible. It is the mechanism that allows the Page to do automatic template parsing. The standard html/template is still the engine used behind the scenes. The code can be found in pkg/services/template_renderer.go.

Here is an example of a complex rendering that uses multiple template files as well as an entire directory of template files:

buf, err = c.TemplateRenderer.
    Parse().
    Group("page").
    Key("home").
    Base("main").
    Files("layouts/main", "pages/home").
    Directories("components").
    Execute(data)

This will do the following:

Using the example from the page rendering, this is will execute:

buf, err = c.TemplateRenderer.
    Parse().
    Group("page").
    Key(page.Name).
    Base(page.Layout).
    Files(
        fmt.Sprintf("layouts/%s", page.Layout),
        fmt.Sprintf("pages/%s", page.Name),
    ).
    Directories("components").
    Execute(page)

If you have a need to separately parse and cache the templates then later execute, you can separate the operations:

_, err := c.TemplateRenderer.
    Parse().
    Group("my-group").
    Key("my-key").
    Base("auth").
    Files("layouts/auth", "pages/login").
    Directories("components").
    Store()

tpl, err := c.TemplateRenderer.Load("my-group", "my-key")
buf, err := tpl.Execute(data)

Custom functions

All templates will be parsed with the funcmap so all of your custom functions as well as the functions provided by sprig will be available.

Caching

Parsed templates will be cached within a sync.Map so the operation will only happen once per cache group and ID. Be careful with your cache group and ID parameters to avoid collisions.

Hot-reload for development

If the current environment is set to config.EnvLocal, which is the default, the cache will be bypassed and templates will be parsed every time they are requested. This allows you to have hot-reloading without having to restart the application so you can see your HTML changes in the browser immediately.

File configuration

To make things easier and less repetitive, parameters given to the template renderer must not include the templates directory or the template file extensions. The file extension is stored as a constant (TemplateExt) within the config package.

Funcmap

The funcmap package provides a function map (template.FuncMap) which will be included for all templates rendered with the template renderer. Aside from a few custom functions, sprig is included which provides over 100 commonly used template functions. The full list is available here.

To include additional custom functions, add to the map in NewFuncMap() and define the function in the package. It will then become automatically available in all templates.

Cache

As previously mentioned, the default cache implementation is a simple in-memory store, backed by otter, a lockless cache that uses S3-FIFO eviction. The Container houses a CacheClient which is a useful, wrapper to interact with the cache (see examples below). Within the CacheClient is the underlying store interface CacheStore. If you wish to use a different store, such as Redis, and want to keep using the CacheClient, simply implement the CacheStore interface with a Redis library and adjust the Container initialization to use that.

The built-in usage of the cache is currently only for optional page caching and a simple example route located at /cache where you can set and view the value of a given cache entry.

Since the current cache is in-memory, there's no need to adjust the Container during tests. When this project used Redis, the configuration had a separate database that would be used strictly for tests to avoid writing to your primary database. If you need that functionality, it is easy to add back in.

Set data

Set data with just a key:

err := c.Cache.
    Set().
    Key("my-key").
    Data(myData).
    Expiration(time.Hour * 2).
    Save(ctx)

Set data within a group:

err := c.Cache.
    Set().
    Group("my-group").
    Key("my-key").
    Expiration(time.Hour * 2).
    Data(myData).
    Save(ctx)

Include cache tags:

err := c.Cache.
    Set().
    Key("my-key").
    Tags("tag1", "tag2").
    Expiration(time.Hour * 2).
    Data(myData).
    Save(ctx)

Get data

data, err := c.Cache.
    Get().
    Group("my-group").
    Key("my-key").
    Fetch(ctx)

Flush data

err := c.Cache.
    Flush().
    Group("my-group").
    Key("my-key").
    Execute(ctx)

Flush tags

This will flush all cache entries that were tagged with the given tags.

err := c.Cache.
    Flush().
    Tags("tag1", "tag2").
    Execute(ctx)

Tagging

As shown in the previous examples, cache tags were provided because they can be convenient. However, maintaining them comes at a cost and it may not be a good fit for your application depending on your needs. When including tags, the CacheClient must lock in order to keep the tag index in sync. And since the tag index cannot support eviction, since that could result in a flush call not actually flushing the tag's keys, the maps that provide the index do not have a size limit. See the code for more details.

Tasks

Tasks are queued operations to be executed in the background, either immediately, at a specfic time, or after a given amount of time has passed. Some examples of tasks could be long-running operations, bulk processing, cleanup, notifications, etc.

Since we're already using SQLite for our database, it's available to act as a persistent store for queued tasks so that tasks are never lost, can be retried until successful, and their concurrent execution can be managed. Backlite is the library chosen to interface with SQLite and handle queueing tasks and processing them asynchronously. I wrote that specifically to address the requirements I wanted to satisfy for this project.

To make things easy, the Backlite client is provided as a Service on the Container which allows you to register queues and add tasks.

Configuration for the Backlite client is exposed through the app's yaml configuration. The required database schema will be automatically installed when the app starts.

Queues

A full example of a queue implementation can be found in pkg/tasks with an interactive form to create a task and add to the queue at /task (see pkg/handlers/task.go). Also refer to the Backlite documentation for reference and examples.

See pkg/tasks/register.go for a simple way to register all of your queues and to easily pass the Container to them so the queue processor callbacks have access to all of your app's dependencies.

Dispatcher

The task dispatcher is what manages the worker pool used for executing tasks in the background. It monitors incoming and scheduled tasks and handles sending them to the pool for execution by the queue's processor callback. This must be started in order for this to happen. In cmd/web/main.go, the task dispatcher is automatically started when the app starts via:

c.Tasks.Start(ctx)

The app configuration contains values to configure the client and dispatcher including how many goroutines to use, when to release stuck tasks back into the queue, and how often to cleanup retained tasks in the database.

When the app is shutdown, the dispatcher is given 10 seconds to wait for any in-progress tasks to finish execution. This can be changed in cmd/web/main.go.

Cron

By default, no cron solution is provided because it's very easy to add yourself if you need this. You can either use a ticker or a library.

Static files

Static files are currently configured in the router (pkg/handler/router.go) to be served from the static directory. If you wish to change the directory, alter the constant config.StaticDir. The URL prefix for static files is /files which is controlled via the config.StaticPrefix constant.

Cache control headers

Static files are grouped separately so you can apply middleware only to them. Included is a custom middleware to set cache control headers (middleware.CacheControl) which has been added to the static files router group.

The cache max-life is controlled by the configuration at Config.Cache.Expiration.StaticFile and defaults to 6 months.

Cache-buster

While it's ideal to use cache control headers on your static files so browsers cache the files, you need a way to bust the cache in case the files are changed. In order to do this, a function is provided in the funcmap to generate a static file URL for a given file that appends a cache-buster query. This query string is randomly generated and persisted until the application restarts.

For example, to render a file located in static/picture.png, you would use:

<img src="{{file "picture.png"}}"/>

Which would result in:

<img src="/files/picture.png?v=9fhe73kaf3"/>

Where 9fhe73kaf3 is the randomly-generated cache-buster.

Email

An email client was added as a Service to the Container but it is just a skeleton without any actual email-sending functionality. The reason is because there are a lot of ways to send email and most prefer using a SaaS solution for that. That makes it difficult to provide a generic solution that will work for most applications.

The structure in the client (MailClient) makes composing emails very easy and you have the option to construct the body using either a simple string or with a template by leveraging the template renderer. The standard library can be used if you wish to send email via SMTP and most SaaS providers have a Go package that can be used if you choose to go that direction. You must finish the implementation of MailClient.send.

The from address will default to the configuration value at Config.Mail.FromAddress. This can be overridden per-email by calling From() on the email and passing in the desired address.

See below for examples on how to use the client to compose emails.

Sending with a string body:

err = c.Mail.
    Compose().
    To("hello@example.com").
    Subject("Welcome!").
    Body("Thank you for registering.").
    Send(ctx)

Sending with a template body:

err = c.Mail.
    Compose().
    To("hello@example.com").
    Subject("Welcome!").
    Template("welcome").
    TemplateData(templateData).
    Send(ctx)

This will use the template located at templates/emails/welcome.gohtml and pass templateData to it.

HTTPS

By default, the application will not use HTTPS but it can be enabled easily. Just alter the following configuration:

To use Let's Encrypt follow this guide.

Logging

By default, the Echo logger is not used for the following reasons:

  1. It does not support structured logging, which makes it difficult to deal with variables, especially if you intend to store a logger in context with pre-populated attributes.
  2. The upcoming v5 release of Echo will not contain a logger.
  3. It should be easy to use whatever logger you prefer (even if that is Echo's logger).

The provided implementation uses the relatively new log/slog library which was added to Go in version 1.21 but swapping that out for whichever you prefer is very easy.

Context

The simple pkg/log package provides the ability to set and get a logger from the Echo context. This is especially useful when you use the provided logger middleware (see below). If you intend to use a different logger, modify these methods to receive and return the logger of your choice.

Usage

Adding a logger to the context:

logger := slog.New(logHandler).With("id", requestId)
log.Set(ctx, logger)

Access and use the logger:

func (h *handler) Page(ctx echo.Context) error {
    log.Ctx(ctx).Info("send a message to the log",
      "value_one", valueOne,
      "value_two", valueTwo,
    )
}

Log level

When the Container configuration is initialized (initConfig()), the slog default log level is set based on the environment. INFO is used for production and DEBUG for everything else.

Middleware

The SetLogger() middleware has been added to the router which sets an initialized logger on the request context. It's recommended that this remains after Echo's RequestID() middleware because it will add the request ID to the logger which means that all logs produced for that given request will contain the same ID, so they can be linked together. If you want to include more attributes on all request logs, set those fields here.

The LogRequest() middleware is a replacement for Echo's Logger() middleware which produces a log of every request made, but uses our logger rather than Echo's.

2024/06/15 09:07:11 INFO GET /contact request_id=gNblvugTKcyLnBYPMPTwMPEqDOioVLKp ip=::1 host=localhost:8000 referer="" status=200 bytes_in=0 bytes_out=5925 latency=107.527803ms

Roadmap

Future work includes but is not limited to:

Credits

Thank you to all of the following amazing projects for making this possible.