Awesome

Phoenix Content Negotiation Tutorial

A tutorial showing how to return different content (format) for the same route based on Accepts header.

Why? 🤷

As a small team of software engineers, we don't have resources (time) to maintain two separate applications (one for our App and another for an API) the way some larger companies do. We need to focus on building features that people using our products want/need. We want to be able to ship our Web UI and a corresponding feature-complete REST API in the same Phoenix App. This way everyone using our App has a "default" UI/UX (the server-rendered client-enhanced Phoenix Web UI) while simultaneously giving people who want/need API access, exactly what they need from day 1. We know from experience that Apps that focus on UI and leave the API for "later" end up producing a poor API experience. We want to avoid that at all costs. People who want to use the @dwyl API exclusively and never look at the web UI, should always be able to do that. If someone wants to use @dwyl from their CLI they should be able to use 100% of the features. If they want to add items to their lists via IFTTT or Zapier they should be able to do that without any obstacles.

The only way to achieve feature parity between our UI and API is by making the API a "first class citizen" and requiring every feature we build to render both HTML and JSON. Building our app with Content Negotiation baked in guarantees that anyone can use their creativity to build any UI/UX to interface with their data. It also ensures that we have 100% accessibility because any device can access the data. We believe this is a more inclusive way to build Apps even if it adds a 5-10% more "work" up-front, it's 100% worth it for achieving our mission! By combining the Web UI and API into the same Phoenix Application, we only have one thing to focus on, deploy, scale and maintain.

This tutorial shows how simple it is to turn any Phoenix Web App into a REST API using the same routes as your Web UI.

Goal? 🎯

Our goal is: to run the same Phoenix Application for both our Web UI and REST API and have the same route handler (Controller) transparently return the appropriate content (HTML or JSON) based on the Accept header.

So a request made in a Web Browser will display HTML whereas a cURL command in a terminal (or request from any other Frontend-only App) will return JSON for the same URL.

That way we ensure that all routes in our App have the equivalent JSON response so every action can be performed programatically. Which means anyone can build their own Frontend UI/UX for the @dwyl App. We believe this is crucial to the success of our product. We think the API is our Product and the Web UI is just one representation of what is possible to build with the API.

What? 💡

This tutorial shows how to do content negotiation in a Phoenix App from first principals. <br /> If you just want to implement content negotiation in your project as fast as possible see: github.com/dwyl/content. <br /> We still recommend following this tutorial as it only takes 20 mins and will ensure you understand how to do it from scratch.

Context

In our App we want to ensure that all requests that can be made in the Web UI have a corresponding JSON response without any duplication of effort. We definitely don't want to have to run/maintain two separate Phoenix Apps as we know (from experience) that the functionality will diverge almost immediately as a contributor who is building their own UI will make an API-focussed addition and forget to add the corresponding web UI (or vice versa). We don't want to have to "police" the PRs or force anyone to have to write the same code twice. We want a JSON response to be automatically available for every route and never have to think about it. We want anyone to be able to build an App/UI using our API.

What Is Content Negotiation? 💭

Content negotiation is the process of selecting the best representation for a given response when there are multiple representations available." ~ RFC 2616/7231

The gist is that depending on the Accept header specified by the requesting agent (e.g. a Web Browser or script), a different representation of the content can be returned.

If the concept of HTTP content negotiation is new to you, we suggest you read the detailed article on MDN (5 mins): https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation

What Are We Building? ✨

The aim of this tutorial is to demonstrate content negotiation in a real-world scenario. <br /> We are going to build a simple interface to display famous quotations, both a basic Web UI and REST API. <br /> When we visit: / in a browser we see a random quotation rendered as HTML. <br /> When we curl the same endpoint, we see a JSON representation.

Try It! 💻

Before you attempt to follow the example, Try the Heroku example version so you know what to expect.

Browser 📱

Visit: https://phoenix-content-negotiation.herokuapp.com

You should see a random inspiring quote:

Terminal ⬛

Run the following command:

curl -H "Accept: application/json" https://phoenix-content-negotiation.herokuapp.com

You should see a random quote as JSON:

Who? 👤

This example is aimed at anyone building a Phoenix App who wants to automatically have a REST API. <br /> For us @dwyl who are building our API and App Web UI simultaneously, it serves as a gentle intro to the topic.

If you get stuck or have any questions, please ask.

How? 💻

Prerequisites? ✅

This example assumes you have Elixir and Phoenix installed on your computer and that you have some basic familiarity with the language and framework respectively. If you are totally new to either of these, we recommend you first read: github.com/dwyl/learn-elixir and github.com/dwyl/learn-phoenix-framework

Ideally follow the "Chat" example for more detailed step-by-step introduction to Phoenix: github.com/dwyl/phoenix-chat-example

Once you are comfortable with Phoenix, proceed with this example!

0. Run the Finished App ⬇️

We encourage everyone to "Begin With the End in Mind" so suggest that you run finished App on your localhost before attempting to build it. Seeing the App working on your machine will give you confidence that we will achieve our objectives (defined above) and it's a good reference if you get stuck.

Clone the Repository 📋

In your terminal, clone the repo from GitHub:

git clone git@github.com:dwyl/phoenix-content-negotiation-tutorial.git

Install The Dependencies 📦

Change into the newly created directory and run the mix command:

cd phoenix-content-negotiation-tutorial
mix deps.get

Run the App 🚀

Run the Phoenix app with the following command:

mix phx.server

You should see output similar to the following in your terminal:

[info] Running AppWeb.Endpoint with cowboy 2.7.0 at 0.0.0.0:4000 (http)
[info] Access AppWeb.Endpoint at http://localhost:4000

Test it in your Browser 🖥️

Visit: http://localhost:4000

You should see a random motivational quote like this:

Test it in your Terminal ⬛

In your terminal, run the following curl command:

curl -H "Accept: application/json" http://localhost:4000

You should see a random quote:

Now that you know the end state of the tutorial works, change out of the directory (cd ..) and let's re-create it from scratch!

1. Create New Phoenix App

In your terminal, run the following command to create a new app:

mix phx.new app --no-ecto --no-webpack

When asked if you want to Fetch and install dependencies? [Yn] Type <kbd>Y</kbd> followed by the <kbd>Enter</kbd> key.

Note: This example only needs the bare minimum Phoenix; we don't need any JavaScript or Database. <br /> For more info, see: https://hexdocs.pm/phoenix/Mix.Tasks.Phx.New.html <br /> The beauty is that this simple use-case is identical to the advanced one. Once you understand these basic principals, you "grock" how to use Content Negotiation in a more advanced app.

Note 2: We default to calling all our apps "App" for simplicity. Some people prefer other more elaborate names. We like this one.

Note 3: We have deliberately made this API "read only", again for simplicity. If you want to extend this tutorial to allow for creating new quotes both via UI and API, please open an issue. We think it could be a good idea to add POST endpoints as a "Bonus Level", but we don't want to complicate things for the first part of the tutorial.

Change into the app directory (cd app) and open the project in your text editor (or IDE). <br /> e.g: atom .

1.1 Check That Everything Works

Before diving in to adding any features to our app, let's check that it works. <br /> Run the server in your terminal:

mix phx.server

Then visit localhost:4000 in your web browser. <br /> You should see something like this (the default Phoenix home page):

Having confirmed that the UI works, let's run the tests:

mix test

You should see the following output in your terminal:

Generated app app
...

Finished in 0.02 seconds
3 tests, 0 failures

2. Add Quotes!

In order to display quotes in the UI/API we need a source of quotes. Here's one we made earlier: https://hex.pm/packages/quotes

As per the instructions: https://github.com/dwyl/quotes#elixir add the quotes dependency to mix.exs:

{:quotes, "~> 1.0.5"}

e.g mix.exs#L47

Then run:

mix deps.get

That will download the quotes package which contains the quotes.json file and Elixir functions to interact with it.

2.1 Try It in iex!

In your terminal type:

iex -S mix

In the iex prompt type: Quotes.random() you will see a random quote.

iex> Quotes.random()
%{
  "author" => "Lao Tzu",
  "text" => "If you would take, you must first give, this is the beginning of intelligence."
}

Great! So we know our quotes library is loaded into our Phoenix App. <br /> Quit iex and let's get back to building the App.

3. Generate the Quotes Controller, View, Templates and Tests

mix phx.gen.html Ctx Quotes quotes author:string text:string tags:string source:string --no-schema --no-context

Note: Ctx is just an abbreviation for Context. We will remove all references to Ctx in step 3.3 (below) because we really don't need a Context abstraction in a simple example like this. ✂️

In your terminal, you should see the following output:

* creating lib/app_web/controllers/quotes_controller.ex
* creating lib/app_web/templates/quotes/edit.html.eex
* creating lib/app_web/templates/quotes/form.html.eex
* creating lib/app_web/templates/quotes/index.html.eex
* creating lib/app_web/templates/quotes/new.html.eex
* creating lib/app_web/templates/quotes/show.html.eex
* creating lib/app_web/views/quotes_view.ex
* creating test/app_web/controllers/quotes_controller_test.exs

Add the resource to your browser scope in lib/app_web/router.ex:

    resources "/quotes", QuotesController

Git commit of files created in this step: 9a37b21

3.1 Add the Quotes Resources to lib/app_web/router.ex

Let's follow the instructions given by the output of the mix phx.gen.html command to add the resources to lib/app_web/router.ex.

Open the router.ex file and locate the scope "/", AppWeb do block:

scope "/", AppWeb do
  pipe_through :browser

  get "/", PageController, :index
end

add the following line to the block:

resources "/quotes", QuotesController

Your router.ex file should now look like this: router.ex#L20

3.2 Tidy Up: Delete Unused Files (Optional)

The mix phx.gen.html command creates a bunch of files that are useful for "CRUD". In our case we are not going to be creating or editing any quotes as we already have our "bank" of quotes. For simplicity we don't want to run a Database for this example so we can focus on rendering the content and not the "management".

Let's delete the files we don't need so our project is tidy:

rm lib/app_web/templates/quotes/edit.html.eex
rm lib/app_web/templates/quotes/form.html.eex
rm lib/app_web/templates/quotes/new.html.eex
rm lib/app_web/templates/quotes/show.html.eex

Commit: 2d4ca13

3.3 Compilation Error ... 🤷‍

Sadly, this mix phx.gen command does not do exactly what we expect. <br /> The --no-context flag does not create a context.ex file, but the quotes_controller.ex#L4-L5 still has references to Ctx and expects there to be an "implementation" of a Context. That means that if we attempt to run the tests now they will fail:

mix test

You will see the following compilation error:

Compiling 18 files (.ex)

== Compilation error in file lib/app_web/controllers/quotes_controller.ex ==
** (CompileError) lib/app_web/controllers/quotes_controller.ex:13:
App.Ctx.Quotes.__struct__/1 is undefined, cannot expand struct App.Ctx.Quotes.
Make sure the struct name is correct. If the struct name exists and is correct
but it still cannot be found, you likely have cyclic module usage in your code
    (stdlib 3.11.2) lists.erl:1354: :lists.mapfoldl/3
    lib/app_web/controllers/quotes_controller.ex:12: (module)
    (stdlib 3.11.2) erl_eval.erl:680: :erl_eval.do_apply/6

We opened an issue to clarify the behaviour: https://github.com/phoenixframework/phoenix/issues/3832 <br /> Turns out that "generators are first and foremost learning tools", fair enough. <br /> If the generator doesn't do exactly what we expect, we just work around it.

Let's make a few of quick updates to the quotes_controller_test.exs, quotes_controller.ex and index.html.eex files to avoid this compilation error.

The tests created by mix phx.gen.html assume we are building a standard "CRUD" interface; we aren't. So we need to delete those irrelevant tests and replace them. Open the file test/app_web/controllers/quotes_controller_test.exs and replace the contents with the following code:

defmodule AppWeb.QuotesControllerTest do
  use AppWeb.ConnCase

  describe "/quotes" do
    test "shows a random quote", %{conn: conn} do
      conn = get(conn, Routes.quotes_path(conn, :index))
      assert html_response(conn, 200) =~ "Quote"
    end
  end

end

Before: quotes_controller_test.exs <br /> After: quotes_controller_test.exs

Open the lib/app_web/controllers/quotes_controller.ex and replace the contents with the following:

defmodule AppWeb.QuotesController do
  use AppWeb, :controller

  # transform map with keys as strings into keys as atoms!
  # https://stackoverflow.com/questions/31990134
  def transform_string_keys_to_atoms(map) do
    for {key, val} <- map, into: %{}, do: {String.to_existing_atom(key), val}
  end

  def index(conn, _params) do
    q = Quotes.random() |> transform_string_keys_to_atoms
    render(conn, "index.html", quote: q)
  end
end

Before: quotes_controller.ex <br/> After: quotes_controller.ex

Finally, open the lib/app_web/templates/quotes/index.html.eex file and replace the contents with this code:

<h1>Quotes</h1>
<p>"<strong><em><%= @quote.text %></em></strong>" ~ <%= @quote.author %></p>

Before: quotes/index.html.eex <br/> After: quotes/index.html.eex

Now re-run the tests:

mix test

You should see them pass:

Compiling 3 files (.ex)
....

Finished in 0.07 seconds
4 tests, 0 failures

Randomized with seed 115090

Let's do a quick visual check. Run the Phoenix server:

mix phx.server

Then visit localhost:4000/quotes in your web browser. <br /> You should see a random quotation:

With tests passing again and a random quote rendering, let's attempt to make a JSON request to the HTML endpoint (and see it fail).

3.4 Content Negotiation Fails

At this stage if we run the server (mix phx.server) and attempt to make a request to the /quotes endpoint (in a different terminal window) with a JSON Accepts header:

curl -i -H "Accept: application/json" http://localhost:4000/quotes

We will see the following error: <br />

HTTP/1.1 406 Not Acceptable
cache-control: max-age=0, private, must-revalidate
content-length: 1915
date: Fri, 15 May 2020 07:44:44 GMT
server: Cowboy
x-request-id: Fg8j6sIqqtAKLiIAAAGB

# Phoenix.NotAcceptableError at GET /quotes

Exception:

    ** (Phoenix.NotAcceptableError) no supported media type in accept header.

    Expected one of ["html"] but got the following formats:

      * "application/json" with extensions: ["json"]

    To accept custom formats, register them under the :mime library
    in your config/config.exs file:

        config :mime, :types, %{
          "application/xml" => ["xml"]
        }

    And then run `mix deps.clean --build mime` to force it to be recompiled.

        (phoenix 1.5.1) lib/phoenix/controller.ex:1313: Phoenix.Controller.refuse/3
        (app 0.1.0) AppWeb.Router.browser/2
        (app 0.1.0) lib/app_web/router.ex:1: AppWeb.Router.__pipe_through0__/1
        (phoenix 1.5.1) lib/phoenix/router.ex:347: Phoenix.Router.__call__/2
        (app 0.1.0) lib/app_web/endpoint.ex:1: AppWeb.Endpoint.plug_builder_call/2
        (app 0.1.0) lib/plug/debugger.ex:132: AppWeb.Endpoint."call (overridable 3)"/2
        (app 0.1.0) lib/app_web/endpoint.ex:1: AppWeb.Endpoint.call/2
        (phoenix 1.5.1) lib/phoenix/endpoint/cowboy2_handler.ex:64: Phoenix.Endpoint.Cowboy2Handler.init/4


## Connection details

### Params

    %{}

### Request info

  * URI: http://localhost:4000/quotes
  * Query string:

### Headers

  * accept: application/json
  * host: localhost:4000
  * user-agent: curl/7.64.1

### Session

    %{}

And in the terminal running the phx.server, you will see: <br />

[debug] ** (Phoenix.NotAcceptableError) no supported media type in accept header.

Expected one of ["html"] but got the following formats:

  * "application/json" with extensions: ["json"]

To accept custom formats, register them under the :mime library
in your config/config.exs file:

    config :mime, :types, %{
      "application/xml" => ["xml"]
    }

And then run `mix deps.clean --build mime` to force it to be recompiled.

    (phoenix 1.5.1) lib/phoenix/controller.ex:1313: Phoenix.Controller.refuse/3
    (app 0.1.0) AppWeb.Router.browser/2
    (app 0.1.0) lib/app_web/router.ex:1: AppWeb.Router.__pipe_through0__/1
    (phoenix 1.5.1) lib/phoenix/router.ex:347: Phoenix.Router.__call__/2
    (app 0.1.0) lib/app_web/endpoint.ex:1: AppWeb.Endpoint.plug_builder_call/2
    (app 0.1.0) lib/plug/debugger.ex:132: AppWeb.Endpoint."call (overridable 3)"/2
    (app 0.1.0) lib/app_web/endpoint.ex:1: AppWeb.Endpoint.call/2
    (phoenix 1.5.1) lib/phoenix/endpoint/cowboy2_handler.ex:64: Phoenix.Endpoint.Cowboy2Handler.init/4

This is understandable given that the app doesn't have any pipeline/route that accepts JSON requests. <br /> Let's get on with the content negotiation part!

4. Create a Content Negotiation Pipeline in router.ex

By default the Phoenix router separates the :browser pipeline (which accepts "html") from the :api (which accepts "json"):

defmodule AppWeb.Router do
  use AppWeb, :router

  pipeline :browser do
    plug :accepts, ["html"]
    plug :fetch_session
    plug :fetch_flash
    plug :protect_from_forgery
    plug :put_secure_browser_headers
  end

  pipeline :api do
    plug :accepts, ["json"]
  end

  scope "/", AppWeb do
    pipe_through :browser

    get "/", PageController, :index
    resources "/quotes", QuotesController
  end

  # Other scopes may use custom stacks.
  # scope "/api", AppWeb do
  #   pipe_through :api
  # end
end

By default the /api scope is commented out. We are not going to enable it, rather as per our goal (above) we want to have the API and UI handled by the same router pipeline.

Let's replace the code in the router.ex with the following:

defmodule AppWeb.Router do
  use AppWeb, :router

  pipeline :any do
    plug :accepts, ~w(html json)
    plug :negotiate
  end

  defp negotiate(conn, []) do
    {"accept", accept} = List.keyfind(conn.req_headers, "accept", 0)

    if accept =~ "json" do # don't do anything for JSON (API) requests:
      conn
    else # setup conn for HTML requests:
      conn
      |> fetch_session([])
      |> fetch_flash([])
      |> protect_from_forgery([])
      |> put_secure_browser_headers([])
    end
  end

  scope "/", AppWeb do
    pipe_through :any

    get "/", PageController, :index
    resources "/quotes", QuotesController
  end
end

In this code we are replacing the :browser pipeline with the :any pipeline that handles all types of content. The :any pipeline invokes :negotiate which is defined immediately below.

In negotiate/2 we simply check the accept header in conn.req_headers. If the accept header matches the string "json", we don't need to do any further setup, otherwise we assume the request expects an HTML response invoke the appropriate plugs that were in the :browser pipeline.

Note: we know this is not "production" code. This is just an "MVP" for how to do content negotiation. We will improve it below!

At the end of this step, your router file should look like this: router.ex

5. Handle JSON requests in QuotesController

Now that our router.ex pipeline is setup to accept any content type, we need to handle the request for JSON in our controller.

Open the lib/app_web/controllers/quotes_controller.ex file and update the index/2 function with the following:

def index(conn, _params) do
  q = Quotes.random() |> transform_string_keys_to_atoms
  {"accept", accept} = List.keyfind(conn.req_headers, "accept", 0)

  if accept =~ "json" do
    json(conn, q)
  else
    render(conn, "index.html", quote: q)
  end
end

Here we use the Phoenix.Controller json/2 to sends a JSON response.

It uses the configured :json_library (Jason) under the :phoenix application for :json to pick up the encoder module.

At this point our rudimentary content negotiation is working. Try it: run the Phoenix server:

mix phx.server

In a different terminal window/tab, run the cURL command:

curl -i -H "Accept: application/json" http://localhost:4000/quotes

You should see output similar to this:

HTTP/1.1 200 OK
cache-control: max-age=0, private, must-revalidate
content-length: 86
content-type: application/json; charset=utf-8
date: Sat, 16 May 2020 14:25:51 GMT
server: Cowboy
x-request-id: Fg-IYvb_4_U9xvYAAASh

{"author":"Johann Wolfgang von Goethe","text":"Knowing is not enough; we must apply!"}

If you prefer to just have the JSON response, omit the -i flag:

curl -H "Accept: application/json" http://localhost:4000/quotes

Now you will just see the quote text and author (and where available, tags and source):

{
  "author":"Ernest Hemingway",
  "source":"https://www.goodreads.com/quotes/353013",
  "tags":"listen, learn, learning",
  "text":"I like to listen. I have learned a great deal from listening carefully. Most people never listen."
}

Confirm that it still works in the browser: http://localhost:4000/quotes

5.1 Fix Failing Tests!

While the content negotiation works for returning HTML and JSON, the changes we have made will break the tests.

If you try to run the tests now you will see them fail:

mix test

1) test /quotes shows a random quote (AppWeb.QuotesControllerTest)
   test/app_web/controllers/quotes_controller_test.exs:5
   ** (MatchError) no match of right hand side value: nil
   code: |> get(Routes.quotes_path(conn, :index))
   stacktrace:
     (app 0.1.0) lib/app_web/router.ex:9: AppWeb.Router.negotiate/2
     (app 0.1.0) AppWeb.Router.any/2
     (app 0.1.0) lib/app_web/router.ex:1: AppWeb.Router.__pipe_through0__/1

This fails because we are attempting to get the "accept" header in the router.ex negotiate/2 function but there are no headers defined in our test!

In Plug (and thus Phoenix) tests, no headers are set by default. <br /> This is the output of inspecting the conn (IO.inspect(conn)):

%Plug.Conn{
  adapter: {Plug.Adapters.Test.Conn, :...},
  assigns: %{},
  before_send: [],
  body_params: %Plug.Conn.Unfetched{aspect: :body_params},
  cookies: %Plug.Conn.Unfetched{aspect: :cookies},
  halted: false,
  host: "www.example.com",
  method: "GET",
  owner: #PID<0.335.0>,
  params: %Plug.Conn.Unfetched{aspect: :params},
  path_info: [],
  path_params: %{},
  port: 80,
  private: %{phoenix_recycled: true, plug_skip_csrf_protection: true},
  query_params: %Plug.Conn.Unfetched{aspect: :query_params},
  query_string: "",
  remote_ip: {127, 0, 0, 1},
  req_cookies: %Plug.Conn.Unfetched{aspect: :cookies},
  req_headers: [],
  request_path: "/",
  resp_body: nil,
  resp_cookies: %{},
  resp_headers: [{"cache-control", "max-age=0, private, must-revalidate"}],
  scheme: :http,
  script_name: [],
  secret_key_base: nil,
  state: :unset,
  status: nil
}

The important line is:

req_headers: [],

req_headers is an empty List.

There are two ways of fixing this failing test:

a. We include the right "accept" header in each test. <br /> b. We set a default value if there is no "accept" header defined.

If we go with the first option, we will need to add an accept header in the test:

test "shows a random quote", %{conn: conn} do
  conn =
    conn
    |> put_req_header("accept", "text/html")
    |> get(Routes.quotes_path(conn, :index))

  assert html_response(conn, 200) =~ "Quote"
end

This is fine in an individual case, but it will get old if we are using content negotiation in a more sophisticated app with dozens of routes.

We prefer to create a helper function that sets a default value if no accept header is set. Open the lib/app_web/controllers/quotes_controller.ex file and add the following helper function:

@doc """
`get_accept_header/1` gets the "accept" header from req_headers.
Defaults to "text/html" if no header is set.
"""
def get_accept_header(conn) do
  case List.keyfind(conn.req_headers, "accept", 0) do
    {"accept", accept} ->
      accept

    nil ->
      "tex/html"
  end
end

We can now use this function in both our AppWeb.QuotesController.index/2 and AppWeb.Router.negotiate/2 functions:

With the lib/app_web/controllers/quotes_controller.ex file still open, update the index/2 function to:

def index(conn, _params) do
  q = Quotes.random() |> transform_string_keys_to_atoms

  if get_accept_header(conn) =~ "json" do
    json(conn, q)
  else
    render(conn, "index.html", quote: q)
  end
end

Your quotes_controller.ex file should look like this: quotes_controller.ex#L10-L32

And in router.ex update the negotiate/2 function to:

defp negotiate(conn, []) do
  if AppWeb.QuotesController.get_accept_header(conn) =~ "json" do
    conn
  else
    conn
    |> fetch_session([])
    |> fetch_flash([])
    |> protect_from_forgery([])
    |> put_secure_browser_headers([])
  end
end

Your router.ex file should look like this: router.ex#L8-L18

Now re-run the tests and they will pass:

mix test

Expect to see:

Compiling 3 files (.ex)
....

Finished in 0.07 seconds
4 tests, 0 failures

Randomized with seed 485

At this point we have functioning content negotiation in our little app. <br />

5.2 Test the JSON Request

At present we don't have a test that executes the json branch of our code. We know it works from our terminal (manual cURL) testing, but we don't yet have an automated test. Let's fix that!

Open the test/app_web/controllers/quotes_controller_test.exs file and add the following test to it:

test "GET /quotes (JSON)", %{conn: conn} do
  conn =
    conn
    |> put_req_header("accept", "application/json")
    |> get(Routes.quotes_path(conn, :index))

  {:ok, json} = Jason.decode(conn.resp_body)
  %{ "author" => author, "text" => text } = json
  assert String.length(author) > 2
  assert String.length(text) > 10
end

Note: we are asserting that the length of author and text is greater than a certain String length because we cannot make any other assertions against a random quotation. This is enough for our needs because we know that we were able to Jason.decode the conn.resp_body indicating that it's valid JSON.

This will indirectly invoke the AppWeb.QuotesController.get_accept_header/1 function that extracts the "accept" header from conn.req_header. So we should have full test coverage for our little project.

Your test/app_web/controllers/quotes_controller_test.exs file should now look like this: quotes_controller_test.exs#L10-L20

6. Tidy Up The Project (Optional)

At this stage we have a working app that shows random quotations. But anyone viewing the app will first be greeted by irrelevant noise:

The home page of the App is the default Phoenix one and has no info about what the app actually does.

The quotes route has the Phoenix Framework logo and Links to get Started, which are irrelevant to the person viewing the quote.

Let's start by removing the Phoenix Framework logo, "Get Started" and "LiveDashboard" links from the layout template.

Open the lib/app_web/templates/layout/app.html.eex file and replace the contents with the following:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8"/>
    <meta http-equiv="X-UA-Compatible" content="IE=edge"/>
    <meta name="viewport" content="width=device-width, initial-scale=1.0"/>
    <link rel="stylesheet" href="https://unpkg.com/tachyons@4.12.0/css/tachyons.min.css"/>
    <title>Random Motivational Quotes App</title>
  </head>
  <body class="container w-100 helvetica tc">
    <%= @inner_content %>
  </body>
</html>

This is a good simplification of the layout template. The only addition is the Tachyons CSS library so that we can have easy control over the layout and typography. If you want to learn more see: /dwyl/learn-tachyons

Before: app.html.eex <br /> After: app.html.eex

If you run the Phoenix App now:

mix phx.server

And visit the /quotes

This is already much tidier.

But we can take it a step further. Next we will remove the "Quotes" heading from the quotes index template. <br /> Open the /lib/app_web/templates/quotes/index.html.eex file and replace the contents with:

<p class="f1 pa2">
  "<strong><em><%= @quote.text %></em></strong>" <br />
  ~ <%= @quote.author %>
</p>

Note: the only two things that might be unfamiliar if you are new to Tachyons CSS are the two classes on the <p> tag. The f1 just means "font size 1" or (H1) and pa2 means "padding all sides 2 units".

The quotes page now looks like this:

6.1 Make QuotesController the Default Route Handler

At present the "homepage" of the App is the PageController (see screenshot above with pink square outlining irrelevant content). The person wanting to see the quotes has to navigate to /quotes. Let's change it so that the quotes are rendered as the home page.

Open the lib/app_web/router.ex file and locate the scope "/" section:

scope "/", AppWeb do
  pipe_through :any

  get "/", PageController, :index
  resources "/quotes", QuotesController
end

Replace the code block with this simplified version:

scope "/", AppWeb do
  pipe_through :any

  resources "/", QuotesController
end

See: router.ex#L21-L25

Now when we visit the home page http://localhost:4000 we see a quote:

Now we just add a picture of sunrise from Unsplash: https://unsplash.com/photos/UweNcthlmDc

See: index.html.eex#L7-L23

And boom we have a motivational quote generator:

6.2 Fix Failing Tests

We made a few changes in the previous step which break our tests.

If you run mix test you will see that page_controller_test.exs are failing:

Compiling 4 files (.ex)
....

  1) test GET / (AppWeb.PageControllerTest)
     test/app_web/controllers/page_controller_test.exs:4
     Assertion with =~ failed
     code:  assert html_response(conn, 200) =~ "Welcome to Phoenix!"
     left:  "<!DOCTYPE html>\n<html lang=\"en\">\n  <head>\n    
     <meta charset=\"utf-8\"/>\n    
     <meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\"/>\n    
     <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\"/>\n    
     <link rel=\"stylesheet\" href=\"https://unpkg.com/tachyons@4.12.0/css/tachyons.min.css\"/>\n    
     <title>Random Motivational Quotes App</title>\n  </head>\n  
     <body class=\"container w-100 helvetica\">\n\n  <p class=\"f1 ph5 tl\">\n  
     \"<strong class=\"fw9\"><em>One who gains strength by overcoming obstacles
     possesses the only strength which can overcome adversity.</em></strong>\"\n     
     <span class=\"fr\"> ~ Albert Schweitzer</span>\n  </p>\n\n  
     <small class=\"fixed right-0 bottom-1 mr3 white\" style=\"font-size: 0.1em;\">\n    
     Sunrise Photo by\n <a class=\"no-underline white\"
     href=\"https://unsplash.com/photos/UweNcthlmDc\">\n      
     Alice Donovan Rouse on Unsplash\n    </a>\n  </small>\n\n<style>\n  
     body {\n    background-image: url(https://i.imgur.com/TIAf9Il.jpg);\n    
       background-repeat: no-repeat;\n    background-size: cover;\n    
       width: 100%;\n    height: 100%;\n    opacity: .8;\n  }\n</style>\n  
       </body>\n</html>\n"
     right: "Welcome to Phoenix!"
     stacktrace:
       test/app_web/controllers/page_controller_test.exs:6: (test)



Finished in 0.1 seconds
5 tests, 1 failure

Randomized with seed 305070

This test will never pass again because we are no longer using PageController in our project.

So, let's delete the controller, view, template and the corresponding test files:

rm lib/app_web/controllers/page_controller.ex
rm lib/app_web/templates/page/index.html.eex
rm lib/app_web/views/page_view.ex
rm test/app_web/controllers/page_controller_test.exs
rm test/app_web/views/page_view_test.exs

Deleting code (and the corresponding tests) is an important part of maintenance in a software project. Don't be afraid of doing it. You can always recover/restore deleted code because it's still there in your git history.

See commit: dcc322a

Now when we run mix test we see them pass (as expected):

Generated app app
....

Finished in 0.1 seconds
4 tests, 0 failures

Randomized with seed 746624

With the tests passing, we are done!

But Wait! There's More ...

So far in the tutorial we have shown from first principals how to render HTML and JSON in the same route/controller using content negotiation.

While this approach is fine for an MVP/tutorial, we feel we can do much better!

Part 2

In the first part of this tutorial, we saw how to add Content Negotiation to a Phoenix App from first principals.

In the next 2 mintues we will refactor our Phoenix App to use the content package.

7. Add the content Package to mix.exs

Open the mix.exs file, locate the deps definition and add the following line:

{:content, "~> 1.3.0"},

e.g: mix.exs#L52-L53

Install the dependency:

mix deps.get

You should see output similar to the following:

New:
  content 1.3.0
* Getting content (Hex package)

8. Add the Plug to router.ex

Open the lib/app_web/router.ex file and replace the line that read plug :negotiate with:

plug Content, %{html_plugs: [
  &fetch_session/2,
  &fetch_flash/2,
  &protect_from_forgery/2,
  &put_secure_browser_headers/2
]}

Note: those & and /2 additions to the names of plugs are the Elixir way of passing functions by reference. The & means "capture" and the /2 is the Arity of the function we are passing. We would obviously prefer if functions were just variables like they are in some other programming languages, but this works. See: https://dockyard.com/blog/2016/08/05/understand-capture-operator-in-elixir and: https://culttt.com/2016/05/09/functions-first-class-citizens-elixir

As we have replaced the negotiate/2 function we can safely remove it from the router.ex file.

Before: router.ex#L4-L19 <br /> After: router.ex

Simple, right? 😉

9. Use the Content.reply/5 in QuotesController

Finally in the lib/app_web/controllers/quotes_controller.ex replace the lines:

if get_accept_header(conn) =~ "json" do
  json(conn, q)
else
  render(conn, "index.html", quote: q)
end

With:

Content.reply(conn, &render/3, "index.html", &json/2, q)

The Content.reply/5 takes the 5 argument:

1. conn - the Plug.Conn where we get the req_headers from.
2. render/3 - the Phoenix.Controller.render/3 function, or your own implementation of a render function that takes conn, template and data as it's 3 params.
3. template - the .html template to be rendered if the accept header matches "html"; in this case "index.html"
4. json/2 - the Phoenix.Controller.json/2 function that renders json data. Or your own implementation that accepts the two params: conn and data corresponding to the Plug.Conn and the json data you want to return.
5. data - in this case the q (or quote) we want to render as HTML or JSON.

With this single line we can render HTML or JSON depending on the accept header.

We can delete the get_accept_header/1 function we created in step 5.1 (above) as it's now baked into the Content.reply/5. <br /> Note: it's still available as Content.get_accept_header/1 if we ever need it in one of our our Controllers.

If you need finer grained control in your controller, you can still write code like this:

if Content.get_accept_header(conn) =~ "json" do
  data = transform_data(q)
  json(conn, data)
else
  render(conn, "index.html", data: q)
end

Commit: 3e4f49d

Note: we also updated our lib/app_web/templates/quotes/index.html.eex file from: @quote.text to @data.text to reflect how Content.reply/5 labels the data.

9.1 Re-Run The Tests!

To confirm that the refactor is successful, re-run the tests:

mix test

Everything still passes:

Compiling 3 files (.ex)
....

Finished in 0.08 seconds
4 tests, 0 failures

Randomized with seed 452478

10. View JSON in a Web Browser

Sometimes while you are testing, you want to view the JSON data in Web Browser. The content package allows you to add .json to any route directly in the browser's URL field and view the JSON representation of that route.

Content will automatically recognise the request update the accept header to be application/json and send back the data as JSON.

There are two steps to enable this:

1. Create a "wildcard" route in your router.ex file:

get "/*wildcard", QuotesController, :redirect

e.g: /lib/app_web/router.ex#L21

2. Create the corresponding handler function in your Controller:

def redirect(conn, params) do
  Content.wildcard_redirect(conn, params, AppWeb.Router)
end

e.g: /lib/app_web/controllers/quotes_controller.ex#L16-L18

You can now visit http://localhost:4000/.json in your web browser to view a random quote in JSON format:

Done.

In this tutorial we learned how to do Content Negotiation from first principals. <br /> Then we saw how to use the content Plug to simplify our code!

If you found this useful, please ⭐ the repo on GitHub!

<br /> <hr /> <br />

Notes & Observations

Q: Is there an Official Way of Doing Content Negotiation?

While there is no "official" guide in the docs for how to do content negotiation, there is an issue/thread where it is discussed: phoenix/issues/1054

Both José Valim the creator of Elixir and Chris McCord creator of Phoenix have given input in the issue. So we have a fairly good notion that this is the acceptable way of doing content negotiation in a Phoenix App.

José outlines the Plug approach (this is what we did in step 4 above):

Chris advises to use Phoenix.Controller.get_format and pattern matching:

This is relevant for the general use case but is not to our specific one.

Chris also created a Gist: https://gist.github.com/chrismccord/31340f08d62de1457454 <br /> Which shows how to do content negotiation based on params.format. We have used this approach into our tutorial.

Note: this issue phoenix/issues/1054 is a textbook example of why we open issues to ask questions. <br /> The thread shows the initial uncertainty of the original poster. <br /> There is a discussion for why content negotiation is necessary and suggested approaches for doing it. <br /> Finally there is a comment from a person who discovered the issue years later and found the thread useful. <br /> 3 years later we are using it as the basis for our solution! <br /> In the future others will stumble upon it and be grateful that it exists. <br /> Conclusion: Open issues with questions! It's the right thing to do to learn and discuss all topics. <br /> Both people in your team and complete strangers will benefit!