Home

Awesome

Backend best practices

Check out my other repositories:

In this readme are presented some of the best practices, tools and guidelines for backend applications gathered from different sources.

This Readme contains code examples mainly for TypeScript + NodeJS, but practices described here are language agnostic and can be used in any backend project.

Architecture

Software architecture is about making fundamental choices of your application structure.

Architecture serves as a blueprint for a system. It provides an abstraction to manage the system complexity and establish a communication and coordination mechanism among components.

Choosing the right architecture is crucial for your application.

We discussed architecture in details in this repository: Domain-Driven Hexagon.

Make sure your architecture is enforced: Enforcing architecture

Read more:

API Security

Software security is the application of techniques that allow to mitigate and protect software systems from vulnerabilities and malicious attacks.

Software security is a large and complex discipline, so we will not cover it in details here.

Instead, here are some generic recommendations to ensure at least basic level of security:

Read more:

Data Validation

Data validation is critical for security of your API. You should validate all the input sent to your API.

Below are some basic recommendations on what data should be validated:

Cheap operations like checking for null/undefined and checking length of data come early in the list, and more expensive operations that require calling the database should be executed afterwards.

Example files:

Read more:

Enforce least privilege

Ensure that users and systems have the minimum access privileges required to perform their job functions (Principle of least privilege).

For example:

Eliminating unnecessary access rights significantly reduces your attack surface.

Read more:

Rate Limiting

Enforce a limit to the number of API requests within a time frame, this is called Rate Limiting or API throttling

By default, there is no limit on how many request users can make to your API. This may lead to problems, like DoS or brute force attacks, performance issues like high response time etc.

To solve this, implementing Rate Limiting is essential for any API.

Also enforce rate limiting for login attempts. Lock a user account for specific period of time after a given number of failed attempts

Read more:

Testing

Software Testing helps to catch bugs early. Properly tested software product ensures reliability, security and high performance which further results in time saving, cost-effectiveness and customer satisfaction.

White box vs Black box

Let's review two types of software testing:

Testing module/use-case internal structures (creating a test for every file/class) is called White Box testing. White Box testing is widely used technique, but it has disadvantages. It creates coupling to implementation details, so every time you decide to refactor business logic code this may also cause a refactoring of corresponding tests.

Use case requirements may change mid-work, your understanding of a problem may evolve, or you may start noticing new patterns that emerge during development, in other words, you start noticing a "big picture", which may lead to refactoring. For example: imagine that you defined a unit test for a class, and while developing this class you start noticing that it does too much and should be separated into two classes. Now you'll also have to refactor your test. After some time, while implementing a new feature, you notice that this new feature uses some code from that class you defined before, so you decide to separate that code and make it reusable, creating a third class (which originally was one), which leads to changing your unit tests yet again, every time you refactor. Use case requirements, input, output or behavior never changed, but tests had to be changed multiple times. This is inefficient and time-consuming.

When we have domain models that change often, tests tend to change with them. Traditional white box unit tests tend to be very coupled to internals of our domain model structure.

To solve this and get the most out of your tests, prefer Black Box testing (Behavioral Testing). This means that tests should focus on testing user-facing behavior users care about (your code's public API), not the implementation details of individual units it has inside. This avoids coupling, protects tests from changes that may happen while refactoring, makes tests easier to understand and maintain thus saving time.

Tests that are independent of implementation details are easier to maintain since they don't need to be changed each time you make a change to the implementation.

Try to avoid White Box testing when possible. However, it's worth mentioning that there are cases when White Box testing may be useful:

Use White Box testing only when it is really needed and as an addition to Black Box testing, not the other way around.

It's all about investing only in the tests that yield the biggest return on your effort.

Black Box / Behavioral tests can be divided in two parts:

Note: some people try to make e2e tests faster by using in-memory or embedded databases (like sqlite3). This makes tests faster, but reduces the reliability of those tests and should be avoided. Read more: Don't use In-Memory Databases for Tests.

For BDD tests Cucumber with Gherkin syntax can give a structure and meaning to your tests. This way even people not involved in a development can define steps needed for testing. In node.js world jest-cucumber is a nice package to achieve that.

Example files:

Read more:

Load Testing

For projects with a bigger user base you might want to implement some kind of load testing to see how program behaves with a lot of concurrent users.

Load testing is a great way to minimize performance risks, because it ensures an API can handle an expected load. By simulating traffic to an API in development, businesses can identify bottlenecks before they reach production environments. These bottlenecks can be difficult to find in development environments in the absence of a production load.

Automatic load testing tools can simulate that load by making a lot of concurrent requests to an API and measure response times and error rates.

Example tools:

Example files:

More info:

Fuzz Testing

Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program.

Fuzzing is a common method hackers use to find vulnerabilities of the system. For example:

There are a lot of examples of a problems like this, for example sending a certain character could crash and disable access to apps on an iPhone.

Sanitizing and validating input data is very important. But sometimes we make mistakes of not sanitizing/validating data properly, opening application to certain vulnerabilities.

Automated Fuzz testing tools can prevent such vulnerabilities. Those tools contain a list of strings that are usually sent by hackers, like malicious code snippets, SQL queries, unicode symbols etc. (for example: Big List of Naughty Strings), which helps test most common cases of different injection attacks.

Fuzz testing is a nice addition to typical testing methods described above and potentially can find serious security vulnerabilities or defects.

Example tools:

Read more:

Documentation

Here are some useful tips to help users/other developers to use your program.

Document APIs

Use OpenAPI (Swagger) or GraphQL specifications. Document in details every endpoint. Add description and examples of every request, response, properties and exceptions that endpoints may return or receive as body/parameters. This will help greatly to other developers and users of your API.

Example files:

Read more:

Use wiki

Create a wiki for collecting and sharing knowledge. Describe common tools, practices and procedures used in your organization. Write down notes explaining peculiarities of your software, how it works and why you made certain decisions.

It must be easy for everyone, especially new team members, to learn about specifics of the project.

According to The Bus Factor, projects can stall if knowledge is not shared properly across team members.

Here are some useful tools for note-taking and creating wikis:

Read more:

Add Readme

Create a simple readme file in a git repository that describes basic app functionality, available CLI commands, how to setup a new project etc.

Write self-documenting code

Code can be self-documenting to some degree. One useful trick is to separate complex code to smaller chunks with a descriptive name. For example:

This makes code easier to understand and maintain.

Read more:

Prefer statically typed languages

Types give useful semantic information to a developer and can be useful for creating self-documenting code. Good code should be easy to use correctly, and hard to use incorrectly. Types system can be a good help for that. It can prevent some nasty errors at a compile time, so IDE will show type errors right away.

Applications written using statically typed languages are usually easier to maintain, more scalable and better suited for large teams.

Note: For smaller projects/scripts/jobs static typing may not be needed.

Read more:

Avoid useless comments

Writing readable code, using descriptive function/method/variable names and creating tests can document your code well enough. Try to avoid comments when possible and try to make your code legible and tested instead.

Use comments only when it's really needed. Commenting may be a code smell in some cases, like when code gets changed but a developer forgets to update a comment (comments should be maintained too).

Code never lies, comments sometimes do.

Use comments only in some special cases, like when writing an counter-intuitive "hack" or performance optimization which is hard to read.

For documenting public APIs use code annotations (like JSDoc) instead of comments, this works nicely with code editor intellisense.

Read more:

Database Best Practices

Backups

Data is one of the most important things in your business. Keeping it safe is a top priority of any backend service.

Here are some basic recommendations:

Read more:

Managing Schema Changes

Migrations can help for database table/schema changes:

Database migration refers to the management of incremental, reversible changes and version control to relational database schemas. A schema migration is performed on a database whenever it is necessary to update or revert that database's schema to some newer or older version.

Source: Wiki

Migrations can be written manually or generated automatically every time database table schema is changed. When pushed to production it can be launched automatically.

BE CAREFUL not to drop some columns/tables that contain data by accident. Perform data migrations before table schema migrations and always backup database before doing anything.

Examples:

Read more:

Data Seeding

To avoid manually creating data in the database, seeding is a great solution to populate database with data for development and testing purposes.

Example packages for nodejs:

Example file: user.seeds.ts

Read more:

Configuration

Example files:

Logging

Read more:

Monitoring

Monitoring is the process to gather metrics about the operations of an IT environment's hardware and software to ensure everything functions as expected.

Additionally, to logging tools, when something unexpected happens in production, it's critical to have thorough monitoring in place. As software hardens more and more, unexpected events will get more and more infrequent and reproducing those events will become harder and harder. So when one of those unexpected events happens, there should be as much data available about the event as possible. Software should be designed from the start to be monitored. Monitoring aspects of software are almost as important as the functionality of the software itself, especially in big systems, since unexpected events can lead to money and reputation loss for a company. Monitoring helps fix and sometimes preventing unexpected behavior like failures, slow response times, errors etc.

Health monitoring tools are a good way to keep track of system performance, identify causes of crashes or downtime, monitor behavior, availability and load.

Some health monitoring tools already include logging management and error tracking, as well as alerts and general performance monitoring.

Here are some basic recommendation on what can be monitored:

Choose health monitoring tools depending on your needs, here are some examples:

Read more:

Telemetry

Telemetry data (metrics, logs, and traces) can help analyze your software’s performance and behavior.

For example, imagine that you have an event handler that listens for events and executes something:

async handleUserCreatedEvent(event: UserCreatedEvent) {
  await scheduleEmailNotification(event);
  await createWalletForNewUser(event);
  await doSomethingElse(event);
}

In production environments, you or your clients can notice that application is unresponsive, or that operations have a long delay, and sometimes it's hard to determine why. Telemetry data can help investigate why. Let's modify our previous example:

async handleUserCreatedEvent(event: UserCreatedEvent) {
  // Initialize tracing with OpenTelemetry
  const tracer = trace.getTracer('EventSubscriber', '0.1.0');
  const span = tracer.startSpan(name);
  span.addAttribute('eventName', event.name);

  span.addEvent(`Received event. Scheduling email notification...`);
  await scheduleEmailNotification(event);

  span.addEvent(`Creating wallet for new user...`);
  await createWalletForNewUser(event);

  span.addEvent(`Starting something else...`);
  await doSomethingElse(event);

  span.addEvent(`Event processed.`);
}

We created a tracer span and logged some events before every function execution. These logs can be integrated with a cloud provider, or tools like Prometheus, that will visualize your telemetry data, create charts and graphs, showing execution times for each span and log. Now, you can check your telemetry graphs at any time and instantly locate exact function call that causes unusual spikes.

Telemetry data can be logged anywhere:

Using telemetry makes it trivial to locate bottlenecks in your application.

Example tools:

Standardization

Standardization is the process of implementing and developing technical standards based on the consensus of different parties.

Define and agree on standards in the development process, for example:

Standards help enforce best practices and simplify both the development process and code.

Create documents that describe your standards and enforce them as much as you can. Ideally there should be only one way of doing a common task. If you don't have standards everybody in the team will be doing things differently and your system will be a mess.

Read more:

Static Code Analysis

Static code analysis is a method of debugging by examining source code before a program is run.

For JavasScript and TypeScript, Eslint with typescript-eslint plugin and some rules (like airbnb / airbnb-typescript) can be a great tool to enforce writing better code.

Try to make linter rules reasonably strict, this will help greatly to avoid "shooting yourself in a foot". Strict linter rules can prevent bugs and even serious security holes (eslint-plugin-security).

Adopt programming habits that constrain you, to help you to limit mistakes.

For example:

Using explicit any type is a bad practice. Consider disallowing it (and other things that may cause problems):

// .eslintrc.js file
  rules: {
    '@typescript-eslint/no-explicit-any': 'error',
    // ...
  }

Also, enabling strict mode in tsconfig.json is recommended, this will disallow things like implicit any types:

  "compilerOptions": {
    "strict": true,
    // ...
  }

Example file: .eslintrc.js

Code Spell Checker may be a good addition to eslint.

Read more:

Code formatting

The way code looks adds to our understanding of it. Good style makes reading code a pleasurable and consistent experience.

Consider using code formatters like Prettier to maintain same code styles in the project.

Read more:

Shut down gracefully

When you shut down your application you may interrupt all operations that are running at that time and lose data. Unless you gracefully shut down your app.

Shutting down gracefully means when you send a termination signal to your app, it should (if it's a web server):

This way you are not interrupting running operations and prevent a lot of related problems.

Read more:

Profiling

Profiling allows us to collect and analyze data on how functions in your code perform when executed. This can help us with identifying bottlenecks in our code and fix them to make application perform faster.

Here are some examples for NodeJS:

Benchmarking

To make sure that your optimizations are working and system run fast, consider using benchmarking tools to analyze execution times and performance of your backend, apps, scripts, jobs, etc.

Read more:

Make application easy to setup

There are a lot of projects out there which take effort to configure after downloading it. Everything has to be set up manually: database, all configs etc. If new developer joins the team he has to waste a lot of time just to make application work.

This is a bad practice and should be avoided. Setting up project after downloading it should be as easy as launching one or few commands in terminal. Consider adding scripts to do this automatically:

Example files:

Deployment

Automate your deployment process. CI/CD tools can help with that.

Deployment automation reduces errors, speeds up deployments, creates a smooth path to deliver updates and upgrades on a regular basis. The process becomes very easy, just pushing your changes to a repository can trigger a build and deploy process automatically so you don't have to do anything else.

During deployment execute your e2e tests, load/fuzz tests, static code analysis checks, check for vulnerabilities in your packages etc. and stop a deploy process if anything fails. This prevents deploying failing or insecure code to production and makes your application robust and secure.

Read more:

Blue-Green Deployment

Blue-green deployment is a release strategy that proposes the use of two servers: "green" and "blue". When deploying, the new build is deployed to one of the servers ("green"). When build is finished, requests are routed to a new build ("green" server), while maintaining an old version of your program ("blue") running. If a "green" server has some bugs or doesn't work properly, you can easily switch back to a "blue" server with zero downtime. This allows to quickly roll back to a previous version if anything goes wrong by simply routing traffic back to a "blue" server with previous version of your program.

Read more:

Code Generation

Code generation can be important when using complex architectures to avoid typing boilerplate code manually.

Hygen is a great example. This tool can generate building blocks (or entire modules) by using custom templates. Templates can be designed to follow best practices and concepts based on Clean/Hexagonal Architecture, DDD, SOLID etc.

Main advantages of automatic code generation are:

Notes:

Version Control

Make sure you are using version control systems like git. Version control systems can track changes you make to files, so you have a record of what has been done, and you can revert to specific versions should you ever need to. It will also help coordinating work among programmers allowing changes by multiple people to all be merged into one source.

Below are some good practices to use together with git.

Pre-push/pre-commit hooks

Consider launching tests/code formatting/linting every time you do git push or git commit. This prevents bad code getting in your repo. Husky is a great tool for that.

Read more:

Conventional commits

Conventional commits add some useful prefixes to your commit messages, for example:

This creates a common language that makes easier communicating the nature of changes to teammates and also may be useful for automatic package versioning and release notes generation.

Read more:

API Versioning

API versioning is the practice of transparently managing changes to your API.

API versioning allows you to incorporate the latest changes in a new version of your API thereby still allowing users to have access to the older version of your API without breaking your users application.

If you need to create a new version of an endpoint, a simple solution would be to create new version of DTOs and a URL like this: /v2/users. Keep old version of an endpoint running for backwards compatibility until your users fully migrate to a new version.

Creating a new version of an endpoint instead of modifying an old one protects users of your API from breaking their app.

Note: if the only user of your API is your own frontend that can change on demand API versioning may be not worth it.

If you are using GraphQL, you can use a @deprecated directive on a field.

For versioning packages, libraries, SDKs etc. you can use semantic versioning.

Example files:

Read more:

Additional resources

Github Repositories