Home

Awesome

Databricks Scala Guide

At Databricks, our engineers work on some of the most actively developed Scala codebases in the world, including our own internal repo called "universe" as well as the various open source projects we contribute to, e.g. Apache Spark and Delta Lake. This guide draws from our experience coaching and working with our engineering teams as well as the broader open source community.

Code is written once by its author, but read and modified multiple times by lots of other engineers. As most bugs actually come from future modification of the code, we need to optimize our codebase for long-term, global readability and maintainability. The best way to achieve this is to write simple code.

Scala is an incredibly powerful language that is capable of many paradigms. We have found that the following guidelines work well for us on projects with high velocity. Depending on the needs of your team, your mileage might vary.

<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>.

<a name='TOC'>Table of Contents</a>

  1. Document History

  2. Syntactic Style

  3. Scala Language Features

  4. Concurrency

  5. Performance

  6. Java Interoperability

  7. Testing

  8. Miscellaneous

<a name='history'>Document History</a>

<a name='syntactic'>Syntactic Style</a>

<a name='naming'>Naming Convention</a>

We mostly follow Java's and Scala's standard naming conventions.

<a name='variable-naming'>Variable Naming Convention</a>

<a name='linelength'>Line Length</a>

<a name='rule_of_30'>Rule of 30</a>

"If an element consists of more than 30 subelements, it is highly probable that there is a serious problem" - Refactoring in Large Software Projects.

In general:

<a name='indent'>Spacing and Indentation</a>

<a name='blanklines'>Blank Lines (Vertical Whitespace)</a>

<a name='parentheses'>Parentheses</a>

<a name='curly'>Curly Braces</a>

Put curly braces even around one-line conditional or loop statements. The only exception is if you are using if/else as an one-line ternary operator that is also side-effect free.

// Correct:
if (true) {
  println("Wow!")
}

// Correct:
if (true) statement1 else statement2

// Correct:
try {
  foo()
} catch {
  ...
}

// Wrong:
if (true)
  println("Wow!")

// Wrong:
try foo() catch {
  ...
}

<a name='long_literal'>Long Literals</a>

Suffix long literal values with uppercase L. It is often hard to differentiate lowercase l from 1.

val longValue = 5432L  // Do this

val longValue = 5432l  // Do NOT do this

<a name='doc'>Documentation Style</a>

Use Java docs style instead of Scala docs style.

/** This is a correct one-liner, short description. */

/**
 * This is correct multi-line JavaDoc comment. And
 * this is my second line, and if I keep typing, this would be
 * my third line.
 */

/** In Spark, we don't use the ScalaDoc style so this
  * is not correct.
  */

<a name='ordering_class'>Ordering within a Class</a>

If a class is long and has many methods, group them logically into different sections, and use comment headers to organize them.

class DataFrame {

  ///////////////////////////////////////////////////////////////////////////
  // DataFrame operations
  ///////////////////////////////////////////////////////////////////////////

  ...

  ///////////////////////////////////////////////////////////////////////////
  // RDD operations
  ///////////////////////////////////////////////////////////////////////////

  ...
}

Of course, the situation in which a class grows this long is strongly discouraged, and is generally reserved only for building certain public APIs.

<a name='imports'>Imports</a>

<a name='pattern-matching'>Pattern Matching</a>

<a name='infix'>Infix Methods</a>

Avoid infix notation for methods that aren't symbolic methods (i.e. operator overloading).

// Correct
list.map(func)
string.contains("foo")

// Wrong
list map (func)
string contains "foo"

// But overloaded operators should be invoked in infix style
arrayBuffer += elem

<a name='anonymous'>Anonymous Methods</a>

Avoid excessive parentheses and curly braces for anonymous methods.

// Correct
list.map { item =>
  ...
}

// Correct
list.map(item => ...)

// Wrong
list.map(item => {
  ...
})

// Wrong
list.map { item => {
  ...
}}

// Wrong
list.map({ item => ... })

<a name='lang'>Scala Language Features</a>

<a name='case_class_immutability'>Case Classes and Immutability</a>

Case classes are regular classes but extended by the compiler to automatically support:

Constructor parameters should NOT be mutable for case classes. Instead, use copy constructor. Having mutable case classes can be error prone, e.g. hash maps might place the object in the wrong bucket using the old hash code.

// This is OK
case class Person(name: String, age: Int)

// This is NOT OK
case class Person(name: String, var age: Int)

// To change values, use the copy constructor to create a new instance
val p1 = Person("Peter", 15)
val p2 = p1.copy(age = 16)

<a name='apply_method'>apply Method</a>

Avoid defining apply methods on classes. These methods tend to make the code less readable, especially for people less familiar with Scala. It is also harder for IDEs (or grep) to trace. In the worst case, it can also affect correctness of the code in surprising ways, as demonstrated in Parentheses.

It is acceptable to define apply methods on companion objects as factory methods. In these cases, the apply method should return the companion class type.

object TreeNode {
  // This is OK
  def apply(name: String): TreeNode = ...

  // This is bad because it does not return a TreeNode
  def apply(name: String): String = ...
}

<a name='override_modifier'>override Modifier</a>

Always add override modifier for methods, both for overriding concrete methods and implementing abstract methods. The Scala compiler does not require override for implementing abstract methods. However, we should always add override to make the override obvious, and to avoid accidental non-overrides due to non-matching signatures.

trait Parent {
  def hello(data: Map[String, String]): Unit = {
    print(data)
  }
}

class Child extends Parent {
  import scala.collection.Map

  // The following method does NOT override Parent.hello,
  // because the two Maps have different types.
  // If we added "override" modifier, the compiler would've caught it.
  def hello(data: Map[String, String]): Unit = {
    print("This is supposed to override the parent method, but it is actually not!")
  }
}

<a name='destruct_bind'>Destructuring Binds</a>

Destructuring bind (sometimes called tuple extraction) is a convenient way to assign two variables in one expression.

val (a, b) = (1, 2)

However, do NOT use them in constructors, especially when a and b need to be marked transient. The Scala compiler generates an extra Tuple2 field that will not be transient for the above example.

class MyClass {
  // This will NOT work because the compiler generates a non-transient Tuple2
  // that points to both a and b.
  @transient private val (a, b) = someFuncThatReturnsTuple2()
}

<a name='call_by_name'>Call by Name</a>

Avoid using call by name. Use () => T explicitly.

Background: Scala allows method parameters to be defined by-name, e.g. the following would work:

def print(value: => Int): Unit = {
  println(value)
  println(value + 1)
}

var a = 0
def inc(): Int = {
  a += 1
  a
}

print(inc())

in the above code, inc() is passed into print as a closure and is executed (twice) in the print method, rather than being passed in as a value 1. The main problem with call-by-name is that the caller cannot differentiate between call-by-name and call-by-value, and thus cannot know for sure whether the expression will be executed or not (or maybe worse, multiple times). This is especially dangerous for expressions that have side-effect.

<a name='multi-param-list'>Multiple Parameter Lists</a>

Avoid using multiple parameter lists. They complicate operator overloading, and can confuse programmers less familiar with Scala. For example:

// Avoid this!
case class Person(name: String, age: Int)(secret: String)

One notable exception is the use of a 2nd parameter list for implicits when defining low-level libraries. That said, implicits should be avoided!

<a name='symbolic_methods'>Symbolic Methods (Operator Overloading)</a>

Do NOT use symbolic method names, unless you are defining them for natural arithmetic operations (e.g. +, -, *, /). Under no other circumstances should they be used. Symbolic method names make it very hard to understand the intent of the methods. Consider the following two examples:

// symbolic method names are hard to understand
channel ! msg
stream1 >>= stream2

// self-evident what is going on
channel.send(msg)
stream1.join(stream2)

<a name='type_inference'>Type Inference</a>

Scala type inference, especially left-side type inference and closure inference, can make code more concise. That said, there are a few cases where explicit typing should be used:

<a name='return'>Return Statements</a>

Avoid using return in closures. return is turned into try/catch of scala.runtime.NonLocalReturnControl by the compiler. This can lead to unexpected behaviors. Consider the following example:

def receive(rpc: WebSocketRPC): Option[Response] = {
  tableFut.onComplete { table =>
    if (table.isFailure) {
      return None // Do not do that!
    } else { ... }
  }
}

the .onComplete method takes the anonymous closure { table => ... } and passes it to a different thread. This closure eventually throws the NonLocalReturnControl exception that is captured in a different thread . It has no effect on the poor method being executed here.

However, there are a few cases where return is preferred.

<a name='recursion'>Recursion and Tail Recursion</a>

Avoid using recursion, unless the problem can be naturally framed recursively (e.g. graph traversal, tree traversal).

For methods that are meant to be tail recursive, apply @tailrec annotation to make sure the compiler can check it is tail recursive. (You will be surprised how often seemingly tail recursive code is actually not tail recursive due to the use of closures and functional transformations.)

Most code is easier to reason about with a simple loop and explicit state machines. Expressing it with tail recursions (and accumulators) can make it more verbose and harder to understand. For example, the following imperative code is more readable than the tail recursive version:

// Tail recursive version.
def max(data: Array[Int]): Int = {
  @tailrec
  def max0(data: Array[Int], pos: Int, max: Int): Int = {
    if (pos == data.length) {
      max
    } else {
      max0(data, pos + 1, if (data(pos) > max) data(pos) else max)
    }
  }
  max0(data, 0, Int.MinValue)
}

// Explicit loop version
def max(data: Array[Int]): Int = {
  var max = Int.MinValue
  for (v <- data) {
    if (v > max) {
      max = v
    }
  }
  max
}

<a name='implicits'>Implicits</a>

Avoid using implicits, unless:

When implicits are used, we must ensure that another engineer who did not author the code can understand the semantics of the usage without reading the implicit definition itself. Implicits have very complicated resolution rules and make the code base extremely difficult to understand. From Twitter's Effective Scala guide: "If you do find yourself using implicits, always ask yourself if there is a way to achieve the same thing without their help."

If you must use them (e.g. enriching some DSL), do not overload implicit methods, i.e. make sure each implicit method has distinct names, so users can selectively import them.

// Don't do the following, as users cannot selectively import only one of the methods.
object ImplicitHolder {
  def toRdd(seq: Seq[Int]): RDD[Int] = ...
  def toRdd(seq: Seq[Long]): RDD[Long] = ...
}

// Do the following:
object ImplicitHolder {
  def intSeqToRdd(seq: Seq[Int]): RDD[Int] = ...
  def longSeqToRdd(seq: Seq[Long]): RDD[Long] = ...
}

<a name='symbol'>Symbol Literals</a>

Avoid using symbol literals. Symbol literals (e.g. 'column) were deprecated as of Scala 2.13 by Proposal to deprecate and remove symbol literals. Apache Spark used to leverage this syntax to provide DSL; however, now it started to remove this deprecated usage away. See also SPARK-29392.

<a name='exception'>Exception Handling (Try vs try)</a>

<a name='option'>Options</a>

<a name='chaining'>Monadic Chaining</a>

One of Scala's powerful features is monadic chaining. Almost everything (e.g. collections, Option, Future, Try) is a monad and operations on them can be chained together. This is an incredibly powerful concept, but chaining should be used sparingly. In particular:

A chain can often be made more understandable by giving the intermediate result a variable name, by explicitly typing the variable, and by breaking it down into more procedural style. As a contrived example:

class Person(val data: Map[String, String])
val database = Map[String, Person]
// Sometimes the client can store "null" value in the  store "address"

// A monadic chaining approach
def getAddress(name: String): Option[String] = {
  database.get(name).flatMap { elem =>
    elem.data.get("address")
      .flatMap(Option.apply)  // handle null value
  }
}

// A more readable approach, despite much longer
def getAddress(name: String): Option[String] = {
  if (!database.contains(name)) {
    return None
  }

  database(name).data.get("address") match {
    case Some(null) => None  // handle null value
    case Some(addr) => Option(addr)
    case None => None
  }
}

Another example is an if-else block that confuses if the chain is for the whole if-else or only else block.

// A monadic chaining approach
val condition: Boolean = ...
if (condition) {
  Seq(1, 2, 3)  // Results in List(1, 2, 3)
} else {
  Seq(1, 2, 3)  // Results in List(2, 3, 4)
}.map(_ + 1)

// A more readable approach.
val ret = if (condition) {
  Seq(1, 2, 3)
} else {
  Seq(1, 2, 3)
}
ret.map(_ + 1)  // Results in List(2, 3, 4)

<a name='concurrency'>Concurrency</a>

<a name='concurrency-scala-collection'>Scala concurrent.Map</a>

Prefer java.util.concurrent.ConcurrentHashMap over scala.collection.concurrent.Map. In particular the getOrElseUpdate method in scala.collection.concurrent.Map is not atomic (fixed in Scala 2.11.6, SI-7943). Since all the projects we work on require cross-building for both Scala 2.10 and Scala 2.11, scala.collection.concurrent.Map should be avoided.

<a name='concurrency-sync-vs-map'>Explicit Synchronization vs Concurrent Collections</a>

There are 3 recommended ways to make concurrent accesses to shared states safe. Do NOT mix them because that could make the program very hard to reason about and lead to deadlocks.

  1. java.util.concurrent.ConcurrentHashMap: Use when all states are captured in a map, and high degree of contention is expected.
private[this] val map = new java.util.concurrent.ConcurrentHashMap[String, String]
  1. java.util.Collections.synchronizedMap: Use when all states are captured in a map, and contention is not expected but you still want to make code safe. In case of no contention, the JVM JIT compiler is able to remove the synchronization overhead via biased locking.
private[this] val map = java.util.Collections.synchronizedMap(new java.util.HashMap[String, String])
  1. Explicit synchronization by synchronizing all critical sections: can used to guard multiple variables. Similar to 2, the JVM JIT compiler can remove the synchronization overhead via biased locking.
class Manager {
  private[this] var count = 0
  private[this] val map = new java.util.HashMap[String, String]
  def update(key: String, value: String): Unit = synchronized {
    map.put(key, value)
    count += 1
  }
  def getCount: Int = synchronized { count }
}

Note that for case 1 and case 2, do not let views or iterators of the collections escape the protected area. This can happen in non-obvious ways, e.g. when returning Map.keySet or Map.values. If views or values are required to pass around, make a copy of the data.

val map = java.util.Collections.synchronizedMap(new java.util.HashMap[String, String])

// This is broken!
def values: Iterable[String] = map.values

// Instead, copy the elements
def values: Iterable[String] = map.synchronized { Seq(map.values: _*) }

<a name='concurrency-sync-vs-atomic'>Explicit Synchronization vs Atomic Variables vs @volatile</a>

The java.util.concurrent.atomic package provides primitives for lock-free access to primitive types, such as AtomicBoolean, AtomicInteger, and AtomicReference.

Always prefer Atomic variables over @volatile. They have a strict superset of the functionality and are more visible in code. Atomic variables are implemented using @volatile under the hood.

Prefer Atomic variables over explicit synchronization when: (1) all critical updates for an object are confined to a single variable and contention is expected. Atomic variables are lock-free and permit more efficient contention. Or (2) synchronization is clearly expressed as a getAndSet operation. For example:

// good: clearly and efficiently express only-once execution of concurrent code
val initialized = new AtomicBoolean(false)
...
if (!initialized.getAndSet(true)) {
  ...
}

// poor: less clear what is guarded by synchronization, may unnecessarily synchronize
val initialized = false
...
var wasInitialized = false
synchronized {
  wasInitialized = initialized
  initialized = true
}
if (!wasInitialized) {
  ...
}

<a name='concurrency-private-this'>Private Fields</a>

Note that private fields are still accessible by other instances of the same class, so protecting it with this.synchronized (or just synchronized) is not technically sufficient. Make the field private[this] instead.

// The following is still unsafe.
class Foo {
  private var count: Int = 0
  def inc(): Unit = synchronized { count += 1 }
}

// The following is safe.
class Foo {
  private[this] var count: Int = 0
  def inc(): Unit = synchronized { count += 1 }
}

<a name='concurrency-isolation'>Isolation</a>

In general, concurrency and synchronization logic should be isolated and contained as much as possible. This effectively means:

<a name='perf'>Performance</a>

For the vast majority of the code you write, performance should not be a concern. However, for performance sensitive code, here are some tips:

<a name='perf-microbenchmarks'>Microbenchmarks</a>

It is ridiculously hard to write a good microbenchmark because the Scala compiler and the JVM JIT compiler do a lot of magic to the code. More often than not, your microbenchmark code is not measuring the thing you want to measure.

Use jmh if you are writing microbenchmark code. Make sure you read through all the sample microbenchmarks so you understand the effect of deadcode elimination, constant folding, and loop unrolling on microbenchmarks.

<a name='perf-whileloops'>Traversal and zipWithIndex</a>

Use while loops instead of for loops or functional transformations (e.g. map, foreach). For loops and functional transformations are very slow (due to virtual function calls and boxing).


val arr = // array of ints
// zero out even positions
val newArr = list.zipWithIndex.map { case (elem, i) =>
  if (i % 2 == 0) 0 else elem
}

// This is a high performance version of the above
val newArr = new Array[Int](arr.length)
var i = 0
val len = newArr.length
while (i < len) {
  newArr(i) = if (i % 2 == 0) 0 else arr(i)
  i += 1
}

<a name='perf-option'>Option and null</a>

For performance sensitive code, prefer null over Option, in order to avoid virtual method calls and boxing. Label the nullable fields clearly with Nullable.

class Foo {
  @javax.annotation.Nullable
  private[this] var nullableField: Bar = _
}

<a name='perf-collection'>Scala Collection Library</a>

For performance sensitive code, prefer Java collection library over Scala ones, since the Scala collection library often is slower than Java's.

<a name='perf-private'>private[this]</a>

For performance sensitive code, prefer private[this] over private. private[this] generates a field, rather than creating an accessor method. In our experience, the JVM JIT compiler cannot always inline private field accessor methods, and thus it is safer to use private[this] to ensure no virtual method call for accessing a field.

class MyClass {
  private val field1 = ...
  private[this] val field2 = ...

  def perfSensitiveMethod(): Unit = {
    var i = 0
    while (i < 1000000) {
      field1  // This might invoke a virtual method call
      field2  // This is just a field access
      i += 1
    }
  }
}

<a name='java'>Java Interoperability</a>

This section covers guidelines for building Java compatible APIs. These do not apply if the component you are building does not require interoperability with Java. It is mostly drawn from our experience in developing the Java APIs for Spark.

<a name='java-missing-features'>Java Features Missing from Scala</a>

The following Java features are missing from Scala. If you need the following, define them in Java instead. However, be reminded that ScalaDocs are not generated for files defined in Java.

<a name='java-traits'>Traits and Abstract Classes</a>

For interfaces that can be implemented externally, keep in mind the following:

// The default implementation doesn't work in Java
trait Listener {
  def onTermination(): Unit = { ... }
}

// Works in Java
abstract class Listener {
  def onTermination(): Unit = { ... }
}

<a name='java-type-alias'>Type Aliases</a>

Do NOT use type aliases. They are not visible in bytecode (and Java).

<a name='java-default-param-values'>Default Parameter Values</a>

Do NOT use default parameter values. Overload the method instead.

// Breaks Java interoperability
def sample(ratio: Double, withReplacement: Boolean = false): RDD[T] = { ... }

// The following two work
def sample(ratio: Double, withReplacement: Boolean): RDD[T] = { ... }
def sample(ratio: Double): RDD[T] = sample(ratio, withReplacement = false)

<a name='java-multi-param-list'>Multiple Parameter Lists</a>

Do NOT use multi-parameter lists.

<a name='java-varargs'>Varargs</a>

<a name='java-implicits'>Implicits</a>

Do NOT use implicits, for a class or method. This includes ClassTag, TypeTag.

class JavaFriendlyAPI {
  // This is NOT Java friendly, since the method contains an implicit parameter (ClassTag).
  def convertTo[T: ClassTag](): T
}

<a name='java-companion-object'>Companion Objects, Static Methods and Fields</a>

There are a few things to watch out for when it comes to companion objects and static methods/fields.

<a name='testing'>Testing</a>

<a name='testing-intercepting'>Intercepting Exceptions</a>

When testing that performing a certain action (say, calling a function with an invalid argument) throws an exception, be as specific as possible about the type of exception you expect to be thrown. You should NOT simply intercept[Exception] or intercept[Throwable] (to use the ScalaTest syntax), as this will just assert that any exception is thrown. Often times, this will just catch errors you made when setting up your testing mocks and your test will silently pass without actually checking the behavior you want to verify.

// This is WRONG
intercept[Exception] {
  thingThatThrowsException()
}

// This is CORRECT
intercept[MySpecificTypeOfException] {
  thingThatThrowsException()
}

If you cannot be more specific about the type of exception that the code will throw, that is often a sign of code smell. You should either test at a lower level or modify the underlying code to throw a more specific exception.

<a name='misc'>Miscellaneous</a>

<a name='misc_currentTimeMillis_vs_nanoTime'>Prefer nanoTime over currentTimeMillis</a>

When computing a duration or checking for a timeout, avoid using System.currentTimeMillis(). Use System.nanoTime() instead, even if you are not interested in sub-millisecond precision.

System.currentTimeMillis() returns current wallclock time and will follow changes to the system clock. Thus, negative wallclock adjustments can cause timeouts to "hang" for a long time (until wallclock time has caught up to its previous value again). This can happen when ntpd does a "step" after the network has been disconnected for some time. The most canonical example is during system bootup when DHCP takes longer than usual. This can lead to failures that are really hard to understand/reproduce. System.nanoTime() is guaranteed to be monotonically increasing irrespective of wallclock changes.

Caveats:

<a name='misc_uri_url'>Prefer URI over URL</a>

When storing the URL of a service, you should use the URI representation.

The equality check of URL actually performs a (blocking) network call to resolve the IP address. The URI class performs field equality and is a superset of URL as to what it can represent.

<a name='misc_well_tested_method'>Prefer existing well-tested methods over reinventing the wheel</a>

When there is an existing well-tesed method and it doesn't cause any performance issue, prefer to use it. Reimplementing such method may introduce bugs and requires spending time testing it (maybe we don't even remember to test it!).

val beginNs = System.nanoTime()
// Do something
Thread.sleep(1000)
val elapsedNs = System.nanoTime() - beginNs

// This is WRONG. It uses magic numbers and is pretty easy to make mistakes
val elapsedMs = elapsedNs / 1000 / 1000

// Use the Java TimeUnit API. This is CORRECT
import java.util.concurrent.TimeUnit
val elapsedMs2 = TimeUnit.NANOSECONDS.toMillis(elapsedNs)

// Use the Scala Duration API. This is CORRECT
import scala.concurrent.duration._
val elapsedMs3 = elapsedNs.nanos.toMillis

Exceptions: