Snapshot Testing in Swift

This article explores “snapshot testing”: what it is, why it’s important, and what you can get from it.

An example of a snapshot failure in Xcode.

A snapshot test is a test case that uses reference data—typically a file on disk—to assert the correctness of some code.1 Where unit tests make fine-grained checks against explicit expectations (really, whatever the engineer anticipates should be right or may go wrong), snapshot tests can zoom out and cover far more surface area and potential regressions.

Facebook’s popular snapshot testing library, FBSnapshotTestCase, records screen shots of iOS views to ensure they don’t change. Writing a snapshot test is a matter of passing a view to an assertion.2

  FBSnapshotVerifyView(viewController.view)

Should a screen render differently than a reference in the future, the associated test will fail and produce a diff for inspection.

Using Kaleidoscope to investigate a screen shot failure. The copy and line height of the bottom-left section have changed.

These tests force us to proactively highlight intentional changes to UI (like updated colors, copy, or constraints), keep us informed of unintended changes, and build living documentation of an app’s UI—these artifacts can be checked into source control and highlight changes in GitHub pull requests. They provide far more coverage than a unit test normally allows—an assertion covers every pixel on the screen—but snapshot tests aren’t limited to screen shots.

Facebook maintains another popular testing library with snapshot support: Jest. Like FBSnapshotTestCase, Jest snapshots ensure that UI doesn’t change unexpectedly, but where the former captures images, the latter captures text. Jest serializes and records data structures, particularly React view trees. This means that even non-visual UI elements, like accessibility attributes, can be tested. In fact, because Jest works with text, it can go beyond UI testing and write snapshots against any serializable data.

It turns out that snapshots are great for testing complex data structures with lots of properties! Snapshots also let us quickly generate coverage for untested code, unlocking the ability to refactor where refactoring seemed impossible.

Snapping Structures in Swift

While FBSnapshotTestCase has popularized the idea of screen shot tests in the iOS community, more generalized snapshot testing hasn’t yet taken hold. What might a snapshot test look like in our world? Let’s walk through a few examples.

It’s common to have an API service in a code base, and somewhere in that service there will be code that is responsible for preparing a URLRequest value out of given data. This preparation work constructs the URL for the request, attaches authentication, some headers, and perhaps sets the POST body. All of this data can be captured easily in a snapshot test!

func testUrlRequestPreparation() {
  let service = ApiService()
  let request = service
    .prepare(endpoint: .createArticle("Hello, world!"))

  assertSnapshot(matching: request)
}

The resulting snapshot may look something like this:

▿ https://api.site.com/articles?oauth_token=deadbeef
  ▿ url: Optional(https://api.site.com/articles?oauth_token=deadbeef)
    ▿ some: https://api.site.com/articles?oauth_token=deadbeef
      - _url: https://api.site.com/articles?oauth_token=deadbeef #0
        - super: NSObject
  - cachePolicy: 0
  - timeoutInterval: 60.0
  - mainDocumentURL: nil
  - networkServiceType: __ObjC.NSURLRequest.NetworkServiceType
  - allowsCellularAccess: true
  ▿ httpMethod: Optional("POST")
    - some: "POST"
  ▿ allHTTPHeaderFields: Optional(["App-Version": "42"])
    ▿ some: 1 key/value pairs
      ▿ (2 elements)
        - key: "App-Version"
        - value: "42"
  ▿ httpBody: Optional(19 bytes)
    ▿ some: "body=Hello%20world!"
  - httpBodyStream: nil
  - httpShouldHandleCookies: true
  - httpShouldUsePipelining: false

That’s a lot of surface area covered with very little effort!

An app may similarly communicate with an analytics service and buffer tracking events. You could use snapshot testing to capture all of these events and their properties.

func testFavorite() {
  let analytics = Analytics()
  analytics.track(event: .favorite(articleId: 1))
  analytics.track(event: .unfavorite(articleId: 1))
  
  assertSnapshot(matching: analytics)
}

The resulting snapshot is a detailed view of the underlying data.

▿ App.Analytics #0
  ▿ buffer: 2 elements
    ▿ 2 key/value pairs
      ▿ (2 elements)
        - key: "event"
        - value: "Favorite"
      ▿ (2 elements)
        - key: "properties"
        ▿ value: 30 key/value pairs
          ▿ (2 elements)
            - key: "article_id"
            - value: 1
          ▿ (2 elements)
            - key: "app_version"
            - value: 42
          ▿ (2 elements)
            - key: "ios_version"
            - value: "11.0"
          …
    ▿ 2 key/value pairs
      ▿ (2 elements)
        - key: "event"
        - value: "Unfavorite"
      ▿ (2 elements)
        - key: "properties"
        ▿ value: 30 key/value pairs
          ▿ (2 elements)
            - key: "article_id"
            - value: 1
          ▿ (2 elements)
            - key: "app_version"
            - value: 42
          ▿ (2 elements)
            - key: "ios_version"
            - value: "11.0"
          …

With server-side Swift maturing, snapshots could cover an entire request/response lifecycle!

func testBasicAuthWhenUnauthorized() {
  let app = basicAuth(user: "admin", password: "secret")
    .writeStatus(.ok)
    .respond(html: "<p>Private!</p>")
  let request = URLRequest(url: URL(string: "/")!)

  assertSnapshot(matching: app.connect(request))
}

A CustomStringConvertible-like protocol (Snapshottable?) might render a snapshot like this:

▿ Request
  GET /

▿ Response
  HTTP/1.1 401 Unauthorized
  Content-Type: text/plain
  WWW-Authenticate: Basic

  Please authenticate.

Once you start writing your first snapshot tests you begin to realize how many places in your code have only partial coverage and could be improved.

How do we make these ideas realities? Let’s explore what goes into writing a snapshot test and build one from first principles.

Snapshots from Scratch

Let’s pretend we’re writing an application that keeps its state in a centralized store.3 For the moment, we have a structure representing our user and a container structure holding our app’s state, namely whether or not a user is logged in or available.

struct User {
  let name: String
}

struct AppState {
  var user: User? = nil
}

We can start by writing a unit test against the initial state of our data structure. In this case, there should be no user attached by default.

final class AppStateTests: XCTestCase {
  func testInitialState() {
    let state = AppState()

    XCTAssertNil(state.user)
  }
}

This test is simple and explicit: it completely covers our expectations of state, and it passes! Over time, though, we add new properties to AppState, and this test still passes. Nothing reminds us to revisit it as associated code changes. Nothing encourages us to write additional assertions. A snapshot test, on the other hand, would capture the entire structure of our data and force us to reflect and return as our app model grows and changes: it would capture default state over time.

Let’s ditch our unit test and write the bookend of a snapshot test instead. We start by preparing our data, as we did in our unit test, but this time we’ll assert against a snapshot using a reference.

final class AppStateTests: XCTestCase {
  func testInitialState() {
    let state = AppState()

    // ???

    XCTAssertEqual(reference, snapshot)
  }
}

The snapshot can be a string describing our data. The reference can be an empty string to satisfy the compiler so that we can run the test.

func testInitialState() {
  let state = AppState()

  let snapshot = String(describing: state)
  let reference = ""

  XCTAssertEqual(reference, snapshot)
}

We expect this test to fail, and it does!

XCTAssertEqual failed: ("") is not equal to ("AppState(user: nil)") -

We can take advantage of this failure message, though, and plug it into our reference data.

func testInitialState() {
  let state = AppState()

  let snapshot = String(describing: state)
  let reference = "AppState(user: nil)"

  XCTAssertEqual(reference, snapshot)
}

It passes! We just test-drove our first snapshot test! It works as-is, but there are a couple problems: the way we take a snapshot doesn’t produce the most human-readable thing in the world (large data structures will spread out further and further over a single, long line), and String.init(describing:) uses the CustomStringConvertible protocol, which can omit information about our data.

Swift provides another function, dump, which solves both of these problems: it takes any data and renders its raw contents over multiple lines. Converting our test to use dump is a succinct exercise involving mutability and an in-out parameter.

func testInitialState() {
  let state = AppState()

  var snapshot = ""
  dump(state, to: &snapshot)

  let reference = "AppState(user: nil)"
  XCTAssertEqual(reference, snapshot)
}

Now we can re-run the test and, on failure, copy over our new reference.

func testInitialState() {
  let state = AppState()

  var snapshot = ""
  dump(state, to: &snapshot)

  let reference = """
▿ SnapshotsTests.AppState
  - user: nil
"""
  XCTAssertEqual(reference, snapshot)
}

Manually writing and updating snapshots isn’t very efficient. FBSnapshotTestCase and Jest write their snapshots to disk for us, so let’s support that next.

We’ll need a place to store our snapshots. FBSnapshotTestCase suggests an Xcode build scheme configuration to determine where screen shots are saved, but Jest automatically saves snapshots adjacent to their test files, nested in a __snapshots__ directory. We can follow Jest’s lead and avoid configuration by using Swift’s #file identifier to produce a directory name relative to our test file.

  let snapshotDirectoryUrl = URL(fileURLWithPath: "\(#file)")
    .deletingPathExtension()

We delete the .swift path extension and have a snapshot directory name at hand!

How do we differentiate each snapshot file? We can use Swift’s #function identifier to name the snapshot after the current test function.

  let snapshotFileUrl = snapshotDirectoryUrl
    .appendingPathComponent(#function)
    .appendingPathExtension("txt")

We append .txt to make the snapshots easy to preview in Finder and wherever else extension determines file kind.

Now that we have our directory and file templates, let’s use them! We can always attempt to create the snapshot directory in the case it doesn’t already exist.

  let fileManager = FileManager.default
  try! fileManager
    .createDirectory(at: snapshotDirectoryUrl,
                     withIntermediateDirectories: true)

We still need to determine whether or not we have an existing snapshot. If we do, we can test that our current data matches. If we don’t, we can record a snapshot for next time.

  if fileManager.fileExists(atPath: snapshotFileUrl.path) {
    let reference =
      try! String(contentsOf: snapshotFileUrl, encoding: .utf8)
    XCTAssertEqual(reference, snapshot)
  } else {
    try! snapshot
      .write(to: snapshotFileUrl, atomically: true, encoding: .utf8)
    XCTFail("Wrote snapshot:\n\n\(snapshot)")
  }

Upon recording a snapshot, we fail the test to encourage us to confirm that the snapshot looks right.

Let’s take a look at our test in full.

func testInitialState() {
  let state = AppState()

  var snapshot = ""
  dump(state, to: &snapshot)

  let snapshotDirectoryUrl = URL(fileURLWithPath: "\(#file)")
    .deletingPathExtension()

  let snapshotFileUrl = snapshotDirectoryUrl
    .appendingPathComponent(#function)
    .appendingPathExtension("txt")

  let fileManager = FileManager.default
  try! fileManager
    .createDirectory(at: snapshotDirectoryUrl,
                     withIntermediateDirectories: true)

  if fileManager.fileExists(atPath: snapshotFileUrl.path) {
    let reference =
      try! String(contentsOf: snapshotFileUrl, encoding: .utf8)
    XCTAssertEqual(reference, snapshot)
  } else {
    try! snapshot
      .write(to: snapshotFileUrl, atomically: true, encoding: .utf8)
    XCTFail("Wrote snapshot:\n\n\(snapshot)")
  }
}

That’s not so bad, but it’s hardly scalable per snapshot. Let’s extract this logic to a function that’s reusable across tests.

// SnapshotTesting.swift

func assertSnapshot(matching any: Any) {
  // ???
}

How much of our logic do we need to change? Not much! We can start by replacing our state variable with our any input.

func assertSnapshot(matching any: Any) {
  var snapshot = ""
  dump(any, to: &snapshot)

  let snapshotDirectoryUrl = URL(fileURLWithPath: "\(#file)")
    .deletingPathExtension()
  let snapshotFileUrl = snapshotDirectoryUrl
    .appendingPathComponent(#function)
    .appendingPathExtension("txt")

  let fileManager = FileManager.default
  try! fileManager
    .createDirectory(at: snapshotDirectoryUrl,
                     withIntermediateDirectories: true)

  if fileManager.fileExists(atPath: snapshotFileUrl.path) {
    let reference =
      try! String(contentsOf: snapshotFileUrl, encoding: .utf8)
    XCTAssertEqual(reference, snapshot)
  } else {
    try! snapshot
      .write(to: snapshotFileUrl, atomically: true, encoding: .utf8)
    XCTFail("Wrote snapshot:\n\n\(snapshot)")
  }
}

Now our test case reduces to this:

final class AppStateTests: XCTestCase {
  func testInitialState() {
    let state = AppState()

    assertSnapshot(matching: state)
  }
}

When we run our suite, the snapshot is recorded to disk, and the test passes on the next run, but we notice a problem. Xcode highlighted our failure inside our assertSnapshot function, specifically on the line that calls XCTFail. If we were to try to write a second snapshot test, we’d hit another issue: snapshots now write out relative to our SnapshotTesting.swift file and are always called assertSnapshot(matching:).txt, named after our new helper!

We can solve these problems by knowing a little more about Swift’s #-prefixed identifiers. When #file and #line are evaluated as default parameters to a function, they refer to the calling function’s file name and line number. (The #function identifier likewise refers to the caller’s function name.) XCTest assertions define default file and line parameters and Xcode uses them to highlight failures. If we use these defaults ourselves, we can pass them along so that Xcode highlights things appropriately, and so that our snapshot files are saved appropriately.

We’re left, finally, with a reusable function that manages snapshots for us and highlights regressions inline.

func assertSnapshot(
  matching any: Any,
  file: StaticString = #file,
  function: String = #function,
  line: UInt = #line)
{
  var snapshot = ""
  dump(any, to: &snapshot)

  let snapshotDirectoryUrl = URL(fileURLWithPath: "\(file)")
    .deletingPathExtension()
  let snapshotFileUrl = snapshotDirectoryUrl
    .appendingPathComponent(function)
    .appendingPathExtension("txt")

  let fileManager = FileManager.default
  try! fileManager
    .createDirectory(at: snapshotDirectoryUrl,
                     withIntermediateDirectories: true)

  if fileManager.fileExists(atPath: snapshotFileUrl.path) {
    let reference =
      try! String(contentsOf: snapshotFileUrl, encoding: .utf8)
    XCTAssertEqual(reference, snapshot, file: file, line: line)
  } else {
    try! snapshot
      .write(to: snapshotFileUrl, atomically: true, encoding: .utf8)
    XCTFail("Wrote snapshot:\n\n\(snapshot)",
            file: file,
            line: line)
  }
}

Let’s take one last before/after look at our test.

 final class AppStateTests: XCTestCase {
   func testInitialState() {
     let state = AppState()
 
-    XCTAssertNil(state.user)
+    assertSnapshot(matching: state)
   }
 }

The snapshot version will provide much better regression test coverage as AppState grows and changes, but we can begin to see some of its shortcomings. The original test documents our expectation in-line, while the snapshot test files our expectation away and out of sight. Granular business logic is probably best tested explicitly, while snapshots can play a supporting role.

We should consider other shortcomings of our approach and see if we can address them. Recording snapshots is automatic and easy, but rerecording them requires a trip to the file system. Snapshot tests are brittle: we expect them to fail when associated code changes, so updating them should be just as easy as recording them.

Our failure output could also be better. We highlight when a snapshot doesn’t match its reference by printing the entire reference and snapshot, but we make no effort to zoom in on the difference, which could be difficult to spot. Usability improvements to these problems (and others) are suggested as exercises at the end of this article.

Let’s wrap up with a final note on tooling, because Xcode could solve some of these problems even better! Imagine a world where our environment offered first-class support for snapshots, providing a way to see them alongside their tests and an interface for viewing and updating them when they change. Xcode could even let the community fill this gap by providing more extensibility in future releases. The activities and attachments APIs help, but they fall a bit short: attachments are hidden away in the report navigator and live locally with build products, so they can’t easily be checked into source control. Xcode improvements will always be dreams before realities, but in the meantime we can file a radar to share them.

Conclusion

We’ve covered what snapshot testing is, what it isn’t, and even implemented our own tool for the job! It’s one of many testing aids to keep in mind when writing durable, regression-proof software, and it can make our lives much easier while we write and refactor our apps.

An expanded version of the code above is available as a library, SnapshotTesting, and contains many improvements, including those suggested as exercises below.

Exercises

If you’ve had fun following along, there are plenty of enhancements that can be made! Here are a few ideas to get you started:

  1. Snapshot tests are sometimes called “characterization tests.”

    Libraries like DVR, which records network responses, are not snapshot testing libraries per se, but they are related: they cache expensive, IO-bound computations to prevent slow, sporadically-failing test runs. ↩

  2. You’ll also need to make sure your test class inherits from FBSnapshotTestCase and that its recordMode is set to true. ↩

  3. It’s a popular choice these days in front-end web development (see Elm and Redux), and is seeing some adoption in our world. Chris Eidhof demonstrates the pattern in “Reducers” (be sure to watch the Swift Talk episode). ↩