Home

Awesome

Twitter Image Pipeline (a.k.a. TIP)

Background

The Twitter Image Pipeline is a streamlined framework for fetching and storing images in an application. The high level concept is that all requests to fetch or store an image go through an image pipeline which encapsulates the work of checking the in memory caches and an on disk cache before retrieving the image from over the network as well as keeping the caches both up to date and pruned.

Goals and Requirements

Twitter Image Pipeline came to fruition as numerous needs rose out of Twitter for iOS use cases. The system for image loading prior to TIP was fragile and inefficient with some severe edge cases. Designing a new framework from the ground up to holistically approach the need for loading images was the best route and led to TIP.

Architecture

Caches

There are 3 separate caches for each image pipeline: the rendered in-memory cache, the image data in-memory cache, and the on-disk cache. Entries in the caches are keyed by an image identifier which is provided by the creator of the fetch request or automatically generated from the image fetch's URL.

The image will simultaneously be loaded into memory (as raw bytes) and written to the disk cache when retrieving from the Network. Partial images will be persisted as well and not replace any completed images in the cache.

Once the image is either retrieved from any of the caches or the network, the retrieved image will percolate back through the caches in its various forms.

Caches will be configurable at a global level to have maximum size. This maximum will be enforced across all image pipeline cache's of the same kind, and be maintained with the combination of time-to-live (TTL) expiration and least-recently-used (LRU) purging. (This solves the long standing issue for the Twitter iOS app of having an unbounded cache that could consume Gigabytes of disk space).

Execution

The architecture behind the fetch operation is rather straightforward and streamlined into a pipeline (hence, "image pipeline").

When the request is made, the fetch operation will perform the following:

Preview Support

In addition to this simple progression, the fetch operation will offer the first matching (based on image identifier) complete image in the In-Memory Cache or On-Disk Cache (rendered and sized to the request's specified target sizing) as a preview image when the URLs don't match. At that point, the fetch delegate can choose to just use the preview image or continue with the Network loading the final image. This is particularly useful when the fetch image URL is for a smaller image than the image in cache, no need to hit the network :)

Progressive Support

A great value that the image pipeline offers is the ability to stream progressive scans of an image, if it is PJPEG, as the image is loaded from the Network. This progressive rendering is natively supported by iOS 8+, the OS minimum for TIP is now iOS 10+. Progressive support is opt-in and also configurable in how scans should load.

Resuming Image Downloads

As already mentioned, by persisting the partial load of an image to the On-Disk Cache, we are able to support resumable downloads. This requires no interface either, it's just a part of how the image pipeline works.

Rendering to Target Sizing

As of 2.20, the image pipeline will load the image from data to the specified target sizing of the fetch request, which avoids the overhead of loading the entire image into a large bitmap just to scale it down to the correct size. If the target sizing is larger than the image data, it will load that image bitmap and scale it up to the target sizing specified by the fetch request. If a request does not provide target sizing (or the sizing indicates to not resize), it will yield the full size image, as one would expect.

Twitter Image Pipeline features

Components of the Twitter Image Pipeline

Usage

The simplest way to use TIP is with the TIPImageViewHelper counterpart.

For concrete coding samples, look at the TIP Sample App and TIP Swift Sample App (in Objective-C and Swift, respectively).

Here's a simple example of using TIP with a UIViewController that has an array of image views to populate with images.


    /* category on TIPImagePipeline */

    + (TIPImagePipeline *)my_imagePipeline
    {
        static TIPImagePipeline *sPipeline;
        static dispatch_once_t sOnceToken;
        dispatch_once(&sOnceToken, ^{
            sPipeline = [[TIPImagePipeline alloc] initWithIdentifier:@"com.my.app.image.pipeline"];

            // support looking in legacy cache before hitting the network
            sPipeline.additionalCaches = @[ [MyLegacyCache sharedInstance] ];
        });
        return sPipeline;
    }

    // ...

    /* in a UIViewController */

    - (void)viewDidLayoutSubviews
    {
        [super viewDidLayoutSubviews];

        if (nil == self.view.window) {
            // not visible
            return;
        }

        [_imageFetchOperations makeAllObjectsPerformSelector:@selector(cancelAndDiscardDelegate)];
        [_imageFetchOperations removeAllObjects];

        TIPImagePipeline *pipeline = [TIPImagePipeline my_imagePipeline];
        for (NSInteger imageIndex = 0; imageIndex < self.imageViewCount; imageIndex++) {
            UIImageView *imageView = _imageView[imageIndex];
            imageView.image = nil;
            id<TIPImageFetchRequest> request = [self _my_imageFetchRequestForIndex:imageIndex];

            TIPImageFetchOperation *op = [pipeline operationWithRequest:request context:@(imageIndex) delegate:self];

            // fetch can complete sync or async, so we need to hold the reference BEFORE
            // triggering the fetch (in case it completes sync and will clear the ref)
            [_imageFetchOperations addObject:op];
            [[TIPImagePipeline my_imagePipeline] fetchImageWithOperation:op];
        }
    }

    - (id<TIPImageFetchRequest>)_my_imageFetchRequestForIndex:(NSInteger)index
    {
        NSAssert(index < self.imageViewCount);

        UIImageView *imageView = _imageViews[index];
        MyImageModel *model = _imageModels[index];

        MyImageFetchRequest *request = [[MyImageFetchRequest alloc] init];
        request.imageURL = model.thumbnailImageURL;
        request.imageIdentifier = model.imageURL.absoluteString; // shared identifier between image and thumbnail
        request.targetDimensions = TIPDimensionsFromView(imageViews);
        request.targetContentMode = imageView.contentMode;

        return request;
    }

    /* delegate methods */

    - (void)tip_imageFetchOperation:(TIPImageFetchOperation *)op
                didLoadPreviewImage:(id<TIPImageFetchResult>)previewResult
                         completion:(TIPImageFetchDidLoadPreviewCallback)completion
    {
        TIPImageContainer *imageContainer = previewResult.imageContainer;
        NSInteger idx = [op.context integerValue];
        UIImageView *imageView = _imageViews[idx];
        imageView.image = imageContainer.image;

        if ((imageContainer.dimension.width * imageContainer.dimensions.height) >= (originalDimensions.width * originalDimensions.height)) {
            // scaled down, preview is plenty
            completion(TIPImageFetchPreviewLoadedBehaviorStopLoading);
        } else {
            completion(TIPImageFetchPreviewLoadedBehaviorContinueLoading);
        }
    }

    - (BOOL)tip_imageFetchOperation:(TIPImageFetchOperation *)op
    shouldLoadProgressivelyWithIdentifier:(NSString *)identifier
                                URL:(NSURL *)URL
                          imageType:(NSString *)imageType
                 originalDimensions:(CGSize)originalDimensions
    {
        // only load progressively if we didn't load a "preview"
        return (nil == op.previewImageContainer);
    }

    - (void)tip_imageFetchOperation:(TIPImageFetchOperation *)op
          didUpdateProgressiveImage:(id<TIPImageFetchResult>)progressiveResult
                           progress:(float)progress
    {
        NSInteger idx = [op.context integerValue];
        UIImageView *imageView = _imageViews[idx];
        imageView.image = progressiveResult.imageContainer.image;
    }

    - (void)tip_imageFetchOperation:(TIPImageFetchOperation *)op
                  didLoadFinalImage:(id<TIPImageFetchResult>)finalResult
    {
        NSInteger idx = [op.context integerValue];
        UIImageView *imageView = _imageViews[idx];
        imageView.image = finalResult.imageContainer.image;

        [_imageFetchOperations removeObject:op];
    }

    - (void)tip_imageFetchOperation:(TIPImageFetchOperation *)op
            didFailToLoadFinalImage:(NSError *)error
    {
        NSInteger idx = [op.context integerValue];
        UIImageView *imageView = _imageViews[idx];
        if (!imageView.image) {
            imageView.image = MyAppImageLoadFailedPlaceholderImage();
        }

        NSLog(@"-[%@ %@]: %@", NSStringFromClass([self class]), NSStringFromSelector(_cmd), error);
        [_imageFetchOperations removeObject:op];
    }

Inspecting Image Pipelines

Twitter Image Pipeline has built in support for inspecting the caches via convenience categories. TIPGlobalConfiguration has an inspect: method that will inspect all registered TIPImagePipeline instances (even if they have not been explicitely loaded) and will provide detailed results for those caches and the images there-in. You can also call inspect: on a specific TIPImagePipeline instance to be provided detailed info for that specific pipeline. Inspecting pipelines is asynchronously done on background threads before the inspection callback is called on the main thread. This can provide very useful debugging info. As an example, Twitter has built in UI and tools that use the inspection support of TIP for internal builds.

License

Copyright 2015-2020 Twitter, Inc.

Licensed under the Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0

Security Issues?

Please report sensitive security issues via Twitter's bug-bounty program (https://hackerone.com/twitter) rather than GitHub.