Awesome

Show Facebook Computer Vision Tags Chrome Extension

Since April 2016, Facebook has been automatically adding alt tags to images you upload that are populated with keywords representing the content of your images:

<img csrc="https://facebook.com/some-url.jpg"
alt="Image may contain: golf, grass, outdoor and nature"
width="316" height="237">

They are labeling your images using a Deep ConvNet built by Facebook's FAIR team.

On one hand, this is really great. It improves accessibility for blind users who depend on screen readers which are only capable of processing text.

But I think a lot of internet users don't realize the amount of information that is now routinely extracted from photographs. Facebook (and Google, Apple, Amazon, etc) can easily tell from your photographs if you have a pet dog, if you collect cameras, if you play golf, if you have children, or if you are just really into sunglasses. There's nothing stopping them from using this information to show you relevant ads just based on photos of you - even if another user uploaded the photo and didn't even directly tag you!

This is a very simple Chrome Extension that I hacked together in a few minutes to make it easy to see the tags that Facebook is automatically applying to you and your friends' photos. Once you install it, all the photos you see on your Facebook timeline will automatically be overlaid with their tags, like this:

The goal is simply to make everyone aware of the kind of information that is routinely extracted from your own images today. This isn't some technology that is 'coming soon', but this is something in wide-spread production use now by every major internet company. You can even learn how to implement this kind of system yourself using widely available open-source software.

Here's some examples of the kinds of things that Facebook detects in your photos:

Objects in your photos

It's possible to tell what kinds of objects are in your photos and identify your interests based on that.

Activities you are doing

Whether you are eating, running, or playing golf, it's possible for an algorithm to tell what you are doing in your photographs.

When and where a photo was taken

You can tell if a photo was taken inside or outside, what the time of day was, what physical landmarks (mountains, trees, water, etc) were around, and so on.

Events you are attending

It's even possible to tell which kind of sporting event you are attending.

These are just a few examples. See what kinds of things Facebook has found in your own photos!

Installing this Chrome Extension

Option 1: From the Chrome Web Store

Update: Sorry, this is extension not currently available in the Chrome Web Store. Facebook filed a Trademark complaint due to the name of the extension having the word "Facebook" in it, so I had to remove it.

Option 2: From Source

Clone this repo so you have a copy in a folder locally.
Open chrome://extensions in the location or go to Tools > Extensions
Enable Developer mode by checking the checkbox in the upper-right corner.
Click on the button labelled Load unpacked extension....
Select the directory where you cloned this repo to.
Visit Facebook!

Installing in Firefox

Option 1: From the Add-ons website:

Click here and choose "+ Add to Firefox"