Home

Awesome

Earth Lens

Please note that Microsoft will not officially support this repository starting Feb 28, 2019. However, contributions from the community are still welcome!

Earth Lens leverages Microsoft’s AI tools to detect objects in aerial imagery and translate raw image data into digestible insights for researchers. With this tool, researchers will be able to automatically extract information from a large dataset and have the analysis presented in a useful and interactive view. As a result, this tool can be useful in various industrial or agricultural applications to further Microsoft’s AI For Earth commitment to sustainability.

The application utilizes Apple's CoreML framework to run the machine learning model locally on the iPad, showcasing the power and capability of the Intelligent Edge. Users are able to upload images, process the locally without the internet, and then scroll through and analyze the data. Earth Lens helps users automatically aggregate, count, classify and label the objects in satellite images. Colored bounding boxes are drawn around detected objects, which users can toggle on and off to control the information displayed. A time series analysis view allows users to look at images over time alongside a data visualization chart to identify trends and patterns.

Device Compatibility

This application runs on iPad devices with iOS 9.0 or above.

Features

Project Structure

Xamarin

This folder contains the application Visual Studio solution, which consists of five separate projects:

EarthLens

EarthLens.iOS

EarthLens.Tests

EarthLens.Tests.iOS

EarthLens.TestUtils

Getting Started with Development

Setup on Windows

Windows 7 or above

  1. Update your Windows if you are on a version older than Windows 7, as specified in the Xamarin System requirements.

Visual Studio 2017 (Community, Professional, or Enterprise) and Xamarin 4.5.0 or above

  1. Follow the Windows installation instructions to install Visual Studio 2017 (Community, Professional, or Enterprise) and the Xamarin component on Windows.

Mac Agent

  1. Follow the official Xamarin documentation to Pair to Mac for Xamarin.iOS Development.

Setup on macOS

macOS High Sierra 10.12 or above

  1. Update your OS version if you are on a version older than 10.12, as specified in the Xamarin System requirements.

Xcode 9 or above

  1. Download and install Xcode from App Store on macOS.
  2. Launch Xcode.
  3. Read and accept the License Agreement.
  4. Let Xcode install additional components (this includes command-line tools that Visual Studio needs).

Visual Studio for Mac

  1. Follow the official Microsoft documentation to Setup and Install Visual Studio for Mac
    • Make sure at least the following options are selected when choosing what to install
      • iOS
      • .NET Core

Getting the Source Code

  1. Clone the repository to your development machine:
git clone https://github.com/Microsoft/Earth-Lens.git
  1. Open Xamarin/EarthLens.sln in Visual Studio (for Windows) or Visual Studio for Mac (for macOS).

Building/Running the App

Running on a simulator

  1. In Visual Studio, select the following configuration:

    ProjectEarthLens.iOS
    ConfigurationDebug
    PlatformiPhoneSimulator
    Device<select iPad>

  2. Run the app on the simulator.

Running on a physical device

  1. Connect the physical iPad on which the app is to run to a Mac or Mac Agent.

  2. If prompted to "Trust This Computer" on the iPad, select "Trust" and enter the iPad passcode.

  3. If iTunes launches, wait for it to identify the device, unplug the device, and then plug the device back in.

  4. For the first time, follow the Xamarin documentation to setup Device Provisioning for Xamarin.iOS.

    • For the product name, enter: EarthLens
    • For the bundle ID, enter: com.companyname.EarthLens

  5. In Visual Studio, select the following configuration:

    ProjectEarthLens.iOS
    ConfigurationDebug
    PlatformiPhone
    Device<your iPad name>

  6. Run the app on the iPad.

App Center (Optional)

Microsoft's Visual Studio App Center is used for the development of Earth Lens. App Center is a collection of services that enable developers to better manage and ship applications on any platform. The following App Center services are used during the development of Earth Lens:

App Center is free to try out. To get started, visit https://appcenter.ms/ to get an App Secret key, and follow the Get Started with Xamarin instructions for Xamarin.iOS.

Earth Lens uses Build scripts and Environment variables to load the App Center App Secret key into Earth Lens during runtime. Under the EarthLens.iOS project, there is a appcenter-pre-build.sh file that is run each time a build is triggered on App Center, and an Environment.txt file where the App Secret key is entered.

App Center supports automated builds and distributions. For more information, check out Building Xamarin apps for iOS.

CoreML Integration

  1. Acquire the compiled .mlmodelc format model of your original .mlmodel format model, as indicated here. Put the folder containing the compiled model under the Resources folder of the EarthLens.iOS project.
  2. Acquire the model's anchors in the training process. Store them in a CSV file, and put the file under the Resources folder of the EarthLens.iOS project.
  3. Make sure that the ResourceName (for the model path) , and AnchorsResrouceName (for the anchor file path) values in the Constants.cs file under the EarthLens.iOS project are correct.
  4. Inspect the original .mlmodel file, and acquire the input and output feature names. Make sure that the DefaultInputFeatureName ,ModelOutputBoxFeatureName and ModelOutputClassFeatureName values in the Constants.cs file under the EarthLens.iOS project correspond to the feature names of your model.

Third-Party Library Dependencies

Contributing

This project welcomes contributions and suggestions. There are many ways to contribute to Earth Lens:

Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

License

Earth Lens is released under the MIT License.

See LICENSE for details.

DISCLAIMERS

The model is intended for demonstration purposes only.

THE SOFTWARE CODE IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL MICROSOFT OR ITS LICENSORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THE SOFTWARE CODE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Acknowledgement

Developer Interns

Designer Intern

Program Manager Intern

Coaches

Program Manager

Technical Advisor