Home

Awesome

<!-- *** Thanks for checking out the Best-README-Template. If you have a suggestion *** that would make this better, please fork the repo and create a pull request *** or simply open an issue with the tag "enhancement". *** Thanks again! Now go create something AMAZING! :D -->

MF-LBM: A Portable, Scalable and High-performance Lattice Boltzmann Code for DNS of Flow in Porous Media

<img src="images/core-dns.png" alt="drawing" width="900"/> <br/> <br/> <!-- TABLE OF CONTENTS --> <details open="open"> <summary>Table of Contents</summary> <ol> <li><a href="#about-the-code">About The Code</a></li> <ul> <li><a href="#features">Features</a></li> </ul> <ul> <li><a href="#components">Components</a></li> </ul> <ul> <li><a href="#technical-details-of-the-main-simulation-code">Technical details of the main simulation code</a></li> </ul> <ul> <li><a href="#citing-mf-lbm">Citing MF-LBM</a></li> </ul> <li><a href="#performance-benchmarking">Performance Benchmarking</a></li> <li><a href="#build-instructions">Build Instructions</a></li> <ul> <li><a href="#prerequisites">Prerequisites</a></li> </ul> <ul> <li><a href="#installation">Installation</a></li> </ul> <li><a href="#usage">Usage</a></li> <ul> <li><a href="#pre-processing-code">Pre-processing code</a></li> <!-- #name must be identical to the bullet name (including dash) except capital letter and space (replaced by dash) --> </ul> <ul> <li><a href="#the-main-simulation-code">The main simulation code</a></li> </ul> <ul> <li><a href="#output-files">Output files</a></li> </ul> <li><a href="#important-notes">Important Notes</a></li> <li><a href="#acknowledgements">Acknowledgements</a></li> <li><a href="#license">License</a></li> <li><a href="#contact">Contact</a></li> </ol> </details> <br/> <!-- ABOUT THE CODE -->

About The Code

MF-LBM 1 2 is a high-performance lattice Boltzmann (LB) code for direct numerical simulation (DNS) of flow in porous media, primarily developed by Dr. Yu Chen (LANL), under the supervision of Prof. Albert Valocchi (UIUC), Dr. Qinjun Kang (LANL) and Dr. Hari Viswananthan (LANL). 'MF' refers to microfluidics or 'Magic Find'. The code was first developed at University of Illinois at Urbana-Champaign based on a mainstream LB color-gradient multiphase model and further improved at Los Alamos National Laboratory by implementing the Continuum-Surface-Force and geometrical wetting models to reduce spurious currents so that the inertial effects in scCO<sub>2</sub> and brine displacement in porous media can be accounted for 2. In addition, a single-phase flow solver is also provided for absolute permeability measurement or DNS of turbulent flow. Only double precision is supported.

Features

Components

Technical details of the main simulation code

Modern manycore processors/coprocessors, such as GPUs, are developing rapidly and greatly boost computing power. These processors not only provide much higher FLOPS (floating-point operations per second) but also much higher memory bandwidth compared with traditional CPU. One of the most attractive features of the lattice Boltzmann method (LBM) is that it is explicit in time, has nearest neighbor communication, and the computational effort is in the collision step, which is localized at a grid node. For these reasons, the LBM is well suited for manycore processors which require a higher degree of explicit parallelism. The data movement in the LBM is much more intensive than for traditional CFD considering that the D3Q19 lattice model has 19 lattice velocities. Given the current state of computational hardware, in particular the relative speed and capacity of processors and memory, the LBM is a memory-bandwidth-bound numerical method. The high memory bandwidth provided by GPUs greatly benefits the LBM.

The code is written on Fortran 90 and employs MPI-OpenACC/OpenMP hybrid programing model. The main reason that we chose OpenACC/OpenMP (directive-based parallel programming models) over CUDA is that we want to keep the code portable across different computing platforms so that we are not limited by the NVIDIA GPU solution. As GPU and Intel Xeon Phi processor (and even latest CPU from Intel with AVX512 instructions) rely heavily on SIMT/SIMD, the optimization strategy for these manycore processors/coprocessors are similar, which enables us to achieve reasonable performance across different platforms:

Citing MF-LBM

Chen, Y., Valocchi, A., Kang, Q., & Viswanathan, H. S. (2019). Inertial effects during the process of supercritical CO2 displacing brine in a sandstone: Lattice Boltzmann simulations based on the continuum‐surface‐force and geometrical wetting models. Water Resources Research, 55, 11144– 11165. https://doi.org/10.1029/2019WR025746

<!-- Build Instructions -->

Build Instructions

Prerequisites

Installation

  1. Clone the repo

    git clone https://github.com/lanl/MF-LBM.git
    
  2. Initiate submodules for external non-code files (geometry files used in the test suites)

    cd path-to-MF-LBM
    git submodule init
    git submodule update
    
  3. Make

    cd path-to-MF-LBM/multiphase_3D
    
    1. CPU version
      # Make necessary changes to the makefile:
      # Choose CPU as the architecture option.
      # Choose a proper compiler.
      # Enabling OpenMP is recommended.
      # See instructions in the makefile for more information.  
      your-preferred-editor makefile
      make
      # MF_LBM.cpu will be generated
      
    2. GPU version
      # Make necessary changes to the makefile:
      # Choose GPU as the architecture option.
      # Choose the NVIDIA compiler (recommended for NVIDIA GPU).
      # OpenMP must be disabled.
      # See instruction in the makefile for more information.  
      your-preferred-editor makefile
      make
      # MF_LBM.gpu will be generated
      
    3. MIC (Intel Xeon Phi) version
      # Make necessary changes to the makefile:
      # Choose MIC as the architecture option.
      # Choose a proper compiler (Intel compiler is recommended).
      # OpenMP and AVX512 must be enabled for the MIC version.
      # See instructions in the makefile for more information.  
      your-preferred-editor makefile   
      make
      # MF_LBM.mic will be generated
      
  4. Configure the run script (for CPU platform)

    cd working_directory
    cp path-to-MF-LBM/multiphase_3D/run_template/template-config.sh ./config.sh
    # Make necessary changes to config.sh (i.e., paths and run command). 
    # See instructions in template-config.sh for more information.
    your-preferred-editor config.sh
    ./config.sh
    
    # A script, irun.sh, will be generated.  
    
    # If OpenMP is enabled (recommended for CPU, MIC, and ARM platform), then run the following command:
    # export OMP_NUM_THREADS=n, where n is recommended to be the core or thread count of the UMA domain of the CPU. Hyperthreading may help in some cases.
    
    # At least one MPI rank per UMA domain is recommended.
    
    # For GPU platform, the number of MPI processes should equal to the total number of GPUs: one MPI rank per GPU.
    
  5. Modify the simulation control file

    # Make necessary changes to simulation_control.txt. See instructions in simulation_control.txt for more information.
    your-preferred-editor simulation_control.txt
    
  6. Run the program

    # See instructions in template-config.sh and irun.sh for more information.
    ./irun.sh new
    
<br/> <!-- USAGE EXAMPLES -->

Usage

Pre-processing code

The main simulation code

Check out template-simulation_control.txt for more information regarding simulation control. The units used in the simulation control file are all lattice units. One can control capillary number, contact angle, absolute values of surface tension and viscosities to link the simulation with a physical system. In particular, the absolute values of surface tension and viscosities will affect Reynolds number even when the capillary number is fixed. The Ohnesorge number is recommended to control the parameters when inertial effects are not negligible 2.

Output files

Three output directories will be created:

<br/>

Important Notes

<br/> <!-- LICENSE -->

Acknowledgements

This work was supported by LANL's LDRD program and was supported as part of the Center for Geologic Storage of CO<sub>2</sub>, an Energy Frontier Research Center funded by the U.S. Department of Energy, Office of Science, Basic Energy Sciences, under Award DE-SC0C12504.

<br/> <!-- LICENSE -->

License

Distributed under the BSD-3 License. See LICENSE for more information.

© 2021. Triad National Security, LLC. All rights reserved.

This program was produced under U.S. Government contract 89233218CNA000001 for Los Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC for the U.S. Department of Energy/National Nuclear Security Administration. All rights in the program are reserved by Triad National Security, LLC, and the U.S. Department of Energy/National Nuclear Security Administration. The Government is granted for itself and others acting on its behalf a nonexclusive, paid-up, irrevocable worldwide license in this material to reproduce, prepare derivative works, distribute copies to the public, perform publicly and display publicly, and to permit others to do so.

<br/> <!-- CONTACT -->

Contact

Dr. Yu Chen - yu_chen_007@outlook.com

MF-LBM-live which is forked from this repo will be used to develop new features by Dr. Yu Chen.

Dr. Qinjun Kang - qkang@lanl.gov

<br/> <!-- MARKDOWN LINKS & IMAGES -->