Awesome
Hadoop File System Forensics Toolkit (HDFS FTK)
View and extract files from an offline image of Hadoop file system.
Supports:
- Support for multiple datanodes
- Support of fsimage XML format
- Search filenames and filter by filetype
- File recovery while preserving metadata
Motivation
Hadoop File Systems is one of the most widely used distributed file systems in the world. However, forensic techniques to analyze and audit the systems remain limited.
In HDFS, metadata is separated from the actual data blocks. The namenode contains metadata (file name, timestamps, permissions); while actual data is stored in the datanodes in blocks. Although HDFS has command client tools to manage the extraction files, it only works with a running cluster of HDFS machines. This tool aims to provide investigators with the ability to perform forensics analysis on offline evidence captures of Hadoop File System images.
Instructions
PreRequisites
- Python 3 and above
- PrettyTable:
pip install prettytable
Evidence Acquisition Procedure
- Obtain metadata from namenode
$namenode: hdfs dfsadmin -safemode enter
$namenode: hdfs dfsadmin –saveNamespace
$namenode: hdfs oiv -i <PATH_TO_FSIMAGE> -o <FILE> -p XML
- Archive data from datanodes
$datanodes: tar czf datanodex.tar.gz $HADOOP_HOME/Hadoop_data
- SCP files to a local forensic workstation and untar the datanodes' data directory.
Usage
Hadoop Forensics File System Forensics Tool Kit (HDFS FTK) ver. 1.0
DEBUG MODE ON: Test values will be used for fsimage, output and datanode directories.
usage: hdfs_ftk.py [-h] [-v] [-displayfsimage] [-showfilesonly] [-showdironly]
[-filterByName FILTERBYNAME] [-d D] [-o O] [-r R] -f F
HDFS Forensics Toolkit.
optional arguments:
-h, --help show this help message and exit
-v, --verbose Verbose mode
-displayfsimage Print out fsimage table
-showfilesonly Show files only
-showdironly Show directories only
-filterByName FILTERBYNAME
Filter fsimage display by filename
-d D Number of Datanodes
-o O, -output O output directory
-r R, -recover R Recover file given ID number
Required named arguments:
-f F, -fsimage F Path to fsimage
Example commands:
- To view contents of HDFS File System:
python hdfs_ftk.py -f fsimage.xml
- Display fsimage:
python hdfs_ftk.py -f test/fsimage.xml -displayfsimage
- Filtering by name:
python hdfs_ftk.py -f test/fsimage.xml -displayfsimage -filterByName tartans
- Extract block id
16386
from HDFS with three datanodes:python hdfs_ftk.py -f test/fsimage.xml -v -r 16386 -o /output -d 3
Demo videos
- Extracting files and displaying fsimage information: https://youtu.be/I1I08ixInxI (Old video)
License
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.
Future works
- Large file recovery: Files more than the blocksize specified in the configurations are broken into several blocks. This tool is unable to recover large data files, yet.
- Recover deleted files: Deleted files are kept on HDFS as long as it is still within
fs.trash.interval
. It might be possible to recover recently deleted files by referencing that property. - Tool to automate evidence collection
- Support for mapred tasks and jobs audit