Home

Awesome

What is NeoPI?

NeoPI is a Python script that uses a variety of statistical methods to detect obfuscated and encrypted content within text/script files. The intended purpose of NeoPI is to aid in the detection of hidden web shell code. The development focus of NeoPI was creating a tool that could be used in conjunction with other established detection methods such as Linux Malware Detect or traditional signature/keyword based searches.

NeoPI recursively scans through the file system from a base directory and will rank files based on the results of a number of tests. It also presents a “general” score derived from file rankings within the individual tests.

Requirements

NeoPI is platform independent and can be run on any system with Python 2.6 or greater installed installed. The user running the script should have read access to all of the files that will be scanned.

How to use it

NeoPI is platform independent and will run on both Linux and Windows. To start using NeoPI first checkout the code from our github repository

git clone https://github.com/Neohapsis/NeoPI.git

The small NeoPI script is now in your local directory. We are going to go though a few examples on Linux and then switch over to Windows.

Let’s run neopi.py with the -h flag to see the options.

[sbehrens@WebServer2 opt]$ ./neopi.py -h
Usage: neopi.py [options] <start directory> <OPTIONAL: filename regex>

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -C FILECSV, --csv=FILECSV
						generate CSV outfile
  -a, --all             Run all tests [Entropy, Longest Word, Compression
  -e, --entropy         Run entropy Test
  -l, --longestword     Run longest word test
  -c, --ic              Run IC test
  -A, --auto            Run auto file extension tests

Let’s break down the options into greater detail.

-C FILECSV, --csv=FILECSV

This generates a CSV output file containing the results of the scan.

-a, --all

This runs all tests including entropy, longest word, and index of coincidence. In general, we suggest running all tests to build the most comprehensive list of possible web shells.

-e, --entropy

This flag can be set to run only the entropy test.

-l, --longestword

This flag can be set to run only the longest word test.

-c, --ic

This flag can be set to run only the Index of Coincidence test.

-A, --auto

This flag runs an auto generated regular expression that contains many common web application file extensions. This list is by no means comprehensive but does include a good ‘best effort’ scan if you are unsure of what web application languages your server is running. The current list of extensions are included below:

valid_regex = re.compile('\.php|\.asp|\.aspx|\.sh|\.bash|\.zsh|\.csh|\.tsch|\.pl|\.py|\.txt|\.cgi|\.cfm')

Now that we are familiar with the flags and we have downloaded a copy of the script from GIT, let’s go head and run it on a web server we think may be infected with obfuscated web shells.

[sbehrens@WebServer2 opt]$ sudo ./neopi.py -C scan1.csv -a -A /var/www/

The resulst of the scan we be displayed to console as well as written to 'scan1.csv'. Here is an example of the scan results:

[root@WebServer2 opt]# python neopi.py -a -A /var/www/html/

[[ Average IC for Search ]]
0.0372337579606

[[ Top 10 IC files ]]
  0.0156    /var/www/html/webmedia/shell3.php
  0.0178    /var/www/html/phpadmin/phpMyAdmin-3.3.8-all-languages/lang/chinese_simplified-utf-8.inc.php
  0.0184    /var/www/html/wordpress/wordpress/wp-admin/weevely.php
  0.0217    /var/www/html/joomla/templates/system/index.php
  0.0217    /var/www/html/joomla/administrator/templates/system/index.php
  0.0225    /var/www/html/wordpress/wordpress/wp-admin/js/revisions-js.php
  0.0229    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-ch.php
  0.0239    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-zh.php
  0.0240    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-zh_cn.php
  0.0248    /var/www/html/phpadmin/shell2.php

[[ Top 10 entropic files ]]
  6.3978    /var/www/html/phpadmin/phpMyAdmin-3.3.8-all-languages/lang/chinese_simplified-utf-8.inc.php
  6.0651    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-ch.php
  6.0061    /var/www/html/webmedia/shell3.php
  5.9870    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-zh.php
  5.9797    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-zh_cn.php
  5.9245    /var/www/html/phpadmin/shell2.php
  5.8895    /var/www/html/wordpress/wordpress/wp-admin/js/revisions-js.php
  5.8580    /var/www/html/phpadmin/phpMyAdmin-3.3.8-all-languages/lang/japanese-utf-8.inc.php
  5.8400    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-ja.php
  5.7602    /var/www/html/wordpress/wordpress/wp-admin/weevely.php

[[ Top 10 longest word files ]]
  111571    /var/www/html/webmedia/shell3.php
	2510    /var/www/html/webmedia/htdocs/templates/main.tpl.php
	1312    /var/www/html/joomla/shell.php
	 728    /var/www/html/wordpress/wordpress/wp-admin/js/revisions-js.php
	 536    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Libs/QuickForm/3.2.11/HTML/QuickForm/Rule/Email.php
	 522    /var/www/html/wordpress/wordpress/wp-includes/functions.php
	 516    /var/www/html/phpadmin/phpMyAdmin-3.3.8-all-languages/libraries/tcpdf/tcpdf.php
	 516    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Libs/PHPExcel/lib/PHPExcel/Shared/PDF/tcpdf.php
	 516    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Libs/TCPDF/tcpdf4/tcpdf.php
	 516    /var/www/html/joomla/libraries/tcpdf/tcpdf.php

[[ Highest Rank Files Based on test results ]]
	 83%    /var/www/html/webmedia/shell3.php
	 56%    /var/www/html/phpadmin/phpMyAdmin-3.3.8-all-languages/lang/chinese_simplified-utf-8.inc.php
	 43%    /var/www/html/wordpress/wordpress/wp-admin/js/revisions-js.php
	 36%    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-ch.php
	 26%    /var/www/html/webmedia/htdocs/templates/main.tpl.php
	 26%    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-zh.php
	 23%    /var/www/html/wordpress/wordpress/wp-admin/weevely.php
	 23%    /var/www/html/joomla/shell.php
	 20%    /var/www/html/joomla/templates/system/index.php
	 20%    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-zh_cn.php

We highly recommend that as a baseline, any file that is displayed in the Highest Rank Files list be investigated at a minimum. We also recommend investigating any files that show up in any of the tests listed above, as some methods are more effective at detecting certain shells than others.

Windows

The tool is cross compatible with windows as well. In the example below we use a regular expressing to just search for php and text files.

python neopi.py -a c:\temp\phpbb "php|txt"

Animal Shell

animal_shell_encoder.php and animal_shell_poc.php are two Proof-of-Concept-type examples scripts to implement an encoding that "should" evade many of the statistical tests NeoPI performs. They are poorly commented and the decoder large such that they are impractical.