Installing MASTIFF

So we have talked about using SecShoggoth 's MASTIFF (here, here, and here), but haven't really gone through the installation. Here goes...

I am assuming a base installation of Ubuntu 12.10 (because its easy and you can run it free on AWS). The first thing that we should do is update the base OS and install ssh for remote management.

sudo apt-get update; sudo apt-get upgrade -y; sudo apt-get dist-upgrade -y; sudo apt-get autoremove -y; sudo apt-get install openssh-server -y; sudo shutdown -r now later

Once that is back online we will begin installing the necessary packages for MASTIFF. I am running through the dependencies as they are introduced in the documentation. First lets get the python dependencies out of the way, as well as an editor (nano is fine...albeit evil)

sudo apt-get install python python-dev python-magic python-sqlite python-setuptools python-pip build-essential vim -y 

Install yapsy from pip

sudo pip install yapsy

Now, I tend to pull the majority of my software to /opt out of habit. You do not need to do the same but if you change the location be sure to update to commands below. First I will ensure that the user and group that I am currently using have access to /opt so I can write to that directory.

sudo chown -R `whoami`:`groups | awk '{print $1}'` /opt

Install TrID, download and run the TrID database updater.

cd /opt

mkdir /opt/trid

cd /opt/trid

wget wget http://mark0.net/download/trid_linux.zip

unzip trid_linux.zip

rm -f unzip trid_linux.zip

chmod +x trid

wget http://goo.gl/RQXV8

unzip RQXV8

rm -f RQXV8

chmod +x tridupdate.py

python tridupdate.py

*Note* if you are running this on a 64 bit machine you will need to install ia32-libs

sudo aptitude install ia32-libs

Next we will pull the dependencies down for ssdeep and pyssdeep and then install those packages

sudo apt-get install subversion libpcre3 libpcre3-dev libpcre++-dev -y

cd /opt/

wget http://sourceforge.net/projects/ssdeep/files/ssdeep-2.9/ssdeep-2.9.tar.gz

tar -xvzf ssdeep-2.9.tar.gz

rm -f ssdeep-2.9.tar.gz

mv ssdeep-2.9 ssdeep

cd /opt/ssdeep

./configure

make

sudo make install

sudo ldconfig

svn checkout http://pyssdeep.googlecode.com/svn/trunk/ pyssdeep

cd /opt/ssdeep/pyssdeep

python setup.py build

sudo python setup.py install

Next up is automake and yara:

sudo apt-get install automake -y

cd /opt

svn checkout http://yara-project.googlecode.com/svn/trunk/ yara

cd /opt/yara

sudo ln -s /usr/bin/aclocal-1.11 /usr/bin/aclocal-1.12

./configure

make

sudo make install

cd yara-python

python setup.py build

sudo python setup.py install

Now install simplejson from soure (NOT the APT repo)

sudo apt-get install git -y

cd /opt

git clone https://github.com/simplejson/simplejson simplejson

cd /opt/simplejson

python setup.py build

sudo python setup.py build install

Pull down Didier Stevens awesome pdf tools

mkdir /opt/pdftools

cd /opt/pdftools

wget http://didierstevens.com/files/software/pdf-parser_V0_3_9.zip

unzip pdf-parser_V0_3_9.zip

rm -f pdf-parser_V0_3_9.zip

chmod +x pdf-parser.py

wget http://didierstevens.com/files/software/pdfid_v0_0_12.zip

unzip pdfid_v0_0_12.zip

rm -f pdfid_v0_0_12.zip

chmod +x pdfid.py

Exiftool

cd /opt

wget http://www.sno.phy.queensu.ca/~phil/exiftool/Image-ExifTool-9.22.tar.gz

tar -xvzf Image-ExifTool-9.22.tar.gz

rm Image-ExifTool-9.22.tar.gz

mv Image-ExifTool-9.22 exiftool

PE-File (again NOT from the apt repo)

cd /opt

svn checkout http://pefile.googlecode.com/svn/trunk/ pefile

cd /opt/pefile

python setup.py build

sudo python setup.py build install

Disitool

mkdir /opt/disitool

cd /opt/disitool

wget http://www.didierstevens.com/files/software/disitool_v0_3.zip

unzip disitool_v0_3.zip

rm disitool_v0_3.zip

Openssl

sudo apt-get install openssl -y

pyOLEScanner

mkdir /opt/pyOLEScanner

cd /opt/pyOLEScanner

wget https://github.com/Evilcry/PythonScripts/raw/master/pyOLEScanner.zip

unzip pyOLEScanner.zip

rm pyOLEScanner.zip

chmod +x pyOLEScanner.py

Distorm

cd /opt

svn checkout http://distorm.googlecode.com/svn/trunk/ distorm

cd /opt/distorm

python setup.py build

sudo python setup.py build install

And finally MASTIFF itself

cd /opt

wget http://downloads.sourceforge.net/project/mastiff/mastiff/0.5.0/mastiff-0.5.0.tar.gz

tar -xvzf mastiff-0.5.0.tar.gz

rm mastiff-0.5.0.tar.gz

mv mastiff-0.5.0/ mastiff

cd /opt/mastiff

sudo make install

Now that MASTIFF is good to go we will want to ensure that the config file is created / edited properly. Ensure that you read through the config file as you will want to add the appropriate VirusTotal API key. Also if you installed the dependencies to different locations now is the time to correct those paths. 

mkdir /etc/mastiff

cd /etc/mastiff

 cat > /opt/mastiff/mastiff.conf.TEST <<EOF

# This is the configuration file for mastiff.

#

# Comments are preceded by a # or ;

#

[Dir]

# log_dir is the base directory where the logs generated will

# be placed in.

#log_dir = /usr/local/mastiff/log

log_dir = ./work/log

# plugin_dir is a list of directories plugins may be present in.

# should be comma-separated.

plugin_dir = ./plugins, /etc/mastiff

[Misc]

# verbose = [on|off]

verbose = off

[Sqlite]

# Sqlite database options

# db_file = Name of the database file

db_file = mastiff.db

[File ID]

# trid is the location of the TrID binary

# trid_db is the location of the TrID database

#trid = /usr/local/bin/trid

trid = /opt/trid/trid

#trid_db = /usr/local/etc/triddefs.trd

trid_db = /opt/trid/triddefs.trd

[Embedded Strings Plugin]

# Options for the Embedded Strings Plugin.

# strcmd is the path to the strings command

strcmd = /usr/bin/strings

[VirusTotal]

# Options for the VirusTotal Submission Plug-in.

# api_key is your API key from virustotal.com

#   - Leave this empty if you wish to disable this plug-in

api_key = GET_YOUR_OWN

# submit [on|off] - submit binary to VirusTotal

submit = off

[pdfid]

# Options to run Didier Stevens pdfid.py script

# pdfid_cmd = Path to the pdfid.py script

#   - Leave blank if you want the script disabled.

# pdfid_opts = Options for program.

#   - Do not put multiple options in quotes.

# Note: pdfid.py has bugs that may cause errors when examining

#       malformed PDFs when using the -e option.

pdfid_cmd = /opt/pdftools/pdfid.py

#pdfid_opts = -e

pdfid_opts =

[pdf-parser]

# Options to run Didier Stevens pdf-parser.py script

# pdf_cmd = Path to pdf-parser.py.

pdf_cmd = /opt/pdftools/pdf-parser.py

[PDF Metadata]

# Options for PDF Metadata script

# exiftool = path to exitfool

exiftool = /opt/exiftool/exiftool

[yara]

# Options for the Yara signature plug-in

# yara_sigs = Base path to Yara signatures. This path will be recursed

#             to find additional signatures.

#             Leave blank to disable the plug-in.

yara_sigs = /opt/yara

[Digital Signatures]

# Options to extract the digital signatures

#

# disitool - path to disitool.py script.

# openssl - path to openssl binary

disitool = /opt/disitool/disitool.py

openssl = /usr/bin/openssl

[Office Metadata]

# Options for Office Metadata script

# exiftool = path to exitfool

exiftool = /opt/exiftool/exiftool

[Single-Byte Strings]

# options for single-byte string extraction plug-in

# length - Minimum length to extract

length = 3

# raw - print raw characters instead of formatted ones (e.g. \\n vs. \n)

raw = False

[ZipExtract]

# options for Zip archive file extraction plug-in

# enabled: [on|off] - Extract files or not

# password: Password to use for zip file. OK to leave blank.

enabled = on

password = infected

[Office pyOLEScanner]

# olecmd = Path to pyOLEScanner.py

olecmd=/opt/pyOLEScanner/pyOLEScanner.py

EOF

Now testing MASTIFF out is as simple as:

cd /opt/mastiff

zwned@malwr:/opt/mastiff$ python mas.py /opt/mastiff/tests/test.exe

[2013-03-10 15:11:47,324] [INFO] [Mastiff] : Starting analysis on /opt/mastiff/tests/test.exe

[2013-03-10 15:11:47,326] [INFO] [Mastiff.Init_File] : Analyzing /opt/mastiff/tests/test.exe.

[2013-03-10 15:11:47,326] [INFO] [Mastiff.Init_File] : Log Directory: ./work/log/c69ffb3057b2077fcaecc99b9f16c7c8

[2013-03-10 15:11:47,417] [INFO] [Mastiff.DB.Insert] : Adding ['Generic', 'EXE']

[2013-03-10 15:11:47,506] [INFO] [Mastiff.Analysis] : File categories are ['Generic', 'EXE'].

[2013-03-10 15:11:47,507] [INFO] [Mastiff.Plugins.Embedded Strings Plugin] : Starting execution.

[2013-03-10 15:11:47,521] [INFO] [Mastiff.Plugins.File Information] : Starting execution.

[2013-03-10 15:11:47,602] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Starting execution.

[2013-03-10 15:11:47,602] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Generating fuzzy hash.

[2013-03-10 15:11:47,681] [INFO] [Mastiff.Plugins.Fuzzy Hashing.compare] : Comparing fuzzy hashes.

[2013-03-10 15:11:47,681] [INFO] [Mastiff.Plugins.VirusTotal] : Starting execution.

[2013-03-10 15:11:48,717] [INFO] [Mastiff.Plugins.VirusTotal.submit] : Submission disabled. Not sending file.

[2013-03-10 15:11:48,717] [INFO] [Mastiff.Plugins.yara] : Starting execution.

[2013-03-10 15:11:48,722] [INFO] [Mastiff.Plugins.Resources] : Starting execution.

[2013-03-10 15:11:48,774] [INFO] [Mastiff.Plugins.Single-Byte Strings] : Starting execution.

[2013-03-10 15:11:48,813] [INFO] [Mastiff.Plugins.PE Info] : Starting execution.

[2013-03-10 15:11:48,926] [INFO] [Mastiff.Plugins.Digital Signatures] : Starting execution.

[2013-03-10 15:11:48,975] [INFO] [Mastiff.Plugins.Digital Signatures] : No signature on the file.

[2013-03-10 15:11:48,976] [INFO] [Mastiff.Analysis] : Finished analysis for /opt/mastiff/tests/test.exe.

zwned@malwr:/opt/mastiff$ python mas.py /opt/mastiff/tests/test.pdf

[2013-03-10 15:12:36,299] [INFO] [Mastiff] : Starting analysis on /opt/mastiff/tests/test.pdf

[2013-03-10 15:12:36,299] [INFO] [Mastiff.Init_File] : Analyzing /opt/mastiff/tests/test.pdf.

[2013-03-10 15:12:36,300] [INFO] [Mastiff.Init_File] : Log Directory: ./work/log/3f53a4bf0097f9075ff641b03bb176f5

[2013-03-10 15:12:36,381] [INFO] [Mastiff.DB.Insert] : Adding ['PDF', 'Generic']

[2013-03-10 15:12:36,468] [INFO] [Mastiff.Analysis] : File categories are ['PDF', 'Generic'].

[2013-03-10 15:12:36,469] [INFO] [Mastiff.Plugins.pdf-parser] : Starting execution.

[2013-03-10 15:12:36,470] [INFO] [Mastiff.Plugins.pdf-parser.uncompress] : Uncompressing PDF.

[2013-03-10 15:12:36,563] [INFO] [Mastiff.Plugins.pdf-parser.get_objects] : Extracting interesting objects.

[2013-03-10 15:12:37,532] [INFO] [Mastiff.Plugins.PDF Metadata] : Starting execution.

[2013-03-10 15:12:37,643] [INFO] [Mastiff.Plugins.pdfid] : Starting execution.

[2013-03-10 15:12:37,729] [INFO] [Mastiff.Plugins.Embedded Strings Plugin] : Starting execution.

[2013-03-10 15:12:37,741] [INFO] [Mastiff.Plugins.File Information] : Starting execution.

[2013-03-10 15:12:37,819] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Starting execution.

[2013-03-10 15:12:37,820] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Generating fuzzy hash.

[2013-03-10 15:12:37,909] [INFO] [Mastiff.Plugins.Fuzzy Hashing.compare] : Comparing fuzzy hashes.

[2013-03-10 15:12:37,910] [INFO] [Mastiff.Plugins.VirusTotal] : Starting execution.

[2013-03-10 15:12:38,386] [INFO] [Mastiff.Plugins.VirusTotal.submit] : Submission disabled. Not sending file.

[2013-03-10 15:12:38,386] [INFO] [Mastiff.Plugins.yara] : Starting execution.

[2013-03-10 15:12:38,392] [INFO] [Mastiff.Analysis] : Finished analysis for /opt/mastiff/tests/test.pdf.

zwned@malwr:/opt/mastiff$ python mas.py /opt/mastiff/tests/test.doc

[2013-03-10 15:12:53,882] [INFO] [Mastiff] : Starting analysis on /opt/mastiff/tests/test.doc

[2013-03-10 15:12:53,883] [INFO] [Mastiff.Init_File] : Analyzing /opt/mastiff/tests/test.doc.

[2013-03-10 15:12:53,883] [INFO] [Mastiff.Init_File] : Log Directory: ./work/log/759f7e53f54df03f2ae06fcec25e8ac3

[2013-03-10 15:12:53,973] [INFO] [Mastiff.DB.Insert] : Adding ['Generic', 'Office', 'ZIP']

[2013-03-10 15:12:54,076] [INFO] [Mastiff.Analysis] : File categories are ['Generic', 'Office', 'ZIP'].

[2013-03-10 15:12:54,078] [INFO] [Mastiff.Plugins.Embedded Strings Plugin] : Starting execution.

[2013-03-10 15:12:54,088] [INFO] [Mastiff.Plugins.File Information] : Starting execution.

[2013-03-10 15:12:54,167] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Starting execution.

[2013-03-10 15:12:54,167] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Generating fuzzy hash.

[2013-03-10 15:12:54,234] [INFO] [Mastiff.Plugins.Fuzzy Hashing.compare] : Comparing fuzzy hashes.

[2013-03-10 15:12:54,234] [INFO] [Mastiff.Plugins.VirusTotal] : Starting execution.

[2013-03-10 15:12:55,239] [INFO] [Mastiff.Plugins.yara] : Starting execution.

[2013-03-10 15:12:55,244] [INFO] [Mastiff.Plugins.Office pyOLEScanner] : Starting execution.

[2013-03-10 15:12:57,497] [INFO] [Mastiff.Plugins.Office Metadata] : Starting execution.

[2013-03-10 15:12:57,681] [INFO] [Mastiff.Plugins.ZipInfo] : Starting execution.

[2013-03-10 15:12:57,682] [INFO] [Mastiff.Plugins.ZipExtract] : Starting execution.

[2013-03-10 15:12:57,683] [INFO] [Mastiff.Plugins.ZipExtract] : Password "infected" will be used for this zip.

[2013-03-10 15:12:57,683] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting [Content_Types].xml.

[2013-03-10 15:12:57,683] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting _rels/.rels.

[2013-03-10 15:12:57,684] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting theme/theme/themeManager.xml.

[2013-03-10 15:12:57,684] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting theme/theme/theme1.xml.

[2013-03-10 15:12:57,685] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting theme/theme/_rels/themeManager.xml.rels.

[2013-03-10 15:12:57,685] [INFO] [Mastiff.Analysis] : Finished analysis for /opt/mastiff/tests/test.doc.

zwned@malwr:/opt/mastiff$ python mas.py /opt/mastiff/tests/test.

test.doc  test.exe  test.pdf  test.zip

zwned@malwr:/opt/mastiff$ python mas.py /opt/mastiff/tests/test.zip

[2013-03-10 15:13:22,856] [INFO] [Mastiff] : Starting analysis on /opt/mastiff/tests/test.zip

[2013-03-10 15:13:22,870] [INFO] [Mastiff.Init_File] : Analyzing /opt/mastiff/tests/test.zip.

[2013-03-10 15:13:22,871] [INFO] [Mastiff.Init_File] : Log Directory: ./work/log/033d488bbe65e8aececb2c55bdfbc2fd

[2013-03-10 15:13:23,035] [INFO] [Mastiff.DB.Insert] : Adding ['Generic', 'ZIP']

[2013-03-10 15:13:23,106] [INFO] [Mastiff.Analysis] : File categories are ['Generic', 'ZIP'].

[2013-03-10 15:13:23,107] [INFO] [Mastiff.Plugins.Embedded Strings Plugin] : Starting execution.

[2013-03-10 15:13:23,115] [INFO] [Mastiff.Plugins.File Information] : Starting execution.

[2013-03-10 15:13:23,178] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Starting execution.

[2013-03-10 15:13:23,178] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Generating fuzzy hash.

[2013-03-10 15:13:23,238] [INFO] [Mastiff.Plugins.Fuzzy Hashing.compare] : Comparing fuzzy hashes.

[2013-03-10 15:13:23,238] [INFO] [Mastiff.Plugins.VirusTotal] : Starting execution.

[2013-03-10 15:13:23,440] [INFO] [Mastiff.Plugins.VirusTotal.submit] : Submission disabled. Not sending file.

[2013-03-10 15:13:23,440] [INFO] [Mastiff.Plugins.yara] : Starting execution.

[2013-03-10 15:13:23,445] [INFO] [Mastiff.Plugins.ZipInfo] : Starting execution.

[2013-03-10 15:13:23,446] [INFO] [Mastiff.Plugins.ZipExtract] : Starting execution.

[2013-03-10 15:13:23,447] [INFO] [Mastiff.Plugins.ZipExtract] : Password "infected" will be used for this zip.

[2013-03-10 15:13:23,447] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting test.bin.

[2013-03-10 15:13:23,447] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting test.txt.

[2013-03-10 15:13:23,448] [INFO] [Mastiff.Analysis] : Finished analysis for /opt/mastiff/tests/test.zip.

If you did not receive any errors / warnings you should be good to go. Now that MASTIFF is up and running we can download MASTIFF2HTML on Github.

wget https://raw.github.com/1aN0rmus/TekDefense/master/MASTIFF2HTML.py

chmod +x MASTIFF2HTML.py

python MASTIFF2HTML.py -f /opt/mastiff/work/log/ -d mastiff.db

cd /opt/mastiff/work/log/www/

python -m SimpleHTTPServer

Now if you browse to 127.0.0.1:8000/mastiff.html you should be seeing the results for your analysis.

 If you run into any issues... please leave a comment so we can address / update as necessary.

Cheers

Comments

[s0mar]

First, thanks for the install guide. I tried this command:
sudo aptitude install ia32-libs
but I received an error. I found this:
sudo apt-get install libc6:i386 libgcc1:i386 gcc-4.6-base:i386 libstdc++5:i386 libstdc++6:i386
That command appeared to work but I am not sure of the ramifications.
I try to execute mas.py but I receive a slew of errors. Let me add, I'm not experienced with Ubuntu. I am running 12.04 LTS 64-bit.

[s0mar]

Traceback (most recent call last):
File "mas.py", line 143, in <module>
main()
File "mas.py", line 139, in main
my_analysis.analyze(fname, opts.plugin_name)
File "/opt/mastiff/mastiff/core.py", line 463, in analyze
ftype = self.set_filetype()
File "/opt/mastiff/mastiff/core.py", line 397, in set_filetype
self.filetype['magic'] = FileType.get_magic(self.file_name)
File "/opt/mastiff/mastiff/filetype.py", line 39, in get_magic
magic_ = magic.open(magic.MAGIC_NONE)
AttributeError: 'module' object has no attribute 'open'

[Joshua Gauthier]

What is the error you are getting when installing ia32-libs (sudo aptitude install ia32-libs)?
also please let me know what is returned when you execute:
dpkg --get-selections | grep python-magic

[s0mar]

sudo aptitude install ia32-libs:
sudo: aptitude: command not found
I was able to run:
sudo apt-get install ia32-libs-multiarch
dpkg --get-selections | grep python-magic:
python-magic install

[Joshua Gauthier]

Run this command, it should print amd64: sudo dpkg --print-architecture
Then run this: sudo dpkg --print-foreign-architectures
If that command doesnt return i386 - run the following:
sudo dpkg --add-architecture i386; sudo apt-get update; sudo apt-get install ia32-libs
Let me know if that works or if sudo dpkg --print-foreign-architectures returned i386.

Also, how did you install python-magic? From the deb repo or from pip?

[s0mar]

Well, seeing your most recent comments, it returns I am running 32-bit. Ultimately, I would like to run Cuckoo and Mastiff on this machine. I think I will reinstall Ubuntu 12.10 64-bit from scratch and retry. I appreciate your assistance! I will report back once I reinstall.

[s0mar]

I think I may have it working now after a reinstall but I still have to perform some testing.
I believe the instructions need to be modified for pyOLEScaner. I think you need a cd /opt/pyOLEScanner and then that plug-in will work.
I don't have a VT key but if I comment that line out or just have it blank, I get an error. That is the only error I see.

[Joshua Gauthier]

Updated to reflect the cd into /opt/pyOLEScanner.
VirusTotal API keys are free to generate - just create an account, go to profile, then API.

[Justin]

The only error I'm currently getting is [ERROR] [Mastiff.Conf.GetVar] : Could not find "Misc" : "copy"
Any idea as to where I could fix this issue? Thanks

[Greg]

For "automake and yara" you now need to add a step per (https://code.google.com/p/yara-project/issues/detail?id=70)
./bootstrap.sh
./configure

[Harry]

Excellent post.
For svn co, if behind a proxy, make sure to edit /etc/subversion/servers

[malm0u53]

Nice post, had to update a bit for the latest version, but am running into a problem with sqlite when running tests:
Traceback (most recent call last):
File "mas.py", line 209, in <module>
main()
File "mas.py", line 196, in main
job_queue.append(fname)
File "/opt/mastiff/mastiff/queue.py", line 119, in append
conn.execute(self._append, (obj_buffer,))
sqlite3.OperationalError: attempt to write a readonly database
Anyone else have this issue?