Security Videos
« Installing Cuckoo | Main | Tektip ep24 - Moloch »

Installing MASTIFF

So we have talked about using SecShoggoth 's MASTIFF (here, here, and here), but haven't really gone through the installation. Here goes...

I am assuming a base installation of Ubuntu 12.10 (because its easy and you can run it free on AWS). The first thing that we should do is update the base OS and install ssh for remote management.

sudo apt-get update; sudo apt-get upgrade -y; sudo apt-get dist-upgrade -y; sudo apt-get autoremove -y; sudo apt-get install openssh-server -y; sudo shutdown -r now later

Once that is back online we will begin installing the necessary packages for MASTIFF. I am running through the dependencies as they are introduced in the documentation. First lets get the python dependencies out of the way, as well as an editor (nano is fine...albeit evil)

sudo apt-get install python python-dev python-magic python-sqlite python-setuptools python-pip build-essential vim -y 

Install yapsy from pip

sudo pip install yapsy
Now, I tend to pull the majority of my software to /opt out of habit. You do not need to do the same but if you change the location be sure to update to commands below. First I will ensure that the user and group that I am currently using have access to /opt so I can write to that directory.
sudo chown -R `whoami`:`groups | awk '{print $1}'` /opt
Install TrID, download and run the TrID database updater.
cd /opt
mkdir /opt/trid
cd /opt/trid
wget wget http://mark0.net/download/trid_linux.zip
unzip trid_linux.zip
rm -f unzip trid_linux.zip
chmod +x trid
wget http://goo.gl/RQXV8
unzip RQXV8
rm -f RQXV8
chmod +x tridupdate.py
python tridupdate.py
*Note* if you are running this on a 64 bit machine you will need to install ia32-libs
sudo aptitude install ia32-libs
Next we will pull the dependencies down for ssdeep and pyssdeep and then install those packages
sudo apt-get install subversion libpcre3 libpcre3-dev libpcre++-dev -y
cd /opt/
wget http://sourceforge.net/projects/ssdeep/files/ssdeep-2.9/ssdeep-2.9.tar.gz
tar -xvzf ssdeep-2.9.tar.gz
rm -f ssdeep-2.9.tar.gz
mv ssdeep-2.9 ssdeep
cd /opt/ssdeep
sudo make install
sudo ldconfig
svn checkout http://pyssdeep.googlecode.com/svn/trunk/ pyssdeep
cd /opt/ssdeep/pyssdeep
python setup.py build
sudo python setup.py install
Next up is automake and yara:
sudo apt-get install automake -y
cd /opt
svn checkout http://yara-project.googlecode.com/svn/trunk/ yara
cd /opt/yara
sudo ln -s /usr/bin/aclocal-1.11 /usr/bin/aclocal-1.12
sudo make install
cd yara-python
python setup.py build
sudo python setup.py install
Now install simplejson from soure (NOT the APT repo)
sudo apt-get install git -y
cd /opt
git clone https://github.com/simplejson/simplejson simplejson
cd /opt/simplejson
python setup.py build
sudo python setup.py build install
Pull down Didier Stevens awesome pdf tools
mkdir /opt/pdftools
cd /opt/pdftools
wget http://didierstevens.com/files/software/pdf-parser_V0_3_9.zip
unzip pdf-parser_V0_3_9.zip
rm -f pdf-parser_V0_3_9.zip
chmod +x pdf-parser.py
wget http://didierstevens.com/files/software/pdfid_v0_0_12.zip
unzip pdfid_v0_0_12.zip
rm -f pdfid_v0_0_12.zip
chmod +x pdfid.py
cd /opt
wget http://www.sno.phy.queensu.ca/~phil/exiftool/Image-ExifTool-9.22.tar.gz
tar -xvzf Image-ExifTool-9.22.tar.gz
rm Image-ExifTool-9.22.tar.gz
mv Image-ExifTool-9.22 exiftool
PE-File (again NOT from the apt repo)
cd /opt
svn checkout http://pefile.googlecode.com/svn/trunk/ pefile
cd /opt/pefile
python setup.py build
sudo python setup.py build install
mkdir /opt/disitool
cd /opt/disitool
wget http://www.didierstevens.com/files/software/disitool_v0_3.zip
unzip disitool_v0_3.zip
rm disitool_v0_3.zip
sudo apt-get install openssl -y
mkdir /opt/pyOLEScanner
cd /opt/pyOLEScanner
wget https://github.com/Evilcry/PythonScripts/raw/master/pyOLEScanner.zip
unzip pyOLEScanner.zip
rm pyOLEScanner.zip
chmod +x pyOLEScanner.py
cd /opt
svn checkout http://distorm.googlecode.com/svn/trunk/ distorm
cd /opt/distorm
python setup.py build
sudo python setup.py build install
And finally MASTIFF itself
cd /opt
wget http://downloads.sourceforge.net/project/mastiff/mastiff/0.5.0/mastiff-0.5.0.tar.gz
tar -xvzf mastiff-0.5.0.tar.gz
rm mastiff-0.5.0.tar.gz
mv mastiff-0.5.0/ mastiff
cd /opt/mastiff
sudo make install
Now that MASTIFF is good to go we will want to ensure that the config file is created / edited properly. Ensure that you read through the config file as you will want to add the appropriate VirusTotal API key. Also if you installed the dependencies to different locations now is the time to correct those paths. 
mkdir /etc/mastiff
cd /etc/mastiff
 cat > /opt/mastiff/mastiff.conf.TEST <<EOF
# This is the configuration file for mastiff.
# Comments are preceded by a # or ;
# log_dir is the base directory where the logs generated will
# be placed in.
#log_dir = /usr/local/mastiff/log
log_dir = ./work/log
# plugin_dir is a list of directories plugins may be present in.
# should be comma-separated.
plugin_dir = ./plugins, /etc/mastiff
# verbose = [on|off]
verbose = off
# Sqlite database options
# db_file = Name of the database file
db_file = mastiff.db
[File ID]
# trid is the location of the TrID binary
# trid_db is the location of the TrID database
#trid = /usr/local/bin/trid
trid = /opt/trid/trid
#trid_db = /usr/local/etc/triddefs.trd
trid_db = /opt/trid/triddefs.trd
[Embedded Strings Plugin]
# Options for the Embedded Strings Plugin.
# strcmd is the path to the strings command
strcmd = /usr/bin/strings
# Options for the VirusTotal Submission Plug-in.
# api_key is your API key from virustotal.com
#   - Leave this empty if you wish to disable this plug-in
api_key = GET_YOUR_OWN
# submit [on|off] - submit binary to VirusTotal
submit = off
# Options to run Didier Stevens pdfid.py script
# pdfid_cmd = Path to the pdfid.py script
#   - Leave blank if you want the script disabled.
# pdfid_opts = Options for program.
#   - Do not put multiple options in quotes.
# Note: pdfid.py has bugs that may cause errors when examining
#       malformed PDFs when using the -e option.
pdfid_cmd = /opt/pdftools/pdfid.py
#pdfid_opts = -e
pdfid_opts =
# Options to run Didier Stevens pdf-parser.py script
# pdf_cmd = Path to pdf-parser.py.
pdf_cmd = /opt/pdftools/pdf-parser.py
[PDF Metadata]
# Options for PDF Metadata script
# exiftool = path to exitfool
exiftool = /opt/exiftool/exiftool
# Options for the Yara signature plug-in
# yara_sigs = Base path to Yara signatures. This path will be recursed
#             to find additional signatures.
#             Leave blank to disable the plug-in.
yara_sigs = /opt/yara
[Digital Signatures]
# Options to extract the digital signatures
# disitool - path to disitool.py script.
# openssl - path to openssl binary
disitool = /opt/disitool/disitool.py
openssl = /usr/bin/openssl
[Office Metadata]
# Options for Office Metadata script
# exiftool = path to exitfool
exiftool = /opt/exiftool/exiftool
[Single-Byte Strings]
# options for single-byte string extraction plug-in
# length - Minimum length to extract
length = 3
# raw - print raw characters instead of formatted ones (e.g. \\n vs. \n)
raw = False
# options for Zip archive file extraction plug-in
# enabled: [on|off] - Extract files or not
# password: Password to use for zip file. OK to leave blank.
enabled = on
password = infected
[Office pyOLEScanner]
# olecmd = Path to pyOLEScanner.py
Now testing MASTIFF out is as simple as:
cd /opt/mastiff
zwned@malwr:/opt/mastiff$ python mas.py /opt/mastiff/tests/test.exe
[2013-03-10 15:11:47,324] [INFO] [Mastiff] : Starting analysis on /opt/mastiff/tests/test.exe
[2013-03-10 15:11:47,326] [INFO] [Mastiff.Init_File] : Analyzing /opt/mastiff/tests/test.exe.
[2013-03-10 15:11:47,326] [INFO] [Mastiff.Init_File] : Log Directory: ./work/log/c69ffb3057b2077fcaecc99b9f16c7c8
[2013-03-10 15:11:47,417] [INFO] [Mastiff.DB.Insert] : Adding ['Generic', 'EXE']
[2013-03-10 15:11:47,506] [INFO] [Mastiff.Analysis] : File categories are ['Generic', 'EXE'].
[2013-03-10 15:11:47,507] [INFO] [Mastiff.Plugins.Embedded Strings Plugin] : Starting execution.
[2013-03-10 15:11:47,521] [INFO] [Mastiff.Plugins.File Information] : Starting execution.
[2013-03-10 15:11:47,602] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Starting execution.
[2013-03-10 15:11:47,602] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Generating fuzzy hash.
[2013-03-10 15:11:47,681] [INFO] [Mastiff.Plugins.Fuzzy Hashing.compare] : Comparing fuzzy hashes.
[2013-03-10 15:11:47,681] [INFO] [Mastiff.Plugins.VirusTotal] : Starting execution.
[2013-03-10 15:11:48,717] [INFO] [Mastiff.Plugins.VirusTotal.submit] : Submission disabled. Not sending file.
[2013-03-10 15:11:48,717] [INFO] [Mastiff.Plugins.yara] : Starting execution.
[2013-03-10 15:11:48,722] [INFO] [Mastiff.Plugins.Resources] : Starting execution.
[2013-03-10 15:11:48,774] [INFO] [Mastiff.Plugins.Single-Byte Strings] : Starting execution.
[2013-03-10 15:11:48,813] [INFO] [Mastiff.Plugins.PE Info] : Starting execution.
[2013-03-10 15:11:48,926] [INFO] [Mastiff.Plugins.Digital Signatures] : Starting execution.
[2013-03-10 15:11:48,975] [INFO] [Mastiff.Plugins.Digital Signatures] : No signature on the file.
[2013-03-10 15:11:48,976] [INFO] [Mastiff.Analysis] : Finished analysis for /opt/mastiff/tests/test.exe.
zwned@malwr:/opt/mastiff$ python mas.py /opt/mastiff/tests/test.pdf
[2013-03-10 15:12:36,299] [INFO] [Mastiff] : Starting analysis on /opt/mastiff/tests/test.pdf
[2013-03-10 15:12:36,299] [INFO] [Mastiff.Init_File] : Analyzing /opt/mastiff/tests/test.pdf.
[2013-03-10 15:12:36,300] [INFO] [Mastiff.Init_File] : Log Directory: ./work/log/3f53a4bf0097f9075ff641b03bb176f5
[2013-03-10 15:12:36,381] [INFO] [Mastiff.DB.Insert] : Adding ['PDF', 'Generic']
[2013-03-10 15:12:36,468] [INFO] [Mastiff.Analysis] : File categories are ['PDF', 'Generic'].
[2013-03-10 15:12:36,469] [INFO] [Mastiff.Plugins.pdf-parser] : Starting execution.
[2013-03-10 15:12:36,470] [INFO] [Mastiff.Plugins.pdf-parser.uncompress] : Uncompressing PDF.
[2013-03-10 15:12:36,563] [INFO] [Mastiff.Plugins.pdf-parser.get_objects] : Extracting interesting objects.
[2013-03-10 15:12:37,532] [INFO] [Mastiff.Plugins.PDF Metadata] : Starting execution.
[2013-03-10 15:12:37,643] [INFO] [Mastiff.Plugins.pdfid] : Starting execution.
[2013-03-10 15:12:37,729] [INFO] [Mastiff.Plugins.Embedded Strings Plugin] : Starting execution.
[2013-03-10 15:12:37,741] [INFO] [Mastiff.Plugins.File Information] : Starting execution.
[2013-03-10 15:12:37,819] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Starting execution.
[2013-03-10 15:12:37,820] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Generating fuzzy hash.
[2013-03-10 15:12:37,909] [INFO] [Mastiff.Plugins.Fuzzy Hashing.compare] : Comparing fuzzy hashes.
[2013-03-10 15:12:37,910] [INFO] [Mastiff.Plugins.VirusTotal] : Starting execution.
[2013-03-10 15:12:38,386] [INFO] [Mastiff.Plugins.VirusTotal.submit] : Submission disabled. Not sending file.
[2013-03-10 15:12:38,386] [INFO] [Mastiff.Plugins.yara] : Starting execution.
[2013-03-10 15:12:38,392] [INFO] [Mastiff.Analysis] : Finished analysis for /opt/mastiff/tests/test.pdf.
zwned@malwr:/opt/mastiff$ python mas.py /opt/mastiff/tests/test.doc
[2013-03-10 15:12:53,882] [INFO] [Mastiff] : Starting analysis on /opt/mastiff/tests/test.doc
[2013-03-10 15:12:53,883] [INFO] [Mastiff.Init_File] : Analyzing /opt/mastiff/tests/test.doc.
[2013-03-10 15:12:53,883] [INFO] [Mastiff.Init_File] : Log Directory: ./work/log/759f7e53f54df03f2ae06fcec25e8ac3
[2013-03-10 15:12:53,973] [INFO] [Mastiff.DB.Insert] : Adding ['Generic', 'Office', 'ZIP']
[2013-03-10 15:12:54,076] [INFO] [Mastiff.Analysis] : File categories are ['Generic', 'Office', 'ZIP'].
[2013-03-10 15:12:54,078] [INFO] [Mastiff.Plugins.Embedded Strings Plugin] : Starting execution.
[2013-03-10 15:12:54,088] [INFO] [Mastiff.Plugins.File Information] : Starting execution.
[2013-03-10 15:12:54,167] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Starting execution.
[2013-03-10 15:12:54,167] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Generating fuzzy hash.
[2013-03-10 15:12:54,234] [INFO] [Mastiff.Plugins.Fuzzy Hashing.compare] : Comparing fuzzy hashes.
[2013-03-10 15:12:54,234] [INFO] [Mastiff.Plugins.VirusTotal] : Starting execution.
[2013-03-10 15:12:55,239] [INFO] [Mastiff.Plugins.yara] : Starting execution.
[2013-03-10 15:12:55,244] [INFO] [Mastiff.Plugins.Office pyOLEScanner] : Starting execution.
[2013-03-10 15:12:57,497] [INFO] [Mastiff.Plugins.Office Metadata] : Starting execution.
[2013-03-10 15:12:57,681] [INFO] [Mastiff.Plugins.ZipInfo] : Starting execution.
[2013-03-10 15:12:57,682] [INFO] [Mastiff.Plugins.ZipExtract] : Starting execution.
[2013-03-10 15:12:57,683] [INFO] [Mastiff.Plugins.ZipExtract] : Password "infected" will be used for this zip.
[2013-03-10 15:12:57,683] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting [Content_Types].xml.
[2013-03-10 15:12:57,683] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting _rels/.rels.
[2013-03-10 15:12:57,684] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting theme/theme/themeManager.xml.
[2013-03-10 15:12:57,684] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting theme/theme/theme1.xml.
[2013-03-10 15:12:57,685] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting theme/theme/_rels/themeManager.xml.rels.
[2013-03-10 15:12:57,685] [INFO] [Mastiff.Analysis] : Finished analysis for /opt/mastiff/tests/test.doc.
zwned@malwr:/opt/mastiff$ python mas.py /opt/mastiff/tests/test.
test.doc  test.exe  test.pdf  test.zip
zwned@malwr:/opt/mastiff$ python mas.py /opt/mastiff/tests/test.zip
[2013-03-10 15:13:22,856] [INFO] [Mastiff] : Starting analysis on /opt/mastiff/tests/test.zip
[2013-03-10 15:13:22,870] [INFO] [Mastiff.Init_File] : Analyzing /opt/mastiff/tests/test.zip.
[2013-03-10 15:13:22,871] [INFO] [Mastiff.Init_File] : Log Directory: ./work/log/033d488bbe65e8aececb2c55bdfbc2fd
[2013-03-10 15:13:23,035] [INFO] [Mastiff.DB.Insert] : Adding ['Generic', 'ZIP']
[2013-03-10 15:13:23,106] [INFO] [Mastiff.Analysis] : File categories are ['Generic', 'ZIP'].
[2013-03-10 15:13:23,107] [INFO] [Mastiff.Plugins.Embedded Strings Plugin] : Starting execution.
[2013-03-10 15:13:23,115] [INFO] [Mastiff.Plugins.File Information] : Starting execution.
[2013-03-10 15:13:23,178] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Starting execution.
[2013-03-10 15:13:23,178] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Generating fuzzy hash.
[2013-03-10 15:13:23,238] [INFO] [Mastiff.Plugins.Fuzzy Hashing.compare] : Comparing fuzzy hashes.
[2013-03-10 15:13:23,238] [INFO] [Mastiff.Plugins.VirusTotal] : Starting execution.
[2013-03-10 15:13:23,440] [INFO] [Mastiff.Plugins.VirusTotal.submit] : Submission disabled. Not sending file.
[2013-03-10 15:13:23,440] [INFO] [Mastiff.Plugins.yara] : Starting execution.
[2013-03-10 15:13:23,445] [INFO] [Mastiff.Plugins.ZipInfo] : Starting execution.
[2013-03-10 15:13:23,446] [INFO] [Mastiff.Plugins.ZipExtract] : Starting execution.
[2013-03-10 15:13:23,447] [INFO] [Mastiff.Plugins.ZipExtract] : Password "infected" will be used for this zip.
[2013-03-10 15:13:23,447] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting test.bin.
[2013-03-10 15:13:23,447] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting test.txt.
[2013-03-10 15:13:23,448] [INFO] [Mastiff.Analysis] : Finished analysis for /opt/mastiff/tests/test.zip.
If you did not receive any errors / warnings you should be good to go. Now that MASTIFF is up and running we can download MASTIFF2HTML on Github.
wget https://raw.github.com/1aN0rmus/TekDefense/master/MASTIFF2HTML.py
chmod +x MASTIFF2HTML.py
python MASTIFF2HTML.py -f /opt/mastiff/work/log/ -d mastiff.db
cd /opt/mastiff/work/log/www/
python -m SimpleHTTPServer
Now if you browse to you should be seeing the results for your analysis.
 If you run into any issues... please leave a comment so we can address / update as necessary.

Reader Comments (12)

First, thanks for the install guide. I tried this command:
sudo aptitude install ia32-libs
but I received an error. I found this:
sudo apt-get install libc6:i386 libgcc1:i386 gcc-4.6-base:i386 libstdc++5:i386 libstdc++6:i386
That command appeared to work but I am not sure of the ramifications.
I try to execute mas.py but I receive a slew of errors. Let me add, I'm not experienced with Ubuntu. I am running 12.04 LTS 64-bit.

March 30, 2013 | Unregistered Commenters0mar

Traceback (most recent call last):
File "mas.py", line 143, in <module>
File "mas.py", line 139, in main
my_analysis.analyze(fname, opts.plugin_name)
File "/opt/mastiff/mastiff/core.py", line 463, in analyze
ftype = self.set_filetype()
File "/opt/mastiff/mastiff/core.py", line 397, in set_filetype
self.filetype['magic'] = FileType.get_magic(self.file_name)
File "/opt/mastiff/mastiff/filetype.py", line 39, in get_magic
magic_ = magic.open(magic.MAGIC_NONE)
AttributeError: 'module' object has no attribute 'open'

March 30, 2013 | Unregistered Commenters0mar

What is the error you are getting when installing ia32-libs (sudo aptitude install ia32-libs)?
also please let me know what is returned when you execute:
dpkg --get-selections | grep python-magic

March 30, 2013 | Registered CommenterJoshua Gauthier

sudo aptitude install ia32-libs:
sudo: aptitude: command not found
I was able to run:
sudo apt-get install ia32-libs-multiarch
dpkg --get-selections | grep python-magic:
python-magic install

March 30, 2013 | Unregistered Commenters0mar

Run this command, it should print amd64: sudo dpkg --print-architecture
Then run this: sudo dpkg --print-foreign-architectures
If that command doesnt return i386 - run the following:
sudo dpkg --add-architecture i386; sudo apt-get update; sudo apt-get install ia32-libs
Let me know if that works or if sudo dpkg --print-foreign-architectures returned i386.

Also, how did you install python-magic? From the deb repo or from pip?

March 30, 2013 | Registered CommenterJoshua Gauthier

Well, seeing your most recent comments, it returns I am running 32-bit. Ultimately, I would like to run Cuckoo and Mastiff on this machine. I think I will reinstall Ubuntu 12.10 64-bit from scratch and retry. I appreciate your assistance! I will report back once I reinstall.

March 30, 2013 | Unregistered Commenters0mar

I think I may have it working now after a reinstall but I still have to perform some testing.
I believe the instructions need to be modified for pyOLEScaner. I think you need a cd /opt/pyOLEScanner and then that plug-in will work.
I don't have a VT key but if I comment that line out or just have it blank, I get an error. That is the only error I see.

April 3, 2013 | Unregistered Commenters0mar

Updated to reflect the cd into /opt/pyOLEScanner.
VirusTotal API keys are free to generate - just create an account, go to profile, then API.

April 8, 2013 | Registered CommenterJoshua Gauthier

The only error I'm currently getting is [ERROR] [Mastiff.Conf.GetVar] : Could not find "Misc" : "copy"
Any idea as to where I could fix this issue? Thanks

May 29, 2013 | Unregistered CommenterJustin

For "automake and yara" you now need to add a step per (https://code.google.com/p/yara-project/issues/detail?id=70)

July 10, 2013 | Unregistered CommenterGreg

Excellent post.
For svn co, if behind a proxy, make sure to edit /etc/subversion/servers

March 6, 2014 | Unregistered CommenterHarry

Nice post, had to update a bit for the latest version, but am running into a problem with sqlite when running tests:
Traceback (most recent call last):
File "mas.py", line 209, in <module>
File "mas.py", line 196, in main
File "/opt/mastiff/mastiff/queue.py", line 119, in append
conn.execute(self._append, (obj_buffer,))
sqlite3.OperationalError: attempt to write a readonly database
Anyone else have this issue?

April 29, 2015 | Unregistered Commentermalm0u53

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>