Automater 1.0 - Passive IP and URL Analysis

Update: Automater gets its own project page http://www.tekdefense.com/automater/

Description:

Automater is an IP and URL analysis tool that I created to assist analyst in pulling data quickly and passively in mass.  This is the first stable release of the tool.

Features:

IPVoid: Pulls blacklist, ISP, and Geo Location

Robtex: Pulls DNS information. *A records only.

Fortiguard: Pulls URL Categorization

Unshotren.me: Will determine if the URL is shortened.  If it is it will display the final destination.

URLVoid - Pulls IP Address, blacklist, ISP, Geo Location, Domain creation date.

Installation:

 1. Download from github: 

wget https://github.com/1aN0rmus/TekDefense/archive/master.tar.gz

 2. Unzip the file:

tar -xvcf master.tar.gz 

 3. Make executable:

cd TekDefense-master

chmod +x *

 4. Now you are ready to run!

*Required Libraries: httplib2, re, sys, argparse, urllib, urllib2

Examples:

Display help information

./Automater.py -h

./Automater.py -t 188.95.52.162

./Automater.py -t securitytube.net

./Automater.py -e bit.ly/XDlV1q

./Automater.py -f hostsss

./Automater.py -f hostsss -o host.out

host.out

Video Demo:

Known Bugs:

• If the IP or URL has not been previously scanned at IPVoid or URLVoid, the script is supposed to submit the IP or URL and then pull results.  This seems to work most of the time, but on occasion it will not wait long enough to pull the appropriate result.  Running the command a second time will work though.
• Can not use the -e and -f switch together.
• URLs with http:// cannot be scanned.  Must take the http:// out for it to work.
• Please submit any other bugs to 1aN0rmus@tekdefense.com

Upcoming Features:

• For those who would like to be able to just query a specific engine or source such as robtex, we will be creating an option to do so.
• Check IP and/or URL against Malwaredomainlist
• Check IP and/or URL against malware sandboxes such as ThreatExpert.
• Summary report that will give statistics on the targets highlighting the known bad information such as blacklists and malicious URL categories.
• Please submit feature requests to 1aN0rmus@tekdefense.com

Automater - IP and URL analysis tool

Update: Automater gets its own project page http://www.tekdefense.com/automater/

One challenge I have faced, as well as seen other analyst face as well is the amount of time it takes to investigate an IP Address or URL.  If you are like most analyst you have probably used at least some if not all of the following web tools to investigate IPs or URLs involved in an incident:

• IPVoid
• URLVoid
• Robtex
• MalwareDomainList
• CleanMX
• ThreatExpert
• FortiGuard
• etc

As I mentioned previously to lookup an IP against all or even some of these takes more time than it should.  Additionally, as they all are formatted differently and have different options, you can not export them in a format that can be attached to a trouble ticket, evidence file, or even an email.  Noticing this issue, I attempted to fix it by creating a python program that will scrape the previously mentioned resources to pull out the information relevant to what the analyst needs.

Meet Automater:

The tool currently only queries IPVoid, Robtex, and Fortiguard currently, but I am working on adding modules from all the resources I mentioned earlier in the article.  The help option will explain where I am heading with the project, all though I am not quite there yet. 

root@bt:~/workspace/Automater# ./Automater.py -h

 

    ONLY -t AND -h WORK CURRENTLY!! 

    -t: target ip or url.  URL must include http://

    -s: source engine (robtex, ipvoid, fortiguard)

    -a: all engines

    -h: help

    -f: import a file of IPs and/or URLs

    -o: output results to file

    -i: Interactive Mode

    Examples:

    ./Automater.py -t 123.123.123.123 -a -o result.txt

    ./Automater.py -f hosts.txt -s robtex -o results.txt

Automater right now only takes the -t and the -h options and only works for one target at a time.  This again will change as I modify it.  URL support has not been added yet either.

I am posting this now in its pre-release form because I would like to hear from the community what types of features they would like to see added.  I would also like to know of any bugs you can find.  Lastly, I am of course interested in anyone who would like to contribute to the project.  If all goes as planned I would like to have the tool fully functional within a couple of weeks.  Once complete I will attempt to pitch the tool to Doug Burks to add to his Security Onion Distro.  I think this could be a really nice tool for analysts.

Contribute to, or download the tool on Github.

Report any Bugs or feature requests to 1aN0rmus@TekDefense.com

DEMO:

TekTip ep12 - Regex Basics

Description: Regular expressions are a way to match specific patterns in strings.

Demo Setup: For the demo, I am using BT5 with eclipse and pydev.  I am demonstrating with python, but there are many other methods that could be used.

Uses:  Singature creation pcre, scripting, programming

"\w": word character

"\W": Not word character

"\d" : digit

"\D" : Not digit

"\s" : space

"\S" : not space

"\" : escape 

"." : Any character except \n (new line)

"|" : or

"{}" : Range

"+" : One or more

Examples:

String: Hi, my name is 1aN0rmus.  My phone number is 555-555-5555 my address is 123 Internet Lane. 

Regex: '\d\w\w\d\w\w\w\w'

Result: 1aN0rmus

Regex: '\d\w+\d\w+'

Result: 1aN0rmus

Regex: '\d{3}.\d{3}.\d{4}'

Result: 555.555.5555 

Regex: '\d{1,5}\s\w+\s\w+'

Result: 123 Internet Lane

String: saDaSDASD/forum/Themes/core/images/sdfs.exe SAdasd

Regex: '\/forum\/Themes\/core\/images\/\w+\.exe'

Result: /forum/Themes/core/images/sdfs.exe

Scripts seen on this episode can be downloaded at https://github.com/1aN0rmus/TekDefense.  Feel free to help out with either script.  I'm a beginner when it comes to python.

Thank you,

1aN0rmus@tekdefense.com

http://www.securitytube.net/user/1aN0rmus

www.youtube.com/user/TekDefense