Raw HTTP Requests to Burp Proxy

On a recent Web application test I encountered a new challenge. The Web application presented a Web API intended to be used by a mobile application, in order for developers to utilise this API the documentation was also served from the Web application.

In order to assess each API function for vulnerabilities I first had to build valid requests from the documentation and then get them into my Web assessment tool of choice Burp Suite Pro.

It would have been possible to accomplish this by reading the documentation and patiently typing the raw HTTP request into Burp repeater. However with over thirty API functions to test and a tight schedule this was not a viable option. I therefore decided to script it.

The first step was to download all of the HTML documentation and parse each page to extract the HTTP method, path, example URL parameters, and, if present, the example body parameters. Using this information I built raw HTTP requests which I stored in text files. (As this first script is quite specific to the client’s application I will not be releasing it at this time).

With a directory full of raw HTTP requests it was time to import them into Burp and start testing proper. However I could not find any method of importing my raw HTTP requests into Burp other than manually copying and pasting them into repeater, an achievable task with the relatively small number of functions I had to test in this instance but a chilling prospect for future, larger tests.

After a coffee I had the idea to simply send the raw HTTP request through Burp by sending them from a Web client with a proxy configured. Since the requests had a variety of HTTP methods and body parameters a Web browser wasn’t an option. I briefly tried using telnet and netcat but these failed for some reason I haven’t identified. I also tried using curl, but this required further processing to issue the request using the curls’s command line options. I therefore turned back to Python and wrote a script to read files from a directory, then for each file: parse them into an object (using BaseHttpRequestHandler), build a request using urllib2 and send this via a proxy.

This resulted in the HTTP request being stored in Burp ready for assessment like any normal request to a Web application – visible in the site map, proxy history and easily sent to Intruder, Repeater, Scanner and Sequencer.

I’ve released this script under the GPLv2 licence in the hope that it will be useful to others, it is available here.

Example Usage

Parse one or more files and send via the default proxy (127.0.0.1:8080):

raw2proxy.py -f FILENAME FILENAME...

Parse a directory of files and send via a proxy running on 192.168.0.1 port 9001:

raw2proxy.py -d DIRECTORY -p 192.168.0.1:9001

Additional options are available:

raw2proxy.py --help

As always, if you have any comments or suggestions please feel free to get in touch.

Python Script to Standalone Executable (with Icon)

When releasing tools, and proof of concepts, to the industry and more often to clients, I find I need to provide a standalone executable that can be run without installing Python and any required modules.

To accomplish this I use py2exe . While other options exist (for example pyinstaller) personally I have found py2exe quicker and easier to use once a few stumbling blocks were overcome. I therefore decided to write a short post describing how I setup and use py2exe for when laptop rebuild time comes around and in the hope it will be useful to others.

First, at the time of writing, py2exe does not support creating a single executable using 64 bit Python, throwing the error:

error: bundle-files 1 not yet supported on win64

So step 1 is to install 32 bit Python (being careful not to overwrite your existing 64 bit installation) and 32 bit versions of any non standard library modules that are required by your script.

Next you need to install py2exe itself. The project home page points to the SourceForge project page. Ensure you download the 32 bit version for the version of Python you have installed.

Now you are ready to create the script that will create your standalone executable. There are many options available, but I find the following minimal script very effective. This script will create a single executable (‘bundle_files’) for script.py.

from distutils.core import setup
import py2exe, sys

sys.argv.append('py2exe')

setup(
        options = {
                    'py2exe': {'bundle_files': 1,
                               'compressed': True
                              }
                  },
        console = [{
                    'script': "script.py"
                  }],
        zipfile = None,
)

The one additional option I sometimes use is to add a custom icon to the executable. To do this I first create my icon image (256×256 pixels) in an image editor and export the required sizes (16×16, 32×32, 48×48, 256×256) in the png image format. I then use png2ico to create a .ico file, note the order in which you add the different size images is important it must be largest to smallest otherwise the icon may not be displayed at all! i.e:

png2ico favicon.ico icon_256.png icon_48.png icon_32.png icon_16.png

With the icon (favicon.ico) created the following script can be used to turn script.py into a standalone executable with an icon.

from distutils.core import setup
import py2exe, sys

sys.argv.append('py2exe')

setup(
        options = {
                    'py2exe': {'bundle_files': 1,
                               'compressed': True
                              }
                  },
        console = [{
                    'script': "script.py",
                    'icon_resources': [(0, 'favicon.ico')]
                  }],
        zipfile = None,
)

Once the setup.py script above has been written, the standalone executable can be created simply by running it using your 32 bit Python installation (my 32 bit installation is at ‘C:\Python27_x86\python’):

C:\Python27_x86\python setup.py

By default the executable will be created in the “dist” directory.


As always, if you have any comments or suggestions please feel free to get in touch.

Installing pytesseract – practically painless

A recent project of mine called for optical character recognition.  After a brief Google search and a personal recommendation I decided to use tesseract because it is cross platform, under active development, and has a Python API (pytesseract).

Installing these was surprisingly easy:

tesseract has a Windows installer which comes with the English language data available here.

pytesseract can be installed using pip:

pip install pytesseract

pytesseract states that it requires Python Imaging Library (PIL) however this project no longer appears to be active, so I used the maintained fork of that project pillow. This can be installed using pip:

pip install pillow

And that’s it!

You should now be able to do some optical recognition with python:

import pytesseract
from PIL import Image
print pytesseract.image_to_string(Image.open('test.jpg'))

 


As always, if you have any comments or suggestions please feel free to get in touch.

Installing rdpy on Windows 7 64bit

At the time of writing I am working on a tool that utilities rdpy.  I encountered some problems installing it so I thought I’d document how I solved them for when I inevitably need to rebuild my laptop and in the hope that it will be useful to others.

Following the instructions in the readme: https://github.com/citronneur/rdpy/blob/master/README.md

pip install rdpy

Failed with error:

error: Unable to find vcvarsall.bat

This cryptic error message means that the compiler required to compile the C/C++ code in the package is not present on the system. Usually in this situation the easiest option is to download a precompiled version of the package. Unfortunately I haven’t been able to find a precompiled version of rdpy, so a compiler is required, preferably the same one used to compile the version of Python you are using. For 2.7 this is Visual Studio C++ 2008 and for 3.4.1 this is Visual Studio C++ 2010.

You can double check which version you need using the Python interactive interpreter. If the version information (shown bold red below) shows MSC v.1500 you need Visual Studio C++ 2008, whereas if it show MSC v.1600 you need Visual Studio C++ 2010.

c:\>python
ActivePython 2.7.8.10 (ActiveState Software Inc.) based on
Python 2.7.8 (default, Jul 2 2014, 19:48:49) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

So where do you get Visual Studio C++ 2008 from? Well you only need the express edition, so you can get for free from here (at the time of writing).

You will also need to install the Windows SDK, again the version is important Windows SDK for Windows 7 and .NET Framework 3.5 SP1, you should be able to uncheck everything except:

Developer Tools >> Visual C++ Compilers

(Note I encountered installation errors when I attempted this, this is probably unique to my machine, but if you also encounter a problem try installing the complete SDK, which worked for me.)

(You should register Visual Studio (for free) within 30 days.)

At this point you should now be able to install the package, however you may get this error (discussed here ):

ValueError: [u'path']

This is because the latest version of Visual Studio has changed the .bat file, but Python has not yet (at the time of writing) updated to account for this.

To fix this error you simply need to copy the <install_location>\VC\bin\vcvars64.bat to <install_location>\VC\bin\vcvarsamd64.bat and <install_location>\VC\bin\amd64\vcvarsamd64.bat

Assuming the default install path is used (C:\Program Files (x86)\Microsoft Visual Studio 9.0) the following will accomplish this if run in an elevated command prompt (i.e. Run as Administrator):

copy "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars64.bat" "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvarsamd64.bat"
copy "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars64.bat" "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\vcvarsamd64.bat"

With Visual Studio installed, rdpy should now install correctly.

There are two dependencies for rdpy, PyWin32 and PyQt4. The easiest way I have found to install PyWin32 is to install ActivePython which comes with this package. An installer for PyQt4 is available here (ensure you select the correct installer for your version of Python and operating system architecture).

With the dependencies met you should now be able to use rdpy. However you are likely to encounter this error:

C:\Python27\lib\site-packages\twisted\internet\_sslverify.py:184: UserWarning: You do not have the service_identity module installed. Please install it from <https://pypi.python.org/pypi/service_identity>. Without the service_identity module and a recent enough pyOpenSSL to support it, Twisted can perform only rudimentary TLS client hostnameverification. Many valid certificate/hostname mappings may be rejected.  verifyHostname, VerificationError = _selectVerifyImplementation()

As this helpful error message indicates, you need to install the service_identity module. This can be done easily using pip:

pip install service_identity


As always, if you have any comments or suggestions please feel free to get in touch.

Esedbxtract

TLDR: Python script to automate the extraction of hashes from ntds.dit and system files. Available here : https://bitbucket.org/grimhacker/esedbxtract.

During an internal Penetration Test, once I’ve gained Domain Administrator access the fun doesn’t stop. In order to test the strength of user account passwords I need to retrieve the password hashes.

There are several ways to do this with either specialist tools or builtin Windows utilities as  @lanjelot discusses here and Inquis discusses here and here.

My preferred method is to use Volume Shadow Copy to extract a copy of the NTDS.dit and SYSTEM files, since this is an administrative task carried out with Windows utilities it does not normal cause alerts or require the Antivirus to be disabled – as is the case with some of the other options.

Once I have these files on my Kali box I use esedbexport from libesedb to export the data and link tables, these tables are used with the SYSTEM file by ntdshashes.py (available here by LaNMaSteR53 based on ntdsusers.py from ntdsxtract) to get the hashes.

This method can take a long time if the Active Directory is very large, but at this point during the assessment I’m not usually in a rush, it is less intensive on the domain controller, and it doesn’t panic the IT staff of the organisation with alerts.

There are a few reasons why I wrote a script to automate the extraction of the hashes from the NTDS.dit and SYSTEM files.

First and foremost, it’s a multi step process that doesn’t actually require any brain power – just for the output of one tool to be fed into another in the correct way. Writing a script to do this means that time isn’t wasted waiting for me to come back to it between each step. And of course it is another success for my continuing mission to replace myself with a small script…

Second ntdshashes.py didn’t work with the the latest version of ntdsxtract when I last rebuilt my machine (because a ntdsxtract added a new required working directory parameter). I patched the script (and raised an Issue on the repository) but decided it wouldn’t take much work to rewrite it as a class and include it in a larger tool.

Finally I hadn’t played with calling external programs as a subprocess in python before and this seemed like a reasonable excuse.

The result is esedbxtraxt.py available on BitBucket: https://bitbucket.org/grimhacker/esedbxtract.

Dependencies are stated in the README.

Normal usage will result in the password hashes being written to ‘hashes.pwdump’:

python esedbxtract.py -n /path/to/NTDS.dit -s /path/to/SYSTEM

Use ‘–help’ or see the README for further options.


As always, if you have any questions, comments or suggestions please feel free to get in touch.

 

Bitfields

I recently had need to interpret bitfields with Python.
I’m quite happy with the 3 lines of code that I came up with so I thought I’d share them in case they are of use to anyone else.

Bitfields are basically a binary number where each bit is assigned a meaning which can either have a value of True ‘1’ or False ‘0’.
Usually they are interpreted using bit shifting and bitwise AND operations but this seemed to be quite involved to get the data into a usable form so I found another way.

Consider the pwdProperties attribute from Active Directory (http://msdn.microsoft.com/en-us/library/ms679431(v=vs.85).aspx) which contains several settings for the account as a bitfield which can be retrieved using an LDAP query.

Each of the bits of this attribute mean the following:
1st bit = DOMAIN_PASSWORD_COMPLEX
2nd bit = DOMAIN_PASSWORD_NO_ANON_CHANGE
3rd bit = DOMAIN_PASSWORD_NO_CLEAR_CHANGE
4th bit = DOMAIN_LOCKOUT_ADMINS
5th bit = DOMAIN_PASSWORD_STORE_CLEARTEXT
6th bit = DOMAIN_REFUSE_PASSWORD_CHANGE

So if the pwdProperties attribute has a value of 17 in decimal, which equals 010001 in binary, the 1st and 5th bits (from the right) are set to 1 indicating that the domain requires complex passwords and stores passwords in cleartext.

Using python-ldap this attribute is returned in a dictionary as a decimal number represented as a string within a list, i.e.

attrs = {'pwdProperties': ['17']}

So the first step is to extract the string of the number and convert it to an integer:

pwd_properties = int(attrs['pwdProperties'][0])

Next the decimal number is converted to a string representation of the binary number with left 0 padding to the correct length:

pwd_properties = format(pwd_properties, "06b")

Then the binary number string is split into a list:

pwd_properties = list(pwd_properties)

For my purposes I needed the bitfields to be represented as a boolean. To do this a the string replace() method is used to replace instances of ‘0’ with an empty string and then the bool() function is used to convert the result to either True or False while iterating over the list. (Note when dealing with strings an empty string is False and everything else is True).

bitfield_values = [bool(w.replace('0', '')) for w in pwd_properties]

Next a list containing the meaning of each bit is defined (make sure you have them in the correct order to match the bits) :

bitfield_keys = ['refuse_password_change', 'password_store_cleartext', 'lockout_admins', 'password_no_clear_change', 'password_no_anon_change', 'password_complex']

The two lists can then be formed into a list of tuples using zip() which is then used to create a dictionary using dict() :

pwd_properties = dict(zip(bitfield_keys, bitfield_values))

Finally this can all be condensed into :

bitfield_keys = ['refuse_password_change', 'password_store_cleartext', 'lockout_admins', 'password_no_clear_change', 'password_no_anon_change', 'password_complex']
bitfield_values = [bool(w.replace('0', '')) for w in list(format(int(attrs['pwdProperties'][0]), '06b'))]
pwd_properties = dict(zip(bitfield_keys, bitfield_values))

Resulting in a dictionary like this:

{'password_store_cleartext': True,
'password_no_anon_change': False,
'lockout_admins': False,
'refuse_password_change': False,
'password_no_clear_change': False,
'password_complex': True}

A limitation of this method is that it is not easy to go from the resulting dictionary back to the bitfield because a dictionary in Python is unordered. This can probably be overcome by using an ordered dictionary from the collections module. However for my current purpose there is no advantage to implementing this.

EDIT:
I have been mulling this over and come up with the following line to convert the dictionary back to a binary number :

int("{refuse_password_change}{password_store_cleartext}{lockout_admins}{password_no_clear_change}{password_no_anon_change}{password_complex}".format(**pwd_properties).replace('True', '1').replace('False', '0'), 2)

This is probably horribly inefficient due to the string replacement, but it works.
It takes advantage of unpacking and referencing keyword arguments to form a string with the values in the correct order, then replaces the strings ‘True’ and ‘False’ with ‘1’ and ‘0’ respectively before using the int() function to convert the string base 2 number (i.e. binary) to a decimal number.

It might be more efficient to avoid string replacement like this :

int("{refuse_password_change}{password_store_cleartext}{lockout_admins}{password_no_clear_change}{password_no_anon_change}{password_complex}".format(**dict(zip(pwd_properties.keys(), ['1' if pwd_properties[key] == True else '0' for key in pwd_properties.keys()]))), 2)

This recreates the dictionary with ‘1’ and ‘0’ by testing each key for True. Then takes advantage of unpacking and keyword arguments to get the bits in the correct order, before converting to a decimal number using int().

At some point I’ll time these two methods to find which is more efficient and update this post.


As always, if you have any comments or suggestions please feel free to get in touch.

Installing python-ldap on Windows 7 64bit

TL;DR: Architecture is important (duh), download the 64bit exe from here.


I recently had a horrendous time trying to install python-ldap on Windows 7 64bit so I decided to write a short post for when my future self needs to install it again in the hope that it will go much more smoothly.

As it turns out I missed one key piece of information, the “preferred point for downloading the ‘official’ source distribution is now the PyPI repository which supports installing via setuptools” (http://www.python-ldap.org/download.shtml) BUT this only has 32bit versions, not 64bit!

Therefore installing using pip or the exe fails (obvious in hindsight).

Here are some of the super helpful error message I encountered while I epically failed to take architectures into account:

1) Installing using pip (pip install python-ldap) requires Visual C++ 2008 Express Edition installed, which I didn’t have so resulted in:

error: Unable to find vcvarsall.bat

To fix, install it free from Microsoft here because it will be required for other modules in the future. Then read 2).

2) Installing using pip (pip install python-ldap) tries to install a 32bit version which isn’t much use when the OS and Python interpreter are 64bit so results in:

ValueError: [u'path']

To fix: there is not a 64bit version in the pip repo as far as I am aware – skip to the punch line below.

3) Installing using the officially distributed executable on PyPI failed to find my Python install in the registry and gave me this error:

Python version 2.7 required, which was not found in the registry.

I use ActiveState Python because it comes with pywin32  but apparently it didn’t put the entry in the registry. Looking at the documentation afterwards it appears as though the installer didn’t run with admin privileges and therefore created entries under KEY_CURRENT_USER and not under HKEY_LOCAL_MACHINE.

4) After fixing 3) the exe was able to complete installation (and for a few brief moments I was happy…), but when I tried to import the module it failed with this error:

ImportError: DLL load failed: %1 is not a valid Win32 application

This is once again because of an architecture mismatch, my Python interpreter is 64bit but the module is 32bit.


After a cup of tea I noticed that the architecture problem and found the alternative installers by Christoph Gohlke here. (The python-ldap download page links to the maintainer’s (Waldemar Osuch) webpage which links to Christoph Gohlke’s page.)

I was then able to download and install the 64bit version with no problems.

 


As always, if you have any comments or suggestions please feel free to get in touch.

Nettynum – A Windows Domain Enumeration Tool

TL;DR: I’ve written a Windows Domain Enumeration Tool, you can get it from here: https://bitbucket.org/grimhacker/nettynum.


Enumerating information from the Windows Domain is nothing new and has been quite extensively covered in books, blog posts and academic papers.

So what is the point off this post you ask? To release my first tool to the community!

There are a lot of very good tools to extract information from the Windows Domain, some of which have been especially created for use by Penetration Testers while others were created for administration.

However when on tests there are a lot of things that I need to be doing and taking the output of one tool, and feeding into another is not one of them. I wanted a tool that I could set running with no information and have it find everything it could about the Windows Domain, including the Domain Name, Domain Controllers, Domain Groups, Domain Admins, and Accounts Policies.  i.e.

“Replace myself with a small shell script.”

This lead me to create Nettynum, a Python script that automates enumeration of information from the Windows Domain for my final year project at Northumbria University. It currently uses the pywin32 to access the Windows API functions and is therefore a Windows only tool – but I have plans to address this.

For those interested all 190 pages of my dissertation including UML diagrams can be found here: ‘Oliver Morton Individual Project Nettynum‘.
It was accompanied by the first version of Nettynum (those au fait with git can checkout the first commit in the main branch of the BitBucket repository).

The automated domain enumeration option will:

  1. Discover the NetBIOS and DNS style domain names on the network.
  2. Discover the domain controllers for these domains (IP address, NetBIOS name, and DNS name).
  3. Authenticate to the domain controller (with a NULL session by default).
  4. Enumerate a List of Domain Groups (and their comment).
  5. Enumerate the members of groups that match a regular expression (.*admin.*) by default.
  6. Also attempt to enumerate the members of certain groups (“Domain Admins” and “Enterprise Admins” by default) regardless of whether they were in the list of groups or not. – This is to work around the 100 group limit of the Windows API.
  7. For each of the discovered users, enumerate the account information (including comment, whether it is disabled, SID,  Bad Password Count, Password Age, and more.)
  8. Enumerate the groups of which these users are a member.  (At the time of writing I am not aware of another tool that does this.)
  9. Enumerate the Accounts Policies (Lockout Threshold, Duration and Observation Window, Minimum/Maximum Password Age, Minimum Password Length, and Password History Length).
  10. Deauthenticate from the domain controller.
  11. Output this information as an XML file (so it can be easily parsed for use by other tools [I have ideas, watch this space]) with a style sheet applied (so that the file can be opened in a browser and read easily by a human).

So in short, if the domain controller has NULL sessions enabled this:

python nettynum.py -A

Results in this:

Nettynum example output in text editor and browser
Nettynum example output in text editor and browser

There is also an automated option for ‘local’ enumeration which discovers hosts and then proceeds to enumerate information on a ‘local’ level before generating a report.

Nettynum also offers targeting options to limit the scope to specified Domain Names, Domain Controllers, or Groups. It also allows the user to specify authentication credentials, or will use a previously established session with the host (and leave it intact afterwards).

Finally there is also a ‘manual’ mode which allows the user to specify a particular category of information to enumerate but requires prerequisite information to be provided.

Where to get it: clone or download from BitBucket https://bitbucket.org/grimhacker/nettynum

Dependencies and usage are in the README file.

I have no doubt that there are bugs hiding in the code partly due the nightmare that is Windows documentation and partly because of my mediocre programming skills, so if you find any please let me know!


As always, if you have any comments or suggestions please feel free to get in touch.

Hello world!

This is my first blog post, following convention it is entitled “Hello world!”.

This blog will be very sporadic, I am by no means a literary genius and do not feel the need to post my thoughts and feelings to the Internet.

However this will be the means by which I document the tools and techniques that I use during my career as a Penetration Tester. Primarily for my own reference, but also in the hope that it is of some use to the wider community.

I will also be using this blog to share any tools and techniques that I develop to make my life easier or exploit vulnerabilities. I am not a software developer so the code is unlikely to be fantastic, but they will be tools that work (for the most part) and I hope are useful.