Merged -r13793:HEAD from nmap-exp/dev/nmap branch now that we're open…

…ing up trunk development

again.  Here are the items which were merged:

------------------------------------------------------------------------
r13971 | jah | 2009-06-29 14:30:27 -0700 (Mon, 29 Jun 2009) | 2 lines

Improved a pattern for matching HTTP status-line, tidy away some variables and
fix a typo.
------------------------------------------------------------------------
r13967 | daniel | 2009-06-29 13:47:04 -0700 (Mon, 29 Jun 2009) | 5 lines

o Added a convenience top-level BSD makefile redirecting BSD make
  to GNU make on BSD systems.  This should help prevent bogus
  error reports when users run "make" instead of "gmake" on BSD
  systems. [Daniel Roethlisberger]

------------------------------------------------------------------------
r13965 | batrick | 2009-06-29 06:50:11 -0700 (Mon, 29 Jun 2009) | 14 lines

[NSE] The NSE Nsock Library binding no longer relies on garbage collection to
monitor the use of socket "slots". A thread (script) attempting to connect must
first obtain one of a limited number of available socket locks (usually 10 or
--max-parallelism). The binding would use garbage collection of sockets to
determine when a thread has finished using its allocated sockets. This is
unfortunately slow and requires us to constantly run the garbage collector to
cause timely reallocation. I have changed the binding to now regularly inspect
allocated sockets in the nsock_loop function. Available sockets slots are now
immediately reallocated and done with far less execution time.

See [1] for benchmarks and further explanation.

[1] http://seclists.org/nmap-dev/2009/q2/0624.html

------------------------------------------------------------------------
r13964 | batrick | 2009-06-29 06:37:49 -0700 (Mon, 29 Jun 2009) | 10 lines

[NSE] Fixed a rare (and usually undetectable) bug that can cause a SEGFAULT.
The NSE nsock library binding may attempt to push values on the stack of
a thread that ended due to an error. It is possible that the internal
Lua stack was completely full and any further pushed values would result
in a segmentation memory violation.

This bug is very hard to reproduce with a SEGFAULT but is usually visible
when Lua assertion checks are turned on. A socket handler routine must be
called AFTER a thread has ended in error.

------------------------------------------------------------------------
r13963 | batrick | 2009-06-29 05:51:20 -0700 (Mon, 29 Jun 2009) | 3 lines

Fixed some global scoped variables to be local. This caused a many scripts to
overwrite each others' sockets, options, etc.

------------------------------------------------------------------------
r13939 | joao | 2009-06-27 16:07:35 -0700 (Sat, 27 Jun 2009) | 2 lines

Fixed port rule to include ssl pop3 port, now that pop3.lua supports SSL connections in function capabilities

------------------------------------------------------------------------
r13938 | joao | 2009-06-27 16:06:28 -0700 (Sat, 27 Jun 2009) | 2 lines

Added transparent SSL support using comm.tryssl

------------------------------------------------------------------------
r13937 | joao | 2009-06-27 16:05:19 -0700 (Sat, 27 Jun 2009) | 2 lines

Added transparent SSL support using comm.tryssl

------------------------------------------------------------------------
r13936 | joao | 2009-06-27 16:03:50 -0700 (Sat, 27 Jun 2009) | 2 lines

Added SSL transparent support using comm.tryssl

------------------------------------------------------------------------
r13935 | joao | 2009-06-27 16:02:39 -0700 (Sat, 27 Jun 2009) | 2 lines

Added SSL transparent support using comm.tryssl

------------------------------------------------------------------------
r13934 | joao | 2009-06-27 16:01:38 -0700 (Sat, 27 Jun 2009) | 2 lines

Added SSL transparent support using comm.tryssl

------------------------------------------------------------------------
r13933 | joao | 2009-06-27 16:00:27 -0700 (Sat, 27 Jun 2009) | 2 lines

SSL transparent support using comm.tryssl

------------------------------------------------------------------------
r13932 | joao | 2009-06-27 15:19:58 -0700 (Sat, 27 Jun 2009) | 2 lines

Included transparent ssl support to function pop3.capabilities using comm.tryssl

------------------------------------------------------------------------
r13931 | joao | 2009-06-27 15:19:06 -0700 (Sat, 27 Jun 2009) | 3 lines

New version of comm.lua with function tryssl, that transparently adds support to ssl connections


------------------------------------------------------------------------
r13930 | joao | 2009-06-27 14:50:38 -0700 (Sat, 27 Jun 2009) | 6 lines

Fixed buffering problem exposed by david on nmap-dev list.
The problem was solved using a buffer to receive the data, making the script work fine in cases where the ssh packets are fragmented.

A very similar solution was applied to ssh1.lua.


------------------------------------------------------------------------
r13928 | batrick | 2009-06-27 04:43:12 -0700 (Sat, 27 Jun 2009) | 18 lines

[NSE] We now propogate a NSE initiated yield on a script through all user
coroutines so that NSE may resume control. Previously, scripts that would yield
in a child coroutine (e.g. a script's child coroutine generated by Lua's
coroutine.create function) would give control back to the script. A script
would yield in this way by making a blocking socket operation. NSE would be
unable to correctly resume child coroutine when the socket operation is
finished processing.

By yielding the chain of coroutines a script has operating, we allow to NSE to
handle the socket operation properly. NSE would then resume the entire chain so
execution may correctly resume at the coroutine which initiated the socket
operation. This restores the "illusion" that a script executes without
interruption.

See [1] for more information, further explanation, and some use cases.

[1] http://seclists.org/nmap-dev/2009/q2/0586.html

------------------------------------------------------------------------
r13817 | david | 2009-06-18 15:57:29 -0700 (Thu, 18 Jun 2009) | 3 lines

Improve an OS fingerprint with a model number and broader matching.
Based on a follow-up report from a submitter.

------------------------------------------------------------------------
r13814 | josh | 2009-06-17 21:34:15 -0700 (Wed, 17 Jun 2009) | 3 lines

[zenmap] Added support to zenmap for the new SCTP options: -PY, -sY and -sZ


------------------------------------------------------------------------
r13797 | ron | 2009-06-17 11:02:18 -0700 (Wed, 17 Jun 2009) | 1 line

Applied a patch from Mak Kolibabi that enhances the output of smb-enum-processes. The output is now modeled after the output of the 'ps' tool for higher verbosity levels.
------------------------------------------------------------------------
r13795 | david | 2009-06-17 09:05:21 -0700 (Wed, 17 Jun 2009) | 6 lines

The configure script now allows cross-compiling by assuming that
libpcap is recent enough. Previously it would quit because a test
program could not be run. libpcap will always be recent enough when
the included copy is used. The patch was contributed by Mike
Frysinger.

BSDmakefile

@@ -0,0 +1,9 @@
+# $Id$
+# Redirect BSD make to GNU gmake for convenience
+
+USE_GNU:
+	@gmake $(.TARGETS)
+
+$(.TARGETS): USE_GNU
+
+.PHONY: USE_GNU

CHANGELOG

@@ -1,5 +1,20 @@
 # Nmap Changelog ($Id$); -*-text-*-
 
+o Added a convenience top-level BSD makefile redirecting BSD make
+  to GNU make on BSD systems.  This should help prevent bogus
+  error reports when users run "make" instead of "gmake" on BSD
+  systems. [Daniel Roethlisberger]
+
+o [Zenmap] Added support to zenmap for the SCTP options: -PY, -sY and -sZ, as
+  well as making a comment in zenmapCore/NmapOptions.py on how to add new
+  options. [Josh Marlow]
+
+o The configure script now allows cross-compiling by assuming that
+  libpcap is recent enough. Previously it would quit because a test
+  program could not be run. libpcap will always be recent enough when
+  the included copy is used. The patch was contributed by Mike
+  Frysinger.
+
 Nmap 4.90RC1 [2009-06-25]
 
 o [Zenmap] Fixed a display hanging problem on Mac OS X reported by

configure

@@ -7089,13 +7089,8 @@ if test $have_libpcap = yes; then
   { $as_echo "$as_me:$LINENO: checking if libpcap version is recent enough" >&5
 $as_echo_n "checking if libpcap version is recent enough... " >&6; }
   if test "$cross_compiling" = yes; then
-  { { $as_echo "$as_me:$LINENO: error: in \`$ac_pwd':" >&5
-$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
-{ { $as_echo "$as_me:$LINENO: error: cannot run test program while cross compiling
-See \`config.log' for more details." >&5
-$as_echo "$as_me: error: cannot run test program while cross compiling
-See \`config.log' for more details." >&2;}
-   { (exit 1); exit 1; }; }; }
+  { $as_echo "$as_me:$LINENO: result: cross-compiling -- assuming yes" >&5
+$as_echo "cross-compiling -- assuming yes" >&6; }; have_libpcap=yes
 else
   cat >conftest.$ac_ext <<_ACEOF
 /* confdefs.h.  */

configure.ac

@@ -386,7 +386,8 @@ int main() {
   exit(0);
 }],
 [AC_MSG_RESULT(yes); have_libpcap=yes],
-[AC_MSG_RESULT(no); have_libpcap=no])
+[AC_MSG_RESULT(no); have_libpcap=no],
+[AC_MSG_RESULT(cross-compiling -- assuming yes); have_libpcap=yes])
 LIBS="$LIBS_OLD"
 fi
 

nmap-os-db

@@ -3496,10 +3496,10 @@ T7(R=Y%DF=Y%T=FA-104%TG=FF%W=0%S=Z%A=S+%F=AR%O=%RD=0%Q=)
 U1(DF=N%T=FA-104%TG=FF%IPL=164%UN=0%RIPL=G%RID=G%RIPCK=G%RUCK=G%RUD=G)
 IE(DFI=N%T=FA-104%TG=FF%CD=S)
 
-# AV-TECH Digital Video Recorder (device connected to a surveillance camera)
-Fingerprint AVtech digital video recorder
+# AV-TECH Digital Video Recorder (device connected to a surveillance camera), AVT AVC 773W 4CH Network DVR
+Fingerprint AVtech AVC 773W digital video recorder
 Class AVtech | embedded || webcam
-SEQ(SP=C-1E%GCD=1-6%ISR=55-5F%TI=I%II=RI%SS=O|S%TS=U)
+SEQ(SP=C-1E%GCD=1-6%ISR=1B-5F%TI=I%II=RI%SS=O|S%TS=U)
 OPS(O1=M59E%O2=M578%O3=M280%O4=M59E%O5=M218%O6=M109)
 WIN(W1=FAF0%W2=FAF0%W3=FAF0%W4=FAF0%W5=FAF0%W6=FAF0)
 ECN(R=Y%DF=Y%T=FA-104%TG=FF%W=FAF0%O=M59E%CC=N%Q=)
@@ -3510,8 +3510,8 @@ T4(R=Y%DF=Y%T=FA-104%TG=FF%W=FAF0%S=A+%A=S%F=AR%O=%RD=0%Q=)
 T5(R=Y%DF=Y%T=FA-104%TG=FF%W=FAF0%S=A%A=S+%F=AR%O=%RD=0%Q=)
 T6(R=Y%DF=Y%T=FA-104%TG=FF%W=FAF0%S=A%A=S%F=AR%O=%RD=0%Q=)
 T7(R=Y%DF=Y%T=FA-104%TG=FF%W=FAF0%S=A%A=S+%F=AR%O=%RD=0%Q=)
-U1(DF=Y%T=FA-104%TG=FF%IPL=38%UN=27B6|306C|3949|425C|4B9A%RIPL=G%RID=G%RIPCK=G%RUCK=G%RUD=G)
-IE(DFI=S%T=24-2E|30-40%TG=40%CD=S)
+U1(DF=Y%T=FA-104%TG=FF%IPL=38%RIPL=G%RID=G%RIPCK=G%RUCK=G%RUD=G)
+IE(DFI=S%T=24-40%TG=40%CD=S)
 
 # Axis 70U Network document server
 Fingerprint AXIS 70U Network Document Server

nse_main.cc

@@ -23,6 +23,8 @@
 #define NSE_TRACEBACK "NSE_TRACEBACK"
 
 /* string keys used in interface with nse_main.lua */
+#define NSE_YIELD "NSE_YIELD"
+#define NSE_BASE "NSE_BASE"
 #define NSE_WAITING_TO_RUNNING "NSE_WAITING_TO_RUNNING"
 #define NSE_DESTRUCTOR "NSE_DESTRUCTOR"
 
@@ -151,10 +153,16 @@ static int dump_dir (lua_State *L)
   return 1;
 }
 
+/* This must call the l_nsock_loop function defined in nse_nsock.cc.
+ * That closure is created in luaopen_nsock in order to allow
+ * l_nsock_loop to have access to the nsock library environment.
+ */
 static int nsock_loop (lua_State *L)
 {
-  if (l_nsock_loop(luaL_checkint(L, 1)) == NSOCK_LOOP_ERROR)
-    luaL_error(L, "an error occurred in nsock_loop");
+  lua_settop(L, 1);
+  lua_getfield(L, LUA_REGISTRYINDEX, NSE_NSOCK_LOOP);
+  lua_pushvalue(L, 1);
+  lua_call(L, 1, 0);
   return 0;
 }
 
@@ -464,6 +472,14 @@ static int run_main (lua_State *L)
   return 0;
 }
 
+int nse_yield (lua_State *L)
+{
+  lua_getfield(L, LUA_REGISTRYINDEX, NSE_YIELD);
+  lua_pushthread(L);
+  lua_call(L, 1, 1); /* returns NSE_YIELD_VALUE */
+  return lua_yield(L, 1); /* yield with NSE_YIELD_VALUE */
+}
+
 void nse_restore (lua_State *L, int number)
 {
   luaL_checkstack(L, 5, "nse_restore: stack overflow");
@@ -506,6 +522,12 @@ void nse_destructor (lua_State *L, char what)
   lua_pop(L, what == 'a' ? 2 : 1);
 }
 
+void nse_base (lua_State *L)
+{
+  lua_getfield(L, LUA_REGISTRYINDEX, NSE_BASE);
+  lua_call(L, 0, 1); /* returns base thread */
+}
+
 static lua_State *L_NSE = NULL;
 
 void open_nse (void)

nse_main.h

@@ -31,8 +31,10 @@ class Target;
 
 
 /* API */
+int nse_yield (lua_State *);
 void nse_restore (lua_State *, int);
 void nse_destructor (lua_State *, char);
+void nse_base (lua_State *);
 
 void open_nse (void);
 void script_scan (std::vector<Target *> &targets);

nse_main.lua

@@ -32,6 +32,8 @@
 
 local NAME = "NSE";
 
+local YIELD = "NSE_YIELD";
+local BASE = "NSE_BASE";
 local WAITING_TO_RUNNING = "NSE_WAITING_TO_RUNNING";
 local DESTRUCTOR = "NSE_DESTRUCTOR";
 
@@ -84,6 +86,45 @@ do -- Append the nselib directory to the Lua search path
   package.path = package.path..";"..path.."?.lua";
 end
 
+-- NSE_YIELD_VALUE
+-- This is the table C uses to yield a thread with a unique value to
+-- differentiate between yields initiated by NSE or regular coroutine yields.
+local NSE_YIELD_VALUE = {};
+
+do
+  -- This is the method by which we allow a script to have nested
+  -- coroutines. If a sub-thread yields in an NSE function such as
+  -- nsock.connect, then we propogate the yield up. These replacements
+  -- to the coroutine library are used only by Script Threads, not the engine.
+
+  local function handle (co, status, ...)
+    if status and NSE_YIELD_VALUE == ... then -- NSE has yielded the thread
+      return handle(co, resume(co, yield(NSE_YIELD_VALUE)));
+    else
+      return status, ...;
+    end
+  end
+
+  function coroutine.resume (co, ...)
+    return handle(co, resume(co, ...));
+  end
+
+  local resume = coroutine.resume; -- local reference to new coroutine.resume
+  local function aux_wrap (status, ...)
+    if not status then
+      return error(..., 2);
+    else
+      return ...;
+    end
+  end
+  function coroutine.wrap (f)
+    local co = create(f);
+    return function (...)
+      return aux_wrap(resume(co, ...));
+    end
+  end
+end
+
 -- Some local helper functions --
 
 local log_write, verbosity, debugging =
@@ -430,16 +471,30 @@ local function run (threads)
     hosts[thread.host][thread.co] = true;
   end
 
+  -- Map of yielded threads to the base Thread
+  local yielded_base = setmetatable({}, {__mode = "kv"});
+  -- _R[YIELD] is called by nse_yield in nse_main.cc
+  _R[YIELD] = function (co)
+    yielded_base[co] = current; -- set base
+    return NSE_YIELD_VALUE; -- return NSE_YIELD_VALUE
+  end
+  _R[BASE] = function ()
+    return current.co;
+  end
   -- _R[WAITING_TO_RUNNING] is called by nse_restore in nse_main.cc
   _R[WAITING_TO_RUNNING] = function (co, ...)
-    if waiting[co] then -- ignore a thread not waiting
-      pending[co], waiting[co] = waiting[co], nil;
-      pending[co].args = {n = select("#", ...), ...};
+    local base = yielded_base[co] or all[co]; -- translate to base thread
+    if base then
+      co = base.co;
+      if waiting[co] then -- ignore a thread not waiting
+        pending[co], waiting[co] = waiting[co], nil;
+        pending[co].args = {n = select("#", ...), ...};
+      end
     end
   end
   -- _R[DESTRUCTOR] is called by nse_destructor in nse_main.cc
   _R[DESTRUCTOR] = function (what, co, key, destructor)
-    local thread = all[co] or current;
+    local thread = yielded_base[co] or all[co] or current;
     if thread then
       local ch = thread.close_handlers;
       if what == "add" then
@@ -456,7 +511,6 @@ local function run (threads)
   -- Loop while any thread is running or waiting.
   while next(running) or next(waiting) do
     local nr, nw = table_size(running), table_size(waiting);
-    cnse.nsock_loop(50); -- Allow nsock to perform any pending callbacks
     if cnse.key_was_pressed() then
       print_verbose(1, "Active NSE Script Threads: %d (%d waiting)\n",
           nr+nw, nw);
@@ -489,7 +543,12 @@ local function run (threads)
             traceback(co, tostring(result)));
         thread:close();
       elseif status(co) == "suspended" then
-        waiting[co] = thread;
+        if result == NSE_YIELD_VALUE then
+          waiting[co] = thread;
+        else
+          thread:d("%THREAD yielded unexpectedly and cannot be rerun.");
+          thread:close();
+        end
       elseif status(co) == "dead" then
         hosts[thread.host][co] = nil;
         if type(result) == "string" then
@@ -516,12 +575,13 @@ local function run (threads)
       end
     end
 
+    cnse.nsock_loop(50); -- Allow nsock to perform any pending callbacks
     -- Move pending threads back to running.
     for co, thread in pairs(pending) do
       pending[co], running[co] = nil, thread;
     end
 
-    collectgarbage "collect"; -- important for collecting used sockets & proxies
+    collectgarbage "step";
   end
 
   progress "endTask";

nse_nmaplib.cc

@@ -241,7 +241,7 @@ static int aux_mutex (lua_State *L)
       }
       lua_pushthread(L);
       lua_rawseti(L, lua_upvalueindex(1), lua_objlen(L, lua_upvalueindex(1))+1);
-      return lua_yield(L, 0);
+      return nse_yield(L);
     case DONE:
       lua_pushthread(L);
       if (!lua_equal(L, -1, lua_upvalueindex(2)))