Merge branch 'develop' into 22-undo-vote due to large number of chang…

…es since initial branch.

Testing progress steemit#22

doc/Python-Debug-Node.md

@@ -0,0 +1,78 @@
+
+Python Debug Node Readme
+------------------------
+
+The Python Debug Node is a wrapper class that automates the creation and maintenance
+of a running Steem Debug Node. The Debug Node is a plugin for Steem that allows realtime
+local modification of the chain state in a way that mimicks real world behaviors
+without corrupting a localally saved blockchain or propogating changes to the live chain.
+
+More information of the debug node can be found [here](debug_node_plugin.md)
+
+Why Should I Use This?
+----------------------
+
+While the Debug Node Plugin is very powerful, it does require intimate knowledge of how
+to run a node in various configurations as well as interfacing with the node over the
+RPC or WebSocket interfaces. Both of these are considered developer level skills. Python
+is a higher level language that many amateur and skilled programmers use. There has already
+been community development of Python libraries to make interfacing with a live node easier.
+This plugin closes the gap by allowing a node to be launched programmatically in Python
+in addition to interfacing with the node. This module utilizes community member Xeroc's
+[Python Steem library](https://github.com/xeroc/python-steemlib).
+
+How Do I Use This?
+------------------
+
+First of all, you need to install the module. Navigate to `tests/external_testing_scripts`
+and run `python3 setup.py install`
+To use the script include `from steemdebugnode import DebugNode`
+
+There are a couple of examples already made that you can try modifying yourself.
+
+[debug_hardforks.py](https://github.com/steemit/steem/python_scripts/tests/debug_hardforks.py)
+This script starts a debug node, replays blocks, schedules a hardfork, and finally generates
+new blocks after the hardfork. The script also communicates via the general purpose rpc
+interface in Xeroc's Library to do a simple analysis of the results. In this case it
+generates a historgram of block producers to verify the witness scheduling algorithm works
+properly. The purpose of the script is it verify any given hardfork does not have a bug that
+could crash the chain entirely.
+
+[debugnode.py](https://github.com/steemit/steem/python_scripts/steemdebugnode/debugnode.py#L212)
+This script is much simpler. It has the same parsing logic, but has much less test logic.
+All it does is replay the blockchain, periodically printing a status update so the user
+knows it is still working. The script then hangs so the user can interact with the chain
+through RPC calls or the CLI Wallet.
+
+What is the important part of these scripts?
+
+``` Python
+debug_node = DebugNode( str( steemd ), str( data_dir ) )
+with debug_node:
+   # Do stuff with blockchain
+```
+
+The module is set up to use the `with` construct. The object construction simply checks
+for the correct directories and sets internal state. `with debug_node:` connects the node
+and establishes the internal rpc connection. The script can then do whatever it wants to.
+When the `with` block ends, the node automatically shutsdown and cleans up. The node uses
+a system standard temp directory through the standard Python TemporaryDirectory as the
+working data directory for the running node. The only work your script needs to do is
+specify the steemd binary location and a populated data directory. For most configurations
+this will be `programs/steemd/steemd` and `witness_node_data_dir` respectively, from the
+git root directory for Steem.
+
+TODO/ Long Term Goals
+---------------------
+
+While this is a great start to a Python testing framework, there are a lot of areas in
+which this script is lacking. There is a lot of flexibility in the node, from piping
+stdout and stderr for debugging to dynamically specifying plugins and APIs to run with,
+there is a lack of ways to interface. Ideally, there would a Python equivalent wrapper
+for every RPC API call. This is tedious and with plugins, those API calls could change
+revision to revision. For the most part, the exposed Debug Node calls are wrappers for
+the RPC call. Most, if not all, RPC API calls could be programatically generated from
+the C++ source. It would also be a good step forward to introduce a simple testing framework
+that could be used to start a debug node and then run a series of test cases on a common
+starting chain state. This would address much of the integration testing that is sorely
+needed for Steem.

example_plugins/hello_api/hello_api_plugin.cpp

@@ -49,6 +49,13 @@ class hello_api_api
 {
    public:
       hello_api_api( steemit::app::application& app );
+
+      /**
+       * Called immediately after the constructor.  If the API class uses enable_shared_from_this,
+       * shared_from_this() is available in this method, which allows signal handlers to be registered
+       * with app::connect_signal()
+       */
+      void on_api_startup();
       std::string get_message();
 
    private:
@@ -91,6 +98,8 @@ std::string hello_api_plugin::get_message()
 
 hello_api_api::hello_api_api( steemit::app::application& app ) : _app(app) {}
 
+void hello_api_api::on_api_startup() {}
+
 std::string hello_api_api::get_message()
 {
    std::stringstream result;

libraries/app/api.cpp

@@ -48,8 +48,12 @@ namespace steemit { namespace app {
     {
     }
 
+    void login_api::on_api_startup() {
+    }
+
     bool login_api::login(const string& user, const string& password)
     {
+       idump((user)(password));
        optional< api_access_info > acc = _app.get_api_access_info( user );
        if( !acc.valid() )
           return false;
@@ -65,19 +69,30 @@ namespace steemit { namespace app {
              return false;
        }
 
+       idump((acc->allowed_apis));
        for( const std::string& api_name : acc->allowed_apis )
        {
           auto it = _api_map.find( api_name );
-          if( it != _api_map.end() )
+          if( it != _api_map.end() ) {
+             wlog( "known api: ${api}", ("api",api_name) );
              continue;
+          }
+          idump((api_name));
           _api_map[ api_name ] = _app.create_api_by_name( api_name );
        }
        return true;
     }
 
     network_broadcast_api::network_broadcast_api(application& a):_app(a)
     {
-       _applied_block_connection = _app.chain_database()->applied_block.connect([this](const signed_block& b){ on_applied_block(b); });
+       /// NOTE: cannot register callbacks in constructor because shared_from_this() is not valid.
+    }
+
+    void network_broadcast_api::on_api_startup()
+    {
+       /// note cannot capture shared pointer here, because _applied_block_connection will never
+       /// be freed if the lambda holds a reference to it.
+       _applied_block_connection = connect_signal( _app.chain_database()->applied_block, *this, &network_broadcast_api::on_applied_block );
     }
 
     void network_broadcast_api::on_applied_block( const signed_block& b )
@@ -94,9 +109,9 @@ namespace steemit { namespace app {
              auto itr = _callbacks.find(id);
              if( itr != _callbacks.end() )
              {
-                const auto& callback = _callbacks.find(id)->second;
+                auto callback = _callbacks.find(id)->second;
                 fc::async( [capture_this,this,id,block_num,trx_num,callback](){ callback( fc::variant(transaction_confirmation{ id, block_num, trx_num, false}) ); } );
-                _callbacks.erase( itr );// safe becaues callback is copied by lambda
+                itr->second = []( const variant& ){};
              }
           }
        }
@@ -110,9 +125,9 @@ namespace steemit { namespace app {
                auto cb_itr = _callbacks.find( trx_id );
                if( cb_itr != _callbacks.end() ) {
                    auto capture_this = shared_from_this();
-                   const auto& callback = _callbacks.find(trx_id)->second; 
+                   auto callback = _callbacks.find(trx_id)->second; 
                    fc::async( [capture_this,this,block_num,trx_id,callback](){ callback( fc::variant(transaction_confirmation{ trx_id, block_num, -1, true}) ); } );
-                   _callbacks.erase( cb_itr ); // safe becaues callback is copied by lambda
+                   _callbacks.erase(cb_itr);
                }
              }
              _callbacks_expirations.erase( itr );
@@ -156,6 +171,8 @@ namespace steemit { namespace app {
     {
     }
 
+    void network_node_api::on_api_startup() {}
+
     fc::variant_object network_node_api::get_info() const
     {
        fc::mutable_variant_object result = _app.p2p_node()->network_get_info();

libraries/app/application.cpp

@@ -227,8 +227,8 @@ namespace detail {
       {
          _self->register_api_factory< login_api >( "login_api" );
          _self->register_api_factory< database_api >( "database_api" );
-         _self->register_api_factory< network_broadcast_api >( "network_broadcast_api" );
          _self->register_api_factory< network_node_api >( "network_node_api" );
+         _self->register_api_factory< network_broadcast_api >( "network_broadcast_api" );
       }
 
       void startup()
@@ -333,6 +333,7 @@ namespace detail {
             wild_access.allowed_apis.push_back( "network_broadcast_api" );
             wild_access.allowed_apis.push_back( "history_api" );
             wild_access.allowed_apis.push_back( "crypto_api" );
+            wild_access.allowed_apis.push_back( "private_message_api" );
             _apiaccess.permission_map["*"] = wild_access;
          }
 
@@ -379,7 +380,10 @@ namespace detail {
       {
          auto it = _api_factories_by_name.find(name);
          if( it == _api_factories_by_name.end() )
+         {
+            wlog( "unknown api: ${api}", ("api",name) );
             return nullptr;
+         }
          return it->second();
       }