Massive overhaul of Python Debug Node. It is now a Python Module! See…

… doc/Python-Debug-Node.md for documentation on current working of the module.

doc/Python-Debug-Node.md

@@ -0,0 +1,78 @@
+
+Python Debug Node Readme
+------------------------
+
+The Python Debug Node is a wrapper class that automates the creation and maintenance
+of a running Steem Debug Node. The Debug Node is a plugin for Steem that allows realtime
+local modification of the chain state in a way that mimicks real world behaviors
+without corrupting a localally saved blockchain or propogating changes to the live chain.
+
+More information of the debug node can be found [here](debug_node_plugin.md)
+
+Why Should I Use This?
+----------------------
+
+While the Debug Node Plugin is very powerful, it does require intimate knowledge of how
+to run a node in various configurations as well as interfacing with the node over the
+RPC or WebSocket interfaces. Both of these are considered developer level skills. Python
+is a higher level language that many amateur and skilled programmers use. There has already
+been community development of Python libraries to make interfacing with a live node easier.
+This plugin closes the gap by allowing a node to be launched programmatically in Python
+in addition to interfacing with the node. This module utilizes community member Xeroc's
+[Python Steem library](https://github.com/xeroc/python-steemlib).
+
+How Do I Use This?
+------------------
+
+First of all, you need to install the module. Navigate to `tests/external_testing_scripts`
+and run `python3 setup.py install`
+To use the script include `from steemdebugnode import DebugNode`
+
+There are a couple of examples already made that you can try modifying yourself.
+
+[debug_hardforks.py](https://github.com/steemit/steem/python_scripts/tests/debug_hardforks.py)
+This script starts a debug node, replays blocks, schedules a hardfork, and finally generates
+new blocks after the hardfork. The script also communicates via the general purpose rpc
+interface in Xeroc's Library to do a simple analysis of the results. In this case it
+generates a historgram of block producers to verify the witness scheduling algorithm works
+properly. The purpose of the script is it verify any given hardfork does not have a bug that
+could crash the chain entirely.
+
+[debugnode.py](https://github.com/steemit/steem/python_scripts/steemdebugnode/debugnode.py#L212)
+This script is much simpler. It has the same parsing logic, but has much less test logic.
+All it does is replay the blockchain, periodically printing a status update so the user
+knows it is still working. The script then hangs so the user can interact with the chain
+through RPC calls or the CLI Wallet.
+
+What is the important part of these scripts?
+
+``` Python
+debug_node = DebugNode( str( steemd ), str( data_dir ) )
+with debug_node:
+   # Do stuff with blockchain
+```
+
+The module is set up to use the `with` construct. The object construction simply checks
+for the correct directories and sets internal state. `with debug_node:` connects the node
+and establishes the internal rpc connection. The script can then do whatever it wants to.
+When the `with` block ends, the node automatically shutsdown and cleans up. The node uses
+a system standard temp directory through the standard Python TemporaryDirectory as the
+working data directory for the running node. The only work your script needs to do is
+specify the steemd binary location and a populated data directory. For most configurations
+this will be `programs/steemd/steemd` and `witness_node_data_dir` respectively, from the
+git root directory for Steem.
+
+TODO/ Long Term Goals
+---------------------
+
+While this is a great start to a Python testing framework, there are a lot of areas in
+which this script is lacking. There is a lot of flexibility in the node, from piping
+stdout and stderr for debugging to dynamically specifying plugins and APIs to run with,
+there is a lack of ways to interface. Ideally, there would a Python equivalent wrapper
+for every RPC API call. This is tedious and with plugins, those API calls could change
+revision to revision. For the most part, the exposed Debug Node calls are wrappers for
+the RPC call. Most, if not all, RPC API calls could be programatically generated from
+the C++ source. It would also be a good step forward to introduce a simple testing framework
+that could be used to start a debug node and then run a series of test cases on a common
+starting chain state. This would address much of the integration testing that is sorely
+needed for Steem.

libraries/plugins/debug_node/debug_node_api.cpp

@@ -25,6 +25,7 @@ class debug_node_api_impl
 
       uint32_t debug_push_blocks( const std::string& src_filename, uint32_t count );
       uint32_t debug_generate_blocks( const std::string& debug_key, uint32_t count );
+      uint32_t debug_generate_blocks_until( const std::string& debug_key, const fc::time_point_sec& head_block_time, bool generate_sparsely );
       void debug_update_object( const fc::variant_object& update );
       //void debug_save_db( std::string db_path );
       void debug_stream_json_objects( const std::string& filename );
@@ -73,6 +74,7 @@ uint32_t debug_node_api_impl::debug_push_blocks( const std::string& src_filename
       ilog( "Completed loading block_database successfully" );
       return count;
    }
+   return 0;
 }
 
 uint32_t debug_node_api_impl::debug_generate_blocks( const std::string& debug_key, uint32_t count )
@@ -107,6 +109,54 @@ uint32_t debug_node_api_impl::debug_generate_blocks( const std::string& debug_ke
    return count;
 }
 
+uint32_t debug_node_api_impl::debug_generate_blocks_until( const std::string& debug_key, const fc::time_point_sec& head_block_time, bool generate_sparsely )
+{
+   std::shared_ptr< steemit::chain::database > db = app.chain_database();
+
+   if( db->head_block_time() >= head_block_time )
+      return 0;
+
+   uint32_t new_blocks = 0;
+
+   if( generate_sparsely )
+   {
+      auto new_slot = db->get_slot_at_time( head_block_time );
+      if( new_slot <= 1 )
+         new_blocks += debug_generate_blocks( debug_key, 1 );
+      else
+      {
+         fc::optional<fc::ecc::private_key> debug_private_key = graphene::utilities::wif_to_key( debug_key );
+         FC_ASSERT( debug_private_key.valid() );
+         steemit::chain::public_key_type debug_public_key = debug_private_key->get_public_key();
+
+         auto scheduled_witness_name = db->get_scheduled_witness( new_slot );
+         auto scheduled_time = db->get_slot_time( new_slot );
+         const auto& scheduled_witness = db->get_witness( scheduled_witness_name );
+         steemit::chain::public_key_type scheduled_key = scheduled_witness.signing_key;
+
+         wlog( "scheduled key is: ${sk}   dbg key is: ${dk}", ("sk", scheduled_key)("dk", debug_public_key) );
+
+         if( scheduled_key != debug_public_key )
+         {
+            wlog( "Modified key for witness ${w}", ("w", scheduled_witness_name) );
+            fc::mutable_variant_object update;
+            update("_action", "update")("id", scheduled_witness.id)("signing_key", debug_public_key);
+            get_plugin()->debug_update( update );
+         }
+
+         db->generate_block( scheduled_time, scheduled_witness_name, *debug_private_key, steemit::chain::database::skip_nothing );
+         new_blocks++;
+      }
+   }
+   else
+   {
+      while( db->head_block_time() < head_block_time )
+         new_blocks += debug_generate_blocks( debug_key, 1 );
+   }
+
+   return new_blocks;
+}
+
 void debug_node_api_impl::debug_update_object( const fc::variant_object& update )
 {
    get_plugin()->debug_update( update );
@@ -159,6 +209,11 @@ uint32_t debug_node_api::debug_generate_blocks( std::string debug_key, uint32_t
    return my->debug_generate_blocks( debug_key, count );
 }
 
+uint32_t debug_node_api::debug_generate_blocks_until( std::string debug_key, fc::time_point_sec head_block_time, bool generate_sparsely )
+{
+   return my->debug_generate_blocks_until( debug_key, head_block_time, generate_sparsely );
+}
+
 void debug_node_api::debug_update_object( fc::variant_object update )
 {
    my->debug_update_object( update );

libraries/plugins/debug_node/include/steemit/plugins/debug_node/debug_node_api.hpp

@@ -34,6 +34,11 @@ class debug_node_api
        */
       uint32_t debug_generate_blocks( std::string debug_key, uint32_t count );
 
+      /*
+       * Generate blocks locally until a specified head block time. Can generate them sparsely.
+       */
+      uint32_t debug_generate_blocks_until( std::string debug_key, fc::time_point_sec head_block_time, bool generate_sparsely = true );
+
       /**
        * Directly manipulate database objects (will undo and re-apply last block with new changes post-applied).
        */
@@ -72,6 +77,7 @@ class debug_node_api
 FC_API(steemit::plugin::debug_node::debug_node_api,
        (debug_push_blocks)
        (debug_generate_blocks)
+       (debug_generate_blocks_until)
        (debug_update_object)
        (debug_stream_json_objects)
        (debug_stream_json_objects_flush)

python_scripts/setup.py

@@ -0,0 +1,12 @@
+from setuptools import setup
+
+setup( name='steemdebugnode',
+       version='0.1',
+       description='A wrapper for launching and interacting with a Steem Debug Node',
+       url='http://github.com/steemit/steem',
+       author='Steemit, Inc.',
+       author_email='[email protected]',
+       license='See LICENSE.md',
+       packages=['steemdebugnode'],
+       #install_requires=['steemapi'],
+       zip_safe=False )

python_scripts/steemdebugnode/__init__.py

@@ -0,0 +1,3 @@
+from steemdebugnode.debugnode import DebugNode
+
+__all__ = [ 'debugnode' ]