Upstream NBT region file support

[pepijndevos]

The guy just committed support for region files. We should try to use the upstream support rather than our own hacking.

https://github.com/twoolie/NBT

[Fenixin]

Good idea. Less work for everyone.

[pepijndevos]

I will have a throw at this.

[pepijndevos]

Waiting for an update on NBT, does not work as-is.

[Fenixin]

maybe we can file a issue in twoolie/NBT

[pepijndevos]

Done: twoolie/NBT#7

[Fenixin]

This is probably the next thing I'll do. The twoolie/NBT is a bit abandoned, but the fork MidnightLightning/NBT is pretty active, and works perfect. I'll use it and see how it works.

[Fenixin]

The MidnightLighting fork has been merged into twoolie, so we can use it right now. Also the pypi install of the nbt module has been updated. Should I add the nbt module code to the repo or add it to dependencies and easy_install will install it without problems? I guess the right thing to do is leave this thing to easy_install. If everybody is ok with this I'll use the easy_install method.

[macfreek]

I had a look at NBT, and found it to be significantly slower then mian.

Perhaps it would be a good idea would first profile NBT before relying on it:
cd NBT/examples ; PYTHONPATH=.. python -m cProfile block_analysis.py <World Folder>

[Fenixin]

@macfreek, you are doing a nice job with NBT and I'm curious. How is mian performing with the new NBT? Any info? Any news?

Also, I've you added you to the credits list in mian! I forgot to do that after the pull request.

[macfreek]

NBT was very slow with block analysis (took 80 seconds for a just generated very small world). The reason was that it looped through all x,y,z coordinates in a specific order. That is very slow, and unnecessary if you just want to have the total block count. Mian was much much faster (2 seconds only!) for two reasons: it did count in order they happen to be stored in memory, and secondly, it could use the string.count() function since it only counted specific blocks. If mian would count all block ids, it took about 10 seconds, roughly the same time as it takes NBT.

Adding data types is possible (NBT had some bugs in that code which I just fixed), but that makes it much slower: 30 second again. Now I just calculate the totals. If I where to do it level-by-level (like mian requires) it will probably takes some more time again.

So clearly there is a trade-off in speed. (1) examining data (e.g. wool color) along with block ID; (2) detecting all block id instead of just a few sample types (likely will no longer gives a speed-up if data is examined too), (3) calculate totals per chunk or per level.

It is certainly possible to let mian rely on NBT, but expect it to be slower, even with the speed-up fixes I made to NBT. Roughly 10 times slower then now. Perhaps this can be improved by making the app multi-threaded (or actually: using subprocesses, since Pyhton can't cope with multiple threads due to something called the Global Interpreter Lock)

Another option is to add the features of mian to NBT as example scripts: creating of graphs and knowing human-readable names. Actually, I'm a bit inclined to do that for a simple reason: having one code rely on the other means version management and having to install dependencies, but I'm fine doing what you think is best.

The other thing I haven't decided upon is what to do with data (sub block types). Certainly possible to support, but again, it makes things slower, and requires near complete rewrite of the block naming library blocks.py. Speed-up may be possible with special cases (no need to check subtype for air and stone - 97.5% of the blocks) or other tricks. I don't know.

Actually work got in the way, so I did not look into this last week. Three questions:

1. Is there an official list of names for all blocks? I did find some in the minecraft code (minecraft.jar/lang/en_US.lang), but that did not contain all block IDs.
2. Is it useful to add mian's functionality as example to NBT or do you prefer to keep them separate and rely on NBT?
3. Do you think it is worth-while to make mian and/or NBT compatible with Python 2 and 3, even though that requires a mayor rewrite of the two, and eliminated Python 2.5 support. (I think the later is no issue, since all OSes support Python 2.6 or later, even Debian, Red Hat and Mac 10.6).

[Fenixin]

Answering your questions:

1.I always use:

http://www.minecraftwiki.net/wiki/Data_values

If follows pretty well (if not perfectly) the official names.

2.I'd like to keep mian development separated, but it could be also added to the NBT as an example. The NBT version could be left static and the mian could be developed forward.

1. I'd say no to this. This is probably going to be time consuming and I think there are other good things to do.

About relaying on nbt and the data block check... I'm not sure either. I guess that if we want a feature complete mian has to check block data, and the best way to do it is through NBT.

Please, feel free to discuss any of the points.