Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OsmAndMapCreator IndexOutOfBoundsException for big extracts #687

Open
feludwig opened this issue Oct 10, 2023 · 7 comments
Open

OsmAndMapCreator IndexOutOfBoundsException for big extracts #687

feludwig opened this issue Oct 10, 2023 · 7 comments

Comments

@feludwig
Copy link

Hello!
I have been trying to create .obf maps of .osm.bz2 extracts, and OsmAndMapCreator generate-obf works well for the following:
Switzerland.osm.bz2 (osm.bz2 692MB, obf 783MB)
But it crashes while writing route index when the input file is bigger than about 1GB (I am giving it -Xmx 25G of memory to use) on the following:
(germany) baden-wuerttemberg-latest.osm.bz2 (osm.bz2 854MB, unfinished obf 696MB)
(germany) bayern-latest.osm.bz2 (osm.bz2 1.2G, unfinished obf 889MB)
Here is the end of the log for baden-wuerttemberg:

INFO: Done 98.965 %.                                                                                                                                                         
Oct 07, 2023 6:15:34 PM net.osmand.impl.ConsoleProgressImplementation print                                                                                                  
INFO: Task index_lo_level_ways: work total has changed to -1                                                                                                                 
Oct 07, 2023 6:15:34 PM net.osmand.impl.ConsoleProgressImplementation print                                                                                                  
INFO: pack_rtree_map started after 2071665 ms - -1                                                                                                                           
Pack.packTree : Size of elmts: 10458811                                               
Pack.packTree : Size of elmts: 3171796                                                
Pack.packTree : Size of elmts: 1758434
Pack.packTree : Size of elmts: 168685                                                                                                                                        
Pack.packTree : Size of elmts: 2345214                                                                                                                                       
Pack.packTree : Size of elmts: 184318                                                                                                                                        
Oct 07, 2023 6:18:17 PM net.osmand.impl.ConsoleProgressImplementation print                                                                                                  
INFO: pack_rtree_transp started after 162578 ms - -1               
Pack.packTree : Size of elmts: 36359                                                  
Oct 07, 2023 6:18:18 PM net.osmand.impl.ConsoleProgressImplementation print                                                                                                  
INFO: Writing map index to binary file... started after 666 ms - -1                                                                                                          
Oct 07, 2023 6:55:14 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapLevelIndex                        
INFO: MAP level SIZE : 469256321                                                                                                                                             
Oct 07, 2023 7:14:33 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapLevelIndex           
INFO: MAP level SIZE : 157953833                                                                                                                                             
Oct 07, 2023 7:26:33 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapLevelIndex           
INFO: MAP level SIZE : 87773842                                                                                                                                              
Oct 07, 2023 7:28:29 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapLevelIndex         
INFO: MAP level SIZE : 8657172                                                                                                                                               
Oct 07, 2023 7:28:29 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapIndex 
INFO: MAP INDEX SIZE : 723805292                                                                                                                                             
Oct 07, 2023 7:28:29 PM net.osmand.impl.ConsoleProgressImplementation print
INFO: Writing route index to binary file... started after 4211731 ms - -1
Oct 07, 2023 7:28:48 PM net.osmand.obf.preparation.IndexCreator generateIndexes
SEVERE: Log exception
java.lang.IndexOutOfBoundsException: Index -26077 out of bounds for length 53395
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
        at java.base/java.util.Objects.checkIndex(Objects.java:372)
        at java.base/java.util.ArrayList.get(ArrayList.java:459)
        at net.osmand.osm.MapRoutingTypes.getTypeByInternalId(MapRoutingTypes.java:335)
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.decodeNames(IndexRouteCreator.java:1337)
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.retrieveObject(IndexRouteCreator.java:1382)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1532)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndexBlocks(IndexRouteCreator.java:1498)
        at net.osmand.obf.preparation.IndexRouteCreator.writeRouteSections(IndexRouteCreator.java:743)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndex(IndexRouteCreator.java:704)
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:631)
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:484)
        at net.osmand.MainUtilities.generateObf(MainUtilities.java:297)
        at net.osmand.MainUtilities.main(MainUtilities.java:127)

Exception in thread "main" java.lang.IndexOutOfBoundsException: Index -26077 out of bounds for length 53395
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
        at java.base/java.util.Objects.checkIndex(Objects.java:372)
        at java.base/java.util.ArrayList.get(ArrayList.java:459)
        at net.osmand.osm.MapRoutingTypes.getTypeByInternalId(MapRoutingTypes.java:335)
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.decodeNames(IndexRouteCreator.java:1337)
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.retrieveObject(IndexRouteCreator.java:1382)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1532)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndexBlocks(IndexRouteCreator.java:1498)
        at net.osmand.obf.preparation.IndexRouteCreator.writeRouteSections(IndexRouteCreator.java:743)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndex(IndexRouteCreator.java:704)
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:631)
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:484)
        at net.osmand.MainUtilities.generateObf(MainUtilities.java:297)
        at net.osmand.MainUtilities.main(MainUtilities.java:127)

(IndexRouteCreator.java:1594 that line does appear 3 consecutive times)
The two extracts from Germany are geofabrik extracts, because I was suspecting that the Bayern input I had was corrupted (it had crashed, at the same point). I have just split Bayern along 49 latitude and will start converting the two halves separately and report how it went.
These extracts take more than 10h to convert so it was a bit of an adventure.
I'm running the latest version from github:

$ git log |head
commit e5b6027c2485a39adf5aadcf2761d2f0200906a4
Author: ivanPyrohivskyi <[email protected]>
Date:   Wed Oct 4 14:24:28 2023 +0300

Do you also want logs for other and/or successful extracts ? Or the complete log ?
Should I run the conversion, or compile MapCreator, again with some specific flags ?
@feludwig
Copy link
Author

The Bayern split accross 49th parallel was successful and the two files are
Bayern-above49.obf 625MB
Bayern-below49.obf 751MB
So the isssue is with input size...

@vshcherb
Copy link
Member

Did you try pbf as input file cause it might work better as it tested more

@feludwig
Copy link
Author

No, but I'll go try that as well

@feludwig
Copy link
Author

Yes, it also crashes:
input: bayern-latest.osm.pbf from geofabrik size: 712M
incomplete output: Bayern-latest.obf size: 890M

Oct 16, 2023 5:26:23 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapLevelIndex                                                                                
INFO: MAP level SIZE : 12645757                                                                                                                                              
Oct 16, 2023 5:26:23 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapIndex                                                                                     
INFO: MAP INDEX SIZE : 927596053                                                                                                                                             
INFO: Writing route index to binary file... started after 5174106 ms - -1                                                                                                    
Oct 16, 2023 5:26:41 PM net.osmand.obf.preparation.IndexCreator generateIndexes                                                                                              
SEVERE: Log exception                                                                                                                                                        
java.lang.IndexOutOfBoundsException: Index -28761 out of bounds for length 58308                                                                                             
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)                                                                                      
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)                                                                            
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)                                                                                      
        at java.base/java.util.Objects.checkIndex(Objects.java:372)                                                                                                          
        at java.base/java.util.ArrayList.get(ArrayList.java:459)                                                                                                             
        at net.osmand.osm.MapRoutingTypes.getTypeByInternalId(MapRoutingTypes.java:335)                                                                                      
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.decodeNames(IndexRouteCreator.java:1337)
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.retrieveObject(IndexRouteCreator.java:1382)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1532)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndexBlocks(IndexRouteCreator.java:1498)
        at net.osmand.obf.preparation.IndexRouteCreator.writeRouteSections(IndexRouteCreator.java:743)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndex(IndexRouteCreator.java:704)
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:631)                                                                                    
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:484)                                                                                    
        at net.osmand.MainUtilities.generateObf(MainUtilities.java:297)                                                                                                      
        at net.osmand.MainUtilities.main(MainUtilities.java:127)                                                                                                             

Exception in thread "main" java.lang.IndexOutOfBoundsException: Index -28761 out of bounds for length 58308
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
        at java.base/java.util.Objects.checkIndex(Objects.java:372)
        at java.base/java.util.ArrayList.get(ArrayList.java:459)
        at net.osmand.osm.MapRoutingTypes.getTypeByInternalId(MapRoutingTypes.java:335)
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.decodeNames(IndexRouteCreator.java:1337)
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.retrieveObject(IndexRouteCreator.java:1382)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1532)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndexBlocks(IndexRouteCreator.java:1498)
        at net.osmand.obf.preparation.IndexRouteCreator.writeRouteSections(IndexRouteCreator.java:743)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndex(IndexRouteCreator.java:704)
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:631)
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:484)
        at net.osmand.MainUtilities.generateObf(MainUtilities.java:297)
        at net.osmand.MainUtilities.main(MainUtilities.java:127)

@vshcherb
Copy link
Member

We don't build bayern separately cause it might be too big, we build as separate regions but the crash is strange. It happens on the last stage...

@feludwig
Copy link
Author

Oh I see, so the workaround is something like

  • osmium extract multiple smaller parts
  • foreach: MapCreator generate-obf
  • MapCreator merge-index --address --poi of all the small parts to get one .obf at the end

@vshcherb
Copy link
Member

Yes that's what we do actually... processing large file is just takes too much memory and we even can run in parallel using 1 mapcreator batch utility and local docker

@pnoll1 pnoll1 mentioned this issue Mar 24, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vshcherb @feludwig