I was trying to store some tedious, repetitive configuration information in a giant map. I had a script generate the configuration information for me, as it was a lot of boilerplate. I ended up with a map that had something like 3,000 keys. I got the “Method code too large” error.
I looked this up and saw this Stackoverflow where someone suggested using a vector instead. (In that case the person asking the question was having a problem with defrecord, but they got the same error.)
I’ve two questions:
What is the maximum number of keys allowed on a map (saved at compile time as a var that never changes during runtime)?
How do I work around this? Should I use a vector? I was hoping for the fast lookup of a map, but I could loop over a vector instead. Any other suggestions?
The problem referenced in the SO response is for records, not arbitary maps. That IIRC is because the macroexpansion for defrecord ultimately expands into deftype, and emits a corresponding jvm class (in bytecode) from the clojure compiler. That is where you can run afoul of this stuff (jvm issue). If you have enough keys, the implementation of valAt (which dispatches on the known static fields internally) probably balloons (among other methods), and trips this sort of stuff.
Clojure hashmaps should not be impacted (by this at least). It would be nice to see a small demo of what is tripping you up.
Thank you for your response I am working on a game. I was hoping to build a large hash map that would have some quick lookup information for the types of units in this game. I wrote a script to generate the redundant data. I put this in its own namespace. The name space has a single var which contains the hash map. The information does not change during run time, it is created at compile time and does not change. The hash map has about 2100 keys in it. The var looks like this:
Looks like the issue is aot compilation, since I can comfortably create and load namespaces with maps that have 10s or 100’s of thousands of keys trivially.
If you don’t need to actually aot everything and can live with an uberjar with a main entry point that resolves stuff at runtime (doesn’t aot it’s dependencies) as the repl does, then a shim can enable this workflow.
There is another easy option I can see to enable aot as is out of the gate:
Generate the map (if all the entries are clojure data structures like keywords and sets and maps) as you did, dump it into a .edn file, and use clojure.edn/read-string to read it at runtime. This is slightly circuitous but should be general enough to handle arbitrary map sizes without a problem (you’re just loading “configuration” data at runtime). Store it as a resource in /resources, and use clojure.java.io/resource to address it (which makes it work when things are bundled in an uberjar later).
A third option could be to break up the big map into smaller maps (programmatically) and then merge those together. You would have to find out what size comes in under the 64kb method limit and then chunk it out that way. This seems lame to me.
A final option is to try to detect compile-files and then unbind it when we load the namespace we want to skip with the big map:
If you are generating this programmatically, the other option is to put that generating function into the logic that initializes the map (instead of the literal map). Then let that logic populate the var at runtime. That is yet another alternative, since there would be no data literals in the static class initialization to clog up the method. Although, you end up with something that isn’t editable in source form (if that matters).
I haven’t run it, but I suspect that the global def, when AOT compiled, gets turned into a static initializer in JVM bytecode. That’s what’s triggering the error.