Hi everyone! I have difficulty wrapping my head around this issue: I think I’m starting to understand transducers, but when I try to use one in a pipeline I’m developing I get a weird result concerning performance. I have x JSON files in a folder - a total of about 50 mb, growing with time - I wan…

Lazy sequences. You never realize any of your values in the first bench. Also: Better use (pmap (comp get-all-maps parse-with-city)) Avoid the apply concat and move it to the transducer cat

You’re right about the lazy sequences, sorry I was in a hurry, now doing: (quick-bench (->> (read-all-files dir) (map #(join-strs :Description " + " %)) (map #(apply dissoc % bad-fields)) (map create-id) doall)) => Execution time mean : 603.928660 ms (quick-bench …

After your advice the sequence actually got worse, but the transducer stayed more or less the same Result with sequence -> Execution time mean : 807.861160 ms Result with transducer -> Execution time mean : 571.074660 ms I think I’m going to get rid of concurrency for benchmarking, because I’m st…

Without knowing what your data looks like it’s tough to know how to speed this up. E.g. the apply isn’t too fast, but it could also be your join-strs function or create-id.You could likely use one pmap and do everything you need to do in that one function? Just a guess…

I have about 110 JSON files with different sizes and they contain structures like this one below (sorry about the Italian, but it doesn’t make much difference): {:Vani ["2,5"], :Genere ["Immobiliare"], :Tipologia ["Appartamento"], :Nome "Appartamento Asta giudiziaria Via II Cortina S. Anna, 7…

Is pmap actually reading the files? It might not be giving you much parallelism for IO. It’s mostly meant for computation. You can try increasing the number of pmap threads (somewhere in the Clojure docs). Or use your own executor.

I will try that thanks! Anyway I had an issue with the running JVM instance, and after some profiling it became clear the overhead coming from the reading function is too high to include it in the evaluation. I was wondering wether iota could help in this case, but I guess I’ll know only by trying…

ClojureVerse

Help with transducers performance

Questions & Help General Questions

clojure

show post in topic

Home
Categories
Guidelines
Terms of Service
Privacy Policy

Powered by Discourse, best viewed with JavaScript enabled