Blog post: Converting CSV to parquet in Clojure

[Harold]

Following some conversations we had at the conj we wrote a short article about using Clojure to convert CSV into parquet files.

Interest is certainly building around parquet as a file format, and we think it’s good tech. This is potentially exciting if you work with tabular/columnar data and want it to load fast.

TechAscent - Converting CSV to parquet in Clojure

[Harleqin]

Did you wrap the Java thing, or did you implement it from scratch?

Is the source code published?

[Harold]

Great questions!

That article is about a new single-jar deps-only lib that simplifies doing the work described in the article:

• On Maven: https://mvnrepository.com/artifact/com.techascent/tmd-parquet
• On Github: GitHub - techascent/tech.parquet: Simple parquet bindings for tech.ml.dataset.
• The deps you would otherwise need to include yourself (but you don’t because of this new library): tech.parquet/deps.edn at master · techascent/tech.parquet · GitHub

The heavy lifting is done by our library for tabular data processing tech.ml.dataset (TMD)

• On Github: GitHub - techascent/tech.ml.dataset: A Clojure high performance data processing system
• And the code that does the parquet: tech.ml.dataset/parquet.clj at master · techascent/tech.ml.dataset · GitHub
• Which does interact with the Java wrapper

hth - keep the good questions coming!

[Harleqin]

Thanks!

That looks like a lot of work that went into it.

[Harold]

You’re welcome.

And indeed, this is just a small moment in a much larger undertaking of trying to understand “functional data science”.

Thanks for looking into it.