Help Needed with Optimizing Clojure Code for High-Performance Data Processing

[BrunoBonacci]

Hi @camilaa,

I would say that the first step is to identify where the bottleneck is. If you think you have found that writing the data to the database is the bottleneck, then that’s already a good step forward.

If you are unsure, please have a look at the comment to this post:

How to Optimize Performance in a Clojure Web Application

Hi, I agree with p-himik comment that it is crucial to identify in production which area of your code is slower and in possibly which inputs make it slower. I’ve highlighted “in production” because the long tail could be different in your offline test bench. To better analyse this sort of issues I’ve developed a library called µ/log which can be used to do structured logging. In particular there is a macro called µ/trace which provides tracing of key parts in your code. The key difference be…

Many of the same suggestions will apply here.

If you are sure that the bottleneck is writing the data to the database then there are a couple of things you could double check:

1. are you using a connection pool for your database connections?
   Opening a connection per query could be rather expensive.
   If you are not using a connection pool, try one of the following:

• GitHub - tomekw/hikari-cp: A Clojure wrapper to HikariCP JDBC connection pool
• GitHub - metabase/connection-pool: Connection pools for JDBC databases. Simple wrapper around C3P0.

2. Do you batch write your data into Postgres?
   Writing data can be done in large batches of thousands of records with a single statement. This is way more efficient than using individual INSERT INTO statements for each record. check PostgreSQL: Documentation: 9.4: INSERT

here is an example of how to bulk insert records

INSERT INTO films (code, title, did, date_prod, kind) VALUES
    ('B6717', 'Tampopo', 110, '1985-02-10', 'Comedy'),
    ('UA502', 'Bananas', 105, '1971-07-13', 'Comedy'),
    ('HG120', 'The Dinner Game', 140, '1961-06-16', 'Comedy');

I hope it helps.
Bruno