# Clojure ML library - advise on model evaluation

**URL:** https://clojureverse.org/t/clojure-ml-library-advise-on-model-evaluation/8211
**Category:** Machine Learning & AI
**Created:** [September 27, 2021, 2:39pm UTC](https://clojureverse.org/t/clojure-ml-library-advise-on-model-evaluation/8211 "2021-09-27T14:39:14Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Carsten\_Behring](https://clojureverse.org/user_avatar/clojureverse.org/carsten_behring/32/2624_2.png) [@Carsten\_Behring](https://clojureverse.org/u/Carsten_Behring)
#### Post date: [September 27, 2021, 2:39pm UTC](https://clojureverse.org/t/clojure-ml-library-advise-on-model-evaluation/8211/1 "2021-09-27T14:39:15Z")

</div>

I am working on a new Clojure library for machine learning (‘scicloj.ml’), based on a lot of existing libraries (tech.ml, tech.ml.dataset, Smile)

One of the few areas where complete new code is needed, is “model evaluation” including cross-validation (eventually nested) in all its variations.

I would like to discuss some aspect of this with anybody interested and knowledgeable in the subject.

It is not easy to find inspirations in this, as the design of the model evaluation is purely functional (so very different then in Python or R)

There is some code to see the current this in action here: [https://scicloj.github.io/scicloj.ml-tutorials/tune-titanic.html](https://scicloj.github.io/scicloj.ml-tutorials/tune-titanic.html)

Further explanations and tutorials are here:

> **[GitHub - scicloj/scicloj.ml: Proposal for a Clojure machine learning library.](https://github.com/scicloj/scicloj.ml)**
>
> Proposal for a Clojure machine learning library. Contribute to scicloj/scicloj.ml development by creating an account on GitHub.

I am welcoming any feedback

---

<div class="post-metadata">

### Author: ![Carsten\_Behring](https://clojureverse.org/user_avatar/clojureverse.org/carsten_behring/32/2624_2.png) [@Carsten\_Behring](https://clojureverse.org/u/Carsten_Behring)
#### Post date: [September 27, 2021, 3:01pm UTC](https://clojureverse.org/t/clojure-ml-library-advise-on-model-evaluation/8211/2 "2021-09-27T15:01:56Z")

</div>

The key question for me is, if the pseudo code below implements so called nested cross validation:

```auto
ps: 2 hyperparameter configs
folds : 6 folds

-> 12 model evaluations

for p in ps:
   for fold in folds :
       model = train (fold.train-data, p)
       prediction = predict(model,fold.test-data)
       metric = calc-metric(prediction, fold.test-data)
   metric-of-p = mean (metrics of all folds)

best-model = p which has best metric-of-p

```

---

<div class="post-metadata">

### Author: ![andycraig](https://clojureverse.org/letter_avatar_proxy/v4/letter/a/ecc23a/32.png) [@andycraig](https://clojureverse.org/u/andycraig)
#### Post date: [October 5, 2021, 10:32am UTC](https://clojureverse.org/t/clojure-ml-library-advise-on-model-evaluation/8211/3 "2021-10-05T10:32:15Z")

</div>

Hi, Python ML libraries like scikit-learn take an OOP approach but R is quite a functional language so there might be some resources there that you can reference. R’s ‘tidymodels’ framework has the function `nested_cv()` for nested cross-validation. It’s implemented as a map of the inner CV function over the outer CV splits:

> <https://github.com/tidymodels/rsample/blob/7fcda23740a9f27b14b46c37d4df909877968b18/R/nest.R#L78>

---

<div class="post-metadata">

### Author: ![Carsten\_Behring](https://clojureverse.org/user_avatar/clojureverse.org/carsten_behring/32/2624_2.png) [@Carsten\_Behring](https://clojureverse.org/u/Carsten_Behring)
#### Post date: [October 5, 2021, 10:12pm UTC](https://clojureverse.org/t/clojure-ml-library-advise-on-model-evaluation/8211/4 "2021-10-05T22:12:00Z")

</div>

Thanks for the link.

I implemented one form of nested\_cv here:

> <https://github.com/scicloj/scicloj.ml-tutorials/blob/train-test-result-change/src/scicloj/ml/nested_cv.clj>

It implements one form of nested cv.  
It does ot do model selection, bu only estimates the accuracy

---

<div class="post-metadata">

### Author: ![system](https://clojureverse.org/uploads/default/original/2X/5/51079bf9e4b7d9466242c06cf1e43b9f8bd6da14.png) [@system](https://clojureverse.org/u/system)
#### Post date: [April 6, 2022, 10:12am UTC](https://clojureverse.org/t/clojure-ml-library-advise-on-model-evaluation/8211/5 "2022-04-06T10:12:56Z")

</div>

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.