(defn memoize-transform
"Returns a function that memoizes the last result.
If the arguments are the same as the last call,
the memoized result is returned."
[f]
(let [mem (atom {:last-x nil
:last-y nil
:last-result nil})]
(fn [& args]
(if (= (first args) (:last-x @mem))
(= (last args) (:last-y @mem))
(:last-result @mem)))))

but now I wonder what is a good clojure way to update the atom with the new values if the values is not in the atom ?

(defn memoize-transform
"Returns a function that memoizes the last result.
If the arguments are the same as the last call,
the memoized result is returned."
[f]
(let [mem (atom {:last-x nil
:last-y nil
:last-result nil})]
(fn [& args]
(if (and (= (first args) (:last-x @mem))
(= (last args) (:last-y @mem)))
(:last-result @mem)
(:last-result (reset! mem {:last-x ... :last-y ... :last-result ...})))))

(Side note: Do the function always takes two arguments, x and y? You could store the args as one variable instead, and compare the argument vector with the saved one). Or use destructuring:

(defn memoize-transform
"Returns a function that memoizes the last result.
If the arguments are the same as the last call,
the memoized result is returned."
[f]
(let [mem (atom {:last-x nil
:last-y nil
:last-result nil})]
(fn [[x y]]
(if (and (= x (:last-x @mem))
(= y (:last-y @mem)))
(:last-result @mem)
(:last-result (reset! mem {:last-x x :last-y y :last-result (f x y)})))))

Also note that (f nil nil) will give nil initially because of the initiatization-values of the atom, even if f returns something else.

(defn memoize-transform
"Returns a function that memoizes the last result.
If the arguments are the same as the last call,
the memoized result is returned."
[f]
(let [mem (atom {:last-x ::sentinel ;; will probably not be the argument to f
:last-y ::sentinel
:last-result nil})]
(fn [[x y]]
(if (and (= x (:last-x @mem))
(= y (:last-y @mem)))
(:last-result @mem)
(:last-result (reset! mem {:last-x x :last-y y :last-result (f x y)})))))

A more general impl would be

(defn memoize-transform
"Returns a function that memoizes the last result.
If the arguments are the same as the last call,
the memoized result is returned."
[f]
(let [mem (atom {})]
(fn [& args]
(if (= args (:last-args @mem ::sentinell)) ;; default value cannot match args
(:last-result @mem)
(:last-result (reset! mem {:last-args args :last-result (apply f args)})))))