# Does Clojure have Scrapy like framework for web scraping
**URL:** https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957
**Category:** General Questions
**Tags:** clojure
**Created:** [March 10, 2019, 11:14am UTC](https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957 "2019-03-10T11:14:54Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@stardiviner](https://clojureverse.org/u/stardiviner)
#### Post date: [March 10, 2019, 11:14am UTC](https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957/1 "2019-03-10T11:14:54Z")
Does Clojure or Java has Scrapy (which is written in Python) -like framework for web scraping. I’m currently learn web scraping, I already can write a basic simple crawler now, but I want it better and functional. I know Scrapy is great, But I want to do it under Clojure. So I ask this question.
If you have any suggestion, welcome to tell me. Thanks in advance.
I wish that framework is powerful, extensive for add new middleware or some kind of concepts. And easy to be distributed and parallel etc?
---
### Author:  [@dimovich](https://clojureverse.org/u/dimovich)
#### Post date: [March 10, 2019, 11:41am UTC](https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957/2 "2019-03-10T11:41:05Z")
I’ve been using JSoup (look [this post here](https://v2.clojureverse.org/t/best-library-for-querying-html/1103/18)) combined with [clj-http](https://github.com/dakrone/clj-http) and mixed with core.async + transducers, makes wonders 🙂
---
### Author:  [@borkdude](https://clojureverse.org/u/borkdude)
#### Post date: [March 10, 2019, 1:49pm UTC](https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957/3 "2019-03-10T13:49:24Z")
I’ve been using the same, except for clj-http + core.async I have been using aleph + manifold.
---
### Author:  [@stardiviner](https://clojureverse.org/u/stardiviner)
#### Post date: [March 10, 2019, 3:05pm UTC](https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957/4 "2019-03-10T15:05:19Z")
Thanks, I use `clj-http` in my previous simple crawler too. I try to use `core.async` in my code, but good at Clojure yet. I have checked out `manifold`, it great and simple. I like this solution `aleph + manifold`. After I get better on Clojure, I will check out `core.async + transducers`. Thanks you both.
What if I want a middle scalable and distributed solution, what do you suggest? Redis? MongoDB? or others? What about Clojure side support?
I checked out Java side project, found Nutch, but I don’t know Java and found Nutch is hard for me to start after I checked out it’s wiki and documents. Does anybody knows how to use Clojure on Nutch? maybe a simple example is great. Also the installation is hard to me. I don’t know which way to install it, Maven? Docker? or others.
---
### Author:  [@didibus](https://clojureverse.org/u/didibus)
#### Post date: [March 11, 2019, 1:37am UTC](https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957/5 "2019-03-11T01:37:39Z")
There’s also crawler4j and Heritrix3 (used by [internetarchive.org](http://internetarchive.org)). But JSOUP is very good as well.
---
### Author:  [@stardiviner](https://clojureverse.org/u/stardiviner)
#### Post date: [March 11, 2019, 5:10am UTC](https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957/6 "2019-03-11T05:10:00Z")
Thanks didibus.
After a closer view of Nutch, Heritrix3, I found both of this are mainly web indexer which mainly used for scraping web pages then used by search engine, of coursed can be used for crawler too. But not very good for web crawler.
---
### Author:  [@didibus](https://clojureverse.org/u/didibus)
#### Post date: [March 11, 2019, 5:43am UTC](https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957/7 "2019-03-11T05:43:24Z")
Hum, can you be more specific what you mean then? I’m confused about the difference you mention between scraping and crawling?
What’s your end goal/use case?
---
### Author:  [@stardiviner](https://clojureverse.org/u/stardiviner)
#### Post date: [March 11, 2019, 9:49am UTC](https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957/8 "2019-03-11T09:49:17Z")
Sorry for my confusing words.
My end goal is find a Clojure/Java library like Python’s Scrapy.
Scraping and crawling, I think they are almost same. The really difference is `web indexer` and `web crawler`.
> After a closer view of Nutch, Heritrix3, I found both of this are mainly web indexer which mainly used for scraping web pages then used by search engine, of coursed can be used for crawler too. But not very good for web crawler.
You can check out Nutch document, it use scraped web pages for search engine like Apache Solr. I don’t need this. I want to save parsed data on web pages to database, and do some data science.
Maybe I can use part of Nutch. I will take a closer look into Nutch about how it works.
---
### Author:  [@madbonkey](https://clojureverse.org/u/madbonkey)
#### Post date: [March 11, 2019, 2:01pm UTC](https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957/9 "2019-03-11T14:01:13Z")
Depends on what you‘re using Redis/MongoDB for, of course, but generally my experience with both of those at medium scale has been great. I‘ve never even needed to shard Mongo, but it’s great to have the option. If you use it as intended, it scales very well, same as Redis (I have no experience whatsoever running Redis in distributed way). I don‘t know if either are good tools if you just want a place to store HTML content. Parsing that and putting it in to Mongo might give you a lot of query power (over Redis), but if you‘re more interested in the linked nature of the web, maybe a graph database would suit your need (ansami/neo4j/daromic/…).
Also, I imagine there‘s loads of Java libraries for this kind of stuff, and many of those have probably considered lots of edge cases. I find an hour of trying out some Java libraries almost always pays off ✌
---
### Author:  [@didibus](https://clojureverse.org/u/didibus)
#### Post date: [March 11, 2019, 3:56pm UTC](https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957/10 "2019-03-11T15:56:56Z")
Ah I see.
Dunno how good it is, but I found this: [https://github.com/maithilish/scoopi](https://github.com/maithilish/scoopi)
And it looks right up your alley.
---
### Author:  [@stardiviner](https://clojureverse.org/u/stardiviner)
#### Post date: [March 13, 2019, 6:03am UTC](https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957/11 "2019-03-13T06:03:28Z")
I agree with that. MongoDB and Redis is enough for medium distributed scale level. I will try MongoDB out. Thanks.
---
### Author:  [@stardiviner](https://clojureverse.org/u/stardiviner)
#### Post date: [March 13, 2019, 6:08am UTC](https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957/12 "2019-03-13T06:08:00Z")
@didibus, Thanks. This one looks really interesting. I considered Dockerize crawer too. One thing I have no idea how to develop Clojure on this Scoopi Java library in Docker. If I solve this problem, I can setup similar environment for my development environment. I personally use Emacs with CIDER to write Clojure code. Emacs have `docker.el` and `docker-tramp.el` support for working with Docker container filesystem. I will try to Google “setup Clojure development in Docker container”.
---
### Author:  [@jackrusher](https://clojureverse.org/u/jackrusher)
#### Post date: [March 13, 2019, 3:29pm UTC](https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957/13 "2019-03-13T15:29:14Z")
In case you need to deal with pages that require a JS runtime to render, there’s also my library [Sparkledriver](https://github.com/jackrusher/sparkledriver).
---
### Author:  [@system](https://clojureverse.org/u/system)
#### Post date: [September 12, 2019, 3:29am UTC](https://clojureverse.org/t/does-clojure-have-scrapy-like-framework-for-web-scraping/3957/14 "2019-09-12T03:29:19Z")
This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.