I was looking for a easy and simple way to crawl multiple web-sites and build a search catalog.
After a few google searches I found sphider I really love it and it does exactly what I was expecting.
You define the URL’s you wish to index, set the level of how deep you wish to follow the links and it gives you a super fast and powerful search engine.
Check it out:
