I have been messing about more with search engines and crawlers. If you take a look at http://www.robotii.co.uk/crawler/ and use the add url function there, it should crawl a bit of your site and report the results.
The results will be stored in the search database. I recommend using the second edition search at http://www.robotii.co.uk/search/ to check the results as it is a lot faster than the previous version.
I will integrate the code and release it as a download together with the database schema I’m using, just in case anyone’s interested – its written in PHP, so you’ll need a PHP enabled webserver and MySQL to run it, although it shouldn’t be too hard to port to Oracle etc… as long as you know what you’re doing.
That’s it for now…
If you enjoyed this post, make sure you subscribe to my RSS feed!
Related posts:
- Experimental Search Engine
- Search Engine
- A few of the more interesting search terms used to reach this site
- links for 2006-03-06
- links for 2006-03-21
