Contact Us

Welcome to ineedhits Blog

Welcome to the ineedhits Search Engine Marketing blog, where we share the latest search engine and online marketing news, releases, industry trends and great DIY tips and advice.

Sunday, April 13, 2008

Google Spider Searches for “Invisible Web”

Posted by @ 10:27 pm

On Friday, the Google webmaster central blog announced that the Google search spider will now attempt to fill out web forms to uncover new links and content. Google claims that the technology will help to uncover the “invisible web” that’s been hiding from search engines for many years.

The invisible web is a collection of webpage’s that have traditionally been hard or impossible for search engines to index. This can be for several reasons including pages that aren’t linked from other pages, sites that require a registration/login and content that is generated dynamically. Because these pages are either unlinked, or require some sort of form action to access, spiders such as the Googlebot have been unable to retrieve them.

Now, Google has introduced a new way to find these pages by completing web forms on high quality websites. From the Google blog post:

In the past few months we have been exploring some HTML forms to try to discover new web pages and URLs that we otherwise couldn’t find and index for users who search on Google. Specifically, when we encounter a <form> element on a high-quality site, we might choose to do a small number of queries using the form…. If we ascertain that the web page resulting from our query is valid, interesting, and includes content not in our index, we may include it in our index much as we would include any other web page.

Google claims the technology should help to increase exposure for sites that use <form> elements on their webpages. Popular Google personality Matt Cutts, has also given a great rundown of how this technology can improve web coverage for businesses with many country specific websites.

If you’ve experienced some unusual crawl patterns on your site, let us know if you’ve got any new pages indexed. If not, G-Boost is a great way to get these tricky pages indexed by Google in only 7 days and comes with a full money-back guarantee.

Matthew Elshaw Matt is a marketing professional at, an international search marketing firm. Matt's passion for online marketing began at university and has proved invaluable in steering product development and marketing initiatives at the company. Matt is a regular contributor to the ineedhits search marketing blog.

View Matthew Elshaw's profile

Discussion (Not Started)

Add Your Comments


Keep up to date with the latest from our blogs.

Subscribe to all blog posts

The Newsletter

  • New Posts
  • Popular
  • Comments


More in Small Business News (91 of 129 articles)

Yesterday, Google made a proposal to the Federal Communications Commission to free up unused TV broadcast channels known as ...