How to access hidden links of websites!

Hey guys!


There are some websites who want to hide things from the public but want to use them for private use (Mind you I haven't tried this method on US websites and I advise you not to. You can end up in jail.). They do so by making a file called "robots.txt" that tells search engines to hide some parts of their website from being indexed.

Now, one problem is that the robots.txt file is available to the public.

To find out the contents of any website's robots.txt file, add "/robots.txt" (without the quotes) to the link.

My blog has a robots.txt file like this:

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://technohackerblog.blogspot.com/feeds/posts/default? 

orderby=UPDATED

Basically, User-agent means the person (or program) that is accessing the site (* means everyone).

Here's the part that's interesting - Disallow means the following part of the website should not be shown in search results while Allow is the opposite.

To access those links, copy the text including the forward-slash (eg, /search) and put it on the URL (eg, http://technohackerblog.blogspot.com/search) (I tried this already, lust leads to the homepage.)

Type in the comments below on what you have found!
Thanks for reading!
-Technohacker