Welcome, Guest
Username Password: Remember me

Help with robots.txt and subdomain setup. Stop Google indexing!
(1 viewing) (1) Guest
  • Page:
  • 1

TOPIC: Help with robots.txt and subdomain setup. Stop Google indexing!

Help with robots.txt and subdomain setup. Stop Google indexing! 12 years, 8 months ago #4824

  • login418
  • OFFLINE
  • Fresh Boarder
  • Posts: 7
  • Karma: 0
I'm trying to setup mobile joomla on a sub domain. I don't want search engines to crawl any of the mobile content. I haven't gone live yet but I think google will still be able to crawl it when I do go live. I think a have a problem with my setup.

The last lines of the robots.txt file at the root of my site says:
Disallow: /iphone/
Disallow: /*&iphone.futuresound.tv

The robot.txt file in the iphone folder says:
User-agent: *
Disallow: /
User-agent: Googlebot
Noindex: /

My CPanel setup is:
Subdomains
(Subdomains.Root Domain) iphone.mywebsite.com
(Document Root) /public_html <-- is this wrong?
(Redirection) not redirected

When I check my robots.txt file in google webmaster tools, it shows the main sites robots.txt file, not the one in my /iphone folder. Please help!
Last Edit: 12 years, 8 months ago by login418.

Re: Help with robots.txt and subdomain setup. Stop Google indexing! 12 years, 8 months ago #4829

  • dryabov
  • OFFLINE
  • Administrator
  • Denis Ryabov, Lead Developer
  • Posts: 4866
  • Karma: 105
I'd use .htaccess file to have different robots.txt file for each of subdomain, smth like
RewriteEngine On
RewriteCond %{HTTP_HOST} ^pda\.yoursite\.com$ [NC]
RewriteRule ^robots\.txt$ pda.robots.txt [L]

(use of pda.robots.txt for pda.yoursite.com).
Last Edit: 12 years, 7 months ago by dryabov. Reason: format

Re: Help with robots.txt and subdomain setup. Stop Google indexing! 12 years, 8 months ago #4831

  • login418
  • OFFLINE
  • Fresh Boarder
  • Posts: 7
  • Karma: 0
Thanks for your help.
Can you explain in more detail? I'm new at this.
Add that code to the .htaccess file in root or /iphone or both?
Should I make a new file named pda.robots.txt at root? Or iphone.robots.txt?
The code says pda but I'm using iphone only so I'm confused.

Sorry for so many questions. I'm a real amateur at this.

Re: Help with robots.txt and subdomain setup. Stop Google indexing! 12 years, 7 months ago #4880

  • login418
  • OFFLINE
  • Fresh Boarder
  • Posts: 7
  • Karma: 0
I tried to add the rule you posted to my root .htaccess and I made a new pda.robots.txt file at root, but it brings down my site. Can you help me more please?

Re: Help with robots.txt and subdomain setup. Stop Google indexing! 12 years, 7 months ago #4881

  • dryabov
  • OFFLINE
  • Administrator
  • Denis Ryabov, Lead Developer
  • Posts: 4866
  • Karma: 105
Send me text of your .htaccess file.

PS. Just noted that this forum supports windows-style end-of-line code only, and two lines in the code above was merged into a single line. So, check that RewriteCond is started with new line in the .htaccess.

Re: Help with robots.txt and subdomain setup. Stop Google indexing! 12 years, 7 months ago #4899

  • login418
  • OFFLINE
  • Fresh Boarder
  • Posts: 7
  • Karma: 0
Thank you so much!

I sent you the htaccess in a PM. I understand now about the windows-style end-of-line code. I pasted the code again and now my main site is not broken. How do I check it? Google Webmaster Tools says I'm using robots.txt for iphone.mysite.com. It should see pda.robots.txt.

Re: Help with robots.txt and subdomain setup. Stop Google indexing! 12 years, 7 months ago #4901

  • dryabov
  • OFFLINE
  • Administrator
  • Denis Ryabov, Lead Developer
  • Posts: 4866
  • Karma: 105
login418 wrote:
I pasted the code again and now my main site is not broken. How do I check it?


To check just request the following files:
yoursite.com/robots.txt - should contain robots.txt for main domain
yoursite.com/pda.robots.txt - it's file that should be used on subdomain
pda.yoursite.com/robots.txt - here you should get content of pda.robots.txt

And it seems I missed leading slash in the rewrite rule, the correct code is:
RewriteCond %{HTTP_HOST} ^pda\.yoursite\.com$ [NC]
RewriteRule ^/robots\.txt$ /pda.robots.txt [L]

Re: Help with robots.txt and subdomain setup. Stop Google indexing! 12 years, 7 months ago #4980

  • login418
  • OFFLINE
  • Fresh Boarder
  • Posts: 7
  • Karma: 0
.htaccess
RewriteCond %{HTTP_HOST} ^iphone\.futuresound\.tv$ [NC]
RewriteRule ^/robots\.txt$ /pda.robots.txt [L]

iphone.futuresound.tv/robots.txt

 Sorry. If you click on the link above you can see that it is not rewriting to pda.robots.txt
btw mobilejoomla plugin is turned off for now. when turned on it still doesn't rewrite.
Last Edit: 12 years, 7 months ago by login418.

Re: Help with robots.txt and subdomain setup. Stop Google indexing! 12 years, 7 months ago #5028

  • dryabov
  • OFFLINE
  • Administrator
  • Denis Ryabov, Lead Developer
  • Posts: 4866
  • Karma: 105
Did you try to write last line "slashless"?
RewriteRule ^robots\.txt$ pda.robots.txt [L]
  • Page:
  • 1
Time to create page: 0.24 seconds

By continuing to use this site you consent to the use of cookies on your device as described in our cookie policy unless you have disabled them. This site will not function correctly without cookies.

I accept cookies from this site.