News:

Welcome to the Unofficial Help Forum



Author Topic: Robots.txt needed  (Read 9883 times)

Offline shubh

  • Jr. Member
  • *
  • Posts: 52
    • View Profile
Robots.txt needed
« on: April 21, 2012, 04:09:06 AM »
Please check this is right or not
Quote
User-agent: Googlebot-Image
Disallow: /

User-agent: YandexImages
Disallow: /

User-agent: msnbot-MM
Disallow: /

User-agent: Googlebot-Mobile
Allow: /*wap
Disallow: /

User-agent: YandexImageResizer
Allow: /*wap
Disallow: /

User-agent: MediaPartners-Google
Allow: /

User-agent: *
Allow: /$
Allow: /*type=rss
Disallow: /*action
Disallow: /*wap
Disallow: /*.msg
Disallow: /*.new
Disallow: /*;
Disallow: /*PHPSESSID
Disallow:/credits
Disallow:/PHPSESSID
Disallow:/?PHPSESSID
Allow: /*board
Allow: /*topic
Allow: /
Disallow:/?sort=views
Disallow:/?sort=subject
Disallow:/?sort=starter
Disallow:/?sort=last_post
Disallow:/?sort=subject
Disallow:/?prev_next=next
Disallow:/?prev_next=prev
Disallow:/msg
Disallow:/reminder
Disallow:/?start=
Crawl-delay: 5
Clean-param: PHPSESSID /index.php
Host: www.yourdomain.com
No words left

Offline Illori

  • Administrator
  • *
  • Posts: 324
    • View Profile
Re: Robots.txt needed
« Reply #1 on: April 21, 2012, 06:46:03 AM »
are you trying to block the bots? if so you will block some with that. you should read www.robotstxt.org/ for more information about robots.txt.

Offline shubh

  • Jr. Member
  • *
  • Posts: 52
    • View Profile
Re: Robots.txt needed
« Reply #2 on: April 21, 2012, 01:58:14 PM »
Yes,
I am trying to stop robot because google shows duplicate description and title.

http://www.yourdomain.com/affiliates/affiliation-gives-me-less-money/?prev_next=next

http://www.yourdomain.com/affiliates/affiliation-gives-me-less-money/?prev_next=next

http://www.yourdomain.com/affiliates/?sort=views

http://www.yourdomain.com/affiliates/?sort=subject


Therefore i have these add today
Quote
Disallow:/?sort=views
Disallow:/?sort=subject
Disallow:/?sort=starter
Disallow:/?sort=last_post
Disallow:/?sort=subject
Disallow:/?prev_next=next
Disallow:/?prev_next=prev
Disallow:/msg
Disallow:/reminder
Disallow:/?start=

In the Vb bulletin,wp .It is clear for robots but in smf it is not clear.
« Last Edit: April 21, 2012, 02:03:44 PM by shubh »
No words left

Offline Illori

  • Administrator
  • *
  • Posts: 324
    • View Profile
Re: Robots.txt needed
« Reply #3 on: April 21, 2012, 02:58:12 PM »
by having

Quote
User-agent: Googlebot-Image
Disallow: /

you are blocking the whole bot so the rest of the text does not matter, read up on what robots.txt actually is and how to use it.

Offline shubh

  • Jr. Member
  • *
  • Posts: 52
    • View Profile
Re: Robots.txt needed
« Reply #4 on: April 21, 2012, 03:21:12 PM »
Confuse me, if the whole bot is stop than how the google show the duplicate meta description and title. According to you what will be my step.
No words left

Offline Illori

  • Administrator
  • *
  • Posts: 324
    • View Profile
Re: Robots.txt needed
« Reply #5 on: April 21, 2012, 04:08:21 PM »
i would say trash the robots.txt file unless you are willing to learn how to make one correctly, and just let things happen as i said before SEO is a joke and not worth your time.

Offline shubh

  • Jr. Member
  • *
  • Posts: 52
    • View Profile
Re: Robots.txt needed
« Reply #6 on: April 21, 2012, 10:03:41 PM »
I can' trash it .My website already go down from first page to third page. If i will trash it, i did not know what the rank.
No words left

Offline Illori

  • Administrator
  • *
  • Posts: 324
    • View Profile
Re: Robots.txt needed
« Reply #7 on: April 22, 2012, 06:26:34 AM »
it is not really in your control what the search engines do, if you think about it. the search engines have not told us what they look for how they rank websites. so just let things go and if you become more popular then you will go up in the search again.

Offline Arantor

  • Team Member
  • *
  • Posts: 82
    • View Profile
Re: Robots.txt needed
« Reply #8 on: April 23, 2012, 03:36:32 PM »
robots.txt has other uses that aren't SEO related. Certainly it can help exclude certain things from reputable search engines (which has a performance concern) and it can prevent duplicate content under certain circumstances (which IS a valid SEO concern)

I would ask, instead of whether to trash the file or 'whether it's right', better to figure out what you're trying to achieve, since this may be exactly what you want or it may be wrong, but in order to know that, we need to know what you're trying to do.

Offline shubh

  • Jr. Member
  • *
  • Posts: 52
    • View Profile
Re: Robots.txt needed
« Reply #9 on: April 24, 2012, 07:44:07 AM »
To avoid this

From the webmaster tools , i got this.
No words left

Offline Arantor

  • Team Member
  • *
  • Posts: 82
    • View Profile
Re: Robots.txt needed
« Reply #10 on: April 24, 2012, 08:03:49 AM »
Robots.txt files will not fix any problems with meta descriptions or title tags.

Offline shubh

  • Jr. Member
  • *
  • Posts: 52
    • View Profile
Re: Robots.txt needed
« Reply #11 on: April 24, 2012, 08:18:57 AM »
No words left

Offline Arantor

  • Team Member
  • *
  • Posts: 82
    • View Profile
Re: Robots.txt needed
« Reply #12 on: April 24, 2012, 08:28:27 AM »
...those are supposed to have canonical URLs on them, though none of those are supposed to be picked up by search engines anyway (which is why I suggested to you to use a dedicated sitemap tool for SMF, rather than a generic sitemap maker that doesn't know about SMF's little oddities)

(Also, please do not PM me unless I have asked, thank you. Support should be publicly available so everyone benefits, not just you.)