Author
Message
DamonHD
Posts:6158
Moderator
Member since: 2006-11-30
:: Quote ::
Subject: Site map for Google...
I'm thinking of finally adding a site map to one of my small sites so that updates to it are picked up quickly, eg by Google new Caffeine/Percolator mechanism.

A few questions to those of you already using it:

1) Does it actually work and help get into the SEs quicker?

2) Does it make your site easier to rip off by others?

3) Is it possible/easy to automatically push an update without needing an interactive password, or just have G concentrate on frequent polling of the site map perhaps hourly?

4) Do images (etc) have to be included? And indeed what's the penalty of leaving something important out of the site map?

Rgds

Damon
September 27, 2010 05:03PM
GegaBit
Posts:3311
Senior member
Member since: 2006-11-30
:: Quote ::
Subject: Re: Site map for Google...
Yes, because of (2) I'd advise you to sticking to uploading your site map to your Google WMT account instead and enjoy your (1)

I think site map is only for pages, no images.
September 27, 2010 06:49PM
DamonHD
Posts:6158
Moderator
Member since: 2006-11-30
:: Quote ::
Subject: Re: Site map for Google...
If (2) is the problem that I have assumed it to be as you seem to suggest, then I think I'll just let things happen the traditional way, automatically, even if I fail to do something!

Rgds

Damon
September 27, 2010 07:09PM
mikeinfrance
Posts:248
Preferred member
Member since: 2009-10-08
:: Quote ::
Subject: Re: Site map for Google...
2)Does it make your site easier to rip off by others?

Are you not blocking all the "common rippers" ( software ) in .htaccess anyway ..and likewise using it to "anti-hotlink"..

That and no archive ..no cache in the headers and robots.txt to tell the way back machine to stay in it's box ..and a framebuster is about the most that one can do without being paranoid or on the stats 24/7

I presume you don't mind if G translates your place ..

You could watermark all your photo's ( with low tech potato print methods and woad dye ;-)..keep you out of mischief for a few months that would ..;-)



Edited 1 time(s). Last edit at 09/27/2010 07:33PM by mikeinfrance.
September 27, 2010 07:30PM
DamonHD
Posts:6158
Moderator
Member since: 2006-11-30
:: Quote ::
Subject: Re: Site map for Google...
No special protection on this entirely static site.

And I'd rather not do noarchive since I do want people to be able to use G's cache if my server happens to be down...

Rgds

Damon



Edited 1 time(s). Last edit at 09/27/2010 08:14PM by DamonHD.
September 27, 2010 08:04PM
mikeinfrance
Posts:248
Preferred member
Member since: 2009-10-08
:: Quote ::
Subject: Re: Site map for Google...
I'd block the rippers and the hotlinkers anway IIWY

re G finding all your stuff ..they seem to do just fine without special
just for them ( I always think that is just to save their CPU cycles )
sitemaps ..

Their spiders are coming from some unusual places these days

I'm picking them up out of Germany and China ( sites that are in dev ..only I know of them ..I have no toolbars and run linux .."clean" and am never signed into anything Googly when I touch the sites ..usually only use boxen that are just for that ..and these like all my domains have privacy ..I still get "reads" from gbot ..I think they are trawling entire IP ranges rather like war-dialers ..it's like hearing asdic or the old time over the horizon radar that we used to call "the woodpecker" that came out of Russia ) ..whatever ..

I dont lose sleep over it as it's all "no archive" from when the first page goes up

But it does demonstrate that they will find anything that isn't behind a firewall ..they'll index it ..whether or not they'll "serp" it depends on if you let them ..and if they want to ..
September 27, 2010 08:26PM
Ian C. Purdie
Posts:2220
Senior member
Member since: 2008-12-12
:: Quote ::
Subject: Re: Site map for Google...
I assume from sitemap we're talking about XML version?

I use it and have it immediately checked with WMT.

I also do noarchive.
September 27, 2010 10:50PM
DamonHD
Posts:6158
Moderator
Member since: 2006-11-30
:: Quote ::
Subject: Re: Site map for Google...
No, in the first instance I was contemplating plain text: trivial for me to generate from my makefile...

Rgds

Damon
September 28, 2010 04:14PM
Ian C. Purdie
Posts:2220
Senior member
Member since: 2008-12-12
:: Quote ::
Subject: Re: Site map for Google...
I'd recommend using the Google preferred XML version. Plain text I think is next to useless to you.

There's a free generator out there somewhere. Having done it when they wanted it way back when, I manually added the handful of additional pages since.

Never add pages without having Google WMT check it [minutes] as it is extremely picky. One tiny minuscule error and...

It stops and goes absolutely no further. Follow strict format or no workee.

Here's my main site:

[www.electronics-tutorials.com]

Pay critical attention to the beginning and to the end. In between are simply repetitious listings of pages.

Trust me, formatting, spaces, line breaks are everything.

An interesting, if at times frustrating exercise.

Guaranteed way to get new pages indexed FAST when you notify update via WMT. The last one I added was searchable within hours.

Bite the bullet.
September 28, 2010 10:19PM
Ian C. Purdie
Posts:2220
Senior member
Member since: 2008-12-12
:: Quote ::
Subject: Re: Site map for Google...
Here's more information and possibly a link to the generator.

[www.sitemaps.org]
September 28, 2010 10:23PM
GegaBit
Posts:3311
Senior member
Member since: 2006-11-30
:: Quote ::
Subject: Re: Site map for Google...
I vouch for text site map
works for me, simple, easy, never had a problem, all gets indexed fast, no fuss.
September 28, 2010 10:44PM
Ian C. Purdie
Posts:2220
Senior member
Member since: 2008-12-12
:: Quote ::
Subject: Re: Site map for Google...
Then if text is fine, why do Google and others push XML?
September 28, 2010 11:18PM
GegaBit
Posts:3311
Senior member
Member since: 2006-11-30
:: Quote ::
Subject: Re: Site map for Google...
because in xml you as a publisher can (if you want) give additional info, like for example priorities of some urls over others, if all you're giving Google in your xml site map is a list of urls then save your time and give them text.
September 28, 2010 11:26PM
mikeinfrance
Posts:248
Preferred member
Member since: 2009-10-08
:: Quote ::
Subject: Re: Site map for Google...
Agree 100% with GegaBit ..whomsoever builds the XML does all the "grunt work" ( unless they use a script to convert from text ..in which case unless they parse what comes out to check that it's what they want ;-) ..or they built their own script and thus know what it is doing and sending ) ..whomsoever receives/ requests the XML has the easy end ..they only have to display it or feed it to a database and interrogate it later ( which is in essence what G do ) ..So it's more unpaid labour for them ..cont below ..

<mini rant on>
And more rule via FUD ..( just for the sake of some of them getting off on having a "fud button" to push ) like the idiots who think you have to shout at / or continually cuff and thus confuse a dog to get it do do stuff ..it does more if you speak quietly .. but the G learnt from religion ( eric appears to think it is humanity's new "salvation" aswell as a money machine ) ..and all religion gets off on fear and FUD and needless lies.
</mini rant off >

cont...

When "they" are handling /parsing said script "they" can do what they want with it ..

So they can easily strip out whatever formatting you put in ( if they don't give a toss about it or your priorities are not theirs ..I'm looking at you Google -->>>:{ ..then they will discard all your "hard work" ..or if you use "auto to XML"it can arrive "munged" unless you re-parsed the XML output to check what "auto" gave you as an output stream ..or the receiver can choose to munge it themselves ( again I'm looking at the "plex" and their "we decide what is relevant / good webmastering " algos ..

<mini rant on >to G in particular ..but SE's in general should leave it like it was bloody well written and squirrel it away that way in your DB ..because I/we can make you do what I/we want with it any way and dance the waltz or whatever I/we choose ..( because you don't allow for the "curious" only the "marketroid" elements in people )..just takes me/us more effort ..which I/we could expend walking my/our dog ( if one has a dog ..which IMO everyone should and one lives by a beach..which again IMO everyone should ) along the beach ..and makes me/us even less likely to *buy your adwords ..'cos you are clumsy with your "racket" </mini rant off>

*unless we can turn such purchases to our advantage ;-))

And yes they read ..here ..via spyders or just plain spies..

Josh must see many long silent members who still "look in" to closed off members only threads from time to time ..n'est-ce pas J ;-)..

And then there are the bots..

mais..

bref ..sitemaps in text will work just fine ..moving them into XML is only going to confuse whoever does it unless they know how to parse the result to verify it ..and anyway all the SE's and anyone else can either ignore your XML "tags" or change them before assimilating them ..or use them verbatim ..

SE's tend to do the first ..

G tends to do the second ..

Save your time ..walk your dogs ..or play with your kids ..whatever ;-)

edited to correct a ( and thereby protect the frail and innocent ) vowel error in the speeling of "munge" ^_^

and 2nd time to correct all the usual speeling stuff ..



Edited 2 time(s). Last edit at 09/29/2010 01:14AM by mikeinfrance.
September 29, 2010 01:05AM
Joshua
Posts:2831
Administrator
Member since: 2007-03-16
:: Quote ::
Subject: Re: Site map for Google...
Since I never update my sitemap, I don't have much input for this thread, other than to say that without updating it G seems to pick-up my content pretty fast.

Quote:
Josh must see many long silent members who still "look in" to closed off members only threads from time to time ..n'est-ce pas J ;-)..

Looking at members that actually posted in the past, there were 2 members 'looking' past 90 days. The list of never posting members is a bit longer.
September 29, 2010 11:11AM
Ian C. Purdie
Posts:2220
Senior member
Member since: 2008-12-12
:: Quote ::
Subject: Re: Site map for Google...
I pay respect to "silent members". [Aboriginal expression I don't just pay lip-service to].

I'll stick with my method though. I trust my instincts of what is right for me.
September 29, 2010 12:06PM

Sorry, you do not have permission to post/reply in this forum.