Since WWF is basically server-side, bots won't be able to see content on pages. What about a mod or option to allow bots (e.g. 'googlebot') to search those pages and index them? It could be googlebot or slurp or whatever. Just spiders...

I can contribute via code, spider names, and spider IP addresses. Google has at least 150 spider IPs that I know of.

Ideas? Comments?

Perhaps I wasn't clear enough.

I only put "restricted" in the question, not the title or body - I mean restricted pages. i.e. Pages that you must have permission to view.

It would make that available in search engine caches, which for some things may not be appropriate, but for a lot of purposes, it should be fine.

e.g. A members only forum is restricted, but we allow Google to peek in on it. Users from search engines would need to login prior to viewing the content though.

For security, this can be done for IP addresses so that users can't spoof their User Agent.

etc... etc...

For the thing about restricted areas being visible - yes - that would be true.

For security problems, I don't agree.

Spoofing a User Agent can work one way (client to server) [ and the reverse is true but not worth discussion here ], and spoofing IP addresses can only work one way (client to server OR attacker to attackee).

The solution around spoofers is to first check the UA, then check the IP. If the IP does not match an approved one, then deny access to the site. Otherwise, allow. That way, only real bots / spiders are allowed.

Spoofing IP addresses is only good for DOS/DDOS/ similar type attacks. It cannot be used to gain access.

I.E. IP address verification is sufficiently secure.

This is NOT something you would want to do for some things, but it would provide an incentive for people to sign up at forums. A similar tactic is common place at some current major sites.

They allow search engine visitors, but disallow internal searches without sign-ups.

The whole thing is quite clever I think.

Often the point of a "restricted" area is not so much to restrict it, but to "dangle a carrot". In this case, you can allow Google to index it, and when users click the Google link, they have to "register". The part about the cache isn't really that important for this situation because the users can only view the cached pages and can't actually "view the site". They still need to register to view the rest.

I still don't see how spoofing IPs and User Agents matters, because once you spoof the IP, the only thing you can do is a DOS or DDOS attack, which is such an immature thing to do, not to mention a waste of time. (User Agent spoofing means nothing for security - i.e. Basing trust on a User Agent is foolhardy for security purposes - it is merely "informational".)

Let's face it, for 99% of web stuff, we really don't need all that much security. Look at the level of security built into WWF - there's quite a bit, but really, how many people actually need that level of security?

I, too, fail to see the point in having a 'restricted' area into which a public search engine can see. It wouldn't be restricted anymore, by definition.

Why don't you just dangle your 'carrots' in a non-restricted area of your forum, open to search bots, and put the juicy stuff into a restricted portion of your forum available only to registered users? That would seem to meet your needs.

I'm having a terribly hard time coming to grips with the absolute inability of anyone to understand the basic concept. This isn't that tough to grasp...

There are times when security is important, and other times when security is not. When it is NOT critical, then why not do it?

And as for posting in other places in the forum, that doesn't work because the content that people are looking for will not be indexed by the bot.

And yes. I have done a lot of security work. I've done B2C webstores and B2B sites as well. And I have NEVER had any problems.

You're all thinking about this in a far to "programmer" like manner. 99.99% of surfers cannot even imagine that when they click the search engine link and get a "Please register" notice, that they can get the actual results from the search engine cache. You're overestimating the general internet population. Most people don't know what the "Home" key is for, much less what a cache is.

The point is, that in order to make yourself visible on the web for people to find out who you are, they have to find out through "indirect means", i.e. search engines, word of mouth, etc. Search engines are by far the most effective way to "advertise" yourself.

A general forum with content restricted to registered members only is not uncommon. There are lots like it. The reason being that they want you to sign up and participate. Once you are signed up, you are far more likely to post. It is generally bad practise to allow anonymous posting because that attracts spammers and other lowlifes.

As for security, it would not add any kind of a hole at all. [ See above. ] It should ONLY be added to portions of a forum where you want people to register. e.g. Imagine a "car audio" forum about subwoofers and all the latest gadgets. How many car audio buffs do you think are accomplished programmers? A lot less than the number of people who visit WWF I would imagine.

As for IP based security... it works. Spoofing is only useful for DOS or DDOS. I'm a whitehat... I'm not interested in blackhat stuff, although I am confident I could. (I've hacked sites by 'accident' on more than one occasion - never doing any damage of course... purely curiousity.)

I'm going to go beat my head against the wall for a bit...

But before I do, let me just state that season 6, "Once More with Feeling" is the absolute best Buffy episode ever  

Spike wrote: First he'll kill her Then I'll save her ... No, I'll save her Then I'll kill her!

Willow wrote: I think this line's mostly filler