Web Wiz - Green Windows Web Hosting

  New Posts New Posts RSS Feed - How about allowing bots/spiders?
  FAQ FAQ  Forum Search   Events   Register Register  Login Login

Topic ClosedHow about allowing bots/spiders?

 Post Reply Post Reply Page  <12
Poll Question: Would you like to allow spiders to index restricted areas?
Poll Choice Votes Poll Statistics
1 [25.00%]
1 [25.00%]
2 [50.00%]
0 [0.00%]
This topic is closed, no new votes accepted

Author
Bluefrog View Drop Down
Senior Member
Senior Member


Joined: 23 October 2002
Location: Korea, South
Status: Offline
Points: 1701
Direct Link To This Post Posted: 14 December 2003 at 2:58am

Often the point of a "restricted" area is not so much to restrict it, but to "dangle a carrot". In this case, you can allow Google to index it, and when users click the Google link, they have to "register". The part about the cache isn't really that important for this situation because the users can only view the cached pages and can't actually "view the site". They still need to register to view the rest.

I still don't see how spoofing IPs and User Agents matters, because once you spoof the IP, the only thing you can do is a DOS or DDOS attack, which is such an immature thing to do, not to mention a waste of time. (User Agent spoofing means nothing for security - i.e. Basing trust on a User Agent is foolhardy for security purposes - it is merely "informational".)

Let's face it, for 99% of web stuff, we really don't need all that much security. Look at the level of security built into WWF - there's quite a bit, but really, how many people actually need that level of security?

 

Back to Top
ljamal View Drop Down
Mod Builder Group
Mod Builder Group


Joined: 16 April 2003
Status: Offline
Points: 888
Direct Link To This Post Posted: 14 December 2003 at 7:46am
I have 2 restricted areas.
1 is solely for moderators, so that we can discuss issues regarding the message board such as user bans and other stuff that NO ONE but moderators and admins should see.

The other one is for XML development and even some mods don't have access to that area as it is only for the XML developers.

In any of those cases if a spider had access to those forums, it would defeat the purpose of restricting them completely. What it sounds like is you want to dangle a carrot in front of non-members by having a restricted area that they can't visit, but one that they benefits from search engine registration. No web board comes with that feature, so if you want it, you will have to leartn WWF (or some other board) well enough to build that mod, pay some one to build the mod for you, or wait until someone develops it for themselves and releashes it as a mod. Personally, I don't think any one would ever release it as a mod there is too much work and done incorrectly would add a security hole to WWF. Even done correctly it could add a security hole to WWF.

If you believe the WWF has "too much security for 99% of web stuff", then you have done NOTHING that needed security. Security is the one of the largest issues on the web and if you can give your users a sense of security, you have gone a long way towards insuring repeat visitors.
Back to Top
eksimba View Drop Down
Groupie
Groupie


Joined: 23 April 2003
Location: United States
Status: Offline
Points: 48
Direct Link To This Post Posted: 15 December 2003 at 11:21am

I, too, fail to see the point in having a 'restricted' area into which a public search engine can see. It wouldn't be restricted anymore, by definition.

Why don't you just dangle your 'carrots' in a non-restricted area of your forum, open to search bots, and put the juicy stuff into a restricted portion of your forum available only to registered users? That would seem to meet your needs.

- eric
Back to Top
Bluefrog View Drop Down
Senior Member
Senior Member


Joined: 23 October 2002
Location: Korea, South
Status: Offline
Points: 1701
Direct Link To This Post Posted: 29 December 2003 at 11:51am

I'm having a terribly hard time coming to grips with the absolute inability of anyone to understand the basic concept. This isn't that tough to grasp...

There are times when security is important, and other times when security is not. When it is NOT critical, then why not do it?

And as for posting in other places in the forum, that doesn't work because the content that people are looking for will not be indexed by the bot.

And yes. I have done a lot of security work. I've done B2C webstores and B2B sites as well. And I have NEVER had any problems.

You're all thinking about this in a far to "programmer" like manner. 99.99% of surfers cannot even imagine that when they click the search engine link and get a "Please register" notice, that they can get the actual results from the search engine cache. You're overestimating the general internet population. Most people don't know what the "Home" key is for, much less what a cache is.

The point is, that in order to make yourself visible on the web for people to find out who you are, they have to find out through "indirect means", i.e. search engines, word of mouth, etc. Search engines are by far the most effective way to "advertise" yourself.

A general forum with content restricted to registered members only is not uncommon. There are lots like it. The reason being that they want you to sign up and participate. Once you are signed up, you are far more likely to post. It is generally bad practise to allow anonymous posting because that attracts spammers and other lowlifes.

As for security, it would not add any kind of a hole at all. [ See above. ] It should ONLY be added to portions of a forum where you want people to register. e.g. Imagine a "car audio" forum about subwoofers and all the latest gadgets. How many car audio buffs do you think are accomplished programmers? A lot less than the number of people who visit WWF I would imagine.

As for IP based security... it works. Spoofing is only useful for DOS or DDOS. I'm a whitehat... I'm not interested in blackhat stuff, although I am confident I could. (I've hacked sites by 'accident' on more than one occasion - never doing any damage of course... purely curiousity.)

I'm going to go beat my head against the wall for a bit...

But before I do, let me just state that season 6, "Once More with Feeling" is the absolute best Buffy episode ever  

Originally posted by Spike Spike wrote:


First he'll kill her
Then I'll save her
...
No, I'll save her
Then I'll kill her!

Originally posted by Willow Willow wrote:


I think this line's mostly filler

 

 

Back to Top
 Post Reply Post Reply Page  <12

Forum Jump Forum Permissions View Drop Down

Forum Software by Web Wiz Forums® version 12.08
Copyright ©2001-2026 Web Wiz Ltd.


Become a Fan on Facebook Follow us on X Connect with us on LinkedIn Web Wiz Blogs
About Web Wiz | Contact Web Wiz | Terms & Conditions | Cookies | Privacy Notice

Web Wiz is the trading name of Web Wiz Ltd. Company registration No. 05977755. Registered in England and Wales.
Registered office: Web Wiz Ltd, Unit 18, The Glenmore Centre, Fancy Road, Poole, Dorset, BH12 4FB, UK.

Prices exclude VAT at 20% unless otherwise stated. VAT No. GB988999105 - $, € prices shown as a guideline only.

Copyright ©2001-2026 Web Wiz Ltd. All rights reserved.