RyanSMcFadyen

Most of the time your scraping won't be a bother. Either they won't know it's happening or won't care because they think being added to a directory is a good thing. but, any kind of rejection is going to harm your site in some way. - A small brand blocking you without you noticing quickly results in dated information in your database. - A small brand blocking you AND requesting removal of all their data causes the above as well as time spent AND a small loss in your appeal (What good is a comparison site that doesn't compare all the things?) - A small brand blocking, requesting removal and taking action does all of the above AND costs you money. - A large brand just blocking you causes all of the above. People can deal with going to a toy comparison website and not finding Dave-Makes-Toys-In-His-Shed LTD. What they won't put up with is a toy comparison website with no Fischer Price on it. - A large brand taking an active interest in blocking you can result in all sorts, such as Mike mentioned. Someone with enough lawyer clout could get your hosting or domain suspended because ultimately scraping content without permission is illegal, it's just frequently ignored because it usually comes with an upside. Large brands are more likely to take issue and action because A.) They aren't desparate for attention, B.) They are much more forceful about their copyright, C.) They are much more likely to have official partnerships in place for that sort of thing and D.) They are more likely to notice/know what you are doing.

Free speech. I can write about any product I want to. It's fair criticism. Directories on the other hand automatically index massive amounts of information for no purpose other than to shift consumer focus away from the business' individual sites and on to the directory, for monetary gain. Here's an analogy. You paint a lovely picture. You put it into your gallery, with all your other paintings and your hope is that the new painting will either attract a buyer for itself or the people who come to see it will browse around and maybe buy one of your other paintings. I turn up at your gallery. I take a picture of all your paintings, I go home and I put them on my website. I've done this quite a lot, I have a fair few artists on there and I encourage people not to visit your gallery but instead to do their browsing on my website. If they see something they like I tell them exactly where it is and how much it costs. Now, if I'm super popular and many many more people are visiting my website than are visiting your gallery.. this could be a good thing for you. It's exposure, your work is reaching a greater number of people than it normally would. If however you pride yourself on your gallery, you've spent a great deal of effort, time and money on building your gallery and you rely quiet heavily on not just selling paintings but little bits and bobs at the gallery gift shop you might not like me making money off of the browsing these people would normally be doing in your gallery. Or maybe you are very particular about the way your paintings are displayed and the information that is displayed alongside them. Or maybe you are a very famous artists, you are the number one name people search for on my website but you simply don't like the idea that your work is being used to draw attention to all the less-famous artists that show up alongside you on my website. Or maybe you are old and don't understand the internet. Or maybe you already started a website with some artist friends because you saw the opportunity a single directory might give to everyone involve but you also saw that if you ran the directory you could give fairer terms to all artists. Or maybe you don't have a reason. It's your art, you can do what you like with it. ..... In this hypothetical situation, would it be within your rights to stop me taking photographs of your painting?

It is not your data. What they do with it elsewhere is not your concern, you need express permission to use it for your own commercial purposes even if those purposes are to direct customers to them. Scraping data is increasing load on their servers when you aren't a customer and thus not potentially resulting in a sale. In many cases this is ignored because there is commercial benefit to allowing it somewhere down the line. Google Search is usually allowed to scrape because it translates directly into better visibility on Google. Adsense is generally allowed to scrape because the more data it has the more relevant the advertisements are and that should increase click rate which increases revenues. Google has also proven itself to be an ethical data scraper. They reduce load as much as possible, they grant you some level of control and access to the scraper and they won't go anywhere you tell them not to (robots.txt) But if the company in question sees no benefit to you scraping their data or has some kind of commercial model that is in opposition to you having that data they are perfectly within their rights to block you. Heck they are perfectly within their rights to block you for no reason at-all. Example, you don't find Flickr image results on Google images. Flickr is owned by Yahoo! and they think that the Flickr data is too valuable to allow it's competitors access to. Who's the biggest company who's data you will be scraping? What if somewhere in their corporate structure they own a directory/comparison type site that deals with the same data. It's well within their interests to deny you that information.

When it comes to web/software startups you are shooting yourself in the foot by keeping your idea close to your chest. Yes, many people on this forum have the skills to duplicate whatever it is you are doing and I understand the fear that you'll get caught out but it's simply not realistic. For a start most of us are pretty busy and any ground-up site like this is a large time investment. Secondly, it is execution and luck that matters. I can say with certainty that your idea is not unique, what will separate you is your ability to spot where improvements can be made, making those improvements and getting the right set of influential people to see, understand and promote those improvements. Facebook was not a new idea. We've had social meeting places since the dawn of humanity. It was better though. Understandably, provably and usefully better and Mark Zuckerberg had the good fortune to have a little insight into a world that could spark the initial spread. My inclination is that you are making a takeaway food directory, like Just-Eat and if I am right data-scraping is the worst way to go about making that a success.

Then you need $65.

*looks over from behind RD's shoulder*. Yeah, you heard! *points*

Using templates for business websites is a fundamental conflict of models and the the exception of just needing a quick & cheap client through your books, it should be avoided at all costs. Your average business should be focusing on USP. They should be doing everything they can to convince customers they aren't like all the other options, they are better, they are cleverer, faster, cheaper... whatever. Templates on the other hand use the opposite model. They are designed to be sold in bulk and they will be sold in bulk to who? Your clients competitors.

You need a certain number of likes before it lets you choose a vanity URL for pages. Facebook -> Use Facebook as [Your Business Page] -> Edit Page -> Update Info -> Basic Information -> Username

I have 8 hours off every night.

Backstretch http://srobbin.com/jquery-plugins/backstretch/ is the best I've seen for doing this. The CSS cover method is nice but some browsers can lag because of it.

I'd ditch the QR code entirely, just increase web address size and put that in the middle instead.

Yes it's dated, but the good news is it's a relatively easy fix. Mainly what makes it look dated is what I like to call 'effects'. Things that are added to elements in an attempt to give them some depth, distance, separation etc. They need to be incredibly carefully used to work well. For your site I would ; Remove underline from links Reduce header font sizes a little bit, add some spacing Remove the border, background, drop shadow and inner shadow from various content boxes. Give the whole sidebar a background colour. Not everything needs its own box. White space can be enough of a separator. Let the content be the main attraction, don't box it in and segregate it from other bits of content. Font-sizes, bolding and placement are much better at keeping stuff apart but also keeping a flow to the site.

Good idea if you can pull it off and deliver with actually useful content. Signed up.

Fine here too..

I can't speak for Facebook but the Twitter API will respond to everything you send it and will tell you if what you tried to do was successful. https://dev.twitter.com/docs/error-codes-responses