[andi01]

Hello, everyone.

I was wondering how you would build a site similar to worldnews.com. Where you would have news articles listed from different agencies such as CNN, BBC, Al Jazeera, etc. The script would have to create a page on your site for the updated news article containing a preview then the visitor would click a link to the full article.

In a way it's similar to RSS feeds but the script is creating a page on your site for each new article.

How would I go about doing this? Is there a script out there where you can do this?


Thanks for your feedback

[Sogo7]

There is no single script as such, however the technique used is well documented. You should also be aware that..

1. there are some copyright and site Terms of Use issues involved
2. it's fiddly enough trying to code this for one site let alone several different ones
3. if the target site changes its layout your script breaks
4. the process is server resource hungry and webhosts don't like that (unless you're on a VPS)
5. if they rumble you then IP bans are the kiss of death because your webserver is probably static

These are just a few of the problems in going after an ordinary website, but if the site(s) in question has defences in place such as IP based redirects, captcha, session ID's, ajax callbacks and viewstate information being passed then it's a way bumpier ride to get the data.

This post has been edited by Sogo7: 13 November 2011 - 08:52 AM

[andi01]

Ok, Thanks!

[Georgew]

Take their news RSS feeds. Do some simpleXML or something on it. Avoids problems 2 and 3 above.