Posted on Tuesday 13th of November 2007 at 05:21 in Tutorials

How to: change your PHP user agent to avoid being blocked when using Curl

Back in March I wrote a tutorial on how to scrape website content with PHP using Curl (rather than the more risky fopen). I have since stumbled across some more useful information on the topic, and here it is.

On my daily travels across the Internet I came across an interesting short piece of content that I think a lot of PHP devs could actually find useful; let me explain. When Curling content with PHP, it's not uncommon for people to get upset because you're either being clever and avoiding paying for something, or you're just flat out stealing someone's content.

The easiest way for them to do this is by checking the user-agent and that's your biggest enemy. If you look in your php.ini file you're probably set to identify as 'PHP' which is not only obvious but it's easy to block. If you've got users visiting your domain identified as PHP; someone is trying to steal your stuff.

Fortunately there are numerous ways round this, you can modify your .htaccess file, set a PHP variable or modify the agent using Curl. Below are examples of how to identify your actions as a Mozilla browser:

.htaccess
Add the following line to your .htaccess file and that should do the trick:

php_value user_agent Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9

PHP set
You could also use an ini_set to define the user agent too, just place this PHP line into the head of the script doing the Curl:

ini_set('user_agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9');

Set it in Curl
You can also set the parameter in the Curl script itself, meaning that only this action is identified as Mozilla. Just add the following line into the Curl script in your PHP (not forgetting to change the $curl variable to whatever you're using:

curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9');

For more information read the original piece of content at TorrentialWebDev as they deserve all the praise for this. If you'd like to learn how to scrape website content using Curl, check out my tutorial.

Enjoy this article? Why not subscribe to the full RSS feed?

Comments

Showing most recent 2 of 2 comments [View all comments]

Awesome! I used file_get_contents() and got trapped by the bot, just ONE line ini_set(’....’) solved the task without switching to curl or coding proper HTTP headers! Thanks :-)

http://www.getaudiq7.com/ (my site about SEO in Russia)

Audi Q7 Lover

Thanks for the tip. It worked a charm.

anonymous

Add a comment

Latest Content

Related Content

Recent Comments

How to: change your PHP user agent to avoid being blocked when using Curl