Crawl Chinese web site with Scrapy

The response return by Scrapy is a Python repr() object thus Chinese character ’display’ in Unicode.

Quick fix is like this:

item['name'][0].encode('utf-8')

Though some characters are still display incorrectly. Find source code here.

More about Unicode in Python:

• http://evanjones.ca/python-utf8.html
• http://www.b-list.org/weblog/2007/nov/10/unicode/