Screening problem with AppStore from iTunes
I am using Scrapy to get some data from iTunes AppStore database. I start with this list of apps: http://itunes.apple.com/us/genre/mobile-software-applications/id36?mt=8
In the following code, I have used the simplest regex that is for all apps in the US store.
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.contrib.spiders import CrawlSpider, Rule
class AppStoreSpider(CrawlSpider):
domain_name = 'itunes.apple.com'
start_urls = ['http://itunes.apple.com/us/genre/mobile-software-applications/id6015?mt=8']
rules = (
Rule(SgmlLinkExtractor(allow='itunes\.apple\.com/us/app'),
'parse_app', follow=True,
),
)
def parse_app(self, response):
....
SPIDER = AppStoreSpider()
When I run it, I get this:
[itunes.apple.com] DEBUG: Crawled (200) <GET http://itunes.apple.com/us/genre/mobile-software-applications/id6015?mt=8> (referer: None)
[itunes.apple.com] DEBUG: Filtered offsite request to 'itunes.apple.com': <GET http://itunes.apple.com/us/app/bloomberg/id281941097?mt=8>
As you can see, when it starts to crawl the first page, it says "Filtered delete request on itunes.apple.com". and then the spider stops. it returns this message as well:
[ScrapyHTTPPageGetter,client] /usr/lib/python2.5/cookielib.py:1577: exceptions.UserWarning: cookielib bug!
Traceback (last call last): File "/usr/lib/python2.5/cookielib.py", line 1575, in make_cookies parse_ns_headers (ns_hdrs), request) File "/usr/lib/python2.5/cookielib.py" , line 1532, at _cookies_from_attrs_set cookie = self._cookie_from_cookie_tuple (tup, request) File "/usr/lib/python2.5/cookielib.py", at line 1451, at _cookie_from_cookie_tuple if version is not (None): version = int ValueError: Invalid literal for int () with base 10: '"1"'
I used the same script for another website and I didn't have this problem.
Any suggestion?
source to share
When I hit this link in my browser, it automatically tries to open iTunes locally. This could be the "external request" mentioned in the bug.
I would try:
1) Remove "? Mt = 8" from the end of the url. It looks like it is not needed and it might be related to the request.
2) Try the same query in the Scrapy Shell. This is a much easier way to debug your code and try new things. More details here: http://doc.scrapy.org/topics/shell.html?highlight=interactive
source to share
I see this post is quite old, if you haven't figured out the reason yet, here it is.
I faced a similar issue working with itunesconnect using mechanize. After a lot of frustration, I discovered that there is a bug in cookielib that does not handle some cookies correctly. It was discussed here: http://bugs.python.org/issue3924
The fix at the bottom of this post worked for me. I will post here for convenience.
Basically you create a custom subclass cookielib.CookieJar
, override _cookie_from_cookie_tuple
and use this one CustomCookieJar
instead of the culinary library
class CustomCookieJar(cookielib.CookieJar):
def _cookie_from_cookie_tuple(self, tup, request):
name, value, standard, rest = tup
version = standard.get("version", None)
if version is not None:
# Some servers add " around the version number, this module expects a pure int.
standard["version"] = version.strip('"')
return cookielib.CookieJar._cookie_from_cookie_tuple(self, tup,request)
source to share