Skip to main content

How to use Scraping Hub? (Step-by-Step)

Do you want to run your Scrapy framework’s spiders without any paid server? Do you want to get clear picture of the generated logs? Do you want to run multiple spiders at once? Do you want to setup cronjobs?

I have used scrapinghub to do so without any price.

Steps:
1.     Create a free account from here: scrapinghub.com
2.     Create an instance of spider
3.     Go to deploy and run spiders section
4.     Note down ID and API key from there
5.     Install the commandline tool shub using pip command
  pip install shub  
6.     Go to the root directory of the scrapy project
7.     Login to the instance of scrapinghub
  shub login  
8.     Enter API key and ID of project (Enter Y to remember the project or N to enter ID on each deployment)
9.     Deploy the latest project
  shub deploy  

Now after each change on the project, you will need to do   shub deploy  .

Comments

Post a Comment

Popular posts from this blog

Trekking to Chandragiri Hills (Nepal)

Fitbit trekking records: Over 13000 steps 300 + storeys climbed 2.75 hours Altitude 2500 m (Approx) Cost Rates:  Lunch per plate: Rs. 550/- (Veg and Non-Veg rates are same) Cable car: One way: Rs. 415/- Student One way: Rs. 315/- Both way: Not sure (Guess Rs. 750/-)                     **(Rates may have changed now.) Hiking to Chandragiri Hills 8 th Jan 2017 (Sunday) We had achieved a milestone in one of our projects at Logpoint Nepal Pvt. Ltd, and hence our management committee offered us hiking as celebration. Me including my team members met at the Cable Car Station around 9: 00 in the morning. It took us 15 minutes to reach the Cable Car Station from Kalanki chowk. Main Gate of Cable Car Station Our office had managed us a bottle of water, a can of redbull, a packet of आलु chips and a snicker. We collected our share and star...
5 comments
Read more

Run different version of chrome using python selenium

Hello, If you are looking for Firefox , please see this link For some purpose I had to run different versions of google chrome using selenium python. I used different versions of chromes from here . A discussion on stackoverflow can be found here I extracted the compressed file and moved the chrome.exe file inside the directory with version name. Please see the screenshot below. Notice the chrome executable in this directory. I downloaded few more versions and did same with them. Then I places those inside the following directory: C:\Program Files (x86)\Google\Chrome Then I had to use binary_location as below. I had four different versions as shown in the image above. Out of them, one was randomly picked using random.choice()  method and the binary_location was created on runtime. from selenium import webdriver import random chrome_options = webdriver.ChromeOptions() available_versions = ['56.0.2924.87', '57.0.2987.133'...
Post a Comment
Read more

Run different version of firefox using python selenium

Hello, If you are looking for Google Chrome , please see this link. For some purpose I had to run different versions of Mozilla Firefox using selenium python. I used different versions of firefoxes from  here . After downloading proper version/os version, I had to simply install the browser using custom mode. Then choose a custom place to install. Like C:\Program Files\FirefoxVersion54 Important: Do not launch the firefox. Go to the installation directory and delete three update files from there namely: 1. updater.exe 2. updater.ini 3. update-settings.ini Then launch the firefox and check the version. Now from the python code do the following: from selenium.webdriver.firefox.firefox_binary import FirefoxBinary from selenium import webdriver import random # Assumed that you have installed 52 and 53 as well in the proper path. version_to_use = random.choice(['54', '52', '53'])  path = r'C:\\Program Files\\FirefoxVersion%s\\fire...
Post a Comment
Read more