scraping javascript tutorial Sunday, Aug 7, 2016

Get Started with Nightmare.js

In this tutorial you will learn to build a screen scraper using node.js, nightmare.js, and jQuery. This will scrape all the gigs posted on craigslist in the computers section for any city you choose. You can hook this up to a cron job and email yourself the results at regular intervals.

Video

Code

Github repo

Dependencies

sudo apt-get install node

sudo apt-get install msmtp ca-certificates

sudo apt-get install gnome-scheduler

npm install --save jquery 

npm install --save nightmare -g

Configuring MSMTP

http://www.absolutelytech.com/2010/07/17/howto-configure-msmtp-to-work-with-gmail-on-linux/

Gnome Scheduler Job

node gigs.js provo > gigs-email.txt && msmtp -t < gigs-email.txt

NOTE: You will need to cd into the correct directory in the job unless you have the files stored in your home directory i.e.

cd [your_app_directory] && node gigs.js provo > gigs-email.txt && msmtp -t < gigs-email.txt