Scraper API – An Introduction

Web Scraping

scraper apiSimplify Your Web Scraping with Scraper API. This library allows you to get up-and-running fast and provides impressive features.

Web Scraping can be challenging. While you can certainly leverage a module such as Axios and handle the details yourself, sometimes it is better to stand on the shoulders of giants.

Scraper API is a giant. This multi-language product makes basic to advanced web scraping a breeze. Scraper API supports Bash, Node, Python/Scrapy, PHP, Ruby and Java. It also features Rendering Javascript, POST/PUT Requests, Custom Headers, Sessions, the ability to specify a Geographic Location and Proxy Mode.

In this article, we will cover basic usage and rendering Javascript. All of the code examples leverage Node.js in order to demonstrate Scraper API.

package.json

Above we have the package.json needed in order to run our application. The only detail we need to concern ourselves with is the inclusion of the scraperapi-sdk NPM module in the dependencies section.

The Web Page to Be Scraped

In the example above, we have the HTML for the web page that we will scrape using the Scraper API. It is important to note that the content mainly contains the paragraph: “I’ve been scraped!”. But, if you visit the target URL: http://examples.kevinchisholm.com/scrape-me, you will see the text: “This text was added with JavaScript.” The reason for this is: there is a JavaScript file that is being executed in the page. That script looks for the element with the ID:”Main” and then replaces the HTML text “I’ve been scraped!” with “This text was added with JavaScript.” This is an important detail that we will touch upon in the next two examples.

Basic Scraper API Usage

Example # 1-A

Example # 1-A is where the web-scraping begins. We set the variable scraperapiClient, which is a reference to the imported scraperapi-sdk NPM module. The reason we have created a scrapePage() function, is that we want to use the JavaScript await expression, so that our script will pause until the asynchronous call to the Scraper API client returns.

Let’s break-down what is happening here:

  1. We set the scrapeUrl variable, which is the page to be scraped.
  2. We set the scrapeResponse variable, which will be the HTML returned when we scrape the page.
  3. We call the scraperapiClient.get() method, passing it the scrapeUrl variable (the page to be scraped).
  4. We use console.log to output the HTML returned from the scraped page (the scrapeResponse variable).

NOTE: See Example # 1-B below for a discussion of the HTML that is returned from the scraped page.
Example # 1-B

In Example # 1-B, we see the HTML that is returned from the scraped page. Ironically, there is not too much to discuss here: the HTML is 100% identical to the actual HTML that was in the web page. This demonstrates the simplicity and power of the Scraper API. In other words: you get exactly what you ask for: the HTML of the URL that you pass to Scraper API.

NOTE: Earlier we discussed the fact that the HTML content is: “I’ve been scraped!”, but what we seen when you visit the URL is: “This text was added with JavaScript.” In the next example, we will discuss the reason for this.

Rendering Javascript

Example # 2-A

Example # 2-A is identical to the code in Example # 1-A with one exception: When we call scraperapiClient.get(), we pass a 2nd argument: {render: true}. This 2nd argument, tells Scraper API to wait until the JavaScript in the page has finished executing. The benefit of this is demonstrated in Example # 2-B: If JavaScript alters the page content, then the HTML that we get back includes the changes that the JavaScript has implemented.

Example # 2-B

In Example # 2-B, you will see that the HTML content returned by Scraper API is: “This text was added with JavaScript.” instead of “I’ve been scraped!”. This is because we passed {render: true} as the 2nd argument when we called scraperapiClient.get().

With Scraper API, you can implement web scraping in minutes. The process is simple and the features are well thought out. Whether your project is for research or commercial purposes, this product provides a robust and reliable way to fetch the source code of any web page.

Node.js File Uploads with Multer

Node.js

Node.js LogoWhen it comes to the UI, file uploads are pretty simple. But on the back-end, there is some work to do. Multer is a Node.js module that simplifies this process.

Uploading a file is a common task for Web applications. Today, most Web pages leverage AJAX — which requires JavaScript — for a smoother user experience, but this can be accomplished using only HTML and zero JavaScript. The truth is, HTML-only file uploads have been possible for more than 20 years. I mention this to point out that in the browser, file uploads are simple and require only a small amount of HTML. This is only half of the equation, however, because a file upload is useless without some back-end code that can process the file. So, in this article I’ll show you how to process a file upload in your Node.js application. To simplify the back-end code needed to handle a file upload, we’ll leverage Multer, a Node.js middleware for handling multipart/form-data.

For this article, our working example code is a simple Node application that accepts a POST request with an “enctype” of “multipart/form-data.” When the user uploads a file, our back-end code will take the uploaded file and put it in the “uploads” folder, right within the root of the project folder. Nothing too fancy here, but it’s worth noting that the examples provide plenty of opportunities for copy/paste. What you do with these examples is up to you, but at least you’ll know how to process a file upload in your Node application.

index.html

In the above example (index.html), we have the HTML file for our application, so take a look at the form element. You’ll see that the “enctype” attribute is set to “multipart/form-data,” which means that we will send images in various formats. This is also important to keep in mind because Multer will only process this kind of file-upload. Note also the input element, which has a type attribute of “file.” This ensures that the browser will take care of implementing a file-upload interface. So, in other words, there will be a “Choose File” button, which allows the user to select a file from his or her hard drive. We certainly don’t need to put any effort into this; simply setting type=“file” takes care of all of it. There is also a name attribute for this input element. This attribute is required so that Multer understands how to handle the request. The Submit button will pass the form to the same exact URL because there is no “action” attribute, so the default behavior is: this is the form submitted to the same exact URL.

Configuring Multer – Example # 1

Example # 1 contains all of the code for our Node application. For this project, we leverage the Express framework. By using Express, we significantly reduce the amount of code needed. One of the most powerful features of Express is the ability to easily create middleware, which is a perfect context for Multer because it needs to intercept the HTTP request for us. The upload variable is used to provide configuration for Multer. In this case, for example, it lets Multer know that we want our uploaded files to be placed in the “uploads” folder. We’re using express.static in order to serve the HTML and CSS files to the user, so when the user goes to the “/” route, index.html and style.css are served by the Express framework.

Adding a Handler for the POST route

On Line # 11, we set up a handler for the POST route. If you’ve ever used the Express framework when building a Node application, this pattern should look familiar to you. But notice that the second argument passed to the app.get() method is upload.single(‘img’). We’re using the upload variable created earlier. The single() method takes a string as an argument, which is the “name” attribute of the form field containing the uploaded file. For demonstration purposes, we output req.file to the console so we can see information on the uploaded file. We call the send method of the response object, passing it some HTML, which simply informs the user that the upload was successful and allows that user to go back to the “/” route.

At this point, it would be a good idea to run the example code yourself, so just follow these steps:

  • git clone
  • git@github.com:kevinchisholm/video-code-examples.git
  • cd /node/file-uploads-with-multer/
  • npm install
  • node index
  • Open this URL in your browser: http://localhost:3000/

Now in your browser, click the “Choose File” button and browse your hard drive for a file to upload. Once you’ve selected a file, click the “Submit” button. You should see the message: “File upload succeeded.” Now, if you look in the “uploads” folder in the root of the project folder, you should see a file with a name similar to: “08e36ff4c9d3dc106e3a9fa2367797c9”.

So, we’ve made good progress here; our example code works and we’re able to upload a file. As you can see, though, the original name of the file is not preserved, and a GUID-like name is provided. This can be helpful in that users will not overwrite a file when uploading the same-named file more than once. The downside, however, is that there’s no connection between the original file name and the one provided. So, let’s fix that.

Show the Original File Name – Example # 2

Stop the Node application and then start it again, using the second example: node index2. Now, upload a file again.
You’ll see that the original file name is preserved.
In Example # 2, we accomplished this by leveraging multer.diskStorage(). When calling that method, we provided a configuration object. The destination property told multer.diskStorage() where the uploaded file will go, and the filename property provided a way for us to specify what the name of the uploaded file will be. This method receives a second argument called file, so we use the “originalname” property of this object to set the file name. But there’s a new problem now: the user can overwrite an uploaded file by uploading a file with the same name. So let’s fix that.

Create a Dynamic File Name – Example # 3

In Example # 3, we have expanded the anonymous function passed to the filename() method. What we’ve done here is use regular expressions to extract the name of the file with and without the extension. We use Date.now() to generate what is essentially a unique value, and we piece the new file name back together. As a result, the user can upload the exact same file over and over, but each uploaded file name will be unique. For example: original-file-name_123456.jpg. So, let’s just confirm this. Stop the Node application and then start it again, using the third example: node index3. Now, upload the same file over and over. You’ll see that each uploaded file has a unique name, but the original file name is included so that it’s easy to reference the actual file that was uploaded.

Set up a Node / Express Static Web Server in Five Minutes

Node.js

node express static web server

Setting up Node and Express as a Simple Lightweight Web Server for Your Single Page Application is Very Easy to Do.

Sometimes you just need a local web server.  Sure, you could use MAMP, but installing Apache, MySQL and PHP seems like overkill in some cases. I have used MAMP for years and it is terrific. In particular, when you need to run PHP locally and / or connect to a MySQL database, it’s the cats pajamas. But that was the standard 10 years ago. Nowadays, it’s very common to build a single page web application where all of the assets are static, data is pulled-in from a REST endpoint and all of the heavy lifting is done in the browser via JavaScript. In these kinds of scenarios, a static Node / Express server is a simple, easy and lightweight approach.

In this article I’ll explain the steps needed to set up a Node / Express Static Web Server. And the good news is that on this high level, the required steps are very simple. First, you’ll need to require the express and path modules. After that, you’ll create an instance of express, then set the port that the web-server will use. The next step, and the key-ingredient here is the express.static method. This tells Express.js that you want to serve static content from a specific folder. In that one line of code, you’ve done the majority of the configuration work.

So, not only will Express serve-up static content from that folder, it can do so for any subfolders as well. You can specify any folder in your project as the static web folder. And the beauty of it is that any folder outside of the one you specify will be hidden from public view, so your application code will be safe. When you pass the the express.static method to the use method of your express instance, you provide the details that express needs to serve your static content. Then finally, you use the listen method of your express instance to start the web server. We’ll take a closer look at the express.static method in Example # 2.

Now, I just want to remind you here that this article pertains to the specific occasions in which you need to serve static web assets locally. In other words, using a Node / Express static web server can be a very simple way to satisfy your need for a local web server, but may not be the best approach for your production needs. Technically, you could take the code that is detailed in this article and deploy it to your production server, and in theory it would work just fine. For this article, however, I’m just going to concentrate on providing a fast and simple way to get a local web server running so that you can test your front-end code (e.g. HTML, CSS or JavaScript).

The code samples can be downloaded here: https://github.com/kevinchisholm/node-express-static-web-server

Example # 1 – package.json

In Example # 1, we have the contents of package.json. Nothing too special going on here. But just note that our only dependency is the express module. Also, in the scripts property, I’ve set up the start command to execute the app.js file in the node web-server folder. This way, we can simply type npm start in the terminal, instead of node web-server/app.js (just a bit less typing).

Example # 2 – The Express Static Web Server

In Example # 2, we have the entire contents of our web server: 15 lines of code (and nearly 25% of that is comments!). The magic happens on line # 10:  We call the app.use method and pass it express.static, which also takes a couple of arguments. So this tells Express that we want to set a static folder. We then use the path.join method to tell Express where all static assets should be served from. In our case, it is the www folder. The two arguments passed to the path.join method are __dirname, which tells us the absolute path to the folder within which the current script is found, and then “../www” which is a relative path to the www folder.

Now, as I mentioned earlier, anything outside of your static folder is protected from public view. This means that while the folder you specify when calling the express.static() method (i.e. “../www”) is publically viewable, any folder that is a sibling or descendant of that folder is not available publically. This is not a critical factor when working locally (i.e, developing), but it does matter in production. In other words, you wouldn’t want your application code to be viewable to the general public. Nor would you want to make available any sensitive information that’s in your application code, such as a secret key or other credentials. So, as you can see, this is one of the key strengths of Express, which is the ability that it provides you to not only define your public/static folder in one line of code, but to also protect all of the other folders by default.

Express does all of the heavy lifting

A little earlier, I used the word magic. We both know that none of this is actually magic, but it sure feels like it. If you’ve ever created a Node web server manually, then you know two things: 1) It’s really easy, 2) It’s really tedious once you get past “Hello World”.  But Express hides all the tedium and makes serving static assets as easy as 1-2-3.

HTTP Headers

There is one downside here. Express does not set the appropriate content-type headers for the HTTP requests.  This is not fatal in most cases because this approach is simply meant to provide a very fast and easy way to set up a static web server. The actual web server works just fine, but keep in mind that content-type headers for files such as JPEG, PNG, CSS or JS will not be set accordingly. If that is a problem, then a simple static web server is probably not what you need and you should consider a more robust approach. So, hopefully, if you do need a simple static web server, this article was what you needed to get up and running quickly.

Summary

There are multiple options when it comes to setting up a static web server. One advantage to leveraging Node and Express.js, however, is that as a developer, you probably already have Node installed on your machine. So, in this case, you won’t need to install any additional software. You can simply import the Express framework, write about a dozen lines of code, and you have a static web server. This is probably not a server that you would use in production, but as you can see, it can easily solve the problem of quickly serving web content on your local machine. If you need to write moderately complex dynamic application logic, then you might need something a bit more advanced than what was discussed here. But for a basic static web server, this approach should get you going (hopefully in less than five minutes : – )

Yikes! AWS Node / NPM ERR! enoent ENOENT: no such file or directory package.json

Node.js

Node.js LogoAWS’s Node deployment keeps telling me that it cannot find package.json, but it’s there! – Fortunately, this problem is easily solved.

AWS makes deploying your Elastic Beanstalk easy. Compress your build files, upload the ZIP and then deploy that application version. Lovely. But sometimes your application goes into a “warning” or “degraded” state, and then a visit to the application with a browser yields: “502 Bad Gateway“. Errrggggg…..

At this point, you look in the logs and see a cryptic message that says something like: “enoent ENOENT: no such file or directory package.json“. You double-triple-quadruple-check and yes, package.json is in-fact very much alive and well. So, of course your next thought it: “WTF???

I have run into this problem a few times and in each case, the problem was me: I zipped-up a folder, instead of the contents of a folder.

Do not compress an entire folder

Compressing the my project folder does not fix package.json problem

Let’s say your Node application is in a folder named: “myProject“. If you are compressing that folder, then this is your problem. You don’t want to compress a folder because when AWS un-zips that file, it will not know to look in the “myProject” folder that is created when the file is un-zipped.

Compress ALL of the items in  your project folder

Compressing the root files fixes package.json problem

What you want to do is: select EVERY file in the root of that folder (i.e. your Node application’s root folder), and then compress THOSE files. This will create a ZIP file that when un-zipped, creates the file structure that AWS expects. Now AWS will find package.json. This should solve the problem.

Compressing the root files fixes package.json problem

In the image above, I have zipped up the contents of the “myProject” folder, and created Archive.zip.

Upload the zipped file

Compressing the root files fixes package.json problem

Now, back in your AWS console, you can use the “Upload and Deploy” button to upload your ZIP file, and then deploy it.

Setting Your AWS-Hosted Node Application’s Port

Node.js

Node.js LogoWhen working locally, using an arbitrary port number is fine, but you need to get that property when deploying your Node application to AWS.

Technically, you can code-up a Node web server in less than ten lines of code. Most likely, your application will require a few more lines of code than that. But my point here is: getting a basic Node web server running is not terribly difficult.

Example # 1

In example # 1, I used port # 3000, but I could have used virtually any valid port number. When working locally this is for the most part a non-issue. As long as no other application is using the port you want to use, you chose one and then use it. Easy. This example does little more than say “Hello!”, but the point I’m trying to make is that your main JS file ends with the server.listen method, and you need to pass it a port number.

But when you attempt to deploy this code to your AWS Elastic Beanstalk instance, you will get a “503 Bad Gateway” error in your browser. The reason for this is: you don’t know which port should be used when calling the server.listen method. The great thing about AWS is that it provides a layer of abstraction for those kinds of details. In other words; AWS takes care of details such as which port to listen on. The downside here is that you have no way of knowing exactly which port that will be when you deploy your code.

Example # 2

In example # 2, we set a variable named: port. We attempt to assign the value of process.env.PORT to that variable. If that value is falsely, then we set it to 3000. The reason this works is; if our code is running on our AWS instance, then process.env.PORT will automatically be set and we will listen on that port. If we are running our code locally, then process.env.PORT will be undefined (or “falsely”). So, then our port variable will have a value of 300. This way, our code can run successfully on our AWS instance, or locally.

Node.js Hosting Links

Node.js

JavaScript LogoThe good news is: there are a lot of Node hosting services out there. The bad news is: there are a lot of Node hosting services out there

Installing Node locally is easy. Cloning an existing Node application from GitHub and running it locally is easy. Creating your own Node application and running it locally is easy. But, choosing a hosting solution for your Node application is definitely not easy.

Below is a list of companies that offer Node hosting services. I do not claim to have every possible company listed here. But I’ve done my best to list the ones that I know about and will update this page any time I learn about another one worth mentioning.


Title: Openshift

Link: openshift.com

Description: I’ve used Openshift.com quite a bit and for the most part have been very happy with their service. They offer a free plan that definitely includes what you need to get up-and-running.


Title: Heroku

Link: heroku.com

Description: Heroku was the first Node hosting service I knew about. I’ve not used it in a while but I was always very happy with it. Setup and deployment was fairly pain-free, as was adding services such as MongoDB.


Title: Amazon Web Services

Link: Deploying Node.js Applications to AWS Elastic Beanstalk

Description: AWS is a big topic. But in general, it’s really easy to get a Node instance up-and-running with Elastic Beanstalk.


Title: Nodejitsu

Link: nodejitsu.com

Description: I’ve not tried Nodejitsu but have heard good things about them.


Title: zeit.co

Link: zeit.co/now

Description: This is a new one to me, but their setup looks super-simple.


Title: Node.js on Google Cloud Platform

Link: cloud.google.com/nodejs

Description: Although Google still has not caught up with Amazon yet, they are serious about their cloud offerings. I’ve not tried their Node hosting but have confidence that it is at worst, solid.


Title: Node.js Hosting

Link: a2hosting.com/nodejs-hosting

Description: Another new one to me, but their packages look very affordable.


Title: Node.js One Click Install | Cloud Hosting – GoDaddy

Link: a2hosting.com/nodejs-hosting

Description: Godaddy now offers a “Cloud” service that supports Node hosting.

Helpful Node.js Education Links

Node.js

JavaScript LogoNode.js is growing fast. This is a great problem. While is means that JavaScript lovers have a rosy future, it can sometimes be difficult to keep up with what is going on with Nodes.js

Here are a list of links that you might find helpful in your Node.js travels. In each case, I’ve provided a brief description of the link / organization / article, so that you have a sense of where you are are headed. If you feel that there is a Node.js link that I should have included in this article, please contact me at: kevin@kevinchisholm.com.

Critical Node.js Links


Node.js Logo

Title: nodejs.org

Link: nodejs.org

Description: Since this is the home page for Node.js, you cannot go wrong here.


Node.js Logo

Title: Node.js v6.6.0 Documentation

Link: nodejs.org/api

Description:The official documentation for Node.js. Very well organized and easy to read. Almost the most important Node.js documentation you can read if you are getting started.


Node.js Logo
Title: npm – Home of the package manager for JavaScript.

Link: www.npmjs.com

Description: It’s hard to imagine doing anything with node without the use of NPM. This is the official home page of NPM, and a great starting point.


Node.js Logo

Title: Homebrew. The better way to install Node.js on Mac OSX

Link: brew.sh

Description: I’m being a little opinionated here (ok. I’m being a lot opinionated). But, for Mac users, Homebrew is the way to go when you install Node.js (sorry Windows users, you are stuck with scoop : – )


Node.js Logo

Title: Built in Node.js – startups, apps, projects using Node

Link: builtinnode.com

Description: A great way to learn about who is using Node.


Node.js Newsletters

Node.js Logo

Title: npm Weekly

Link: www.npmjs.com/npm-weekly

Description: Find out what npm has been working on, thinking about, and talking about every week. A great newsletter if you are into NPM.


Node.js Logo

Title: node weekly

Link: nodeweekly.com

Description: A free, once–weekly e-mail round-up of Node.js news and articles. Another awesome newsletter if you are into Node.


Other Helpful Node.js Links

Node.js Logo

Title: Node Tutorials on scotch.io

Link: scotch.io/tag/node-js

Description: scotch.io tutorials are very easy to read. The site is in general a great resource for learning about a number of web development technologies. Fortunately, they are passionate about Node!

Setting AWS-Node.js Stormpath keys

Node.js

Stormpath Logoprocess.env can be used to set the environment variables you need when using the stormpath api in your aws-hosted node application

Stormpath provides amazing abstraction when it comes to authentication. There are certainly other services like this, but when it comes to security, Stormpath is not only popular, but well respected. This comes as no surprise as they simply make authentication easy.

I have to say that their documentation is for the most part very good. If Node is  your thing, they make it very easy to get up-and-running with their API. Their mailing list is also quite helpful.  At least a once or twice per week, I receive emails that link to interesting articles on their blog.

Recently, I was trying to setup a Node.js/Express.js application, leveraging their express-stormpath Node module. I was thinking to myself: “…hmmmm. There must be a step where I have to configure my secret key or something like that”. After some quality time with Google, I came across this article, that suggested the following:

Unix/Linux/Mac:

Windows:

(where “xxx” is your actual key)

Well, that is fine for working locally, but I knew if I wanted to deploy this as an AWS Elastic Beanstalk application, I needed to actually set these values somewhere.

Using process.env

I set the three environment variables I needed to be properties of process.env:
I

(where “XXX” is your actual key)

I took a look in the source code for the express-stormpath Node module and could see that it seemed to want to find these on process.env, so I think this approach should be fine. I’m still in the process of getting this Node.js/Express.js application up and running, but if you are faced with the same challenge, hopefully this helped you.

Share Node.js code with JSApp.us

Node.js

JavaScript LogoJSApp allows you to write Node.js code in a browser, run it, and also share it with others

One of the things that makes front-end development so much fun is that you can easily create and share code. With tools such as JSFiddle, you can create an example web page and then send that JSFiddle URL to someone. Or you can even send someone the URL of a JavaScript file that you created so that they can just run $.getScript(yourJavaScriptURL) to inject your code in their page. And there are plenty of other clever ways to share / demo front-end code without a lot of fuss.

But what about Node?

Well, it’s not always so easy with Node, right? It’s server-side code, so you can’t just send someone a URL of your Node.js file to inject in their page. Github really saves the day in this case; you can create a public repo, and then send someone the Github repo URL. But that still requires the recipient to have at least git installed on their computer. And as we all know, once something takes more than 2 clicks, you start to lose your audience. That said, anyone with a reasonable attention span and a genuine interest in your code will follow the few clicks needed to clone your repo and run your code, but for quick little snippets, it sill feels like overkill sometimes.

For example, I like to write blog posts here about Node. In some cases, it does make sense to create a Github repo, especially if you have to leverage package.json, and the app requires file access, etc. But what about little examples? Just 10-20 lines of code to demonstrate a concept? Or even a simple working example?

Enter JS App!

When you navigate to jsapp.us, you immediately see some sample Node.js code. You can delete it and write your own. Then,  you simply click “test” in the sidebar (or CTRL + b), and a new browser window opens with your Node.js code running!

If you create a profile (free), you can save your code and share it with others. This is one of the most clever things I’ve seen in a long time. You can also go back and edit your files, re-name them, delete them. Really fun stuff.

If you need to create a quickie Node.js app and a Github repo would be overkill, JSApp might be just the tool you need. It’s been a while since I was this impressed but something I stumbled upon.

Bravo!

Getting Started with Gulp.js – Creating Multiple Tasks

Gulp.js

Gulp.js Logo
Learn how to create multiple Gulp.s tasks

In the article: “Getting Started with Gulp.js – Introduction,” I discussed the absolute basics needed to set up a Gulp.s task.  In that article’s gulpfile.js, we had only one task named “default”. One of the great features of Gulp, is that it will look for a task named “default”, and execute it automatically. This is fine if you have only one task, but as soon as you have two or more, it makes sense to give each one its own name.

When you have one or more named Gulp tasks, you’ll want to execute those tasks from the default task.

Figure # 1 – The folder structure before running Gulp

File structure - before

In Example # 1, you’ll see the folder structure before running gulp. So, if you look in the BUILD folder, you’ll see two sub-folders: JS and CSS. The file main.scss will be compiled into CSS and the output will go into the BUILD/CSS folder. The file: SRC/JS/main.js will be uglified and the output will go in the BUILD/JS folder. The file SRC/COFFEE/global.coffee will be compiled and the output will also go in the BUILD/JS folder.

Example # 1 – gruntfile.js

In Example # 1, we have the contents of gruntfile.js. You’ll notice that there are four tasks: default, uglify, sass and coffeescript. The default task is executed automatically. So that task simply executes the other three tasks.

How to Demo the Code Example

  1. Clone this repository: https://github.com/kevinchisholm/gulp-basics-tutorial-multiple-tasks
  2. Install node_modules with this command: npm install
  3. Run gulp: gulp
  4. Look in the following folder: BUILD/CSS, you will see the file: main.css
  5. Look in the following folder: BUILD/JS, you will see the files: main.js and global.js

Figure # 2 – The folder structure after running Gulp

File structure - after

Summary

One of the key features of Gulp is the ability to have a default task. This task is always executed by default. In this article, I demonstrated how to execute one or more named Gulp tasks from within the default task. While I chose to uglify JavaScript, compile SASS and compile coffeescript, you can create Gulp tasks for any need you might have. I hope that this article has made it easy for you to understand how to run multiple Gulp tasks.

Getting Started with Gulp.js – Introduction

Gulp.js

Gulp.js LogoLearn how to automate your front-end build process using this streaming build system

A while back, I posted an introduction to Grunt, the JavaScript task-runner. I also posted an article about the basics of concatenation and minification with Grunt. Grunt is an excellent tool, and still enjoys a large audience. That said, the most common complaint against Grunt is that its configuration-based syntax can become tedious and cryptic. In this article, I will introduce you to Gulp.js, an excellent streaming JavaScript built tool that has become quite popular in recent years. For this article, I will not discuss the details of installing Node or Gulp. There are plenty of articles available that will provide full details on that. Instead, I will provide a very gentle introduction to Gulp and how to create a simple Grunt task.

Code over configuration

Gulp’s success has to a large degree been based on the fact that it provides a powerful alternative to Grunt’s configuration-based approach. Gulp leverages actual JavaScript code in order to accomplish its tasks. With Gulp, you read files into memory, do things to the files, and then output the files from memory to a specified destination folder.

Easy Setup

Gulp is a node module. Installation and setup could not be simpler. On a very high-level, the steps needed are:

  1. Using npm (node package manager), install Gulp
  2. Create a file named: gulpfile.js or gulpfile.coffee (coffeescript)
  3. Execute the following command in a terminal: gulp

That’s it!

Gulp is simple

One of the things that amazed me most when first looking into Gulp was the fact that there are only four APIs. Yep, four. But there is a great deal of power lurking beneath the hood.

gulp.task – Defines a task
gulp.src – Reads files into memory
gulp.dest – Writes files from memory to disk
gulp.watch – Watches the files defined by gulp.src for changes

Note: The official Gulp documentation states that there are four APIs, but I find it odd that the .pipe method is not counted amongst these.

A Simple Example

I think many people might wonder: “…what would I use Gulp for?” A very common task in front-end tooling is concatenation; you may have three JavaScript files and want them to be combined into one JavaScript file that will be used in your web page. In this example, we will take three JavaScript files, concatenate them, and then output one file that consists of those three files.

Where to Get the Example Code

Clone this repository: https://github.com/kevinchisholm/gulp-basics-tutorial-introduction

Example # 1A – package.json

In Example # 1A, we have the contents of package.json. This file tells Node that we need the following modules: gulp, and gulp-concat.

Figure # 1: Project File Structure

Project File Structure

In Figure # 1, we have the folder structure of our first example. Notice that in the SRC/JS folder there are three JavaScript files. These are the files that we will concatenate into one file. The BUILD/JS folder is empty, but that is where the final concatenated file will be written.

Now, before going any further, let’s install the node modules which our code will need. Navigate to the example-1 folder with your terminal application and then execute the following command: npm install

When running npm install, you’ll notice some activity in the console (don’t worry about the “warn” message), and then there will be a “node_modules” folder. These are the node modules specified in package.json. npm has downloaded them for us and put them in the “node_modules” folder. A detailed explanation for npm and the “node_modules” folder is beyond the scope of this article. A few google searches on either topic will yield plenty of links for further reading.

Figure # 2: Project File Structure with “node_modules” folder.

Project file structure after installing node dependencies

In Figure # 2, you’ll see that we now have a “node_modules” folder. Let’s take a look at gulpfile.js.

gulpfile.js

This is the file where the Gulp code goes. Gulp does support Coffeescript, so gulpfile.coffee is also a valid file name, but for the sake of simplicity, I will only cover the JavaScript implementation.

Example # 1B – gulpfile.js

In Example # 1B, there are two things happening: First we create to variables, each representing a module that we need. Second, we create a gulp “task”. The gulp.task method takes two arguments: 1) a task name, which is a string, and 2) a callback function, which contains the code that defines the actual task. Here is where Gulp’s real power lies: a gulp task is driven by JavaScript code (i.e. code over configuration).

Returning a File Stream

A Gulp task always returns a file stream. This is to say that gulp will read a file into memory and you want to return that in-memory file object from your task’s callback function. In-between those two tasks, you “pipe” that file to one or more plugins that manipulate the file in some way.

gulp.src

In Example # 1B, we use the gulp.src method to read one or more files into memory. In this case, it is the three JavaScript files in our SRC/JS folder. We then chain the pipe method, passing a call to gulp.dest as an argument. The call to gulp.dest takes a string as its sole argument: the path to our output directory: BUILD/JS.

Executing the Gulp Task

In order to actually execute our Gulp task, simply type the following in your terminal: gulp

Yep, that’s it! Because our task is named “default”, we do not need to specify a task name. Gulp assumes that we want to run the “default” task, looks for it, and then executes it. Now when you look in the JS/BUILD folder, you should see three files: file-1.js, file-2.js, and file-3.js.

Figure # 3: Non-Concatenated Files in the BUILD/JS Folder.

Build output

In Figure # 3, you’ll see that there are now three files in the JS/BUILD folder.

You may be wondering why our output is three files, and not one concatenated file. This is because we did not actually concatenate the files inside of our task. In Example # 1, I wanted to demonstrate the basic flow of a Gulp task: using gulp.src to read files into memory, and then using gulp.dest to write those files from memory to disk. Now let’s update our Gulp task so that it actually concatenates the files.

Example # 2 A – Add the Contact Module to Our Gulp Task

In Example # 2 A, we have added a new line to our Gulp task: .pipe(concat(‘scripts-all.js’)). This line takes the in-memory files, pipes them to the concat module (which concatenates them into one file named: “scripts-all.js”), and returns that in-memory file. That’s really it. Now, navigate to the folder: “example-2” in your terminal, and then run Gulp again, so see the output: gulp

Figure # 4: Concatenated Files in the BUILD/JS Folder.

The concatenated JavaScript file

In Figure # 4, you’ll see that instead of three files, there is one file: scripts-all.js.

Example # 2 B – scripts-all.js

Example # 2B shows the contents of scripts-all.js. The details of the actual code are not important. What matters is that by piping the three source files to the concat module, our output is now one file that consists of the contents of all three source files.

Summary

The fact that there are only four APIs is a testament to the fact that Gulp.js is a simple yet powerful tool for running JavaScript tasks. There is a strong and growing community behind Gulp with thousands of existing plugins. The beauty of Gulp is that since it is code, you can leverage plain old JavaScript to make your gulpfile as powerful and efficient as needed. You are only limited by your imagination. While the examples in this article were very simple, there is a great deal of depth to Gulp and plenty of details / features that you can look into. I hope that this article was a helpful introduction and provided the tools you need to understand Grunt and easily start implementing it in your project.

Helpful Links for Gulp.js Basics

http://gulpjs.com/

https://github.com/gulpjs/gulp

https://www.npmjs.com/package/gulp

https://github.com/gulpjs/gulp/blob/master/docs/API.md

https://github.com/gruntjs/grunt-contrib-concat

https://www.codefellows.org/blog/quick-intro-to-gulp-js

Renaming a file with Node.js

Node.js

Node.js LogoLearn how to rename a file with Node.js, in ten lines of code

I was fleshing out a few ideas on a project today and found myself trying to figure out how to rename a file with Node.js. I have no doubt that there are better ways to go about this, but I thought I’d document my findings.

The first thing I realized is that the low-level nature of Node.js offers a great deal of power, but also with that power comes the need to handle all of the details. For me, this meant getting references to four things:

  1. The name of the old file
  2. The name of the new file
  3. The full path to the old file
  4. The full path to the new file (which does not exist yet)

The first two items are easy: I just had to provide a name for the old file, and decide what to call the new file. The last two items involved a bit more effort. I needed to do three things:

  1. Get the path of the folder that contains the old file
  2. Add a trailing slash to that path
  3. Set a permanent reference to the old file

So, to accomplish all of these tasks, I decided to use the filepath Node.js module.

Example # 1A

 Example # 1B

Example # 1A shows the contents of package.json. There is only one dependency: the filepath Node.js module.

In Example # 1B, I first set references to the file system module, as well as the filepath module. Next, I provided strings for the names of the old and new files. The filepath module is then used to get the path to the current folder; I set that to the fpFolder variable (adding a trailing slash to that string, which will be needed when we append file names to that string).

The variable fpFile is used as a permanent reference to the old file (this will come in handy for Example # 3.) Finally, I build the full file paths for the old and new files. After that, a couple of console.log statements to make sure all of this work is correct.

Example # 1C

Example # 1C shows the output of the two console.log statements. Each path will vary for you a bit, so I simply put “[YOUR LOCAL PATH TO]” for the folder structure that leads up to that file in the github repo that you cloned (see “How to Demo” below).

This example does not actually rename a file. So, now I will use the file system module to make that change.

How to Demo:

  • Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  • Navigate to: JavaScript/node-js/node-modules/fs/fs-rename
  • Execute the following command in a terminal prompt: npm install
  • Execute the following command in a terminal prompt: node filepath-1.js

Example # 2

In Example # 2, I use the file system module’s rename method to rename the file: “re-name-me.txt.” This method takes three arguments: a path to the old file, a path to the new file and a callback. The callback takes one argument: an error object. Inside of the callback, I check for the error object, and then output the path of the new file. So now, follow the instructions below to see this code in action. After you execute the code, the file: “re-name-me.txt” will be renamed to: “ive-been-renamed.txt.”

In order to rename Example # 2 again, you’ll need to manually rename the file: “ive-been-renamed.txt” back to: “re-name-me.txt”. After a few times back and forth, this got pretty tedious and I started to think that there must be a way to toggle the file back and forth. Meaning: If the file has been renamed, change it back to the original name, and so forth.

How to Demo:

  • Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  • Navigate to: JavaScript/node-js/node-modules/fs/fs-rename
  • Execute the following command in a terminal prompt: npm install
  • Execute the following command in a terminal prompt: node filepath-2.js

Example # 3A

In Example # 3A, I use the ternary operator when setting the final path for the old and new files. In each case, I check to see if the old file exists, and then depending on the answer, I set each path accordingly.

Example # 3B

Example # 3B is the full code for the final version of this file. I combined all var statements and cleaned up the code a bit. When you follow the instructions below, you’ll see that you can keep executing node filepath-2.js over and over, and the text file will toggle between the old name and the new name.

How to Demo:

  • Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  • Navigate to: JavaScript/node-js/node-modules/fs/fs-rename
  • Execute the following command in a terminal prompt: npm install
  • Execute the following command in a terminal prompt: node filepath-2.js

Summary

As I mentioned, there are probably a number of ways to do this that are more efficient. Everything I detailed here was the result of a few minutes with Google. Hopefully, this article either got you where you needed to go, or pointed you in the right direction.

Helpful Links for Renaming a File with Node.js

http://nodejs.org/api/fs.html

https://www.npmjs.com/package/filepath

Getting started with the filepath Node.js module

Node.js

Node.js LogoWhen you need to reference and work with the local file system in your Node.js program, the filepath module is quite a handy tool.

Even if your Node.js program is a web-server of some sort, working with the local file system is somewhat inevitable. While Node.js does provide low-level file system access (see the Node.js fs module), abstraction is always helpful, particularly when dealing with absolute paths.

The filepath Node.js module is a very helpful utility for simple access to file paths. You’ll need only a package.json file with this module as a dependency, an “npm install” command, and then you are up and running. This article provides a quick introduction to a few of the most common methods.

Example # 1A

Example # 1B:

In Example # 1, we first create the FP variable, which references the filepath module. Then we create the path variable, which holds the return value of the FP object’s newPath method. And finally, we output the path in the console. Example # 1B shows the terminal output when we use console.log to view the path variable. This path will vary for each user so I simply put “[YOUR LOCAL PATH TO]” for the folder structure that leads up to that file in the github repo that you cloned (see “How to Demo” below).

How to Demo:

  1. Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  2. Navigate to: JavaScript/node-js/filepath
  3. Execute the following command in a terminal prompt: node filepath-1.js

Example # 2

Example # 2 demonstrates the list method. The only real difference between this code and Example # 1, is the new variable “files”, which receives the value of the list method, when called on our path variable. The files variable ends up as an array. Each element in the array is an object whose “path” property is a string that points to a file in the current directory.

How to Demo:

  1. Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  2. Navigate to: JavaScript/node-js/filepath
  3. Execute the following command in a terminal prompt: node filepath-2.js

Example # 3A

Example # 3B

Example # 3C

Example # 3D

In Example # 3A, we see the recurse method in action. Just as the name implies, the recurse method will recursively list all of the files in the current directory. As a result, if one of those files is a folder, then it will list all of the files in that folder, and so on. This method differs from the previous two examples in that it takes a callback. The callback is a bit like a forEach call; it iterates over all of the files or folders in the path, and calls the callback for each one. Inside of the callback, the path variable is the current path being iterated over.

Example # 3C is the output from the code in Example # 3A.

In Example # 3C, we use the toString() method of the path object so that instead of a bunch of objects that we would need to handle, we just get the values we are after; the string representation of the path to that file or folder.

Example # 3D is the output from the code in Example # 3C.

How to Demo:

  1. Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  2. Navigate to: JavaScript/node-js/filepath
  3. Execute the following command in a terminal prompt: node filepath-3.js

Summary

The filepath Node.js module has much more to offer than was demonstrated here. Hopefully, this article has demonstrated how easy it is to get started with filepath.

Helpful Links for the filepath Node.js module

https://www.npmjs.com/package/filepath

http://nodejs.org/api/fs.html

Getting started with the uglify-js Node.js module

Node.js

Node.js LogoLearn how to easily implement minification and file concatenation right in your Node.js program.

There is no doubt that tools such as grunt and gulp provide powerful front-end tooling, particularly for large-scale applications. But in some cases, you may want to “roll your own”. If you want to minify and / or concatenate files from your Node.js application, the uglify-js module offers a simple syntax yet plenty of muscle-power.

So, if you want to get serious, a quick package.json file and “npm install” command are all you need to get started. Once these two tasks are taken care of, you can minify one or more files, and concatenate the output to a new file. In this article, I will show you how to do just that in less than 20 lines of code.

Example # 1A

Example # 1B

Example # 1C

 Example # 1D

In Example # 1A we have the contents of the file: package.json. This tells npm that our program depends on the “uglify-js” module. Examples # 1B, 1C and 1D are the contents of the files we will “uglify”. The actual code has no significance. We just want to have a reference so that once we have uglified the files, we can see the difference in the output.

Example # 2A

 Example # 2B

In Example # 2A, we minify the file: file-1.js. In this case, the minified code is simply shown in the console. Example # 2B shows the minified code. It’s hard to imagine a case where we would want to minify code, but only show the result in a terminal window. Realistically, we need to write the output of the minified file to a new file.

How to Demo:

  1. Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  2. Navigate to: JavaScript/node-js/uglify-js
  3. Execute the following command in a terminal prompt: npm install
  4. Execute the following command in a terminal prompt: node uglify-1.js

Example # 3

 

In Example # 3, we have the content of uglify-2.js. Here, we’ve moved things into a more real-world context; we save the result of the minification to a physical file. Now notice that after you execute node uglify-2.js in your terminal, there is a new file named: output.min.js, which is a minified version of file-1.js.

The first change we made was to add a reference to the “fs” module, which provides access to the file system in Node.js. The console.log statement was left in, just so you can still see the output in the console. Below that, we call the writeFile method of the fs object. We pass it three arguments:

  1. the name of the file that will contain the result of the minification process (i.e. the minified code)
  2. the content for that file (i.e. the minified code), and
  3. a callback. The callback takes one argument: an error object. In the callback, we check to see if there was an error, and if not, we send a success message to the console.

In this Example, the callback is optional as it has nothing to do with the minification process and only provides messaging as to the status of the minification attempt.

Although Example # 3 is more of a real-world context than Example # 2, it is still a bit unrealistic as we only minify one file. In a typical production workflow, one would likely want to minify and concatenate more than one file.

How to Demo:

  1. Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  2. Navigate to: JavaScript/node-js/uglify-js
  3. Execute the following command in a terminal prompt: npm install
  4. Execute the following command in a terminal prompt: node uglify-2.js

Example # 4

Example # 4 shows the contents of uglify-3.js. The only change we have made is in the call to UglifyJS.minify. Instead of passing it a string, we pass an array. Each element in the array is a path to a file we want to minimize and the concatenate to the output file. In this case all of the files are in the same folder as our program, so there is no folder structure (i.e. just the file names). You can take the same exact steps to demo this example, and when you do, you will see that the file output.min.js contains the minified code of file-1.js, file-2.js and file-3.js.

Summary

The uglify-js offers a ton of options, parameters and features. For this article, I wanted to demonstrate how easy it is to set up and use this Node,js module. But if you want to understand the true power of uglify-js, there is a ton of documentation available online. Hopefully this article got you to first base!

Helpful Links for the uglify-js Node.js module

https://www.npmjs.com/package/uglify-js

https://www.npmjs.com/package/uglify-js#api-reference

http://lisperator.net/uglifyjs/

Node.js Basics – Command-Line Arguments

Node.js

Node.js LogoLearn how to access command-line arguments passed to your node.js file

When you use Node.js as a command-line tool, you most likely type the following into your terminal application: node some-file.js. Once your application grows to even a modest level of complexity or sophistication, you will probably want to accept arguments on the command line. So, in this article, I will explain the basics of how to gain access to the arguments that are passed to your node application on the command line.

The key to accessing command-line arguments lies in the argv property of the process global. The process object is a global object, which means that you can access it from anywhere in your program. The argv property is an array that contains all arguments provided when you executed your program on the command line.

Example # 1A

Example # 1B

Example # 1C


In Example # 1A, we have the contents of the file: showArgs-1.js. Then this line simply inspects the argv property of the process objet. The output will go straight to the terminal window the moment you execute your program.

In Example # 1B, we have the actual command you use to execute the program; first node (the executable for Node.js), and then the name of the file we want to execute: showArgs-1.js.

Example # 1C shows the output of this program. So, in the console, you will see an array. The first element of that array is node, and the second element is the path to the program. This path will vary for each user so I simply put “[PATH TO]” for the folder structure that leads up to that file in the github repo that you cloned (see “How to Demo” below).

How to Demo:

  1. Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  2. Navigate to: node/cli-arguments-basics
  3. Execute the following command in a terminal prompt: node showArgs-1.js

Example # 2A

Example # 2B

 Example # 2C


In Example # 2B, you’ll see that we passed some additional arguments to our program: “arg2”, “arg3” and “arg4”. I used the numbers 2, 3, and 4 because they make more sense, due to the 0-based nature of this arguments array. Remember: arguments 0 and 1 are node and the file that contains your program, so the additional arguments here are indexed as 2, 3 and 4.

Since we know that the first two arguments will be the same every time, let’s try to get rid of those first two arguments.

How to Demo:

  1. Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  2. Navigate to: node/cli-arguments-basics
  3. Execute the following command in a terminal prompt: node showArgs-1.js arg2 arg3 arg4

Example # 3A

 Example # 3B

 Example # 3C


In Example # 3A, we use the Array.prototype.slice method to remove the first two elements of the arguments array. Specifically, process.argv is an array, so we use its slice method to remove the first two elements of the array, and assign that return value to the variable args. We then inspect the args array using a console.dir statement.

In Example # 3B, we execute our program, passing three arguments: “arg0”, “arg1” and “arg2’. Example # 3C shows the output, which is an array containing only the arguments that we passed to the program.

How to Demo:

  1. Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  2. Navigate to: node/cli-arguments-basics
  3. Execute the following command in a terminal prompt: node showArgs-2.js arg0 arg1 arg2

Example # 4A

Example # 4B

Example # 4C


In Example # 4A, we make the output a bit easier on the eyes. Instead of a simple console.dir statement, we use the forEach method of the args array for iteration. Inside of each iteration, we use a console.log statement to output a more human-friendly message that shows the argument that is contained in that element.

How to Demo: 

  1. Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  2. Navigate to: node/cli-arguments-basics
  3. Execute the following command in a terminal prompt: node showArgs-3.js arg0 arg1 arg2

Summary

While the examples here were quite basic, they are perfectly valid and could be used in your code. There are Node.js modules that provide powerful abstractions when it comes to Node.js command-line arguments. The one I have probably heard about the most is minimist. So, if you are writing an application that has even a moderate level of complexity, you will likely want to use something like minimist. That said, understanding how things work on a low level is always a good idea. Hopefully, this article has provided a helpful introduction to this concept.

Helpful Links for Node.js Command-Line Basics

http://nodejs.org/api/process.html

http://nodejs.org/api/process.html#process_process_argv

https://www.npmjs.com/package/minimist

When did Walmart become so hip?

Node.js

WalmartLabs LogoWalmartLabs is doing some very cool things with Node.js. When the heck did all this happen?

Did you know that Walmart supports nearly 30 open-source modules, most of which are used in production, or that they created their own “private npm” to prevent hacks? Nope, neither did I.

I must admit, Walmart is not a company that comes to mind when I think of leading-edge web development. But like many older large-scale organizations, they have realized that they need to better leverage technology, or lose market share to companies such as Amazon. Well, it sure seems like they are very focused.

I had never heard of WalmartLabs until very recently. I kept noticing that in my Node.js-specific web-surfing, their name started to pop-up. So I took a look, and what I found was impressive.

It seems to me that Walmart has been hiring top engineering talent, and putting them to good use. They are doing some pretty cool stuff with Node and making serious contributions to the open-source community. Below are two videos I have recently viewed that are very much worth checking out. If you are interested in Node.js in the enterprise, these folks have a lot to share.

Cheerios and Fruit Loops: Frontend Node At Walmart by Kevin Decker

Walmart Senior Mobile Web Architect Kevin Decker talks about how Walmart threw out their legacy Java stack, and the many challenges of SPAs. In particular, he provides an in-depth discussion on pre-caching with Phantom.js, Thorax, the Cheerios Library, Fruit Loops (“a sugary cheerios) and Contextify.

Node.js at Walmart

Walmart Sr. Architect Eran Hammer talks about the server stack that they built on smartOs, hapi – their open-source Node framework, and custom “server partials”. He also discusses their use of Node as an orchestration layer, and some of the challenges of migrating from their legacy Java back-end.

http://nodejs.org/video/

Interesting Links related to WallmarLabs

http://www.walmartlabs.com/

https://github.com/walmartlabs

http://en.wikipedia.org/wiki/@WalmartLabs

http://www.walmartlabs.com/the-blog/

http://techcrunch.com/tag/walmartlabs/

Book Review: Node for Front-End Developers, by Garann Means

Node.js

Node for Front-End Developers, by Garann Means - CoverIf you are just getting started with server-side JavaScript, “Node for Front-End Developers” offers a fast, high-quality introduction.

The ubiquity of front-end JavaScript is undeniable. Not only has the appetite for web-based content increased dramatically, but so has the appetite for sophisticated user interfaces. More and more, visitors expect web-based content to offer complex interaction and high-performance. The explosion of mobile device use has only exacerbated this dynamic. Ryan Dahl’s Node.js turned the whole concept of JavaScript on its head by providing an open-source tool that allows the language to be leveraged on the server-side, significantly expanding the potential of this language.

Node for Front-End Developers, by Garann Means is a fast introduction to this incredibly powerful technology. The concept of creating a web-server provides a door through which clear and concise explanations present the basic concepts of server-side JavaScript. I found it particularly helpful that for such a short book, topics such as the query string, post data, path data routing, asynchronous events, templating, databases and MVC are well handled.

The book’s length is deceptive; readers will find a wealth of useful information here. While each topic represents a thread that deserves further reading, anyone who is new to Node.js will find Ms. Means’ introduction helpful. Her writing style is both relaxed and professional. From using NPM to install modules, to real-time communication with WebSockets, Node for Front-End Developers offers a range that is just enough to excite the reader, yet never too much detail. Any of the examples can be typed into your favorite text editor and fired-up with minimal effort. This is critical when delving into a new topic, and makes your introduction to Node.js disarming and fun.

  • Title: Node for Front-End Developers
  • Author: Garann Means
  • Publisher: O’Reilly Media
  • Date Published: February 7, 2012
  • Language: English
  • ISBN-10: 1449318835
  • ISBN-13: 978-1449318833

JavaScript Concatenation and Minification with the Grunt.js Task Runer

Node.js

Grunt.js LogoCombine multiple JS files into one using JavaScript, Node.js and Grunt.

In the article: “Getting Started with Grunt: The JavaScript Task Runner,” we covered the bare-bones basics of how to set up and use Grunt, to automate and manage your JavaScript build task. In that article, we used a Grunt plugin called “grunt-contrib-uglify” to minify and obfuscate our JS code. But while minification is a common task for any build system, file concatenation is also a technique used to minimize the number of http requests. This helps to improve web page performance. In this article, we will look at the Grunt plugin: grunt-contrib-concat. It exposes a number of very useful methods for simple, to advanced file concatenation.

Example # 1

In Example # 1, we have the contents of “package.json“. Naturally, Grunt is listed as a dependency, but there is a reference to the “grunt-contrib-concat” plugin as well.

The Project File Structure – And Files to be Concatenated

 

The Empty Distribution Folder
The Empty Distribution Folder

 

 

Example # 2

In Example # 2, we have the JavaScript code that will set up and run the Grunt tasks. There are two sections to this code: the call to the grunt.initConfig() method, and the two commands that load the plugin and register the task.

Grunt Init configuration

The method: initConfig belongs to the grunt object. For this article, we are concatenating three JavaScript files. As you can see, the sole property to the object-literal passed to the initConfig method is concat. Its value is another object with two properties: “options” and “dist“. The “dist” property provides an object with critical details for this operation: the location of the source files (“src“). It also provides the location of the file to be created that contains all of the source files (“dist“). Note that the value of the “src” property is an array. This ensures that any number of files can be specified. The path is relative to the Gruntfile. In this example, the “options” property specifies the string that will be used to separate each concatenated file.

Loading the Grunt Plugin and Registering the task

Under the call to the grunt.initConfig() method, we load the “grunt-contrib-concat” plugin by passing it as a string to the grunt.loadNpmTasks method. This makes the plugin available to the rest of our code. Next, we register the “default” task with a call to the grunt.registerTask() method. The first argument is the name of the task (in this case it is “default“). The second argument is an array containing one or more named tasks.

That is all we need to run our tasks. So now, in the terminal, we simply type: “grunt“. After grunt has completed running, you should see a new file in the “dist” folder named: “allScripts.js.

The Built File
The Built File

Summary

In this article, we talked about JavaScript minification using Grunt. We discussed configuring the project dependencies in the file: package.json, which is needed in order for our application to work properly. We also reviewed the file: Gruntfile.js, which is where we did set configuration properties for the build, and then run the tasks. There is plenty to dive into on the subject of minification with Grunt. This article provided the most basic elements of how to set, configure and run a task that combines multiple JavaScript files into one.

Helpful Links for: Grunt.js file concatenation

http://stackoverflow.com/questions/16377674/using-grunt-to-concatenate-all-vendor-javascript-files

https://github.com/gruntjs/grunt-contrib-concat

http://stackoverflow.com/questions/12722855/using-grunt-concat-how-would-i-automate-the-concatenation-of-the-same-file-to-m

Getting Started with Grunt: The JavaScript Task Runner

Node.js

Grunt.js LogoAutomate and manage your JavaScript build tasks with JavaScript, Node.js and Grunt.

Today, even the most straightforward web-based application will involve JavaScript, and chances are that JS code will not be too simple. In-fact, it is often inevitable that your JavaScript could start to grow over time and / or span multiple files. When these dynamics are in-play you’ll find yourself spending more and more time having to organize and test your code. Regardless of the details, this kind of maintenance quickly becomes tedious. I think many will agree that when humans have to perform repetitive / tedious tasks, the chance of error increases. Fortunately, this is the exact kind of work that a computer is thrilled to do, and do well. Using a built tool to manage your JavaScript build tasks is not only efficient, but also a smart way to go.

Grunt is promoted as a “Task Runner.” Semantics aside, it is an aptly named tool that provides a significant level of flexibility and power, when it comes to managing and automating your build tasks.

In this article, I will intentionally “scratch the surface.” While there is a great deal of depth to Grunt, getting started is a critical first step. My focus here is to provide you with information that simply enables you to set the minimum configuration required for your first (albeit simple) automated task with Grunt.

Prerequisites

In order to use Grunt, you’ll need Node.js and the The Grunt CLI (Command-Line Interface). Installing Node is beyond the scope of this article. If you have not installed Node yet, you can start here: http://nodejs.org/

The Grunt Command-Line Interface

The Grunt CLI is used to launch Grunt. While this may sound a bit confusing, installing the Grunt CLI does not actually install the Grunt task runner. The CLI is used to launch whatever version of Grunt is defined in your package.json file (more on that file later). The advantage of this is that your projects can all use different versions of Grunt without concern for conflicts. The CLI does not care, it simply launches whatever version your project uses.

(You might need to use the sudo command when executing npm install, depending on your account priviliges.)

Note that in the example above the “-g” flag is used. This is recommended. It installs the module globally so that it can be executed from any folder on your machine, and need not ever be installed again.

package.json

Once you have installed the grunt-cli globally, you can put your project together. The first step is to create a package.json file. The package.json file is a feature of node.js. You can learn more about it here: https://npmjs.org/doc/json.html

In this example, we have named this project: “Grunt-Test-1”, and given it a version #. Both of these values are arbitrary, and completely up to you. But keep in mind that the “name” property is potentially used by Grunt in some cases, so choose a name that makes sense. The “devDependencies” property is an object that whos whose keys contain node modules that your project depends on. The value for each key is the exact (or minimum) version number required for that module.

The Project File Structure

Image # 1: The project’s file structure

Once your package.json file is ready, simply run the following command: “npm install”. Node will refer to your package.json file in order to understand what it needs to install so that your project works correctly. Keep in mind that the “installation” is simply a download that is local to this folder. Its is not “installed” on your computer, like a native / compiled application.

Example # 1

For this example, we have only two dependencies: grunt, and grunt-contrib-uglify. The “grunt” requirement is critical because we want to use Grunt. Everything else is optional. But if grunt is the only dependency, then there won’t be too much we can do. Grunt plugins allow us to define tasks that can be managed by Grunt. The “grunt-contrib-uglify” plugin provides JavaScript minification and obfuscation. For this article, we will simply minify a JavaScript file and save it with a new name.

Once you have created your package.json file, you can install all required modules with two easy words: “npm install”. Entering that command in your terminal will allow you to leverage the Node Package Manager, otherwise known as: “npm” (make sure you are in the root folder of your project). The Node Package Manager will look for a file named package.json in the current folder, and read the devDependencies object. For each property of the “devDependencies” object, npm will download whatever file is specified by the version # you have provided as a value for that property.

One of the ways in which this approach is so helpful is that it negates the need to commit large files to your version control. Node modules are often a combination of text files (e.g. “.js”, “package.json”, “README”, “Rakefile”, etc.) and executables. Sometimes a combination of just a few dependencies can add several megabytes to your project, if not more. Adding this kind of weight to your code repository not only eats up disk space, but it is somewhat unintuitive because version control systems are really meant to store and manage text files that you (or your teammates) will change over time. Node.js modules are managed by the contributors of those modules, and you probably won’t need (or want) to change their source code. So, as long as your project contains the package.json file, whenever you (or anyone on your team) check out that repo, you can run “npm install” to the exact versions of the modules you need.

The Gruntfile

The Gruntfile is the JavaScript code you create to leverage the power of Grunt. Grunt always looks for a file named: “Gruntfile.js” in the root of your project. If this file does not exist, then all bets are off.

Inside of Gruntfile.js, all of your code will be wrapped in one function: module.exports.

Example # 2

In Example # 2, we have defined the “exports” method of the “module” object. Right now it does nothing, but all of your grunt code will go inside of this function.

Example # 3

In Example # 3, we make a call to the grunt.initConfig method. This method takes an object as its sole argument. In this object, we set properties that provide the details Grunt needs in order to work the way we want. Here we set one property: “pkg”. In order to set the value for that property, we tell grunt to read the contents of our “package.json” file. The main reason for this is that we can use the metadata that is stored in package.json, and use it in our tasks. The two delimiters: <% %> can be used to reference any configuration properties. For example: <%= pkg.name %> would display the name that you have given your project. In this case it is: “Grunt-Test-1”, but we could also access the version number of our project in this manner: <%= pkg.version %>.

Since we will be using the “grunt-contrib-uglify” plugin, lets update Gruntfile.js so that it is configured to leverage this plugin.

The Empty Build Folder
The Empty Build Folder

Image # 2: The empty build folder

Example # 4:

In Example # 4, we have added a new property to the config object: “uglify”, which provides the detail that the “grunt-contrib-uglify” plugin needs. This property is an object which, in-turn, has two properties that are objects. As the name suggests, the “options” property contains options for the “grunt-contrib-uglify” plugin. The “build” property is an object that provides two properties that are critical to the “grunt-contrib-uglify” plugin:

  • src: The location of the file to “uglify”
  • dest: The location where the “uglified” file will be placed

While we have provided some important details, running Grunt at this point will produce no results because there are no tasks specified. Let’s take care of that next:

Example # 5:

In Example # 5, we have made two changes: We load the “grunt-contrib-uglify” plugin, and then we register the default task, which in this case is: “uglify”. Notice that the “uglify” task is passed to the registerTask() method, as the sole element of an array. This array can contain as many tasks as you like, always represented as a string. The purpose of the “default” task is to let Grunt know that outside of any other named tasks that are registered, this is the list of tasks that should be executed.

Now that we have provided the minimum details needed, we can run Grunt by simply typing “grunt” in the terminal. Once Grunt completes, you should see a new file in the “build” folder that is in the root of your project. That file should be named as per the details provided in the “uglify” section of the object passed to the grunt.initConfig() method. You’ll see that file has been “uglified,” meaning that it has been minified and obfuscated. The end result should be a significant reduction in file size.

The Built File
The Built File

Image # 3: The  built file

Summary

The example used in this article was very basic (about as basic as you can get!). My goal was to provide the critical base information needed in order to understand, setup and use Grunt for the first time. There is a great deal of power available from Grunt. While it may not fit everyone’s needs, it can, in many cases, provide a robust and actively maintained open-source framework with which to create simple or complex build processes.

Helpful Links for Getting Started with Grunt

http://gruntjs.com/

http://gruntjs.com/getting-started

http://net.tutsplus.com/tutorials/javascript-ajax/meeting-grunt-the-build-tool-for-javascript/

http://blog.mattbailey.co/post/45519082789/a-beginners-guide-to-grunt