How to test HTTP POST with the Node.js request Module

Node.js

Node.js Logo - test HTTP POSTTesting HTTP POST requests is usually tedious. Bit with a few lines of JavaScript, you can spin-up your own HTTP POST testing tool.

In web development, GET and POST requests are quite common. GET requests are the ones more frequently seen, and in fact, when you load most web pages, the majority of the requests that make up what you see in the page are GET requests. For example, you request the initial HTML file, CSS files, JavaScript files and images. But sometimes, you need to make a POST request.

Making a GET request is easy, as is testing one. Testing a POST request is not always so simple, though, because the HTTP request body must include the data you want to send. One approach is to create a simple HTML page with a form. The problem here is that you need to create an input element for each data property that you want to send in the POST request, which can be tedious for a simple test. But then there’s Node.js, which can be leveraged to solve this problem.

In this article, we will see how a small JavaScript file can make an HTTP POST request. Now this approach may not be appropriate for use in a production application, but the idea behind this article is to point out that any time you need to test a POST endpoint, you can set up a quick test using Node.js.

Get the example code from GitHub

If you clone this repo: github.com/kevinchisholm/video-code-examples/tree/master/node/testing-http-post-with-request-module, you can clone the example code locally and edit the code yourself.

package.json

The package.json for this project contains references to the modules needed. We’re using the request module, the body-parser module, and the express module.

Example # 1 – The Web Server

In Example # 1, we have the server code. (Creating the server code is not the focus of this article, but it’s still good to review.) We need the express module and the body-parser module, and once we’ve set the references to those, we set up the POST route. So, when the user sends an HTTP POST request to /form, our code will handle this request. The requestAsJson variable allows us to set up the round-trip – that is, the exact same data from the POST request that we return to the user as JSON. We then set the Content-Type header to be application/json so that the HTTP header will be correct. Note the “log the output” comment; this is just for demonstration purposes. We then send the response using the res.end method.

Example # 2 – Testing the POST Request

In Example # 2, we have the test client, which is the focus of the article. We want an easy way to test POST requests, so instead of mocking up an HTML page with a form, we can use the file test-post.js to test an HTTP POST request. We set a reference to the request module, and no other module is needed in this file.

The postData variable is an object containing the data for the HTTP POST request. The postConfig variable contains the URL for the HTTP POST request, and a reference to the postData variable. The postSuccessHandler variable is a success handler for the HTTP POST request. Inside of that success handler, you can see a console.log statement, which completes the proof of concept. Whatever data sent for the HTTP POST request should be output in that console.log statement.

<h2>How to test the example code</h2>

Open two terminal windows (terminal A and terminal B), and make sure that you are in the root of the repository folder. In terminal A, execute this command: node post-server.js. In terminal B, execute this command: node test-post.js. In terminal A, you should see the message: The POST data received was XXX. In terminal A, you should see the message: JSON response from the server: XXX. (In each case, XXX represents the data from the HTTP POST request).

NOTE: Go ahead and change the properties of the postData object. You can create more properties if you wish. No matter what you do, you can see the data that you set in that object in the two console.log statements.

Fat Arrow Function Basics – Node Modules

Node.js

JavaScript LogoJavaScript Fat Arrow Functions solve the “this” problem by maintaining a reference to the object to which a method belongs. This is the case even with nested functions.

One of the most popular aspects of JavaScript is the fact that functions are first-class citizens. So, this aspect of the ECMAScript specification provides a great deal of power. Now when a function is a property of an object, it is also considered a method. That is, it is a method of that object. And inside of a method, the JavaScript “this” keyword is important, because it allows us to access the object to which the method belongs, as well as its other properties.

Now, when nesting functions, the JavaScript “this” keyword, one of the more frustrating aspects of the language, can be a bit tricky to deal with. So, in this article, I will discuss this very problem and how to solve it using fat arrow functions. If you’d like to run the code examples locally on your computer, clone the following github repository: Using fat arrow functions in your Node module.

(Instructions on how to run the code are available in the Github page.)

One important note about the code examples: the title of this article references “…Node Modules” to keep things simple, so I did not use a node module for the context of the code examples. Most Node applications keep the main file code minimal. Taking a modular approach is almost always a best practice, but for this article, I have put the code in the main JavaScript file.

The problem with “this” – Example # 1

Run Example # 1 in your terminal with the following command: node example-1.js. The result of this is: “THE MESSAGE IS: undefined“.

We have created a tools object in Example # 1, and that name is “tools“, which is arbitrary. It could have been any name, we just need an object to work with. The “tools” object has a “message” property, and there is also a method named “asyncTask“. The asyncTask method simulates an asynchronous task by using the setTimeout method. There is a reference to the JavaScript “this” keyword inside of the anonymous function passed to the setTimeout method. Now here’s where it gets a little dicey: the anonymous function passed to the setTimeout method is not executed in the context of the “tools” object, and therein lies the problem. The resulting console.log message is: “THE MESSAGE IS: undefined“.

So, we need a way to reference the “tools” object inside of the anonymous function that we passed to the setTimeout method. Well, the best approach is still to reference the “this” keyword. A common and popular approach in the past has been to set a reference to “this” before calling the setTimout method. For example: “var me = this;”. Okay, so while that is still a possible technique, there now is a far more elegant approach.

Fat arrow functions solve the “this” problem – Example # 2

Run Example # 2 in your terminal with the following command: node example-2.js. The result of this is: “THE MESSAGE IS: Hello from this.message!”

We made a small change in Example # 2. We converted the anonymous function passed to the setTimeout method to a fat arrow function. Fortunately, this action solved our problem. One of the advantages of fat arrow functions is that they preserve the meaning of the JavaScript “this” keyword. Because of this, when we reference this.message we no longer have an error, and we also see the expected message in the console.

Fat Arrow Function – One Argument – Example # 3A

Fat Arrow Function – Multiple Arguments – Example # 3B

A few things to keep in mind:

  • In Example # 2, the fat arrow function takes no arguments, but, it still has a pair of opening and closing parentheses. This is because when a fat arrow function takes no arguments, you must include a pair of opening and closing parentheses.
  • In Example # 3A, there are no parentheses in the fat arrow function. This is because when there is one argument, you do not need to include parentheses.
  • In Example # 3B, there are two arguments contained inside of parentheses. This is because when there is more than one argument, you must include parentheses.

Summary

In this article we saw that fat arrow functions solve the “this” problem because they provide access to the object to which the containing function belongs, and you can access that object at all times by using the “this” keyword. And even when nesting fat arrow functions, the “this” reference is preserved, eliminating the need to set a temporary reference to “this”. Just keep in mind the importance of how the syntax can differ, depending on the number of arguments that the fat arrow function takes. In other words, with zero or multiple arguments, parentheses are required, and with only one argument parentheses are not required. Pretty simple, once you get used to it.

Node.js Templating with EJS – Basics

Node.js Templating

JavaScript LogoEJS Makes Templating in your Node.js application a breeze. Just supply a template string or .ejs file and some data.

The moniker says it all: “Effective JavaScript templating.” If you haven’t already discovered it, you’ll soon find that as front-end web developers have been transitioning to more of a full-stack role, templating has quickly become an important topic. In other words, this is no longer an unusual front-end task for JavaScript developers. And when working with Node.js, EJS has become the standard for server-side templating.

In this article, I will cover the bare-bones steps needed to get up and running with EJS, and in doing so, I’ll show you how to render data in an EJS template. First, I’ll explain the vanilla JavaScript approach. Then, we’ll move on to rendering your EJS template when using the Express.js framework. And finally, we’ll cover the syntax for EJS template code as well as how to use “if” logic in your template.

Now the power in EJS templates is the separation of concerns. Your data is defined (and possibly manipulated) in your server-side code, and your template simply declares what data will be rendered. This approach embraces the concept of “loose coupling”. With EJS templates, you can leverage that same “loose coupling” design pattern in your Node application. This scenario is, of course, fairly common to back-end developers, who have experience with languages such as Java, PHP, Python or .NET. For a front-end developer, however, this may be new territory. So, to illustrate, let’s take a look at some examples.

Example # 1-A

Example # 1-B: The Rendered HTML

In Example # 1 – A we first require the ejs module. This will be the case with every example, so I won’t cover that again. Just know that we need the ejs module in order to render our EJS templates, so we set a variable named “ejs” via require first. Next, we set the days variable; it’s just an array that contains the five days of the work week. Here, too, this will be the case in every example, so no need to cover this again. Just know that in each code example, there is a days variable – an array that contains the five days of the work week. We also set a variable named “http” which is an instance of the Node http module. We’ll need this in order to run our web server.

Okay, so let’s take a look at line # 3 in Example # 1. We’re using the ejs.render method here to create HTML that we will send to the user. The ejs.render method takes two arguments: a string template and the data for that template. In this case, our string template has the “<%=” and “%>” delimiters to indicate to EJS the start and end points for our template. And inside of those delimiters, we can write JavaScript code. So, let’s use the join() method of the days array to convert the array to a string. Then, inside of the execution of the http.createServer method, we’ll call the end method of the result object (i.e. res.end), passing the html variable to that method. And since the res.end() will send the response to the client and end the connection, the contents of our html variable will be sent to the user’s browser. Now, in Example # 1 – B, we have the HTML that is rendered in the user’s browser. This HTML happens to be very simple, and in fact, is not markup that we’d want to use in production. But what I wanted to demonstrate here is that rendering HTML in an EJS template is as simple as defining the template, then providing data to that template.

Example # 2-A: Setting the view engine for Express.js

Example # 2-B

Example # 2-C: The Rendered HTML

In Example # 2-A we’re leveraging the Express.js framework, so there’s a new require statement at the top of our code which sets the Express variable. On line # 3, we create the app variable which is an instance of the Express.js framework. And on line # 9, we use the app.set method to tell Express that we’re using EJS as our view engine. Note that this is required when leveraging EJS templates in your Express application. Now, on line # 12, we set up a handler for the “/” route. And inside that handler callback, we use the render method of the response object. This render method is available to use because of what we did on line # 9: using the app.set method to let Express know that EJS is our view engine. Okay, so let’s go back to line # 13, where we’ll pass two arguments to the render method: the string “example-2” and the data that our EJS will consume.

Now, you may be scratching your head as to what the first argument in “Example # 2-A” means. Well, it’s important to note that when you leverage EJS as your view engine, Express.js assumes that you will have view templates. These view templates are text files with an “.ejs” extension. So, it’s also important to note that Express.js assumes that these files will be in a folder named “views” that resides in the same folder as the file that is currently being executed. You can specify a different folder for your views, but the default folder that Express will look for is “views”. And in the “views” folder, Express.js will look for a file named XXX.ejs, where “XXX” represents the string that you pass as the first argument to the render method. So in our example, we want to use a template that resides in the file: “views/example-2.ejs”.

Here in Example # 2-B, we have the contents of the file “views/example-2.ejs”. And in this template file, there are two locations for data; the title tag and the body tag. In the title tag, we have a binding for the headerTitle property. In other words: we’ve provided some data to the res.render() method on line # 13 of Example # 2-A. That data was an object literal, and it had a property named: “headerTitle”. So, on line # 3 of our “views/example-2.ejs” file, we’ve told the template to inject the value of the “headerTitle” property of the data object that was provided to it. And the same thing is happening in line # 6 of our “views/example-2.ejs” file. In other words, we’ve asked EJS to inject the value of the “welcomeMessage” property of the data that was provided to the template. And then in Example # 2-C, you see the HTML that is returned to the user’s browser as a result of our template in Example # 2 B. In this HTML, the “headerTitle” property binding is replaced by the actual value: “EJS Demo Page” and the “welcomeMessage” property binding is replaced by the actual value: “This message was rendered on the server.”

Now, Example # 3-A is very similar to Example # 2-A, except that the data we provide to the template is an array, instead of just an object literal. If you look at Example # 3-B, you’ll see that the way we bind to the data differs from example # 2-A. In example # 2-A, we bound to a single property: “welcomeMessage”, but here we are using a loop to iterate over each element in the “days” array. Specifically, we use the forEach() method of the “days” array and in each iteration of the callback function, we have access to a variable named “day”. Then we generate a list item and output the value of “day”. So, if you look at Example # 3-C, you’ll see the HTML that is rendered by the server and sent to the user’s browser. Voila! As expected, we have the HTML with the unordered list rendered with each day of the week (i.e. the “days” array).

Example # 4-A is virtually identical to Example # 3-A; the only difference is the value of the “welcomeMessage” property. Take a look at Example # 4-B. You’ll see that on line # 4, we have some custom CSS in a set of style tags. This will make more sense in a few minutes. Now look at line # 20. Here we are looping over the “days” array, just as we did in Example # 3-B. But on line # 22, we use a basic JavaScript “if” block, to determine if this is the fourth element in the array. We do that by using the index variable, which is the 2nd argument passed to the callback function that we provide to the days.forEach() method. So, if index is equal to 3, then we generate the following in our HTML: class=”selected”. What we are doing here is, we are telling our EJS template that the 4th element in the list (i.e. the element with the index of 3) should have the CSS class “selected”. So, in Example # 4-C, you can see in the rendered HTML that the fourth list item has class=”selected“. As a result, the CSS that we added at the top of the EJS template kicks-in and “Thursday” is dark red text with a yellow background.

Summary

So, in this article, you learned the most basic steps needed to leverage an EJS template in your Node.js application. You started by learning how to render data in an EJS template using vanilla JavaScript, and also when using the Express.js framework. Then we went on to cover how to bind a single data property, as well as how to iterate an array in your template. And finally, we wrapped it up by illustrating how to use “if” logic in your EJS template.

Now this article only scratched the surface of what is possible with EJS templates. My goal here was simply to provide the information needed to get up and running quickly, and to illustrate the most basic concepts so that you can dig in further on your own, because, believe me, there is plenty more to discover on this topic!

Node.js – What is the Difference Between response.send(), response.end() and response.write() ?

Express JS

JavaScript Logoresponse.send() sends the response and closes the connection, whereas with response.write() you can send multiple responses.

In this article, I will explain the difference between response.send(), response.end() and response.write(), and when to use each one. When you’re working with the Express.js framework, you’ll probably most frequently be sending a response to a user. This means that regardless of which HTTP verb you’re handling, you’ll pass a function as the handler for that endpoint. This function will receive two arguments: the request object and the response object. The response object has a send() method, an end() method and a write() method, and in this article we’ll get to know the differences between them.

So, let’s start with the main issue, which is that the response.send() method is used to send the response to the server. Now this makes sense and in some cases, it’s actually the perfect tool. Problems can arise, though, if you’re not entirely sure what the response.send() method actually does. Well, in a nutshell, it does two things; it writes the response and also closes the connection. So, this seems like a win-win, right? Well, in some cases it is, but if you don’t want to close the connection on your first write, then the response.send() method may not be the right tool. When this happens, you’ll need to use a combination of response.write() and response.close(). So, let’s take a look at a few examples, to see just how this works.

Get the example code from GitHub

If you clone this repo: github.com/kevinchisholm/video-code-examples/tree/master/node-express/response-send-end-write-difference, you can clone the example code locally and edit the code yourself.

Trying to use the response.send method more than once per request – Example # 1

Run Example # 1 in your terminal with the following command: node example-1.js, then point your browser to: http://localhost:5000/. Now you’ll see this: “This is the response #: 1“. There are two problems here, however, the first of which is that any responses after the first one are never sent. This is because the send method of the Express.js response object ends the response process. As a result, the user never sees the messages “This is the response #: 2” or “This is the response #: 3”, and so forth.

The second problem is that the send method of the Express response object sets the Content-Type header, which is an automatic action. So, on the first iteration of the for-loop, the Content-Type header is set (i.e. “This is the response #: 1”). Then on the next iteration of the for-loop, the Content-Type header is set again because once more, we are using the response.send() method (i.e. “This is the response #: 2). But, we have already set the Content-Type header in the first iteration of the for-loop.
Because of this, the send method will throw this error: “Error: Can’t set headers after they are sent”. So, our application is essentially broken, but we don’t want users to have an error in their consoles. And more importantly; our back-end logic is not working correctly.

Using the result.write method – Example # 2

So, using the result.write method run Example # 2 in your terminal with the following command: node example-2.js. Now point your browser to: http://localhost:5000/. As you can see, there is still a problem with our code. Depending on your browser, either you will see only the first message or you will see none of them. This is because the response has not been completed. So, I’ll just mention here, that not every browser handles this case the same, which is the reason why you may see one message, all of the messages or none of them. But you should see that the request is “hanging” as your browser will stay in that “loading” state.

So, open your developer tools (e.g. FireBug or Chrome Dev Tools), and then look at the network tab. You’ll see that all five responses did, in fact, come back to the client. The problem is, the browser is waiting for more responses.
At some point, the request should time out and you can see all messages in the browser. This behavior can vary between browsers, but it is not the correct experience.

result.end fixes the problem – Example # 3

Run Example # 3 in your terminal with the following command: node example-3.js, then point your browser to: http://localhost:5000/. You will now see all of the messages in the browser, which means that here, in Example # 3, the problem has been fixed. We see all of the messages generated by the for-loop and the response completes successfully with no console errors. So, we’ve solved the problem by using a combination of of response.write() and response.close().

First we set the Content-Type header, just to get that task out of the way. Then, in each iteration of the for-loop, we used response.write() to send a message back to the client. But since response.write() does not set any headers or close the connection, so we were free to call response.write(), to send another response to the client. And once the for-loop was completed, we used the result.end() method to end the response process (i.e. we closed the connection). This said to the browser: “we’re done; go ahead and render the response now and don’t expect anything more from me.”

Summary

In this article, we learned about the difference between response.send(), response.end() and response.write(). During this discussion, we found that response.send() is quite helpful in that it sends the response and closes the connection. We saw that this becomes problematic, however, when we want to send more than one response to the client. But, fortunately, we discovered that this is easily solved by using a combination of response.write() and response.close(). We used response.write() to send more than one response, and then used response.end() to manually end the response process and close the HTTP connection. So, useful steps and easily solved problems.!

Node.js File Uploads with Multer

Node.js

Node.js LogoWhen it comes to the UI, file uploads are pretty simple. But on the back-end, there is some work to do. Multer is a Node.js module that simplifies this process.

Uploading a file is a common task for Web applications. Today, most Web pages leverage AJAX — which requires JavaScript — for a smoother user experience, but this can be accomplished using only HTML and zero JavaScript. The truth is, HTML-only file uploads have been possible for more than 20 years. I mention this to point out that in the browser, file uploads are simple and require only a small amount of HTML. This is only half of the equation, however, because a file upload is useless without some back-end code that can process the file. So, in this article I’ll show you how to process a file upload in your Node.js application. To simplify the back-end code needed to handle a file upload, we’ll leverage Multer, a Node.js middleware for handling multipart/form-data.

For this article, our working example code is a simple Node application that accepts a POST request with an “enctype” of “multipart/form-data.” When the user uploads a file, our back-end code will take the uploaded file and put it in the “uploads” folder, right within the root of the project folder. Nothing too fancy here, but it’s worth noting that the examples provide plenty of opportunities for copy/paste. What you do with these examples is up to you, but at least you’ll know how to process a file upload in your Node application.

index.html

In the above example (index.html), we have the HTML file for our application, so take a look at the form element. You’ll see that the “enctype” attribute is set to “multipart/form-data,” which means that we will send images in various formats. This is also important to keep in mind because Multer will only process this kind of file-upload. Note also the input element, which has a type attribute of “file.” This ensures that the browser will take care of implementing a file-upload interface. So, in other words, there will be a “Choose File” button, which allows the user to select a file from his or her hard drive. We certainly don’t need to put any effort into this; simply setting type=“file” takes care of all of it. There is also a name attribute for this input element. This attribute is required so that Multer understands how to handle the request. The Submit button will pass the form to the same exact URL because there is no “action” attribute, so the default behavior is: this is the form submitted to the same exact URL.

Configuring Multer – Example # 1

Example # 1 contains all of the code for our Node application. For this project, we leverage the Express framework. By using Express, we significantly reduce the amount of code needed. One of the most powerful features of Express is the ability to easily create middleware, which is a perfect context for Multer because it needs to intercept the HTTP request for us. The upload variable is used to provide configuration for Multer. In this case, for example, it lets Multer know that we want our uploaded files to be placed in the “uploads” folder. We’re using express.static in order to serve the HTML and CSS files to the user, so when the user goes to the “/” route, index.html and style.css are served by the Express framework.

Adding a Handler for the POST route

On Line # 11, we set up a handler for the POST route. If you’ve ever used the Express framework when building a Node application, this pattern should look familiar to you. But notice that the second argument passed to the app.get() method is upload.single(‘img’). We’re using the upload variable created earlier. The single() method takes a string as an argument, which is the “name” attribute of the form field containing the uploaded file. For demonstration purposes, we output req.file to the console so we can see information on the uploaded file. We call the send method of the response object, passing it some HTML, which simply informs the user that the upload was successful and allows that user to go back to the “/” route.

At this point, it would be a good idea to run the example code yourself, so just follow these steps:

  • git clone
  • git@github.com:kevinchisholm/video-code-examples.git
  • cd /node/file-uploads-with-multer/
  • npm install
  • node index
  • Open this URL in your browser: http://localhost:3000/

Now in your browser, click the “Choose File” button and browse your hard drive for a file to upload. Once you’ve selected a file, click the “Submit” button. You should see the message: “File upload succeeded.” Now, if you look in the “uploads” folder in the root of the project folder, you should see a file with a name similar to: “08e36ff4c9d3dc106e3a9fa2367797c9”.

So, we’ve made good progress here; our example code works and we’re able to upload a file. As you can see, though, the original name of the file is not preserved, and a GUID-like name is provided. This can be helpful in that users will not overwrite a file when uploading the same-named file more than once. The downside, however, is that there’s no connection between the original file name and the one provided. So, let’s fix that.

Show the Original File Name – Example # 2

Stop the Node application and then start it again, using the second example: node index2. Now, upload a file again.
You’ll see that the original file name is preserved.
In Example # 2, we accomplished this by leveraging multer.diskStorage(). When calling that method, we provided a configuration object. The destination property told multer.diskStorage() where the uploaded file will go, and the filename property provided a way for us to specify what the name of the uploaded file will be. This method receives a second argument called file, so we use the “originalname” property of this object to set the file name. But there’s a new problem now: the user can overwrite an uploaded file by uploading a file with the same name. So let’s fix that.

Create a Dynamic File Name – Example # 3

In Example # 3, we have expanded the anonymous function passed to the filename() method. What we’ve done here is use regular expressions to extract the name of the file with and without the extension. We use Date.now() to generate what is essentially a unique value, and we piece the new file name back together. As a result, the user can upload the exact same file over and over, but each uploaded file name will be unique. For example: original-file-name_123456.jpg. So, let’s just confirm this. Stop the Node application and then start it again, using the third example: node index3. Now, upload the same file over and over. You’ll see that each uploaded file has a unique name, but the original file name is included so that it’s easy to reference the actual file that was uploaded.

An Introduction to NPM Scripts

NPM

Node.js LogoLearn how to leverage npm scripts to create commands that, in turn, execute more than one other npm script command, allowing you to simplify your builds.

As the default package manager for Node.js, npm has seen a rise in popularity because JavaScript’s is just everywhere! This certainly makes sense – npm is well-designed, well documented, and makes Node.js development more seamless. I think most web developers would have a hard time imagining using Node.js without npm, but they often have to turn to technologies such as grunt and gulp to simplify local development and front-end tooling. But with npm scripts, you have an opportunity to move some of your front-end tooling and local development tasks away from third party tools. The beauty of this approach is that it allows you to simplify your setup.

In order to explain npm scripts, I have created a simple project that leverages Gulp. So, to run the code locally, clone the following git hub repository: Getting started with npm scripts.

Instructions on how to run the code are available in the Git hub page.

This project has four features:

  1. It compiles a coffeescript file to JavaScript.
  2. It compiles a SASS file to CSS.
  3. It uglifies a JavaScript file.
  4. It starts a Node.js web server.

This is a very simple example and it’s mostly Gulp compiling and minifying files. I chose this project because it requires some manual steps. Now, it’s possible to automate these tasks using Gulp, but what if you needed to switch to tools such as Grunt, or Broccoli.js? In such a case, your commands would change. For example, “gulp coffee” would become “grunt coffee”. While this is not fatal, it be nice if we could have a consistent set of commands. So the question is, how can we build our local development assets and start the Node.js server with one command? Also, how can we ensure that this one command never changes? Well, this is where npm scripts come in!

Project Folder Structure – Example # 1

In Example # 1, we have the folder structure for our project. There is an src folder that contains three sub folders:

  • The coffee folder has a coffeescript file.
  • The js folder has a JavaScript file.
  • The sass folder has a SASS file.

These three files are used by our build. The built versions of these files are placed in the build/css and build/js folders accordingly.

package.json (no npm scripts) – Example # 2

The package.json so far allows us to use Gulp. We’re using the gulp-coffee module to compile coffeescript, the gulp-sass module to compile SASS, and the gulp-uglify module to uglify JavaScript. So, we have the following commands available to us:

  • gulp sass: This command will compile the file src/sass/main.scss and create build/css/main.css
  • gulp coffee: This command will compile the file src/coffee/global.coffee and create build/js/global.js
  • gulp uglify: This command will uglify the file src/js/main.js and create build/js/main.js
  • node index.js: This command will start the Node.js web server on port # 3000.

You can run each command and it will work just fine, but the problem is that each time you change any of the files, you will want to build them again, and then restart the web server.

Adding npm scripts to package.json – Example # 3

In Example # 3, we have added a scripts object to package.json. Here is a breakdown of the script commands:

  • build:sass : This command is a shortcut to: gulp sass.
  • build:coffee : This command is a shortcut to: gulp coffee.
    build:js : This command is a shortcut to: gulp uglify.
    build : This command will execute the previous three commands. It executes three steps in one command.
    serve : This command is a shortcut to: node ./index.js (it starts the Node.js web server).
    start : This command builds all three files, and then starts the web server
    clean : This command will delete every the built file (these are files created by all previous commands).

What to expect when you run the example code locally

  • npm start – The build places the three built files in the build/css and build/js folders accordingly. And then, it starts the Node.js web server. You will see messages in your terminal that indicating these outcomes.
    npm run clean – npm deletes the three built files in the build/css and build/js folders. (This is helpful if you want to “start from scratch” when running the npm start command. This way you see the built files created each time.

Summary

This article is a basic introduction to, and high-level overview of npm scripts and its ability to create commands that, in turn, execute more than one other npm script command. As you can see, there’s a great deal of power here, and depending on your needs, they can streamline your front-end tooling process significantly. There’s much more detail available about npm scripts, and a great place to start is: https://docs.npmjs.com/misc/scripts. In the meantime, I hope that this article has provided you with enough information to get you up and running.

Web Scraping with Node and Cheerio.js

Node.js

Node.js LogoCheerio.js allows you to traverse the DOM of a web page that you fetch behind the scenes, and easily scrape that page.

There are security rules that limit the reach of client-side JavaScript, and if any of these rules are relaxed the user may be susceptible to malicious activity. On the server side, however, JavaScript is not subject to these kinds of limitations. And, in fact, in the absence of them there’s a great deal of power, particularly in the area of web scraping, which, as it turns out, allows for one of the cool upsides of this awesome freedom.

To get started, clone the following github repository: Basic web scraping with Node.js and Cheerio.js.

You’ll find instructions on how to run this code in the Github.

The page we will target for web scraping

Lets’ take a moment to look at the example web page that we will scrape: http://output.jsbin.com/xavuga. Now, if you use your web developer tools to inspect the DOM, you’ll see that there are three main sections to the page. There’s a HEADER element, a SECTION element, and a FOOTER element, and we will target those three sections later, in some of the code examples.

The request NPM module

One of our key tools is the request NPM module, which allows you to make an HTTP request and use the return value as you wish.

The cheerio NPM module

The cheerio NPM module provides a server-side jQuery implementation, and its functionality mirrors the most common tasks associated with jQuery. There isn’t a 1:1 method replication; that was not their goal. The key point is: you can parse HTML with JavaScript on the server-side.

Caching an entire web page – Example # 1

In Example # 1, we set some variables. The fs variable references the file system node module, which provides access to the local file system. We’ll need this to write files to disk. The request variable refers to the request node module, which we discussed earlier, and the cheerio variable refers to that cheerio node module that we also discussed. The pageUrl variable is the URL of the web page that we will scrape. Now, at the highest level, there are two things that happen in this code: we define a function named scrapePage, and then we execute that function. So, now, let’s take a look at what happens inside of this function.

First, we call the request function, passing it two arguments, the first of which is the URL of the request. The second argument is a callback function, which takes three arguments. The first argument is an error object, and this “error first” pattern is common in Node.js. The second argument is the response object, and the third argument is the contents of the request, which is HTML.

Inside of the request callback, we leverage the file-system module’s writeFile method. The first argument we pass is the full path of the file name, which tells the fs module what file to write. For the second argument we pass the responseHtml variable, which is the content that we want to write to the file; this is what was returned by the request function. The third argument is a callback function, which we are using to log a message indicating that the file write to disk was successful. When you run Example # 1, you should see a new file in the HTML folder: content.html. This file contains the entire contents of the web page that we make a request to.

Caching only a part of a web page – Example # 2

In Example # 2, we have an updated version of the scrapePage function, and for the sake of brevity, I have omitted the parts of the code that have not changed. The first change to the scrapePage function is the use of the cheerio.load method, and I assigned it to the $ variable. Now we can use the $ variable much the same way we would jQuery. We create the $header variable, which contains the HTML of the HTML header element. We then use the file-system module’s writeFile method to write the HTML header element to the file: header.html.

Now, when you run Example # 2, you should see another new file in the HTML folder called header.html, which contains the entire contents of the web page that we make a request to.

Example # 3

In Example # 3, we have updated the scrapePage function again, and the new code follows the same pattern as the one in Example # 2. The difference is that we have also scraped the content and footer sections, and in both cases, we’ve written the associated HTML file to disk. So, now, when you run Example # 3, you should see four files in the HTML folder, and they are entire-page.html, header.html, content.html and footer.html.

Summary

In this article, took a look at what is possible when scraping web pages. Now, even though we only scratched the surface, we did work in some high-level areas, focusing on making a request and then parsing the HTML of that request. We used the request module to make the HTTP request, and the cheerio module to parse the returned HTML. We also used the fs (file-system) module, in order to write our scraped HTML to disk.

My hope is that this article has opened up some new possibilities in your work, and has pointed you in the right direction for pulling this all off. So, happy web page scraping!

Handling HTTP POST Requests with Express.js

Express JS

Node.js LogoLearn how to access the body of an HTTP POST request using the Express.js framework and body-parser module.

Forms are a common component in web applications. When a user submits a form, that data is sent to the back-end for processing. To process that data, the web server must understand how to access it. Popular web server languages include Java, .NET, PHP, Python and Node.js. In this article, we’ll learn how to access the POST data sent to a Node.js web server using the Express.js framework. To get started, you can go ahead and clone the following github repository: Handling POST requests with Express and Node.js.

And you’ll find instructions on how to run the code in the Github page.

package.json

The package.json for this project is pretty straightforward, and we’ll only need the body-parser and express Node.js modules. We also create a scripts property so that running the example code requires a simple command: npm start.

Requiring the modules we need – Example # 1:

In Example # 1, we’ve imported the Node.js modules that we need. The Express module takes care of the heavy lifting with regard to fulfilling web requests. NOTE: If you’re not familiar with the Express Node.js module, please see my earlier blog post on this subject:  Set up a Node / Express Static Web Server in Five Minutes.

We also import the body-parser Node.js module, which has the critical role of parsing the body of an HTTP request. When it comes to processing a POST request, this is important. And the path Node.js module helps express to construct a file path.

bodyParser.json and bodyParser.urlencoded – Example # 2:

Now, here in Example # 2, we tell express to use the bodyParser.json middleware, which provides support for parsing of application/json type post data. We also tell express to use the bodyParser.urlencoded middleware, which provides support for the parsing of application/x-www-form-urlencoded type post data.

Creating the node.js web server – Example # 3:

In Example # 3, we use express.static to set up the static assets folder, the main purpose of which is to help the working example function in a browser, with minimal effort. For more information on express.static, please see my earlier blog post in Express mentioned above. In this example, we use the app.post method, which tells the Express module to wait for an HTTP request at the /form route that leverages the POST HTTP verb. So, when the user sends a POST request to the /form route, Node.js executes the provided callback, which is the second argument passed to the app.post method.

The app.post callback takes two arguments, the first of which is the request object (i.e. “req”). The second is the result argument (i.e. “res”). We use the res.setHeader method to set the Content-Type header to application/json, which tells the user’s browser how to properly handle the returned data from the request.

NOTE: We wrap the rest of the callback code in a setTimeout, the purpose of which is to mimic a slow internet connection. Otherwise, the working example will move too fast for most to comfortably follow.

Inside the setTimeout, we use the res.send method to send the result body back to the user, and here we’re sending a serialized JSON object. To construct this object, we access the body property of the req object (i.e. the request object), which is why we have implemented the bodyParser.json middleware. And this is what allows us to parse the properties of the request body. In this example, we are expecting firstName and lastName POST parameters, which will allow us to access the req.body.firstName and req.body.lastName properties, to build the JSON for our result object.

To see this code in action, just follow these steps :

  1. Clone the git hub repository: https://github.com/kevinchisholm/video-code-examples/tree/master/node-express/handling-POST-requests-with-express
  2. Follow the instructions in the readme to set up the code
  3. Point your browser to: http://localhost:3000
  4. In the web page, enter some text into the two input boxes, and then click the “Submit” button
  5. Notice the logging statement in your node.js terminal
  6. Notice that the text you entered displayed in a browser message

You might also want to take a look at the Network tab in your Web Developer Tools, which allows you to see the actual network request that goes to the web server. You’ll be able to inspect the POST data sent, and the JSON data returned.

Viewing the working code example

Here’s what happens when you submit the data in the browser:

  1. The JavaScript in www/js/form-handler.js makes an AJAX POST call to the route: /form.
  2. The object sent in the POST request is: {firstName: XXX. lastName: XXX}. (NOTE: “XXX” is whatever value entered into the form’s text inputs.)
  3. Our Node.js web server intercepts the HTTP request to /form.
  4. Our Node.js web server parses the body of the HTTP request and constructs a JSON object.
  5. The XMLHttpRequest for the AJAX call is this JSON object.
  6. The browser displays the data from this JSON object in the browser.

Nothing too fancy here, just illustrating the “round trip” of our HTTP POST request.

Summary

In this article, we learned how to handle POST requests with the Express node.js module, and we talked about the need for bodyParser.json and bodyParser.urlencoded. We also learned how to listen for a POST request to a specific route, and how to access the POST parameters in the HTTP request body. Now, while the working example is simple, it does allow you to inspect every step of the process. If you look at your browser’s network tab, you can see the HTTP POST request go out, and then return. What happens during the server-side processing of that request is what you see in our Node.js code: server.js.

So, a lot to digest at first, but I’m hoping that this it will get you started with your next form-based Node.js application!

Introduction to Express.js, the Node.js web application framework

Express JS

Node.js Logo - expressExpress.js provides the kind of abstraction that lets you stay out of the weeds and focus on your application code.

While the low-level nature of Node can be an asset, it can also be somewhat of a curse because when you’re serving static assets, it can be tedious to detect routes and serve the correct static assets for an entire web page. Some examples of static assets are CSS files, images or JavaScript files. Now, the good news is, Express is a Node module that provides abstraction for these kinds of challenges. It’s popular, it’s used by large companies, and there’s strong community support for it, all of which make this a solid choice.

Why Express?

The main goal of Express is to provide an application framework, and getting started is simple. Take a look at the code samples, which you can clone at the following Github repository: Introduction to Express.js, the Node.js web application framework. You’ll find instructions on how to run the code in the Github

package.json

The package.json for this project is simple: the single dependency is express.

The get() Method – Example # 1

In Example # 1, we call the get() method of the app variable, and we pass the string “/” to the get method. This tells Express.js how we want to handle any GET request to the root of the application. (NOTE: GET is an HTTP verb, other examples are POST and PUT.) In Example # 1, we are sending back a simple HTML page, and we have created the HTML by constructing a string that represents that HTML code. This happens in the “HTML” variable. We then call the send() method of the result object, sending our HTML back to the user. Now run Example # 1 in your terminal with the command node example-1.js, then navigate to http://localhost:3000/ in your browser. There you will then see “This is Example # 1”.

The use() method – Example # 2

Example # 2 is much shorter than Example # 1 because we have not hard-coded our HTML. Instead, we have used the use method of the app object, which tells Express which folder we want to use for serving static assets. As a result, our JavaScript code is cleaner, and easier to read. Also, we’ve separated concerns. In other words, instead of hard-coding our HTML in our JavaScript file, we’ve put HTML where it belongs: in an HTML file.

Now notice how the web page in Example # 2 has an image. I included that to point out how Express handles this for us, even though we never had to write any JavaScript code that specifically waits for an image request. There is also a CSS file being served. In both cases, Express understands that WWW is our public web folder and it serves up static assets as needed, which certainly saves us a lot of time. Now run Example # 2 in your terminal with the command node example-2.js, then navigate to http://localhost:3000/ in your browser. There you will see “This is www/index.html”, which is a major improvement, as the HTML that the user sees is actually served from a static HTML file.

Adding Handlers for a Second Route – Example # 3

In Example # 3, we use the GET method to add a handler for when the user requests “/about“. In this case, we serve-up “/www/about.html“, which is just one example, but we could have added any specific route handlers needed. Now run Example # 3 in your terminal with the command node example-3.js, and navigate to http://localhost:3000/ in your browser. There you will see “This is www/index.html”. Now, click “About” in the upper-right-hand corner, to display the “About Us” page. You can then click “Home” and “About” over and over, to switch routes, because our JavaScript code in Example-1.js handles the alternation of these two routes.

Summary

In this article, we learned the absolute basics of Express, but in doing so, we also got to see how simple it is to use. In our discussion we saw examples of the get() method, as well as the use() method. I’m hoping that this was enough to illustrate the power behind this Node.js web application framework, and that it will get you well on your way to enjoying its usefulness.

Create a Node Websocket Server in Five Minutes

Node.js

Node.js Logo - node websocket serverLeveraging Express.js and the ws NPM module, it is possible to create a Node Websocket Server in less than ten lines of code.

The Websocket protocol provides full-duplex communication channels over a single TCP connection. In the past, web clients had to employ long-polling or the repeated pinging of a server in order to achieve this kind of “push” functionality. Now, Websocket technology eliminates the need for such outdated techniques. So, when a Websocket client connects to the server, a persistent connection is created, and the Websocket server can then push notifications to all connected clients. It is possible for the clients to send messages to the Websocket server as well, but I’ll cover that in a later article.

In this article, I’ll explain the bare-minimum code needed to create a Node Websocket server that can broadcast all incoming messages to connected clients. This should take us about five minutes, and less than ten lines of code. The beauty of Express.js is that it takes care of the heavy lifting with regard to the actual web server. The ws NPM module also plays a starring role in that it handles the Websocket communication layer, allowing us to expose an endpoint that accepts connections and messages from clients. Plus, we can broadcast messages to connected clients.

package.json

Above is the contents of package.json. There are only two dependencies: the Express.js framework and the ws module.

The Node Websocket Server – Example # 1

So, here in Example # 1 we have the entire working application. On line #s 3 through 9 we create our dependencies. I’ve grouped things in a way that I hope makes sense, but I’ll just point out that on a high level there are two things happening here. We require the modules that we need as constants: http, express, and WebSocket. Also, we create the constants app, server and websocketServer. These constants are the results of expressions. Now if you’ve ever worked with Express.js before, the app constant should be familiar to you; it’s simply an instance of the Express framework. The server constant is the result of calling http.createServer(), passing it our express.js instance (ie. “app”). And finally, the constant websocketServer represents our Websocket server.

Now let’s jump ahead for a moment to line # 30, where we start our web server. It’s not that there’s much going on here; it’s just that I wanted to point out that the server is started by calling the server.listen method, passing it the port to listen on (i.e. 3000). The second argument (the anonymous function) is optional.

Now let’s go back up to the top of the file. As you can see, the rest of the code is surprisingly simple. We create two event handlers, the first of which takes care of each Websocket client connection, and the second one processes each message that it receives from that client. On line # 12, we have the first event handler. We use the “on” method of the websocketServer instance to handler an incoming connection. This is somewhat similar to creating a handler for a GET or POST request in Express.js.

We pass the event as the first argument (i.e. “connection”), and then a function as the 2nd argument. The anonymous function that we provide contains the code that we want executed for each new Websocket client connection. This function also receives a Websocket client as its first argument. We have named this variable: “webSocketClient”. On line # 14 we provide some feedback to the Websocket client by sending it the first Websocket message: { “connection” : “ok”}. This is for demonstration purposes only, just so that we can see right away that the connection has been established.

Now inside of the anonymous callback, we set up the second event handler, which will process each message that this client receives. And similar to the connection event handler, we use the “on” method of the webSocketClient variable to handler an incoming message. We pass the event as the first argument (i.e. “message”), and then a function as the 2nd argument. The anonymous function that we provide contains the code that we want executed for each message received by this Websocket client.

Broadcasting the Message to All Websocket Clients

On line # 20, we start the process of broadcasting the incoming message to all Websocket clients. Using the forEach method of the Websocket.clients list, we iterate the list of websocket clients. And for each iteration, we provide a callback function. This callback function receives the currently iterated Websocket client as its first argument. So, we then use the send method of that client object, and send the incoming message (i.e. by sending one message to many recipients, we are “broadcasting” that message).

Testing the Websocket Server

Now if you copy all of the code from Example # 1 into a file and then execute it, you’ll have a running Websocket server on port # 3000. But that isn’t enough. Now we want to test our websocket server, and an easy way to do this is to use the Smart Websocket Client plugin for Google Chrome.

So go ahead and click this link to install the plugin, and once you’ve installed it, start the plugin by clicking the icon in the upper-right-hand corner of your Chrome browser.

smart-websocket-client-icon
smart-websocket-client-icon
smart-websocket-client-1
Smart Websocket Client

Once the Smart Websocket Client is running, enter http://localhost:3000 in the address bar and then click the “Connect” button. You should see { “connection” : “ok”} in the lower window, indicating that a Websocket connection was successfully established (see example # 2).

smart-websocket-client-2
Connecting
smart-websocket-client-3
Connecting Success Message

Example # 2

In the top window, enter any text, click the “Send” button, then you’ll see your message appear in the lower window. Now open a few more instances of the Smart Websocket Client and follow the same steps. If you place your Chrome browser tabs side by side, you’ll see that every message you’ve sent has been broadcast to every Websocket client. Congratulations!  You’ve just built a working Node Websocket server.

Example # 3

Now earlier in this article, I promised that we could create our Websocket server in less than ten lines of code. Example # 1 clocks in at 32 lines of code, but this is because I used whitespace and comments to make the code as readable as possible. So, in Example # 3, I’ve provided the condensed version of our Node Websocket server. This code is not very pretty, but as promised, it is a fully functional Node Websocket server that’s set up in less than ten lines

Set up a Node / Express Static Web Server in Five Minutes

Node.js

node express static web server

Setting up Node and Express as a Simple Lightweight Web Server for Your Single Page Application is Very Easy to Do.

Sometimes you just need a local web server.  Sure, you could use MAMP, but installing Apache, MySQL and PHP seems like overkill in some cases. I have used MAMP for years and it is terrific. In particular, when you need to run PHP locally and / or connect to a MySQL database, it’s the cats pajamas. But that was the standard 10 years ago. Nowadays, it’s very common to build a single page web application where all of the assets are static, data is pulled-in from a REST endpoint and all of the heavy lifting is done in the browser via JavaScript. In these kinds of scenarios, a static Node / Express server is a simple, easy and lightweight approach.

In this article I’ll explain the steps needed to set up a Node / Express Static Web Server. And the good news is that on this high level, the required steps are very simple. First, you’ll need to require the express and path modules. After that, you’ll create an instance of express, then set the port that the web-server will use. The next step, and the key-ingredient here is the express.static method. This tells Express.js that you want to serve static content from a specific folder. In that one line of code, you’ve done the majority of the configuration work.

So, not only will Express serve-up static content from that folder, it can do so for any subfolders as well. You can specify any folder in your project as the static web folder. And the beauty of it is that any folder outside of the one you specify will be hidden from public view, so your application code will be safe. When you pass the the express.static method to the use method of your express instance, you provide the details that express needs to serve your static content. Then finally, you use the listen method of your express instance to start the web server. We’ll take a closer look at the express.static method in Example # 2.

Now, I just want to remind you here that this article pertains to the specific occasions in which you need to serve static web assets locally. In other words, using a Node / Express static web server can be a very simple way to satisfy your need for a local web server, but may not be the best approach for your production needs. Technically, you could take the code that is detailed in this article and deploy it to your production server, and in theory it would work just fine. For this article, however, I’m just going to concentrate on providing a fast and simple way to get a local web server running so that you can test your front-end code (e.g. HTML, CSS or JavaScript).

The code samples can be downloaded here: https://github.com/kevinchisholm/node-express-static-web-server

Example # 1 – package.json

In Example # 1, we have the contents of package.json. Nothing too special going on here. But just note that our only dependency is the express module. Also, in the scripts property, I’ve set up the start command to execute the app.js file in the node web-server folder. This way, we can simply type npm start in the terminal, instead of node web-server/app.js (just a bit less typing).

Example # 2 – The Express Static Web Server

In Example # 2, we have the entire contents of our web server: 15 lines of code (and nearly 25% of that is comments!). The magic happens on line # 10:  We call the app.use method and pass it express.static, which also takes a couple of arguments. So this tells Express that we want to set a static folder. We then use the path.join method to tell Express where all static assets should be served from. In our case, it is the www folder. The two arguments passed to the path.join method are __dirname, which tells us the absolute path to the folder within which the current script is found, and then “../www” which is a relative path to the www folder.

Now, as I mentioned earlier, anything outside of your static folder is protected from public view. This means that while the folder you specify when calling the express.static() method (i.e. “../www”) is publically viewable, any folder that is a sibling or descendant of that folder is not available publically. This is not a critical factor when working locally (i.e, developing), but it does matter in production. In other words, you wouldn’t want your application code to be viewable to the general public. Nor would you want to make available any sensitive information that’s in your application code, such as a secret key or other credentials. So, as you can see, this is one of the key strengths of Express, which is the ability that it provides you to not only define your public/static folder in one line of code, but to also protect all of the other folders by default.

Express does all of the heavy lifting

A little earlier, I used the word magic. We both know that none of this is actually magic, but it sure feels like it. If you’ve ever created a Node web server manually, then you know two things: 1) It’s really easy, 2) It’s really tedious once you get past “Hello World”.  But Express hides all the tedium and makes serving static assets as easy as 1-2-3.

HTTP Headers

There is one downside here. Express does not set the appropriate content-type headers for the HTTP requests.  This is not fatal in most cases because this approach is simply meant to provide a very fast and easy way to set up a static web server. The actual web server works just fine, but keep in mind that content-type headers for files such as JPEG, PNG, CSS or JS will not be set accordingly. If that is a problem, then a simple static web server is probably not what you need and you should consider a more robust approach. So, hopefully, if you do need a simple static web server, this article was what you needed to get up and running quickly.

Summary

There are multiple options when it comes to setting up a static web server. One advantage to leveraging Node and Express.js, however, is that as a developer, you probably already have Node installed on your machine. So, in this case, you won’t need to install any additional software. You can simply import the Express framework, write about a dozen lines of code, and you have a static web server. This is probably not a server that you would use in production, but as you can see, it can easily solve the problem of quickly serving web content on your local machine. If you need to write moderately complex dynamic application logic, then you might need something a bit more advanced than what was discussed here. But for a basic static web server, this approach should get you going (hopefully in less than five minutes : – )

Yikes! AWS Node / NPM ERR! enoent ENOENT: no such file or directory package.json

Node.js

Node.js LogoAWS’s Node deployment keeps telling me that it cannot find package.json, but it’s there! – Fortunately, this problem is easily solved.

AWS makes deploying your Elastic Beanstalk easy. Compress your build files, upload the ZIP and then deploy that application version. Lovely. But sometimes your application goes into a “warning” or “degraded” state, and then a visit to the application with a browser yields: “502 Bad Gateway“. Errrggggg…..

At this point, you look in the logs and see a cryptic message that says something like: “enoent ENOENT: no such file or directory package.json“. You double-triple-quadruple-check and yes, package.json is in-fact very much alive and well. So, of course your next thought it: “WTF???

I have run into this problem a few times and in each case, the problem was me: I zipped-up a folder, instead of the contents of a folder.

Do not compress an entire folder

Compressing the my project folder does not fix package.json problem

Let’s say your Node application is in a folder named: “myProject“. If you are compressing that folder, then this is your problem. You don’t want to compress a folder because when AWS un-zips that file, it will not know to look in the “myProject” folder that is created when the file is un-zipped.

Compress ALL of the items in  your project folder

Compressing the root files fixes package.json problem

What you want to do is: select EVERY file in the root of that folder (i.e. your Node application’s root folder), and then compress THOSE files. This will create a ZIP file that when un-zipped, creates the file structure that AWS expects. Now AWS will find package.json. This should solve the problem.

Compressing the root files fixes package.json problem

In the image above, I have zipped up the contents of the “myProject” folder, and created Archive.zip.

Upload the zipped file

Compressing the root files fixes package.json problem

Now, back in your AWS console, you can use the “Upload and Deploy” button to upload your ZIP file, and then deploy it.

Setting Your AWS-Hosted Node Application’s Port

Node.js

Node.js LogoWhen working locally, using an arbitrary port number is fine, but you need to get that property when deploying your Node application to AWS.

Technically, you can code-up a Node web server in less than ten lines of code. Most likely, your application will require a few more lines of code than that. But my point here is: getting a basic Node web server running is not terribly difficult.

Example # 1

In example # 1, I used port # 3000, but I could have used virtually any valid port number. When working locally this is for the most part a non-issue. As long as no other application is using the port you want to use, you chose one and then use it. Easy. This example does little more than say “Hello!”, but the point I’m trying to make is that your main JS file ends with the server.listen method, and you need to pass it a port number.

But when you attempt to deploy this code to your AWS Elastic Beanstalk instance, you will get a “503 Bad Gateway” error in your browser. The reason for this is: you don’t know which port should be used when calling the server.listen method. The great thing about AWS is that it provides a layer of abstraction for those kinds of details. In other words; AWS takes care of details such as which port to listen on. The downside here is that you have no way of knowing exactly which port that will be when you deploy your code.

Example # 2

In example # 2, we set a variable named: port. We attempt to assign the value of process.env.PORT to that variable. If that value is falsely, then we set it to 3000. The reason this works is; if our code is running on our AWS instance, then process.env.PORT will automatically be set and we will listen on that port. If we are running our code locally, then process.env.PORT will be undefined (or “falsely”). So, then our port variable will have a value of 300. This way, our code can run successfully on our AWS instance, or locally.

Node.js Hosting Links

Node.js

JavaScript LogoThe good news is: there are a lot of Node hosting services out there. The bad news is: there are a lot of Node hosting services out there

Installing Node locally is easy. Cloning an existing Node application from GitHub and running it locally is easy. Creating your own Node application and running it locally is easy. But, choosing a hosting solution for your Node application is definitely not easy.

Below is a list of companies that offer Node hosting services. I do not claim to have every possible company listed here. But I’ve done my best to list the ones that I know about and will update this page any time I learn about another one worth mentioning.


Title: Openshift

Link: openshift.com

Description: I’ve used Openshift.com quite a bit and for the most part have been very happy with their service. They offer a free plan that definitely includes what you need to get up-and-running.


Title: Heroku

Link: heroku.com

Description: Heroku was the first Node hosting service I knew about. I’ve not used it in a while but I was always very happy with it. Setup and deployment was fairly pain-free, as was adding services such as MongoDB.


Title: Amazon Web Services

Link: Deploying Node.js Applications to AWS Elastic Beanstalk

Description: AWS is a big topic. But in general, it’s really easy to get a Node instance up-and-running with Elastic Beanstalk.


Title: Nodejitsu

Link: nodejitsu.com

Description: I’ve not tried Nodejitsu but have heard good things about them.


Title: zeit.co

Link: zeit.co/now

Description: This is a new one to me, but their setup looks super-simple.


Title: Node.js on Google Cloud Platform

Link: cloud.google.com/nodejs

Description: Although Google still has not caught up with Amazon yet, they are serious about their cloud offerings. I’ve not tried their Node hosting but have confidence that it is at worst, solid.


Title: Node.js Hosting

Link: a2hosting.com/nodejs-hosting

Description: Another new one to me, but their packages look very affordable.


Title: Node.js One Click Install | Cloud Hosting – GoDaddy

Link: a2hosting.com/nodejs-hosting

Description: Godaddy now offers a “Cloud” service that supports Node hosting.

Helpful Node.js Education Links

Node.js

JavaScript LogoNode.js is growing fast. This is a great problem. While is means that JavaScript lovers have a rosy future, it can sometimes be difficult to keep up with what is going on with Nodes.js

Here are a list of links that you might find helpful in your Node.js travels. In each case, I’ve provided a brief description of the link / organization / article, so that you have a sense of where you are are headed. If you feel that there is a Node.js link that I should have included in this article, please contact me at: kevin@kevinchisholm.com.

Critical Node.js Links


Node.js Logo

Title: nodejs.org

Link: nodejs.org

Description: Since this is the home page for Node.js, you cannot go wrong here.


Node.js Logo

Title: Node.js v6.6.0 Documentation

Link: nodejs.org/api

Description:The official documentation for Node.js. Very well organized and easy to read. Almost the most important Node.js documentation you can read if you are getting started.


Node.js Logo
Title: npm – Home of the package manager for JavaScript.

Link: www.npmjs.com

Description: It’s hard to imagine doing anything with node without the use of NPM. This is the official home page of NPM, and a great starting point.


Node.js Logo

Title: Homebrew. The better way to install Node.js on Mac OSX

Link: brew.sh

Description: I’m being a little opinionated here (ok. I’m being a lot opinionated). But, for Mac users, Homebrew is the way to go when you install Node.js (sorry Windows users, you are stuck with scoop : – )


Node.js Logo

Title: Built in Node.js – startups, apps, projects using Node

Link: builtinnode.com

Description: A great way to learn about who is using Node.


Node.js Newsletters

Node.js Logo

Title: npm Weekly

Link: www.npmjs.com/npm-weekly

Description: Find out what npm has been working on, thinking about, and talking about every week. A great newsletter if you are into NPM.


Node.js Logo

Title: node weekly

Link: nodeweekly.com

Description: A free, once–weekly e-mail round-up of Node.js news and articles. Another awesome newsletter if you are into Node.


Other Helpful Node.js Links

Node.js Logo

Title: Node Tutorials on scotch.io

Link: scotch.io/tag/node-js

Description: scotch.io tutorials are very easy to read. The site is in general a great resource for learning about a number of web development technologies. Fortunately, they are passionate about Node!

Setting AWS-Node.js Stormpath keys

Node.js

Stormpath Logoprocess.env can be used to set the environment variables you need when using the stormpath api in your aws-hosted node application

Stormpath provides amazing abstraction when it comes to authentication. There are certainly other services like this, but when it comes to security, Stormpath is not only popular, but well respected. This comes as no surprise as they simply make authentication easy.

I have to say that their documentation is for the most part very good. If Node is  your thing, they make it very easy to get up-and-running with their API. Their mailing list is also quite helpful.  At least a once or twice per week, I receive emails that link to interesting articles on their blog.

Recently, I was trying to setup a Node.js/Express.js application, leveraging their express-stormpath Node module. I was thinking to myself: “…hmmmm. There must be a step where I have to configure my secret key or something like that”. After some quality time with Google, I came across this article, that suggested the following:

Unix/Linux/Mac:

Windows:

(where “xxx” is your actual key)

Well, that is fine for working locally, but I knew if I wanted to deploy this as an AWS Elastic Beanstalk application, I needed to actually set these values somewhere.

Using process.env

I set the three environment variables I needed to be properties of process.env:
I

(where “XXX” is your actual key)

I took a look in the source code for the express-stormpath Node module and could see that it seemed to want to find these on process.env, so I think this approach should be fine. I’m still in the process of getting this Node.js/Express.js application up and running, but if you are faced with the same challenge, hopefully this helped you.

Share Node.js code with JSApp.us

Node.js

JavaScript LogoJSApp allows you to write Node.js code in a browser, run it, and also share it with others

One of the things that makes front-end development so much fun is that you can easily create and share code. With tools such as JSFiddle, you can create an example web page and then send that JSFiddle URL to someone. Or you can even send someone the URL of a JavaScript file that you created so that they can just run $.getScript(yourJavaScriptURL) to inject your code in their page. And there are plenty of other clever ways to share / demo front-end code without a lot of fuss.

But what about Node?

Well, it’s not always so easy with Node, right? It’s server-side code, so you can’t just send someone a URL of your Node.js file to inject in their page. Github really saves the day in this case; you can create a public repo, and then send someone the Github repo URL. But that still requires the recipient to have at least git installed on their computer. And as we all know, once something takes more than 2 clicks, you start to lose your audience. That said, anyone with a reasonable attention span and a genuine interest in your code will follow the few clicks needed to clone your repo and run your code, but for quick little snippets, it sill feels like overkill sometimes.

For example, I like to write blog posts here about Node. In some cases, it does make sense to create a Github repo, especially if you have to leverage package.json, and the app requires file access, etc. But what about little examples? Just 10-20 lines of code to demonstrate a concept? Or even a simple working example?

Enter JS App!

When you navigate to jsapp.us, you immediately see some sample Node.js code. You can delete it and write your own. Then,  you simply click “test” in the sidebar (or CTRL + b), and a new browser window opens with your Node.js code running!

If you create a profile (free), you can save your code and share it with others. This is one of the most clever things I’ve seen in a long time. You can also go back and edit your files, re-name them, delete them. Really fun stuff.

If you need to create a quickie Node.js app and a Github repo would be overkill, JSApp might be just the tool you need. It’s been a while since I was this impressed but something I stumbled upon.

Bravo!

Renaming a file with Node.js

Node.js

Node.js LogoLearn how to rename a file with Node.js, in ten lines of code

I was fleshing out a few ideas on a project today and found myself trying to figure out how to rename a file with Node.js. I have no doubt that there are better ways to go about this, but I thought I’d document my findings.

The first thing I realized is that the low-level nature of Node.js offers a great deal of power, but also with that power comes the need to handle all of the details. For me, this meant getting references to four things:

  1. The name of the old file
  2. The name of the new file
  3. The full path to the old file
  4. The full path to the new file (which does not exist yet)

The first two items are easy: I just had to provide a name for the old file, and decide what to call the new file. The last two items involved a bit more effort. I needed to do three things:

  1. Get the path of the folder that contains the old file
  2. Add a trailing slash to that path
  3. Set a permanent reference to the old file

So, to accomplish all of these tasks, I decided to use the filepath Node.js module.

Example # 1A

 Example # 1B

Example # 1A shows the contents of package.json. There is only one dependency: the filepath Node.js module.

In Example # 1B, I first set references to the file system module, as well as the filepath module. Next, I provided strings for the names of the old and new files. The filepath module is then used to get the path to the current folder; I set that to the fpFolder variable (adding a trailing slash to that string, which will be needed when we append file names to that string).

The variable fpFile is used as a permanent reference to the old file (this will come in handy for Example # 3.) Finally, I build the full file paths for the old and new files. After that, a couple of console.log statements to make sure all of this work is correct.

Example # 1C

Example # 1C shows the output of the two console.log statements. Each path will vary for you a bit, so I simply put “[YOUR LOCAL PATH TO]” for the folder structure that leads up to that file in the github repo that you cloned (see “How to Demo” below).

This example does not actually rename a file. So, now I will use the file system module to make that change.

How to Demo:

  • Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  • Navigate to: JavaScript/node-js/node-modules/fs/fs-rename
  • Execute the following command in a terminal prompt: npm install
  • Execute the following command in a terminal prompt: node filepath-1.js

Example # 2

In Example # 2, I use the file system module’s rename method to rename the file: “re-name-me.txt.” This method takes three arguments: a path to the old file, a path to the new file and a callback. The callback takes one argument: an error object. Inside of the callback, I check for the error object, and then output the path of the new file. So now, follow the instructions below to see this code in action. After you execute the code, the file: “re-name-me.txt” will be renamed to: “ive-been-renamed.txt.”

In order to rename Example # 2 again, you’ll need to manually rename the file: “ive-been-renamed.txt” back to: “re-name-me.txt”. After a few times back and forth, this got pretty tedious and I started to think that there must be a way to toggle the file back and forth. Meaning: If the file has been renamed, change it back to the original name, and so forth.

How to Demo:

  • Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  • Navigate to: JavaScript/node-js/node-modules/fs/fs-rename
  • Execute the following command in a terminal prompt: npm install
  • Execute the following command in a terminal prompt: node filepath-2.js

Example # 3A

In Example # 3A, I use the ternary operator when setting the final path for the old and new files. In each case, I check to see if the old file exists, and then depending on the answer, I set each path accordingly.

Example # 3B

Example # 3B is the full code for the final version of this file. I combined all var statements and cleaned up the code a bit. When you follow the instructions below, you’ll see that you can keep executing node filepath-2.js over and over, and the text file will toggle between the old name and the new name.

How to Demo:

  • Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  • Navigate to: JavaScript/node-js/node-modules/fs/fs-rename
  • Execute the following command in a terminal prompt: npm install
  • Execute the following command in a terminal prompt: node filepath-2.js

Summary

As I mentioned, there are probably a number of ways to do this that are more efficient. Everything I detailed here was the result of a few minutes with Google. Hopefully, this article either got you where you needed to go, or pointed you in the right direction.

Helpful Links for Renaming a File with Node.js

http://nodejs.org/api/fs.html

https://www.npmjs.com/package/filepath

Getting started with the filepath Node.js module

Node.js

Node.js LogoWhen you need to reference and work with the local file system in your Node.js program, the filepath module is quite a handy tool.

Even if your Node.js program is a web-server of some sort, working with the local file system is somewhat inevitable. While Node.js does provide low-level file system access (see the Node.js fs module), abstraction is always helpful, particularly when dealing with absolute paths.

The filepath Node.js module is a very helpful utility for simple access to file paths. You’ll need only a package.json file with this module as a dependency, an “npm install” command, and then you are up and running. This article provides a quick introduction to a few of the most common methods.

Example # 1A

Example # 1B:

In Example # 1, we first create the FP variable, which references the filepath module. Then we create the path variable, which holds the return value of the FP object’s newPath method. And finally, we output the path in the console. Example # 1B shows the terminal output when we use console.log to view the path variable. This path will vary for each user so I simply put “[YOUR LOCAL PATH TO]” for the folder structure that leads up to that file in the github repo that you cloned (see “How to Demo” below).

How to Demo:

  1. Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  2. Navigate to: JavaScript/node-js/filepath
  3. Execute the following command in a terminal prompt: node filepath-1.js

Example # 2

Example # 2 demonstrates the list method. The only real difference between this code and Example # 1, is the new variable “files”, which receives the value of the list method, when called on our path variable. The files variable ends up as an array. Each element in the array is an object whose “path” property is a string that points to a file in the current directory.

How to Demo:

  1. Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  2. Navigate to: JavaScript/node-js/filepath
  3. Execute the following command in a terminal prompt: node filepath-2.js

Example # 3A

Example # 3B

Example # 3C

Example # 3D

In Example # 3A, we see the recurse method in action. Just as the name implies, the recurse method will recursively list all of the files in the current directory. As a result, if one of those files is a folder, then it will list all of the files in that folder, and so on. This method differs from the previous two examples in that it takes a callback. The callback is a bit like a forEach call; it iterates over all of the files or folders in the path, and calls the callback for each one. Inside of the callback, the path variable is the current path being iterated over.

Example # 3C is the output from the code in Example # 3A.

In Example # 3C, we use the toString() method of the path object so that instead of a bunch of objects that we would need to handle, we just get the values we are after; the string representation of the path to that file or folder.

Example # 3D is the output from the code in Example # 3C.

How to Demo:

  1. Clone this github repo: https://github.com/kevinchisholm/video-code-examples
  2. Navigate to: JavaScript/node-js/filepath
  3. Execute the following command in a terminal prompt: node filepath-3.js

Summary

The filepath Node.js module has much more to offer than was demonstrated here. Hopefully, this article has demonstrated how easy it is to get started with filepath.

Helpful Links for the filepath Node.js module

https://www.npmjs.com/package/filepath

http://nodejs.org/api/fs.html