Learn Node.js and MongoDB by Building a URL Shortener App
In this article, you’ll learn how to build a URL Shortener application with Node.js and MongoDB. Here’s a live demo of what we’ll be building. You can find the complete source code for this project in this GitHub repo.
Prerequisites
I assume basic familiarity with JavaScript as well as the command line. If you haven’t built a basic Node application before, you might want to start here first, then return to this tutorial at a later time.
You also need to have Node.js and npm installed on your computer. You can visit the Node.js website to view installation instructions for your operating system. npm comes bundled with Node, so once you install Node, you’ll have access to the npm command too.
The versions I used while building this project are as follows:
- Node.js v11.2.0
- npm v6.6.0
You can view the version of Node and npm you have installed by running the following commands in your terminal:
node -v
npm -vGrab the starter files
Grab the starter files for this project at this GitHub repository. Clone the repo to your computer and cd into the created directory. Then run npm install from the project root to install all the dependencies specified in the package.json file. I’ll get into what each of the dependencies do later on as we progress.
Install MongoDB
MongoDB is a free and open-source NoSQL document database commonly used in modern web applications. You’ll need to have it installed on your machine. At the time of writing, the latest stable version is 4.0.5. This is the version I used throughout this tutorial.
Here are the installation instructions for Linux, macOS and Windows. If you’re on Ubuntu like myself, you can install MongoDB using apt:
sudo apt update
sudo apt install -y mongodbYou can checkout what version of mongoDB you have installed using mongo --version.
The database server should be automatically started after the installation process, but you should verify this before moving on from this step. On Ubuntu, you can use the following command to check the status of the mongoDB server:
sudo systemctl status mongodbYou should see this output:
If not, you can start it using the following command:
sudo systemctl start mongodbSet up a basic Node server
Looking through the src folder within the project directory, you’ll see that we’ve got a server.js file and a public folder containing the markup and styles for the application frontend. The server.js file is where the bulk of the application code will be written in.
Unlike the previous Node tutorial where I used the built-in http module to set up the Node server, we’ll be using Express, a popular Node.js web application framework in this instance.
There are other web frameworks out there, but Express is simple enough, well documented and well supported so you shouldn’t run into many issues when using it in your applications.
If you look through the package.json file, you will see the express package is part of the dependencies that we installed earlier. Let’s go ahead and use it to set up the Node server in server.js:
const express = require('express');
const app = express();
app.set('port', process.env.PORT || 4100);
const server = app.listen(app.get('port'), () => {
console.log(`Express running → PORT ${server.address().port}`);
});You can start the server by running npm start in the terminal. I am making use of the Nodemon package to auto restart the Node server when changes are made to it so we don’t have to do so ourselves.
Set up the application frontend
As mentioned earlier, the app frontend lives in the public folder. We need to set up a new route on the server so that when a user visits the application, the HTML file will be sent and rendered in the browser.
Change the server.js file to look like this:
const express = require('express');
const path = require('path');
const app = express();
app.get('/', (req, res) => {
const htmlPath = path.join(__dirname, 'public', 'index.html');
res.sendFile(htmlPath);
});
app.set('port', process.env.PORT || 4100);
const server = app.listen(app.get('port'), () => {
console.log(`Express running → PORT ${server.address().port}`);
});path is a built-in module in Node.js. It allows us to link to directories and file paths in Node.js. The sendFile() method takes an absolute path to the file, so __dirname is used to avoid hardcoding the path. __dirname is the directory in which the executing file is located, so path.join(__dirname, 'public', 'index.html') will resolve to src/public/index.html.
Navigate to http://localhost:4100 in your browser. Notice that the HTML is rendered correctly. However, the styles are missing even though style.css was linked correctly in index.html.
When the browser encounters the reference to style.css, it fires off a request to the server for that file. But since we haven’t configured our server to handle requests for static files (such as images, CSS and JavaScript), the server does nothing, and the request fails leaving the page without styles.
To fix this situation, we need to configure express to handle requests for static files correctly. We can do this using the a built-in middleware function in express as follows:
// beginning of the file
const app = express();
app.use(express.static(path.join(__dirname, 'public')))
// rest of the file
Now reload the page. It should work correctly:
Submit the form to the server
We need to write client side JavaScript to submit the contents of the form input to the server when a user submits the form.
We can do this without using any client side JavaScript by setting the action attribute of the form to a route on the server and setting the method attribute to POST, but I’ve opted to use JavaScript here so that we can handle the response and display the shortened url to the user without a full page refresh.
Create a new main.js file in the public directory and add the following code into it:
const form = document.querySelector('.url-form');
const result = document.querySelector('.result-section');
form.addEventListener('submit', event => {
event.preventDefault();
const input = document.querySelector('.url-input');
fetch('/new', {
method: 'POST',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify({
url: input.value,
})
})
.then(response => {
if (!response.ok) {
throw Error(response.statusText);
}
return response.json();
})
.then(data => {
while (result.hasChildNodes()) {
result.removeChild(result.lastChild);
}
result.insertAdjacentHTML('afterbegin', `
<div class="result">
<a target="_blank" class="short-url" rel="noopener" href="/${data.short_id}">
${location.origin}/${data.short_id}
</a>
</div>
`)
})
.catch(console.error)
});This code listens for the submit event on the form, prevents the form submission and fires of a POST request to the server with the value of the form input in request body. The reason we wrap the body object in JSON.stringify is so that we can consume it as JSON on the server.
Note that we’re posting the data to the /new route which hasn’t been created on the server yet. We’ll create it in the next section. Before that, make sure you reference main.js in your index.html file before the closing body tag:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="ie=edge">
<title>URL Shortener</title>
<link rel="stylesheet" href="style.css" type="text/css">
</head>
<body>
// rest of the code
<script src="main.js"></script>
</body>
</html>Access the form body on the server
Let’s go ahead and create the /new route that will process the URLs to be shortened. Add this below the root route in server.js:
app.post('/new', (req, res) => {
});The first thing we need to do is access the JSON data that was sent from the client in the request body. To do this, we need to use of the body-parser package. This package parses all incoming request bodies and makes them accessible on req.body.
Since this package has been installed already, we can use it right away in server.js:
const express = require('express');
const path = require('path');
const bodyParser = require('body-parser');
const app = express();
app.use(express.static(path.join(__dirname, 'public')))
app.use(bodyParser.json());
// rest of the file
Now, we should be able to access the request body in the req.body property in the /new route:
app.post('/new', (req, res) => {
console.log(req.body);
});You can try this out by entering a URL into the form and submitting it. Then navigate to the terminal where your server is running to see the JSON data printed in the terminal.
This means we can access the url using req.body.url.
Validate the URL
Before we shorten the URL, we need to validate if the URL that was submitted is valid. Client side validation is handled for us in the browser because we’ve set the type of the input to url so the form won’t be submitted if the value doesn’t have a valid URL structure.
To make the application more robust, we need to validate the URL on the server as well. There are several npm packages that can handle this, but I’ve opted to do so using a few built-in Node modules.
The first thing we need to do is check if the URL has a valid structure, then we perform a DNS lookup to see if the domain is operational. A domain like https://google.com will pass both tests, but http://jidfsdm.com will fail the second one since that site does not exist.
Require the built-in dns module at the top of server.js:
const dns = require('dns');Then change up the /new route as follows:
app.post('/new', (req, res) => {
let originalUrl;
try {
originalUrl = new URL(req.body.url);
} catch (err) {
return res.status(400).send({ error: 'invalid URL' });
}
dns.lookup(originalUrl.hostname, (err) => {
if (err) {
return res.status(404).send({ error: 'Address not found' });
};
});
});The URL class returns a new URL object with several properties if the input URL has a valid structure. Otherwise, it throws an error which we can catch and send back to the client.
If the URL input passes the first test, we then check to see if the domain is operational by passing the hostname portion of the URL (the domain) to dns.lookup which checks if the domain is live. If so, we can connect to our MongoDB instance and create the shortened version of the url as you’ll see.
Set up environmental variables
Environmental variables are a great way to configure how your program should work. They are key-value pairs that are stored on the local system where your program is being run, and are accessible from within your code.
It’s considered best practice to set app configuration data such as API keys, tokens, passwords and other sensitive details as environmental variables instead of hardcoding it into the program itself. This prevents you from accidentally exposing it to others, and also makes the values really easy to change without having to touch your code.
In Node.js, you can access variables defined in your environment via the process.env object. You can check the contents of this project via the Node.js REPL as shown below:
Aside from the operating system variables that are present by default, we can create project specific variables using a .env file.
Create a .env file at the root of your project directory and paste the following code into it:
DATABASE=mongodb://localhost:27017Here, we’ve added the URL of our local MongoDB instance as an environmental variable. Port 27017 is the port that MongoDB runs on. The next thing to do is load the values defined in .env into process.env. The easiest way to do this is with the dotenv package which is already part of our app dependencies.
Add the following at the very top of server.js:
require('dotenv').config();
// rest of the file
This will read the contents of the .env file in the root of your project, parse its contents and initialize the values on process.env. Now, you’ll be able to access any set variable under process.env.<KEY>.
Connect to MongoDB
Let’s go ahead and connect to our local MongoDB instance in server.js as shown below:
require('dotenv').config()
const express = require('express');
const bodyParser = require('body-parser');
const path = require('path');
const dns = require('dns');
const { MongoClient } = require('mongodb');
const databaseUrl = process.env.DATABASE;
const app = express();
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());
app.use(express.static(path.join(__dirname, 'public')))
MongoClient.connect(databaseUrl, { useNewUrlParser: true })
.then(client => {
app.locals.db = client.db('shortener');
})
.catch(() => console.error('Failed to connect to the database'));
// rest of the file
First, we’re importing MongoClient from mongodb which is the the native driver for interacting with a MongoDB instance in Node.js. Then we connect to the MongoDB instance specified in the DATABASE environmental variable.
If the connection is successful, we get a reference to the MongoDB instance client and can select a database using the client.db() method. Note that this method creates the database if it does not already exist.
Here, we are selecting a reference to the shortener database and storing that reference in app.locals which is an object provided by express. This object allows us to set local variables that persist throughout the life of the application, and can be accessed in other middleware functions (functions that have access to the req or res objects) via req.app.locals.
The reason we’re storing a reference to the database in app.locals.db is so that we can reuse the db object without having to open another connection to the MongoDB instance.
Shorten URLs
The next step is to actually shorten the URL and store it in the database. To create a unique short id for each url, we will be making use of the nanoid package.
Require it at the top of server.js below the other require statements:
const nanoid = require('nanoid');Then create a new shortenURL function in server.js as follows:
// beginning of the file
MongoClient.connect(databaseUrl, { useNewUrlParser: true })
.then(client => {
app.locals.db = client.db('shortener');
})
.catch(() => console.error('Failed to connect to the database'));
const shortenURL = (db, url) => {
const shortenedURLs = db.collection('shortenedURLs');
return shortenedURLs.findOneAndUpdate({ original_url: url },
{
$setOnInsert: {
original_url: url,
short_id: nanoid(7),
},
},
{
returnOriginal: false,
upsert: true,
}
);
};
// rest of the file
We need to reference a collection before we can add any data into the database. We can do so using the using the db.collection() method. If the collection does not exist yet, it’s created.
Before we shorten the URL and add it to the database, we need to check if the URL has not been shortened already to prevent duplicate database entries for one URL. We can do this using the findOneAndUpdate() method on the collection which allows us to modify a document that already exists in the database collection or create it if it doesn’t exist.
This method takes a few arguments. The first one is an object that is used to filter the collection. Here we are passing an object whose original_url property matches the url we are about to shorten. If a document in the database matches this filter, it will be returned and updated according to the update operators set in the second argument.
The $setOnInsert operator allows us to set the value of the document only if its being inserted. This means that the document will not be modified if it already exists, but if it doesn’t, it will be created with its value set to whatever we set in $setOnInsert.
In this case, the document will have two properties: original_url which is the url to be shortened, and short_id which is a unique 7 character id for that url.
We also need to set the upsert option to true. This ensures that the document is created if it doesn’t exist. Otherwise, $setOnInsert has no effect. Finally, setting returnOriginal to false ensures that findOneAndUpdate returns the new document if one is upserted, which is what we want in this case.
We can make use of the shortenURL() function in the /new route like this:
app.post('/new', (req, res) => {
let originalUrl;
try {
originalUrl = new URL(req.body.url);
} catch (err) {
return res.status(400).send({error: 'invalid URL'});
}
dns.lookup(originalUrl.hostname, (err) => {
if (err) {
return res.status(404).send({error: 'Address not found'});
};
const { db } = req.app.locals;
shortenURL(db, originalUrl.href)
.then(result => {
const doc = result.value;
res.json({
original_url: doc.original_url,
short_id: doc.short_id,
});
})
.catch(console.error);
});
});At this point, the document that was inserted will be sent to the client as JSON. It will be displayed on the page like this:
Set up a catch-all route for all shortened URLs
Apart from the / and /new routes, we need to handle the other requests for the shortened URLs. Specifically, we need to redirect them to the original URLs. Here’s how we can do so using Express:
app.get('/:short_id', (req, res) => {
const shortId = req.params.short_id;
});Here, we’re using named route parameters to capture the value of the short_id part of the URL. This value can then be accessed in the req.params object under the short_id property.
Once we have the short id, we need to check if a url with that short id exists in the database. Let’s create a new function for this purpose just below shortenURL:
const checkIfShortIdExists = (db, code) => db.collection('shortenedURLs')
.findOne({ short_id: code });The findOne method returns a document that matches the filter object passed to it or null if no documents match the filter.
We can then use the function in our catch-all route like this:
app.get('/:short_id', (req, res) => {
const shortId = req.params.short_id;
const { db } = req.app.locals;
checkIfShortIdExists(db, shortId)
.then(doc => {
if (doc === null) return res.send('Uh oh. We could not find a link at that URL');
res.redirect(doc.original_url)
})
.catch(console.error);
});If the short id exists in our database collection, we will redirect the user to the original_url associated with that short id. Otherwise, an error message is sent to the user.
Now you should be able to shorten long urls, visit the shortened url and be redirected to the original url.
Visualise your database with a GUI
A GUI will allow us to connect to our MongoDB instance and visualise all the data that is present. You also be able to create new data, update existing data and perform other similar operations.
There are several MongoDB GUIs out there, but the one I like to use is NoSQLBooster. It is available for Linux, macOS and Windows, and you can download it here.
Once you install and open the app, it should connect to your local MongoDB instance by default. Otherwise, you can click the Connect button in the top left and connect to it from there.
Once connected, you’ll able to view all the collections present in the database, and interact with the data present in those collections.
Deploy to Heroku
Before we deploy the app to Heroku, we need to create a cloud hosted MongoDB instance first. We can do so using MongoDB Atlas. Create an account at that link and create a new cluster when you are redirected to the Create New Cluster page.
Next, click the Security tab and hit the Add new user button. Give the user a name and password, then click the Add user button.
Following that, click IP Whitelist and then Add IP Address. This allows you to choose what IP addresses can access your cluster. Since this is just a demo application, you can allow access from anywhere by clicking the Allow Access From Anywhere button.
Next, go back to the Overview, and hit the Connect button. Choose Connect your Application and then Short SRV connection string. Take note of that string as we’ll be using it shortly on Heroku.
Head over to the Heroku website and sign up for a free account. Once your account is activated, follow this link to create a new app. Give it a unique name. I called mine freshman-shortener.
Next, follow the instructions here to install the Heroku CLI on your machine. Then run the heroku login command in the terminal to login to your Heroku account.
Make sure you’ve initialised a git repository for your project. If not, run the git init command at the root of your project directory, then run the command below to set heroku as a remote for your git repo. Replace <app name> with the name of your application.
heroku git:remote -a <app name>Next, create a Procfile in the root of your project directory (touch Procfile) and paste in the following contents:
web: node src/server.jsFollowing that, specify the version of Node you are running in your package.json file under the engines key. I specified version 11.2.0 since that’s the version I’m running on my computer. You should change that value to match the version of Node you have on your machine.
{
"name": "url-shortener",
"version": "1.0.0",
"description": "URL Shortener Demo",
"main": "src/server.js",
"scripts": {
"start": "npx nodemon src/server.js"
},
"repository": {
"type": "git",
"url": "git+https://github.com/freshman-tech/url-shortener.git"
},
"keywords": [
"url shortener",
"mongodb",
"nodejs"
],
"author": "Ayo Isaiah",
"license": "MIT",
"bugs": {
"url": "https://github.com/freshman-tech/url-shortener/issues"
},
"homepage": "https://github.com/freshman-tech/url-shortener#readme",
"devDependencies": {
"nodemon": "^1.18.9"
},
"dependencies": {
"body-parser": "^1.18.3",
"dotenv": "^6.2.0",
"express": "^4.16.4",
"mongodb": "^3.1.10",
"nanoid": "^2.0.1"
},
"engines": {
"node": "11.2.0"
}
}Before you deploy the app, head over to the Settings tab in the Heroku dashboard and hit Reveal Config Vars. This is where you will set the environmental variables for your app.
As we did earlier on the local .env file, enter DATABASE as the KEY and the SRV string from MongoDB Atlas as the value, then click Add. Remember to replace <PASSWORD> in that string with password for the user you created earlier.
Notice how easy it is to change the database location depending on the environment the program is running without touching the application code. This is one huge advantage of using environmental variables for project configuration.
Finally, commit your code and push it to the Heroku remote using the following commands:
git add .
git commit -m "Initial commit"
git push heroku masterOnce the deployment process is done, you can open https://<your-app-name>.herokuapp.com to view and test your project.
Conclusion
We’ve successfully created a fully featured URL shortener and learnt the basics of Express and MongoDB along the way. We also learnt how to set up a cloud based MongoDB instance on MongoDB Atlas and how to deploy the app to Heroku.
I hope this exercise was helpful to you. If you have any questions regarding this tutorial, please leave a comment below and I’ll get back to you.
Thanks for reading!