tl;dr — This is the fourth article of the Node is Simple article series. In this article series, I will be discussing how to create a simple and secure NodeJS, Express, MongoDB web application.
To follow the past tutorials, Node is Simple Part 1, Node is Simple Part 2 and Node is Simple Part 3.
Hello fellow developers, now you’ve come across this article series, let’s get to it, shall we. In the past articles, I have discussed how to create simple CRUD endpoints, with MongoDB as the database and how to use Postman to test the endpoints. So in this tutorial, I will discuss how to upload files to MongoDB using MongoDB GridFS and view (or stream) them.
So what is this GridFS?
We can upload files to a folder easily with Multer-Middleware. This is a good reference for that.
Uploading Files and Serve Directory Listing Using NodeJS
But I am going to discuss how to use MongoDB as the file storage. However, there is a little hiccup, where you can only store 16MB as a document in MongoDB BSON format. To overcome this issue, there is a feature called GridFS.
Instead of storing a file in a single document, GridFS divides the file into parts, or chunks [1], and stores each chunk as a separate document.
Enough with the theory
Yes, let’s move on to implementation. First, we have to create a GridFS driver. Let’s create gridfs-service.js file inside /database directory.
const mongoose = require("mongoose");
const config = require("../config");
const dbPath = config.MONGO_URI;
const chalk = require("chalk");
const GridFsStorage = require("multer-gridfs-storage");
const Grid = require("gridfs-stream");
Grid.mongo = mongoose.mongo;
const conn = mongoose.createConnection(dbPath, {
useNewUrlParser: true,
useUnifiedTopology: true,
useCreateIndex: true,
useFindAndModify: false
});
conn.on("error", () => {
console.log(chalk.red("[-] Error occurred from the database"));
});
let gfs, gridFSBucket;
conn.once("open", () => {
gridFSBucket = new mongoose.mongo.GridFSBucket(conn.db, {
bucketName: "file_uploads"
});
// Init stream
gfs = Grid(conn.db);
gfs.collection("file_uploads");
console.log(
chalk.yellow(
"[!] The database connection opened successfully in GridFS service"
)
);
});
const getGridFSFiles = id => {
return new Promise((resolve, reject) => {
gfs.files.findOne({ _id: mongoose.Types.ObjectId(id) }, (err, files) => {
if (err) reject(err);
// Check if files
if (!files || files.length === 0) {
resolve(null);
} else {
resolve(files);
}
});
});
};
const createGridFSReadStream = id => {
return gridFSBucket.openDownloadStream(mongoose.Types.ObjectId(id));
};
const storage = new GridFsStorage({
url: dbPath,
cache: true,
options: { useUnifiedTopology: true },
file: (req, file) => {
return new Promise(resolve => {
const fileInfo = {
filename: file.originalname,
bucketName: "file_uploads"
};
resolve(fileInfo);
});
}
});
storage.on("connection", () => {
console.log(chalk.yellow("[!] Successfully accessed the GridFS database"));
});
storage.on("connectionFailed", err => {
console.log(chalk.red(err.message));
});
module.exports = mongoose;
module.exports.storage = storage;
module.exports.getGridFSFiles = getGridFSFiles;
module.exports.createGridFSReadStream = createGridFSReadStream;
If you can see line 63, the “bucketName” is set to “file_uploads”. This means the collection that we are using for storing the files is named as file_uploads. After running the application if you go to the MongoDB Compass, you can see there are two new collections are created.
file_uploads.chunks //contains the file chunks (one file is divided in to chunks of 255 kiloBytes.
file_uploads.files //contains the metadata of the file (such as lenght, chunkSize, uploadDate, filename, md5 hash, and the contentType)
Now we need to install several packages for this. Let’s install them.
$ npm install multer multer-gridfs-storage gridfs-stream
Multer is the middleware that handles multipart/form-data. This is a good reference if you are not familiar with them.
Understanding HTML Form Encoding: URL Encoded and Multipart Forms
The other two packages are for the file upload handling with MongoDB GridFS.
After creating this file, let’s create the GridFS middleware. It is really easy to integrate with Express since Express is handy with middleware. Let’s create /middleware/gridfs-middleware.js file.
const multer = require("multer");
const { storage } = require("../database/gridfs-service");
const upload = multer({
storage
});
module.exports = function GridFSMiddleware() {
return upload.single("image");
};
There is something to mention here. If you can see line 9, I set “image” as the name of the form field name. You can set it to anything you like, but you have to remember it for later use. FYI, There are several methods of uploading files.
upload.single("field_name"); //for uploading a single file
upload.array("field_name"); //for uploading an array of files
upload.fields([{name: 'avatar'}, {name: 'gallery'}]); //for uploading an array of files with multiple field names
upload.none(); //not uploading any files but contains text fields as multipart form data
Here I used, upload.single(“field_name”) because I want to upload only one image (Probably I’ll change this to uploading the student’s profile picture). However, the things I’ve described in the above code-block are not something I brew up myself 🤣. They are all described beautifully in the multer documentation.
Now we can set up the controller to upload files to MongoDB GridFS. Let’s update the /controllers/index.js file.
const GridFSMiddleware = require("../middleware/gridfs-middleware");
/** @route POST /image
* @desc Upload profile image
* @access Public
*/
router.post(
"/image",
[GridFSMiddleware()],
asyncWrapper(async (req, res) => {
const { originalname, mimetype, id, size } = req.file;
res.send({ originalname, mimetype, id, size });
})
);
Now let’s try to upload an image, shall we? Let’s start the web application as usual.
$ pm2 start
As in figure 1, you have to select the request body as form-data and then you have to set the key as image. Now select the key type as “File”. (As seen in Figure 2)
Now if you see the response as figure 3, it means everything worked out. Now let’s see how the MongoDB Compass shows the uploaded file.
As you can see in figure 4, the file metadata is shown. In figure 5, you can see only one chunk. That is because the image we uploaded is less than 255kB in size.
Now that we have uploaded the file, how do we view it? We cannot see the image from the DB, now can we? So let’s implement a way to view the uploaded image.
Now let’s update the /controllers/index.js file.
const { getGridFSFiles } = require("../database/gridfs-service");
const { createGridFSReadStream } = require("../database/gridfs-service");
/** @route GET /image/:id
* @desc View profile picture
* @access Public
*/
router.get(
"/image/:id",
asyncWrapper(async (req, res) => {
const image = await getGridFSFiles(req.params.id);
if (!image) {
res.status(404).send({ message: "Image not found" });
}
res.setHeader("content-type", image.contentType);
const readStream = createGridFSReadStream(req.params.id);
readStream.pipe(res);
})
);
What we are doing here is, we get the image id from the request and check for the image in the DB and if the image exists, we stream it. If the image is not found we return a 404 and a not found message. It is as simple as that.
Now let’s try that out.
Remember the image id is what gets returned as the id in figure 3.
If we send the GET request from a web browser, we’ll see the image in the browser. (If you get a “Your connection is not private” message, ignore it.) It will be the same as in figure 7.
So that was a lot of work, isn’t it? But totally worth it. Now you can save your precious images inside a MongoDB database and view them from a browser. Go show off that talent to your friends and impress them 😁.
This is the full /controllers/index.js file for your reference.
/controllers/index.js file (Updated to reflect the view and upload image endpoints)
So, this is the end for this tutorial and in the next tutorial, I will tell you how to create and use custom middleware with Express. (Remember I told you Express is great with middleware). As usual, you can see the whole code behind this tutorial in my GitHub repository. (Check for the commit message “Tutorial 4 checkpoint”.)
So until we meet again, happy coding…
