Node Js

CHAPTER-4

A Complete Server: Persistence

Our server is off to the races. We have a functioning API that can service a web front-end and other clients.

This is a great start, but as of right now our data is static. While it’s true that we can dynamically return data to a client with paging and filtering, the source of the data can never change.

If we were building a company on top of this service, it wouldn’t be very useful. What we need now is persistence. A client should be able to add, remove, and modify the data that our API serves.

Currently our data is stored on disk. Our app uses the filesystem to retrieve data. We could continue to use the filesystem when we add persistence. We would do this by using fs . writeFile( ) and other fs methods, but that’s not a good idea for a production app.

When we run apps in production, we want to have multiple servers running at the same time. We need this for a variety of reasons: redundancy, scale, zero-downtime deploys, and handling server maintenance. If we continue to use the filesystem for persistence, we’ll run into a problem when we try to run on multiple servers.

Without special setup, each app would only have access to its local filesystem. This means that if a new product is created on server A, any client connected to server B wou1dn’t be able to access it. Effectively, we’d have multiple copies of the app, but they would all have different data.

We can use a database to get around this issue. We can have as many instances of our apps as we’d like, and they can all connect to the same database, ensuring they all have access to the same data.

Databases allow us to separate the data from our running code, so that our app does not have to run at the same location where our data is stored. This allows us to run instances of our apps wherever we’d like. We could even run a local copy that is identical to the production version (assuming our local version has access to the production database).

Databases do add additional complexity to an app. This is why we started with the filesystem — it’s the simplest way to handle data. However, the small increase in complexity comes with a lot more power and functionality. In the previous chapter, our code was inefficient. When we wanted to find products matching specific criteria, we had to iterate over our full collection to find them. We’ll use database indexes so that these lookups are much more efficient.

There are many different databases to choose from when creating a production app, and they each have their own tradeoffs and are popular for different reasons. When the LAMP stack was popular, MySQL was the go-to. Over time, developers gravitated to other databases that did not use SQL like MongoDB, Redis, DynamoDB, and Cassandra.

SQL databases (MySQL, PostgreSQL, MariaDB, SQLite, Microsoft SQL Server, etc…) store data with tables and rows. If we were to think of the closest analog in JavaScript terms, each table would be an array of rows, and each row would be an array of values. Each table would have an associated schema that defines what kinds of values belong in each index of its rows. By storing data this way, SQL databases can be very efficient at indexing and retrieving data.

Additionally, SQL databases are easy for people to work with directly. By using SQL (structured query language), one can write mostly human readable commands to store and retrieve data with a direct interface to the database server.

While these two attributes have advantages, they also come with some tradeoffs. Namely, to store a JavaScript object in a SQL database we need code to translate that object into a SQL insert command string. Similarly, when we want to retrieve data, we would need to create a SQL query string, and then for each row we’d need to use the table’s schema to map fields to key/value pairs. For example, our product objects each have an array of tags. The traditional way to model this in a SQL database is to maintain a separate table for tags, where each row represents a connection between a product and a tag. To retrieve a product and its tags.

These tradeoffs are by no means a deal-breaker, most production apps will use an ORM (Object Relational Mapping) like Sequelize ‘ to handle these translation steps automatically. However, these additional conversion steps highlight that while very mature, capable, and efficient, SQL databases were not created with JavaScript apps in mind.

Conversely, MongoDB was created to closely match how we work with data in JavaScript and Node.js. Rather than using a table and row model, MongoDB has collections of documents. Each document in a collection is essentially a JavaScript object. This means that any data we use in our code will be represented very similarly in the database.

This has the distinctive property of allowing us to be flexible with what data we decide to persist in the database. Unlike with SQL, we would not need to first update the database’s schema before adding new documents to a collection. MongoDB does not require each document in a collection to be uniform, nor is it required that object properties need to be defined in a schema before being added.

Of course, this additional freedom is itself a tradeoff. It is often very nice to have this flexibility. When creating an app, it is useful to evolve the data as changes are made. Unfortunately, if we aren’t careful, we can wind up with a collection of documents that are inconsistent, and our app will need to contend with collections of objects that vary in subtle (or not so subtle) ways. For this reason, we will use Mongoose to get the best of both worlds.

Over time, developers have come to rely on SQL databases to enforce data validation. If a row is inserted with the wrong types of data in its fields, the database can refuse with an error. The app doesn’t need to worry about handling that type of validation logic. MongoDB has no opinions on what types of data belongs in a document, and therefore data validation needs to be done in the app.

Additionally when persisting data with a SQL database, its common to have store relationships between rows. When retrieving data, the database can automatically “join” related rows to conveniently return all necessary at once. MongoDB does not handle relationships, and this means that an app using MongoDB would need its own logic to fetch related documents.

Mongoose is an ODM (object data modeling) library that can easily provide both of these features (and more) to our app. Later in this chapter, we’ll show how to add custom validation logic that goes well beyond what a SQL database can provide and how to store and retrieve related documents with ease.

Getting Started

We’re now ready to convert our app to use MongoDB to store our data. Luckily, we’ve built our app so that switching the persistence layer will be very easy.

We don’t need to change anything about server . js or api . js because the only difference is where data is stored and how it is loaded. api . js can continue to call the same exact methods from products . js. To get to the same functionality we have using the file system, the only thing we need to do is change the functionality of the Products . list( ) and Products . get( ) methods so that they load data from MongoDB instead of the filesystem.

That said, if we start by making those modifications, it will be difficult to check if they’re working correctly. For Products . list( ) and Products . get( ) to pull data from MongoDB, we’ll need to make sure there are documents in the database. It will be more helpful to first add the Products . create( ) method to make that easier.

Once we add the Products . create( ) method, we can easily import our existing products from the products . j son file. From there we can update Products . list( ) and Products . get( ) , and we can verify that they are working as expected. Finally, after we have those three methods, we’ll add Products . ed i L( ) and Product As . remove( ) .

Creating Products

In the last chapter we left off with the ability to send new products to our service using an HTTP POST. However, in api . js we don’t yet use the products . js model to do anything with the received product object.

Updating the API

We’ll first add a call to Products . create( ) in this route handler. Then, we’ll create that method in our model, and have it use mongoose to persist data in MongoDB.

This is our previous route handler:

async function createProduct (req, res, next) { 
  console.log('request body: ', req.body) 
  res.json(req.body)

And this is what we’ll change it to:

async function createProduct (req, res, next) { 
  const product - await Products.create(req.body) 
  res.json(product)

If we use this route now, we’d expect to see an error because Products . create( ) does not yet exist, but we can fix that now.

In our products . js file, we are no longer going to use the fs module. Instead of using the filesystem as our data source, we’re going to use MongoDB. Therefore, instead of using fs, we’re going to create a new module to use instead, db . js.

db . js will use Mongoose to create and export a client instance that is connected to MongoDB. Before we can use Mongoose to connect to MongoDB, we’ll need to make sure that MongoDB is installed and running”. After that, we’ll use n pm to install mongoose with npm i mongoose, and then create our new module.

const mongoose = require('mongoose')

mongoose.connect(
  process.env.MONGO_URI l | 'mongodb: //localhost:27017/printshop',
  { useNewUrlParser: true, useCreateIndex: true }


module.exports - mongoose

There isn’t too much to this module; it’s so short we might be tempted to inline it in products . js. However, we’d have to pull it out again if we wanted to use this connection in other models, utility scripts, and tests.

mongoose . connect( ) accepts a MongoDB connection string that controls which database server to connect to. If the floNG0_UR I environment variable is available, we’ll use that as the connection location, and if not we’ll use loca1host as the default. This allows us to easily override which database our app connects to.

NOTE: The useNewUrl Parser and useCreate Index options are not required, but they will prevent deprecation warnings when using the current versions of Mongoose and MongoDB.

Now that we’ve created db . js, we can begin to use it in products . js, and the first thing we need to do is to create a mongoose model.

Models and Schemas

With Mongoose, we create a model by using mongoose . model ( ) and passing it a name and a schema object. A schema maps to a collection, and it defines the shape of documents within that collection. Here’s how we can create the Product collection:

const cuid - require('cuid')

const db - require(' /db')

const Product = db.model( 'Product' ,
  _id: { type: String, default: cuid }, 
  description: String,
  imgThumb : String, img: String, 
  link: String, 
  userId: String, 
  userName: String, 
  userLink: String,
  tags: { type: [String], index: true

This command tells mongoose which collection in MongoDB to use, and it controls what kinds of properties documents in that collection should have. By default, Mongoose will prevent us from persisting any properties absent from the schema object.

Each of our product objects should have the following properties: _i d, description, imgThumb, img, link, user Id, user Name, userLink, and tags. If we try to save any additional properties, mongoose will ignore them. For all properties except _id and tags, we simply tell mongoose that we expect the value to be a string. We do this by using mongoose schema shorthand where the type (e.g. String) is the value of the property key.

These two schemas are equivalent:

  description: String }


{ description: { type: String } }

In MongoDB, _id is special property that is always indexed and must be unique. By default, id is an instance of Object I d, a custom MongoDB class with particular methods — not a string. Unfortunately, using a special object for our aid makes working with them more cumbersome. For example, we can’t receive custom objects with methods in query strings or in JSON, and we would need to have extra logic to convert them. To avoid this issue we’ll use cuid⁶⁰ strings that have all of the benefits of Object Id (collision-resistant, time-ordered, and time-stamped), but are just strings .

By providing the default option, we are specifying the function we want to call to create the default value for _id. In this case the function we use is cuid ( ) (we have to be sure to npm install cuid first), which when called will return a unique string (e.g.c jvo24y9o88889wg1dxo7a fr 1).

We don't use it here, but default can also be useful in other cases such as automatically providing a timestamp (e.g. ( times tamp :  ( type :  Number , default : Date . now } ) ).

An advantage of MongoDB is that generally anything that can be represented with JSON can be stored similarly in MongoDB. Each of our product objects contains an array of tags. Here’s an example:

MongoDB allows us to store this object as-is. If we were using a SQL database we would either have to serialize/deserialize the tags before saving/after retrieving (e.g. tags . join( ‘ , ‘ ) and tags . split( ‘ , ‘ )), have difficulties with indexing, and or we would have to create a separate table to store relationships between product and tag IDs.

When creating the schema we tell mongoose that we want the tags property to contain an array of strings. We also want to make sure that MongoDB creates an index for our tags so that we can easily find all product documents that have a specified tag. For example, we don’t want MongoDB to have to do a full scan to figure out which products have the “beach” tag.

mongoose has a pretty clever way to specify that a property is an array of strings: [String] . If instead we expected that to be an array of other types it could be [Number] , [Date] , or [Boolean] . We could even have it be an array of objects with a particular shape: [ { tag: String,  updatedAt:  Date } ] .

For more information on Mongoose schemas check out their excellent documentation“

Mongoose Model Methods

We’ve now created our product model with mongoose, Product. This model instance will allow us to easily save and retrieve product objects from the database.

Here’s what our create( ) method looks like:

async function create (fields) {
  const product = await new Product(fields).save() 
  return product

To persist a new product to the database we first create one in memory using new Product( ) and we pass it any fields we’d like it to have (description, img, etc…). Once it’s created in memory we call its async save( ) method which will persist it to the database.

Once we export this method, Products . create( ) will be available in api . js:

module.exports = { 
  get,
  list,
  create

And now we can start our server with node BI/server -01 . js test our new method using curl:

curl - X POST \
  http://localhost:133T/products \
  -H 'Content-Type: application/json' \
  -d '(
  ”description": ”Rug that really ties the room together”, 
  ”imgThumb”'
-t.2.4&q=80&fm=jpg&crop=entropy&cs=tinysrgb&w=200&fit=max&ixid=eydhcHBfaWQiOjY0MjAxf\

  ”img”’
1&ixid-eyJhcHBfaWQiOjY0MjAxfQ”,
  ”link”’ ”https://unsplash.com/photos/Vra_DPrrBlE”, 
  ”userId” : ”GPlq8En0xhg”,
  ”userName”’ ”Ryan Christodoulou”,
  ”userLink”: ”https://unsplash.com/Omisterdoulou”, 
  ”tags” [

   "room",
   "home",
   "bowling",

We can now use the mongo CLI tool to verify that our document was created. First run mongo from the command line. Then use the following commands to select our database and retrieve all products:

We can see that we only have one product in there, and it’s the one that we just sent to our app via cur 1. Now that we know we have a product in the database, we can modify our Products . list( ) and Products . get( ) methods to use mongoose and test them.

We’ll first change Products . list( ) so that it no longer uses the filesystem. Instead, we’ll use our new Mongoose model to perform a query. We’ll use skip( ) and limit( ) for paging functionality, and find( ) to look for a specific tag (if provided).

async function list (opts = (}) (
  const ( offset = 0, limit - 25, tag } = opts

  const query - tag ? ( tags: tag } ’ (}
  const products = await Product. find(query)
    .sort({ _id : 1 })
    .skip(offset) 
     limit(limit)

return products

We also use sort( { _id : 1 } ) to make sure that we keep a stable sort order for paging. Each of our products uses a cuid“ as its _id, and if we use this to sort, we'll return our products in the order of their creation.

Next, we’ll update our Products . get( ) method. Similarly, we’ll use our mongoose model instead of the filesystem, and most notably, we no longer need to inefficiently iterate through all of the products to find a single one. MongoDB will be able to quickly find any product by its _id:

async function get ( _id) {
  const product - await Product.findById(_id) 
  return product

We’ll start our server with node 81/server -82 . js, and when we use curI to list all products, we should now only see the single one in the database:

Now that we have Products . Iist( ) and Products . get( ) working, we can add functionality for Products . edit( ) and Products . remove( ) and test them easily with curl.

First, let’s update api . js to use our to-be-created Products . edit( ) and Products . remove( ) functions.

async function editProduct (req, res, next) { 
  const change - req.body
  const product = await Products.edit(req.params.id, change)

 res.json(product)

async function deleteProduct (req, res, next) ( 
  await Products remove(req params id) 
  res.json({ success: true })

Then, we add those new methods to products . js and export them:

async function edit (_id, change) { 
  const product - await get({ _id })
  Object.keys(change).forEach( function (key) ( 
  product[key] - change[key]

  await product.save() 
  return product

To modify a product, we first fetch it using Products . get( ), then we change each field with an update, and finally we save it.

In theory, we could do this in a single command with Model.findByIdAndUpdate()“. However, it is not recommended to use this approach with mongoose because it limits the use of hooks and validation (more on this later).

async function remove (_id) { 
  await Product.deleteOne({ _id })

Removing a product is simple. We just use the deIeteone( ) model method.

Let’s test our new methods. We’ll start the server with node 01/server . js, and here’s the cur I command to change the description:

We can then verify that the description was changed:

0 curl -X DELETE http://localhost:133T/products/cjvo3vikw0003n8gl0tq3l8zo (”success”:true}

And finally, let’s make sure that it’s not available:

We’ve now verified that our API supports creating, reading (list and single), updating, and deleting products — all backed by MongoDB.

Validation

Currently, we have very loose controls over the data we’ll accept when creating and updating products. In fact, the only restriction we have is that we won’t accept superfluous properties.

If we were to try to create a product with a property that isn’t listed in our schema, it would not persist to the database (e.g. we tried to create a product with the attribute ( color: ‘ red ‘ }). When we retrieve that product from our API, that property would be missing. Not only that, but if we were to create a product with no other properties, our API wouldn’t stop us. We might be surprised with the result:

As expected, mongoose did not persist the color property. However, it still created the product even though the properties from the schema were missing. The only properties on the object are _id which defaults to the value of cu id( ), tags which defaults to [ ] , and v an automatic mongoose property that counts the number of revisions (i.e. version number) to a document.

In a production app, we want to be careful not to create invalid documents in our database. Luckily, mongoose validation makes this easy.

To make a field required, all we need to do is to specify that when creating the schema:

const Product = db.model('Product',
  _id:	type' String, default' cuid }, 
  description: { type: String, required: true }, 
  imgThumb : ( type : String , required : true ) ,
  img : (  type :  String , required :  true ) ,
  link: String,
  userId: { type: String, required: true }, 
  userName: { type' String, required: true }, 
  userLink : String,
  tags:	type' [String], index' true }

We make descript ion, imgThumb, i mg, user I d, and user Name required fields by using an object for their values and setting required to true. link and userLink are still optional so we continue to use the compact form by simply passing String instead of an options object.

If we run the server now (node 82/server -81 . js) and attempt to create a product with missing fields, we’ll see an error:

0 curl -sX POST \ 
  http://localhost:133T/products \
  -H 'Content-Type : application/json' \
  -d '{}' l jq

  "error" : "Product validation failed", 
  "errorDetails":
    "userName"’
      "message": "Path userName is required.", 
      "name”’ "ValidatorError",
      "properties" : (
        "message”’ "Path 'userName' is required ", 
        "type": "required",
        "path”’ "userName"
     "kind"’ "required",
     "path " :  "user Name"

  ” user Id ” : (
"message": "Path 'userId' is required ",

This is pretty cool. Just by adding a simple option to our schema, we can prevent invalid data from sneaking into our database. We also get descriptive error messages for free — the response automatically includes information about all missing fields. Writing the logic to handle these kinds of error objects by hand can be tedious.

In addition to checking existence, mongoose can help us make sure that a field’s value has the right format. For example, our images need to be proper URLs. If a URL is invalid, we want to know right away so that we can fix it; we don’t want a user to encounter a bug production.

To check whether or not a URL is valid were going to use a new module,validator. After instaling it with npm install validator we can use it like this:

const ( isURL } = require('validator')

isURL( * not a url ' ) //  lab se
isURL( https : // fullstack . io ’ ) /7  have

The great thing about this is that mongoose accepts arbitrary validation functions. This means that we can simply pass i sURL to mongoose for our URL fields:

userLink: { 
  type: String, 
  validate: {
    validator: isURL,
    message: props =› ’${props.value} is not a valid URL’

Two things to notice here: first, optional fields can have validation (we would have set required: true if it was required). If this field is omitted, mongoose won’t check to see if it’s a valid URL. Second, we can pass a function as message that will generate validation error messages for us. In this case we dynamically insert the provided (invalid) value into the message.

After adding custom validation to img, imgThumb, link, and userLink, our schema definition has become quite long. We can clean this up a bit by creating a urlSchema( ) function that will generate the object for us:

function urlschema (opts = (}) ( 
  const ( required } = opts 
  return (
    type: String, 
    required: ! !required, 
    validate: {
      validator: isURL,
      message: props =› ’${props.value} is not a valid URL’

Now our schema can be much simpler:

const cuid - require('cuid')
const ( isURL } - require('validator')

const db = require( ' . /db  ' )

const Product - db.model('Product', {
  _id: ( type: String, default: cuid }, 
  description: { type: String, required: true }, 
  imgThumb: urlschema( { required: true }),
  img: urlschema(( required: true }), 
  link: urlschema(),
  userId: { type: String, required: true }, 
  userName: { type: String, required: true }, 
  userLink: urlschema(),
  tags: ( type: [String], index: true }

And now to try it out with curl:

Looking good! It’s great to have an API return detailed information about why a client request fails. Even better that we get this functionality for free with Mongoose.

At this point we have all the tools we need to ensure our database has valid documents and that we keep it that way. Most of the time, we only need to make sure that specific fields are present and that they follow specific formats.

mongoose has additional schema options that we can use. Check out the documentation on SchemaTypes“ for other useful features like trim and lowercase.

Relationships

Our app is in great shape. We can create and update products via our API, and we can put controls in place to make sure every product is valid.

What do we do when someone wants to buy something?

In this section we’re going to show how to create relationships between two different models. We’ve already seen how to create a model, but we’re going to show how we can create links between them.

When a customer wants to buy a product, we’re going to want to create an order that tracks the products that they’ve purchased. If we were using a SQL database we would create a new table and use no I N queries to handle these relationships.

With MongoDB and mongoose we’ll create our schema for the order model in such a way that this can be handled automatically for us. Each order will contain an array of product IDs, and we can have mongoose convert those into full product objects.

For our app we’ll create a new set of endpoints for our orders. A client should be able to create and list orders using the API. We should also support the ability to find orders by product ID and status. Implementing these features is similar to what we did for products.

Here are our new routes:

app.get('/orders', api.listorders) 
app.post('/orders', api.createOrder)

And route handlers:

async function createorder (req, res, next) { 
  const order - await Orders.create(req.body) 
  res.json(order)

async function listorders (req, res, next) {
  const ( offset = 0, limit = 25, productId, status } = req.query
 
  const orders = await Orders list({ 
    offset: Number(offset),
    limit: Number(limit),
    productId,
    status

res.json(orders)

Next up, we’ll create our new orders model, but before we get into that, it’s time to do a small bit of housekeeping. It’s important to evolve the file structure of our apps as they evolve. Our app started simple, and there we kept our file structure simple. However, now that our app is growing and is about to have two models, we should introduce a little more hierarchy to our directory.

Let’s create a model s directory and move our products . js model into it. After that’s done we can create our orders . js model.

Here’s what that looks like:

const cuid - require('cuid')
const ( isEmail } - require('validator')

const db = require('  /db')

const Order - db.model('Order', {
  _id: ( type: String, default: cuid }, 
  buyerEmail: emailschema( { required : true }), 
  products: [
 
      type: String, 
      ref: 'Product', 
      index: true, 
      required ' true

  status: {
    type: String, 
    index: true, 
    default: 'CREATED',
    enum: ['CREATED', 'PENDING', 'COMPLETED']

If we look at the products field, we can see that this schema expects an array of strings. This works the same way as our tags array in the schema for products. What’s different here is that we use the ref option. This tells rnongoose that each item in this array is both a string and the _id of a product. mongoose is able to make the connection because in products . js the name we pass db . model ( ) exactly matches the ref option. This allows mongoose to convert these strings to full objects, if we choose to.

By using the ref feature, mongoose is able to automatically fetch associated products for us. However, it won’t do this by default. To take advantage of this feature we need to use the populate( ) and exec( ) methods. Here they are in action:

async function get (_id) {
  const order = await Order. findById(_id)
    .populate( 'products')
    .exec() return order

If we did not use populate( ) and exec( ) , the products field would be an array of product IDs — what is actually stored in the database. By calling these methods, we tell mongoose to perform the extra query to pull the relevant orders from the database for us and to replace the ID string with the actual product object.

What happens if the associated product has been deleted or can't be found? In our example we're using an array of products. If an associated product can't be found (e.g. it's been deleted) it will not appear in the array. If the association was a single object, it would become nut 1. For more information see the mongoose documentation on populate( ) “

Let’s see our new orders endpoints in action. First, we’ll create a product, then we’ll create an order using that product ID, and finally we’ll get a list of our orders.

Using the product ID from the response, we create the order:

We can already see that even though we’re using an array of product ID strings, mongoose is automatically expanding them into full product objects for us. We’ll see the same thing when we request a list of orders:

If we were to use the mongo command line client, we can see that the order is stored differently:

Data For Development And Testing

When working locally it’s very useful to have data in our database. Before we switched to MongoDB, we were using a JSON file that had products to test with. However, databases start empty, and ours will only have products and orders that we create using the API.

To make things easier on ourselves, we’ll create a simple script that uses our products . js module to create documents in our database to work with:

const db = require(' ./db')
const Products - require(' /models/products')

const products - require(' /. /products json' )

;(async function () {
  for (var i = 0; i products.length; i++) 
   console.log(await Products.create(products[i]))

db.disconnect()

All we’re doing here is using the Products . create( ) method directly by iterating over all the products in the JSON file. The only trick is that we want to close the database connection when we’re finished so that Node.js will exit; Node.js won’t terminate while there is a connection open.

We shouldn’t need to run this very often, but when we do, all we need to do is to run node 03/script/import-products.js.

If for some reason we want to clear out our database, we could do that using the mongo CLI:

> db.dropDatabase()
{ ”dropped”  "printshop, ”ok” 4 }

Of course, we need to be very careful that we’re connected to the correct database when we do this.

File Uploads

Another form of persistence is file uploads. These are very common in web apps and can range from profile photos to spreadsheets to videos. Sometimes it’s appropriate to store file contents in a database, but 999 of the time, it’s better to store them on a dedicated file server.

For example, profile photos are typically public and do not require any security. Additionally, they tend to be accessed frequently; therefore, storage should be optimized for retrieval. For this use-case, we would store the images using a service like Amazon S3, Google Cloud Storage, or Azure Blob Storage, and users would access them through a Content Delivery Network (CDN) like Amazon Cloudfront, Google Cloud CDN, or Azure CDN.

Most of the bigger name storage services will have modules available on npm to use. We’re not going to integrate this into our app, but here’s an example of how to upload an image from the filesystem to Amazon S3:

const fs - require('fs')
const AWS - require('aws-sdk')
const ( promisify } - require(' util ')

const s3 - new  AWS.S3() 
s3. uploadP - promisify(s3. upload)

const params - {
  Bucket :  ' full stack - printshop ' , 
  Key: ’ profile - photos/thedude . j pg ’ ,
  Body:  is . createReadS dream( ' thedude . j pg ' )


(async function () {
  await s3.uploadP(params)

Assuming this bucket has public permissions set, this image would be publicly available at https://fullstack-printshop.s3.us-east-1.amazonaws.com/profi1e-photos/thedude.jpg. For more infor- mation about how to use Node.js with Amazon services, see their documentation‘⁶.

In this example, we’re using a stream from the filesystem, but HTTP requests are also streams. This means that we don’t have to wait for a client to finish transferring a file to our server before we start uploading it to object storage. This is the most efficient way to handle uploaded files because they don’t have to be kept in memory or written to disk by the server.

It’s typically better to serve frequently accessed files from a CDN. Therefore we would also set up a CDN host backed by this S3 bucket. The CDN acts as a cache in front of the bucket. The CDN speeds things up dramatically by serving files from servers close to the user.

So how would we incorporate this knowledge into our app? Our app currently expects that product images are URLs. We make a small change so that the image URL is no longer required to save a product. We would allow a client to create a product without an image — however, we would then add another endpoint that allows a client to upload an image to be associated with a particular product.

We’re using middleware to parse request bodies, but currently this will only attempt to parse JSON (when the content-type header is application/json). If we set up a new route handler and the client makes an HTTP POST with an image body, the JSON body parser would ignore it. This means that our route handler would be able to use the HTTP request stream directly when uploading to object storage.

This means that before a product could go live, the client would have to complete a two-step process. However, we gain advantages from this approach. Storing images this way is way less expensive than in a database, and we would never run out of disk space.

async function setProductImage (req, res) { 
  const productId = req.params.id

  const ext = (
    ' image/png ' ' png ' , 
    ’ image/j peg ’ ' j pg '
  ) [req . headers [ ' content - type ' ] ]

  i I  ( ! ext ) throw new Error ( ' Invalid Image Type ' )

  const params = {
    Bucket :  ' fullstack - printshop ’ ,
    Key: product - images/$(product  Id} . $(ext)	,
    Body: req , //  seq  s a s  dream,  st n	as  to	fs . createPeadStrean()
    ACL : ’ public — read

 const object - await s3. uploadP(params) // our custom promise version 
 
 const change - { img: object.Location }
 const product = await Products.edit(productId, change)

 res.json(product)

We haven’t added this to our app, but if we did, we could use a curl command like the following to use it:

curl -X POST -H “Content-Type: image/jpeg” --data-binary Athedude.jpg http://localhost:133T/products/cjvzbkbv00000n2glbfrfgelx/image

We can optimize things even further if we set up a CDN server that sits in between the user and our object storage server. Any object URL would then map directly to a CDN url. When the CDN URL is accessed, the CDN will return the file if it has it available, and if not, it will grab it from the object storage server first (and store a copy locally for next time). This means that instead of getting the image from https://fullstack-printshop.s3.us-east-1.amazonaws.com/profile-photos/thedude.jpg we would access it at https://printshop-cdn.fullstack.io/profile-photos/thedude.jpg.

What do we do if the files are not public and we want to prevent sharing object storage URLs? The approach that we'd take would vary slightly depending on the service that we decided to use. However, most object storage services and CDNs support authorization schemes. By default the URLs would not be publicly accessible, but we would generate expiring access tokens to be given to users right before they need to access the file in question. This would be the method we'd use if we were selling downloadable software, eBooks, or video courses.

Wrap Up

In this chapter we’ve taken our API from static to dynamic. Hitting the same endpoint will not always return the same data. Clients can now create, edit, and remove documents. We’ve learned how to create models backed by MongoDB, enforce validation, and map relationships.

Using what we’ve learned up to this point, we can create sophisticated APIs that are very useful. However, we do have one big problem. Our service does not have any controls over who can use it. Sometimes this can be OK if the service is only internally accessible, but for a publicly available e-commerce server, this won’t work. We can allow anyone to create or remove products.

In the next chapter we’re going to cover authentication and authorization. We’ll see how to identify users and clients by their credentials and how to use that identification to control access.