The workflow with Docker in industrial projects

I found the following chart that perfectly explain the workflow with Docker in practice Docker Tutorial for Beginners [FULL COURSE in 3 Hours].

As shown above, the steps of the workflow are (all Docker related terms will be explained in the next section):

• After you complete the application development and commit the code to Git, Jenkins will be notified via webhooks

• Jenkins will buid the docker images according to the dockerfile provide
• Jenkins will push the built images to the Docker repository, like Docker Hub or AWS ECR (Elastic Container Registry)
• The server will pull the images from the Docker repository, then create and run containers according to the Docker Compose files (*compose.yaml files)

The Core Concepts of Docker

In the above workflow, I mentioned many essetianls in Docker. In this section, I will explain them in details.

Docker Sever (Docker Deamon)

It’s responsible for managing Docker containers, images, networks, and volumes. It is run on the same machine where Docker is installed. The Docker Client will send all requests to Docker Server.

Docker Client (CLI)

The Docker Client is the interface that users interact with when using Docker.

Docker Hub

Docker Hub is a Docker Repository. It is the default public registry for Docker images, created and maintained by Docker, Inc.

Docker Container

An instance of an image is called a container. It is the running image.

Docker Image

Stopped container. Docker image are read-only layers, and the docker container is the writable layer on top of a image.

Docker Compose

Docker Compose is a powerful tool for defining and managing multi-container Docker applicaitons. In practice, you can sepcify your applicaton’s services, networks, and volumes in a docker-compose.yml file, and then use a single command to start
them all.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

version: '3.9'

services:
  mongodb:
    image: mongo:6.0 # Replace with the desired MongoDB version
    container_name: mongodb-container
    restart: always
    ports:
      - '27017:27017' # Expose MongoDB's default port
    environment:
      MONGO_INITDB_ROOT_USERNAME: admin # MongoDB root username
      MONGO_INITDB_ROOT_PASSWORD: password # MongoDB root password
    volumes:
      - mongodb_data:/data/db # Persistent data storage

volumes:
  mongodb_data:
    driver: local

Dockerfile

A Dockerfile is a script that contains a series of instructions to build a Docker image, inccluding the base image,
dependencies, configurations, and commands to run the application.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

# Use the official Node.js LTS image as the base image
FROM node:18-alpine

# Set the working directory inside the container
WORKDIR /app

# Copy package.json and package-lock.json files to the container
COPY package*.json ./

# Install dependencies
RUN npm install --production

# Copy the rest of the application files
COPY . .

# Expose the port the app runs on
EXPOSE 3000

# Define the command to run the application
CMD ["npm", "start"]

About CMD ["npm", "start"]: Specifies the command to start the Node.js application. Make sure your package.json has a start script defined (e.g., “start”: “node index.js”).

Private Docker Registry

Normally, we push our Docker images to pricate docker registry rather than a public one. A commonly used private Docker registry is AWS ERC (Elastic Container Registry).

Docker Volumes

Docker Volumes is used to persist data in Docker and prevent data loss in stateful applications. They allow folders from the host’ physical file system (e.g. /home/mount/data) to be mounted into the virtual file system (e.g. /var/lib/mysql/data) of Docker.

There are three types of Docker volumes:

• Docker-managed volumes: Managed entirely by Docker and stored in its default location.
• Anonymous volumes: Automatically created without a specific name, typically for temporary use.
• Named volumes: Explicitly named and can be referenced by their names. Named volumes are the most common choice in production environments due to their reusability and ease of management.

Summary

The above covers the core concepts of Docker, providing a quick start for beginners. You can explore each part in greater details based on your interests.

What do we need to keep in mind when considering using Promise.all

• Order matters. Promise.all returns the results in the same order as the source.
• Fail one, Fail all. If one of the promise rejects, then the Promise.all will not wait for the remaining promise to execute but rejects right away.

When do we need Promise.all

In real project, normally when we have an array of data, for each element in the array, we need to do some async/await operations, this is where we need Promise.all. For example, in the following code snippets from a real project, we have an array called idsForDelete, for each id in this array, we need to call an async method removeMonitorTask(apiKey, id). Therefore, we use Promise.all to do async/await calls on all ids in the array.

1
2
3
4
5
6
7
8

...
const deleteIds = await Promise.all(
  idsForDelete?.map(async (id) => {
    const response = await removeMonitorTask(apiKey, id);
    return response?.data[0]?.trackerId?.toString() === id.toString();
  })
...
);

How do we handle exceptions with Promise.all

There are two ways of handling exceptions with Promise.all:

• catch an exception at the end of Promise.all;
• catch an exception in each individual promise.

Only catching exceptions in the latter way when we need to:

• Keep the results of other promise
• Want to know the reason for each rejected promise

Check the following example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

p1 = Promise.resolve(10);
p2 = Promise.resolve(20);
p3 = new Promise(function (resolve, reject) {
  setTimeout(reject, 100, 'test');
}).catch((err) => {
  console.log(`internal err: ${err}`);
});

(async () => {
  const res = await Promise.all([p2, p3, p1]);
  console.log(res);
})().catch((err) => {
  console.error(`err: ${err}`);
});

Execution result:

1
2
3

-- internal err: test
-- [ 20, undefined, 10 ]

As we can see from the execution result that even p3 is rejected, we still get the results of previous p1 and the p2 afterwards.

What if we only catch the exception after Promise.all?

1
2
3
4
5
6
7
8
9
10
11
12

p1 = Promise.resolve(10);
p2 = Promise.resolve(20);
p3 = new Promise(function (resolve, reject) {
  setTimeout(reject, 100, 'test');
});

(async () => {
  const res = await Promise.all([p2, p3, p1]);
  console.log(res);
})().catch((err) => {
  console.error(`err: ${err}`);
});

Result:

1

-- err: test

As shown above, Promise.all rejects all, we are not able to see the result of p2.

That is it, that is all you need to know about Promise.all in most of the time when developing real-world projects.

Context of the Issue

First, let us go through the context of the issue.

Store the Language Settings in React Redux

In the i18next, it has a global variable i18n.language to store language choice and is applied to the whole app. However, it is not enough, if we do not store the language setting, once the app is closed, the user choice will lost. Therefore, I used React Redux to store the language setting, as shown below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

...
const initialState = {
  language: '',
}

export const appSettingSlice = createSlice({
  name: 'appSettings',
  initialState,
  reducers: {
    getLanguageSetting: (state) => {
      return state.language
    },
    setLanguageSetting: (state, action) => {
      if (!action.payload) return
      state.language = action.payload
    },
    resetLanguageSetting: (state) => {
      state.language = ''
    },
  },
})
...

The Expected Scenario

The project is structured as blow. The language in redux is set under set-language/index.jsx, along with i18n.language. And set-language is routed from my-account. What I expected is that if I change the language, once I switch back to the home screen, it should re-render and present with the changed language.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

-- root
  -- app
    -- (tabs)
      -- home
        ...
        -- index.jsx
        ...
      -- my-account
      ...
    -- set-language
      -- index.jsx
      -- index.style.js
    ...
    -- index.js // entry
    -- _layout.js
    ...
  ...
-- index.js
...

The Re-render Issue

Things did no go as I expected. Once I change the language setting and switch back to the home screen, it changes nothing, the scren still show the preivous language.

RCA (Root Cause Analysis)

I added a useEffct hook into (tabs)/home/index.jsx, and would like to see why the home screen does not re-render. The code is shown below:

1
2
3
4
5

useEffect(() => {
  console.log(`language: ${language}`);
  console.log(`i18n.language: ${i18n.language}`);
  i18next.changeLanguage(i18n.language);
}, []);

Then I changed the language and i18n.language from the set-language again, I found out that as soon as I changed the variables, the logs were shown on the terminal (means that the useEffect hook in the home screen was triggered). When I switched back to the home screen, the terminal did not print out the logs again (means that useEffect hook was not trigered).

Hence, I reached to the conclusion that under the tabs navigation of Expo Router, the tabs are somehow binded together, i.e. change states in one tab, will cause the useEffect hooks in the other tabs triggered. In this case, the other tabs consider themselves are re-rendered already. For example, in my case, I changed the language state under my-account/set-language, the useEffect hook in the home screen is triggerd, since I am still in the set-language screen, the components in the home screen will not be re-endered. When I switch back to the home screen, as there is no state change, the useEffect hook will not be triggered, and the language in the home screen stay with the old one.

The Solution

After identifying the root cause, the thought for the solution is obvious: find a method to ‘force’ the (tabs)/home screen to be re-rendered when change the language state in the (tabs)/my-account/set-language screen.

Enlightened by the information provided from Device context doesn’t work with Expo Router, I use the i18n.language as the key value for the Stack component under app/_layout.js (which is the parent componet for (tabs) ). Therefore, everytime the i18n.language is changed, the key value of the Stack component of app/_layout.js will be changed, thus cause all tabs under the (tabs) directory be re-rendered. The source code is shown below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

...

const Layout = () => {
  const { i18n } = useTranslation()

  return (
    <Provider store={store}>
      <PersistGate loading={null} persistor={persistor}>
        <Stack key={JSON.stringify(i18n?.language)}>
          <Stack.Screen name='(tabs)' options={{ headerShown: false }} />
        </Stack>
      </PersistGate>
    </Provider>
  )
}
...

Compile and run the app again. Bingo! The feature works as expcted :)

I searched a bit regarding integrating Redux into Expo, however, in my Expo project, I used tabs navigation structure, so there is no app.js as the main entry (The file structure is listed blow). And all the solutions that I found only works on the Expo project that has an App component (normally app.js) as an entry point. After a few hours of trouble shooting, I finally found the solution myself and decided to share the method in this article, hopefully it will help someone in the future.

1
2
3
4
5
6
7
8
9
10
11

-- root
  -- app
    -- (tabs)
      -- home
      ...
    -- index.js // entry
    -- _layout.js
    ...
  ...
-- index.js
...

Step by Step Guide

Although the way to the destination is rugged, the solution is quite simple. To embed Redux into an Expo project with the tabs navigation structure, we only need to add the redux provider into two files: app/index.js and app/_lauout.js.

app/index.js

As shown below, we wrap the <Redirect href="/home" /> with the Redux provider and the PersistGate.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

import { useRootNavigationState, Redirect } from 'expo-router';

import { Provider } from 'react-redux';
import { PersistGate } from 'redux-persist/es/integration/react';
// store and persistor are self-defined common redux component, nothing special
import { store, persistor } from '../services/redux/store';

const Home = () => {
  return (
    <Provider store={store}>
      <PersistGate loading={null} persistor={persistor}>
        <Redirect href="/home" />
      </PersistGate>
    </Provider>
  );
};

export default Home;

app/_layout.js

This is the tricky part, I could not image that to make Redux work, I need to wrap the Stack with the Redux provider and the PersistGate until I found out that this is a key step.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

import { Stack } from 'expo-router';

import { Provider } from 'react-redux';
import { PersistGate } from 'redux-persist/es/integration/react';
import { store, persistor } from '../services/redux/store';

const Layout = () => {
  return (
    <Provider store={store}>
      <PersistGate loading={null} persistor={persistor}>
        <Stack>
          <Stack.Screen name="(tabs)" options={{ headerShown: false }} />
        </Stack>
      </PersistGate>
    </Provider>
  );
};

export default Layout;

After adding the two, I am able to use the Redux in the files under /app. Happy coding: )

The Definition of Overengineering

On Wikipedia, Overengineering is defined as “the act of designing a product or providing a solution to a problem in an elaborate or complicated manner, where a simpler solution can be demonstrated to exist with the same efficiency and effectiveness as that of the original design.” I would like to simplify it as: choose a more complicated solution to resolve a problem when there exist simpler ones.

How to avoid overengineering

The tricky thing is, in practice, normally it is hard to identify overengineering in the system that is designed by yourself.

From my own personal experience, except harness yourself with rich knowledge and hands on experiences, the best option is to leverage collective wisdom, i.e. consult as many people as you can.

Case Study

A few months ago, my team had a design decision to make for a web system that is under developing. The scenario can be simplified as:

• A group of users, on peak time, the QPS was estimated as around 5K, would request and compete for a limited number of resources (the number of the resources would be less than 500) in a pool.
• Users would request the resources via RestAPI call. The users’ requests would be served in First Come First Serve style, non pre-emptive.
• The individual resource itself has a rate limiting, if it receives more than 20 requests per minute, then it will return an error and become unavailable for a few minutes.
• VIP users could be assigned a resource exclusively and still under the “20 requests per minute” rule.
• According to internal tests (from the ones who were not from the dev team), the possibility that requests from the user might take more than 3 seconds to finish was high (roughly more then 98%).

The design is targeted for resolving two issues of the scenarios:

Resources competition.

How should the system handle the case that multiple users competing for the limited number of resources?

A single resource received more then 20 requests per minute.

How should the system handle the situation that a resource receives more then 20 requests per minute from a VIP user?

One of my colleague instantly proposed using message queues as the core of the solution. To be specific:

for “Resources Competition”

• All the user requests should be put into a message queue and be pulled out in FIFO style.
• A worker would pick a request and assign a resource once there is any available ones in the pool.

for “More than 20 Requests issue”

• Assign a message queue for each resource, store the requests sent from the user into a message queue, a worker pull requests from the queue in FIFO style.
• Only let the requests be pulled out from the queue under the rate of 20 per minute.

The solution is shown below.

Add graph

Everything seems perfectly “reasonable” right now. However, in the evening of the same day, I gave the problem a second thought, then I realised that message queues might be an overengineering solution.

Resources Competition

From User Experiences Perspective

• Resources Competition. On the one hand, let us assume if a message queue is not used here, then once all the resources are assigned, a user will receive an error for the resource request and the user will be informed by a message that he/she should retry after a few minutes. On the other hand, if a message queue is adopted here and all the resources are assigned, then a user’s request will store in the queue and the user will need to wait for the response for unforeseeable time, which does not improve user experiences compare to the first case.

From System Performance Perspective

Well, it is obvious that introducing message queue will not improve system performance.

Therefore, for “Resources Competition”, adopting a message queue is an overengineering option.

More than 20 Requests issue

I was thinking about forcing a user to wait for 3 seconds as an alternative option, i.e. each time the user consumes the resource and takes less than 3 seconds (60 seconds / 20 = 3), then force the user to wait until 3 seconds passed (3 seconds count from taking the ownership of the resource to release it), it ensures that the user will not trigger the 20 requests per minutes error.

From User Experiences Perspective

Consider the fact that more then 98% of chance that a user’s request will take more than 3 seconds to complete, it means that forcing a user to wait for 3 seconds passed if a request takes less than 3 seconds will not significantly impact user experiences.

From System Performance Perspective

One resource per message queue add heavy burden on the backend system while it does not improve the system performance in any means compare to the forcing wait method.

Therefore, for “More than 20 Requests issue”, using message queue solution is an overengineering option too.

With the arguments and conclusions in mind, the other day, I successfully convince the team to avoid adopting message queue which relief the backend dev team :).

The payment platform manages everything for you

There are some payment platforms that manage everything for you. You do not need to bother to design the frontend UIs of “placing order” page (choosing the products listed on the page) and “paying” page (filling credit card information and actually transferring the money), not to mention the backend logics behind the UIs. The platforms do all these things for you. Let us take lemon squeezy as an example(Well, I swear I did not receive ads fee from lemon squeezy, but maybe they should do so :D).

For integrating lemon squeezy into a system, all a software engineer needs to do is:

• Set a store page on lemon squeezy, fill the page with information includes: product information and payment information, as shown below. The customers should complete all the steps regarding buying products on the store page: select products, filling payment information and transferring money.

  

• Add a link to the frontend page where you would like to redirect the customers to the store page of lemon squeezy

• Add a page for customers to activate the keys recieved from lemon squeezy. After a customer complete payment on the store page, lemon squeezy will return an activation key to the customer, the customer should use the key to claim the ownership of the product on the website. On the backend of the website, when activating the key, the software engineer needs to do two things: 1) validate the key with lemon squeezy; 2) activate the key with lemon squeezy.

With such payment platform, the software engineer only need to save the activation key and the order id returned from lemon squeezy. lemon squeezy provide a orders page for the store owners to check order information, as shown below.

Cons:

There is no perfect solution in reality. The cons of using lemon squeezy as I observed from my own experiences are:

• User experience. Rediect to a totally seperate payment page affect user experiences regarding online purchasing in a bad way. In some countries, the page loading speed may need up to 30 seconds.

• Trust issue. As the domain name for the lemon squeezy store page is different from the source site, it may create trust issue for the potential customers

• Less control. High managed platform means little spaces for customization. For instance, the layout of the store page UIs are not in the control.

Type 2: Manage everything on your own

For this type of platform, a typical work flow is:

1

select products --> select payment method --> place the order to the platform via RestAPI --> the platform returns a qrcode link for the customer to pay --> nortify the website the payment has completed

in which you have everything in control, i.e. design UIs of products page and payment page, handling all the backend logics.

Pros and Cons

The cons of adopting the type of platform is that it may takes time to build it as everything is under your control. And the pros are clearly the opposite side of the cons of type 1 platform.

Calling Async Method in ‘useEffect’

There are some cases that we need to call async method in useEffct. Can we do things like below:

1
2
3
4
5

useEffect(async () => {
  ...
  await axios.post(...)
  ...
}, [])

No, we cannot. Why? Because the async axios call returns promise and the useEffect does not expect callback function to return promise. The useEffect waits for the return of nothing or a function.

Then what we should do? There are two methods

• Define an async function inside useEffect

1
2
3
4
5
6
7

useEffect(() => {
  ...
  async () => {
    await axios.post(...)
  }()
  ...
}, [])

• Define an async function outside useEffect

1
2
3
4
5
6
7
8
9
10

const getData = useCallBack (async () => {
  ...
  await axios.post(...)
  ...
}, [])

useEffect(() => {
  getData()
}, [getData])

Use Async in ‘setInterval’

1
2
3

setInterval(async () => {
    await axios(...) 
}, 1000)

Can we do things like above in setInterval? It is questionable, as the async call may cost more than the time set in setInterval method. So what is the better option? Use Promise with setTimeout, like below:

1
2
3
4
5
6
7
8
9
10
11

function delay(ms) {
  return new Promise(resolve => setTimeout(resolve, ms))
}

async function loop() {
  while (/* condition */) {
    /* code to wait on goes here (sync or async) */    

    await delay(100)
  }
}

Reference: https://stackoverflow.com/questions/51830200/how-to-await-inside-setinterval-in-js

The above are the two scenarios that I met in my project, when I look back as a new full-stack engineer (I was heavily involved in backend dev, not frontend):)

Business Requirements

The system needs to accept subcriptions from users with certain period time. After the paid subscription expires, the subscription status in user profile should be downgraded from a subscriped user to a normal user.

Make A Choice among Two Options

MongoDB TTL Index vs Node Schedule

As introduced in one of my previous writing regarding the System Architect , the system uses MongoDB to store user data. To add the function into the system, there were two choices that immediately came to my mind: MongoDB TTL Indexes and Node Schedule

• Node Schedule
  Node schedule runs on the server side, for example, node-schedule. But since the requirement is to change user’s subscription status after certain amount of time (relatively long, like a week, a month etc.), and it will only run once, putting such burden on the server side for such task is not worth it.

• MongoDB TTL Indexes
  Compare to Node Schedule, MongoDB TTL Indexes run on the database side, it will not create extra jobs on the server, it is a better choice for such requirement.

MongoDB TTL Indexes

Directly create MongoDB TTL Indexes

Clearly, in my case, I need to create MongoDB TTL Indexes with give parameters after the schema is generated. And after searching for a while, I realised that in mongoose, I can only create TTL indexes when create the schema, it is not possible to create TTL indexes with give parameters when the app is running (please correct me if I am wrong here). Within mongoose, it only can introduce TTL Index within schema definition, as shown below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

import mongoose from 'mongoose';
const { Schema } = mongoose;

const XXXSchema = new Schema(
  {
    username: {
      type: String,
      required: true,
      unique: true,
    },
  },
  // timestamp of the creation data of the user
  { timestamps: true }
);

XXXSchema.index(
  { createdAt: 1 },
  {
    expireAfterSeconds: 30,
    partialFilterExpression: { username: { $eq: 'xxx' } },
  }
);

Indirectly create MongoDB TTL Indexes

Luckily, there is an alternative method to save my life: using expireAt in the schema. Under the table, the expireAt generates TTL indexes in MongoDB. Below are how I did it:

1. introduce expireAt in the schema with expires property

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

import mongoose from 'mongoose'
const { Schema } = mongoose

const XXXSchema = new Schema(
  {
    username: {
      type: String,
      required: true,
      unique: true,
    },
    expireAt: {
      type: Date,
      default: Date.now,
      expires: 0,
    },
  },
  // timestamp of the creation data of the user
  { timestamps: true },
)
...

2. when receiving the parameters, create the data with expireAt is set.

1
2
3
4
5
6
7
8

...
const newXXX = new XXX({
  username: user.username,
  expireAt: Date.now() + duration * 1000,
});

await newXXX.save();
...

3. After the code in step 2 is ran, in the MongoDB, a TTL index should be seem. Let us take MongoDB Atlas as an example:
   

Job done :)

Business Idea

As it is a chit-chat about system design in real case, I will briefly introduce the business without mentioning too many details.

The system provides a kind of service for registered users that is based on purchased usage quota, i.e., everytime a user consume the service, the number of service usage limit belongs to the user will be reduced by ONE, until it reaches to ZERO. Then the user needs to further purchase the service.

In essence, the system needs:

• a user management service
• a mechanism that is able to change the user data in ‘real time’

System Design

User Management Service

A typical user management module, I chose mongodb for the following 3 reasons:

• the user schema design was not fixed at the beginning, to avoid data schema changing troubles with relational DB, I went with NoSQL

• The better sharding and scalability provied by mongodb as it is json document based.

• To change the user data in ‘real-time’, I consider caching user data in memory for fast I/O, and I only want to cache partial user data, not all of them, therefore, NoSQL is a better option.

Change User Data in ‘Real-time’

As I mentioned, everytime a user consumes the service, the service usage quota will minus ONE. Since the service could be used by multiple users at the same time, the system must not spend too much time on I/O regading changing the user data, hence, I consider use in-memory cache here in stead of updating data directly in mongodb. That is the place where Redis will play.

As shown in the figure, when a user login, the user data will be loaded into Redis, and everytime the user consumes the service, the backend side will update the data in he Redis, and only write back to mongoDB when the user logout.

Sync user data between frontend and backend

At the frontend side, the user may need to see the number of service usage in ‘real-time’. There are two choices:

• Frontend always keey the data sync with backend, means that everytime the service is used, the frontend will invoke REST API of backend and wait for returned result.
• Frontend and backend use different data set. To be more specific, the frontend caches the user data in react-redux (only work with data from react-redux), and everytime the user consumes the service, on the frontend side, it change the number in react-redux, at the same time, invoke REST API to update the number in redis on the backend side.

For the 1st choice, the frontend side will always show the correct data but it sacrifices time. For the 2nd one, the frontend may show different data from backend (if something wrong happens on the backend side regarding updating data in redis), but the frontend side does not need to wait for the REST API call result.

I went for the 2nd choice for speed.

Locking the Serivce

The service will be used by multiple users at the same time and it is not sharable. Therefore, I need a distributed lock here.

Since I have introduced redis for caching, I used redis redlock for distributed locking.

These are the design decisions I have made during the project. System design is always about trade-offs: space, time, cost. The most import one: do not over design, the priority is to meet the business requirements not to create a technically perfect product.

Step.1 Export A Collection of Requests from Postman

As shown below, left click the “…” button at the right side of the requests collection. Then choose “Export” in the popup menu.

Step.2 Write Code for Load Testing

We need to create a sepearte xxx.js file that tells AutoCannon what to do.

• Load request data from exported JSON file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

const autocannon = require('autocannon');
const fs = require('fs/promises');

// read array of items from exported .json file from postman
let entries = undefined;

async function getRequests() {
  const data = await fs.readFile(
    './youtube_clone_proj.postman_collection.json',
    'UTF-8'
  );

  entries = JSON.parse(data).item;
  return true;
}

• Set up AutoCannon for Each Request

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

entries.map(async (entry) => {
    // there are multi request in item
    entry.item.filter((ele) => {
      // filter the empty request
      return ele.request.url !== undefined;
    }).map(async (ele) => {
      console.log(ele.request.method + " " + ele.request.url.raw);
      const result = await autocannon({
        url: ele.request.url.raw,
        method: ele.request.method,
        connections: 100,
        workers: 50,
        duration: 5,
        body: ele.request.body === undefined? null : JSON.stringify(ele.request.body),
         // read other options here: https://github.com/mcollina/autocannon#autocannonopts-cb
      }, finishedBench);

      // track process
      autocannon.track(result, {renderProgressBar: false});

      // this is used to kill the instance on CTRL-C
      process.once('SIGINT', () => {
        result.stop()
      })

      function finishedBench (err, res) {
        console.log('finished bench', err, res)
      }
    });

Launch the test

In the terminal window, run the following

1

node xxx.js

Then we should able to see output like this for each invidual request:

reference

Benchmark express apis with autocannon from postman collection

You are more then welcome to site the content of this blog, but please do indicate the original reference link.

Project Structure

I normally put the configuration file of winston logger like the following:

1
2
3
4
5
6
7
8
9
10

Project
| ...
├───middleware
├───models
├───config
│   ├───winston.js
│ ...
├───app.js
├───package.json
| ...

Then in *.js files, it will use the winston config file like the following:

1
2
3
4

const winstonLogger = require('./config/winston');
...
winstonLogger.error(`${err.status || 500} - ${err.message} - ${req.originalUrl} - ${req.method} - ${req.ip}`);
...

Winston Configuration File Explained

The conent of the configuration file winston.js mentioned in the previous section is shown below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

const appRoot = require('app-root-path');
const winston = require('winston');

const { format, transports } = winston;
const path = require('path');

const colors = {
  error: 'red',
  warn: 'yellow',
  info: 'green',
  http: 'magenta',
  debug: 'white',
};

winston.addColors(colors);

// define the custom settings for each transport (file, console)
const logger = winston.createLogger({
  level: 'http',
  format: format.combine(
    format.timestamp({
      format: 'YYY-MM-DD HH:mm:ss',
    }),
    format.errors({ stack: true }),
    format.splat(),
    format.printf(
      (info) => `${info.timestamp} ${info.level}: ${info.message}`,
    ),
    format.json(),
  ),
  defaultMeta: { service: 'quickpost' },
  transports: [
    new transports.File({ filename: path.join(appRoot.toString(), '/logs/error.log'), level: 'error' }),
    new transports.File({ filename: path.join(appRoot.toString(), '/logs/combined.log') }),
  ],
});

if (process.env.NODE_ENV !== 'production') {
  logger.add(new transports.Console({
    format: format.combine(
      // print all the message colored
      format.colorize({ all: true }),
      format.printf(
        (info) => `${info.timestamp} ${info.level}: ${info.message}`,
      ),
      format.simple(),
    ),
  }));
}

logger.stream = {
  write: (message) => logger.http(message),
};

module.exports = logger;

To explain the above sourcecode in details:

Set Colors for Logger Levels

1
2
3
4
5
6
7
8
9
10

const colors = {
  error: 'red',
  warn: 'yellow',
  info: 'green',
  http: 'magenta',
  debug: 'white',
};

winston.addColors(colors);

Config Other Settings of Winston

For the following settings, please refer to the official site of winston logger. I would emphasize two things:

• On the transports part. In the following configuration, I put loggers into files under the directory /log/*.log files. That is a very basic settings, you can of course adding settings like, generate new log files after the current log file reach to certain size, and further config organizing log files according to their generated date.

• If not in “production” environment, I set the logger still output on the console for facilitating debugging.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

// define the custom settings for each transport (file, console)
const logger = winston.createLogger({
  level: 'http',
  format: format.combine(
    format.timestamp({
      format: 'YYY-MM-DD HH:mm:ss',
    }),
    format.errors({ stack: true }),
    format.splat(),
    format.printf(
      (info) => `${info.timestamp} ${info.level}: ${info.message}`,
    ),
    format.json(),
  ),
  defaultMeta: { service: 'quickpost' },
  transports: [
    new transports.File({ filename: path.join(appRoot.toString(), '/logs/error.log'), level: 'error' }),
    new transports.File({ filename: path.join(appRoot.toString(), '/logs/combined.log') }),
  ],
});

if (process.env.NODE_ENV !== 'production') {
  logger.add(new transports.Console({
    format: format.combine(
      // print all the message colored
      format.colorize({ all: true }),
      format.printf(
        (info) => `${info.timestamp} ${info.level}: ${info.message}`,
      ),
      format.simple(),
    ),
  }));
}

Merge Morgan into Winston

The following code will merge morgan into winston.

1
2
3

logger.stream = {
  write: (message) => logger.http(message),
};

Config Morgan

To make winston working with morgan, we will need to add the following morgan settings:

• File directory:

  1 2 3 4 5 6 7 8

  Project | ... ├───middleware │ ├───morgan.js │ ... ├───app.js ├───package.json | ...

• Config morgan:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

  const morgan = require('morgan'); const logger = require('../config/winston'); const morganMiddleware = morgan( // Define message format string (this is the default one). // The message format is made from tokens, and each token is // defined inside the Morgan library. // You can create your custom token to show what do you want from a request. 'combined', // Options: in this case, I overwrote the stream and the skip logic. // See the methods above. { stream: logger.stream }, ); module.exports = morganMiddleware;

• Use in app.js

  1 2 3 4 5 6

  ... const morganMiddleware = require('./middleware/morgan'); ... app.use(morganMiddleware); ...

If things go well …

You should be able to see logger in the console like the following:

And also the same records can be find in the log files: /log/*.log files.

You are more then welcome to cite the content of this blog, but please do indicate the original reference link.

While talking about TDD within developing web systems, we cannot avoid testing RESTful APIs. I have seen some turtorials online (e.g. the articles which discuss about how to do unit test for REST API via leveraging certain npm packages under Nodejs environment) which mention that for Unit Test (hereafter, UT) it will hit database.

Well, in my humble option, in UT we should not hit database. The core idea of unit test is “Isolation“, i.e. isolate the function/methods under testing, presume everthing that the function/methods will interact with works as expected, and see if the function/methods generate the result we are hoping for. Therefore, if hitting database in UT, the testing result of the UT will depend on the status of the database, the coupling between the function/methods and the database makes the test is not a real “Unit Test”. Tests that requires access database should categories as Integration Test (hereafter IT) or End-to-End test. In the rest of this article, I will explain in details regarding how to do UT and IT for RESTful APIs with sourcecode samples provided under Nodejs environment.

Integration Test for RESTful API under Nodejs Environment: using MongoDB and supertest as an example

Assume that we want to test the following REST API: “POST /api/auth/signup“, we have the following code in our project:

app.js

1
2
3
4

import authRouter from './routes/auth-route.js';
...
app.use('/api/auth', authRouter);
...

auth-route.js

1
2
3
4
5
6
7
8

...
import {
  signup,
  signin,
} from '../controllers/auth-controller.js';
...
router.post('/signup', signup);
...

To test the api in integration test, we will need to complete two steps:

1. connect to database for testing purpose only, not database in production environment
2. test whether the api output expected results

1. Set a Mongodb connection selector between production environment and testing environment

app.js

1
2
3
4
5
6
7
8
9
10
11
12

// According to the value of process.env, choose the connection between prod env and test env
const mongodbConnect = process.env.NODE_ENV === "test"? 
  process.env.MONGODB_CLOUD_TEST : process.env.MONGODB_CLOUD_PROD;

const dbConnect = async () => {
  try {
    await mongoose.connect(mongodbConnect);
    winstonLogger.info(`Connect to mongodb of ${process.env.NODE_ENV}`);
  } catch (err) {
    winstonLogger.error(`Failed to connect to mongodb due to ${err}`);
  }
};

2. Using supertest to complete the test

auth-route.test.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

describe('POST signup', () => {
  const tmpUser = {
    name: 'testaa',
    email: 'testaa@gmail.com',
    password: '1234'
  };

  it('post a new user and respond with 200 and a msg showes that user has been created', async () => {
    // a real call to the database in testing environment
    const { text, status } = await request(app)
      .post('/api/auth/signup')
      .set('Accept', 'application/json')
      .send(tmpUser);

    expect(status).to.be.equal(200);
    expect(text).to.be.equal('User has been created!')
  });
  ...
});

As it can be seen from the above code snippet, the supertest makes a call to the database for creating a user profile, and the user profile is actually created in the database under test environment. Since the test involving testing the connection with a real database, I would like to categories this type of tests as integration test.

Then, how the unit test for the REST API should look like?

Unit Test for RESTful API under Nodejs Environment: using MongoDB and supertest + nock as an example

To clearly tell the difference between Integration Test and Unit Test, in this part I will still use supertest as the agent (alternative option can be axios, for example) to file a request to the “POST /api/auth/signup“, but in Unit Test, I will not let the request hitting datacase via http server, instead, I use nock to intercept the request and return the expected result, as shown blow:

describe('POST signup', () => {  const tmpUser = {    name: 'testaa',    email: 'testaa@gmail.com',    password: '1234'  };  it('post a new user and respond with 200 and a msg showes that user has been created', async () => {    // file http request to the REST API    const signupUser = async () => {      return await request('http://test.com')        .post('/api/auth/signup')        .set('Accept', 'application/json')        .send(tmpUser);    };    // mock http server: check if the server received expected params in request.body, then reply expected response     nock('http://test.com')      .post('/api/auth/signup', body => {        expect(body.name).to.be.equal('testaa');        expect(body.email).to.be.equal('testaa@gmail.com');        expect(body.password).to.be.equal('1234');        return true;      })      .reply(200, 'User has been created!');    const { text, status } = await signupUser();    expect(status).to.be.equal(200);    expect(text).to.be.equal('User has been created!')  });  ...});

In the above unit test, I use nock to mock a http server. The supertest still file a request, but the request will be intercepted by the mock server and return expected results, instead of reaching a real server and further hitting a real database. This is the way how isolation is achieved: I am not coupling the test with a real database, based on the assumption that all other parts works okay (via mocking), everything is completed within the test method.

A further questions may be asked is that what is point to do a Unit Test like this? Should I include the unit tests in my project? The answer is Yes and No.

• No: if you are developing a relatively small system as software vendor for a small business, or you are structuring a new software product that are targeting a small number of users, not aiming for millions or even billions users in the future, then I would say do not bother to add such unit test in your system. The ROI is not worth it, keep the integration test that check the connection with real database is enough.

• Yes: if you are developing a system either big or small in a big company, or if you have a software product that is aiming for millions of daily users in the future (which means the product will be owned by a big company), I would say you probably will need to add the unit tests in your system. The reason? I would quote a saying from a movie called <Kingdom of Heaven>: nothing, everything.

I have experiences of both working in academia (at PhD level) and industry (for some of the big names). The transform between the two was a lesson for me and I would like to share it with the ones who are in the situation of the following:

• Do not know if should choose academia or industry after undergraduate study or receiving Master’s degree
• Thinking of leaving adademia (after completing a PhD - Congratulations! That is a huge achievement already :), or quiting a PhD) and go to the industry, but do not know if it is the right choice.

Academia Way of Getting Things Done

Academia way of getting things done is to learn first (learn things related to your research area), then try to find the breakthrough. That means, you will need some time to do the preparations before actually begin to do something.

All the secrets behind getting success in acedemia, either as a PhD candicate or a PhD degree owner who are in the way of getting a tenure, to sum up in one sentence is: finding a“niche” (i.e. the so called breakthrough).

So how can I find the “niche” ? You may ask. In my experience, that is two steps work:

1. First, you will need to build solid foudation in the researcing area. Basically, you will need to have a solid understanding of the common theories in the area you are working on. The stage can be completed by reading a grate a mount of publishes, books… It takes time and is a must step. Imaging you are presenting in a conference, in the Q&A session, someone critisize what you just said by quoting a famous theory in the area, you do not know much about the theory and then you do not know “how to fight back”…

2. Second, after harnesss yourself with the knowledge you need after stage 1, then you can begin to do your actual work: try to find something new or “breakthrough” in the area that can work on.

In short, in academia, you will be allowed time to ramp up before doing something.

Industrial Way of Getting Things Done

The industrial way of achieving something is not like the academia way. There is very limited time for praparation, it is more like learning by practicing.

While working in the industry, you will have to move much faster, build something tangible that can be evaluate by stakeholders, collect feedbacks, then improve it, then re-evalueate … Therefore, there is no time to do decent preparation, you learn things in the process of solving problems. The key in the industry is to have something that can prove the feasibility of an idea as soon as possible and push to the market.

Conclusion

Thinking about how the GoF Design Patterns was discovered. If it was in Academia, the way of find the patterns would be: tons of readings, then propose some hypothesis, verify it via experiments and finally publish the findings (i.e. design patterns). However, in reality it was not like that, the GoF reviews the previous practices in software industry, then found those patterns. Exactly, the patterns were invented by software engineers in practice without a deliberate purpose (they were found by “variable”. some of most impactful things was invented by accidents, check Anti Fragile.

There are not right or wrong in these two ways of resolveing issues. Because they are in different contexts. In industry, most of the time, we are dealing with practical problem and need to see impacts right away; but in academia, we are dealing with the issues for the future in many case.

I spent a few weeks to develop and set up my own website. The registered domain is root domain (i.e. example.com) and the DNS setting on GoDaddy is to redirect the root domain(example.com) to the www subdomain (e.g. www.example.com). The last step before publishing is to add SSL certificates to my root domain and www subdomain. I found a nice tutorial online regarding how to do it Let’s Encrypt on Heroku with DNS Domain Validation. However, it turned out that follow the intructions I could not generate certificate and private key for my www subdomain. I will show the issue and explain how I resolve it in details.

The Issue

My ultimate goal is to let https://www.example.com work on browsers. I have three custom domain in my Heroku app: *.exmaple.com, exmaple.com, www.example.com.

Following the instructions of Let’s Encrypt on Heroku with DNS Domain Validation, when Let’s Encrypt asked me to provide the domain name (as shown below), no matter I use example.com or www.example.com

1
2

Please enter the domain name(s) you would like on your certificate (comma and/or
space separated) (Enter 'c' to cancel):

later at the stage of applying the certificate to Heroku “sudo heroku certs:add –type=…/certificate …/privatekey”, the heroku will not provide domain name www.example.com as an option for the certificate but only example.com. This leads to the result of www.example.com is not secured.

Resolution

The resolution is easy but took me a lot of time to find it. The trick is using wildcard subdomain. At the stage of providing domain name to Let’s Encrypt, I used *.exmample.com, then later when I apply the generated certificate and private key to Heroku, it offers www.exmample.com as an option.

1
2

=== Almost done! Which of these domains on this application would you like this certificate associated with?
? Select domains www.jingjies-blog.com

Then check on Heroku, I can see the following:

On GoDaddy, change the redirect from http://www.example.com to https://www.example.com

Finally, I can see the “lock” on browser :)

You are more then welcome to cite the content of this blog, but please do indicate the original reference link.

• The Hexo blog is integrated into an existing Express server, i.e. the blog should be visited at express_site/blog
• The theme I used for the blog does not have such feature added, means I cannot just add something to the _config.yml of the theme and the view counter will work as a charm. I will need to customize it.

There was several options that I had in mind:

• Only customize Hexo blog and the theme –> I realize it is not feasible. I do not know how to let Hexo or the theme load customised .js file under Hexo + Express structure;
• Load js from Express side –> I finally use this method. Let Express loading related js file as Express static files, the js file will add the data to the view counter in Hexo pages. Blow is what I got in the end.

  

Now I will introduce how I achive it in details.

Set up view counter

• Set up Firebase (There are tons of tutorial out there, so I am not going to repeat it:))

  There is only one thing I would like to emphasize is the database set up. The database should be Realtime Database, and the “rules” is set as below. I will explain in details later why this will not cause security issues.

• After register and setting up the Firebase, add the following code to the hexo_blog_directory/themes/{your theme}/layout/xxx.ejs where you want to show the view counter.

1
2

<span id="site-visits">Sitevisits: <span class="count">--</span></span>
<span id="page-views">Pageviews: <span class="count">--</span></span>  

• Add the following .js code to express_proj/public/javascripts/:

  Note:

  • I used Firebase SDK 9
  • The details of database read/write APIs could be found at Firebase Database APIs
  • You could of course test data read / write sepeartely before using the whole block

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

import { initializeApp } from 'https://www.gstatic.com/firebasejs/9.8.3/firebase-app.js';
import {
  getDatabase, ref, get, set, child,
// eslint-disable-next-line import/no-unresolved
} from 'https://www.gstatic.com/firebasejs/9.8.3/firebase-database.js';

const viewCounter = async () => {
  // The web app's Firebase configuration
  // For Firebase JS SDK v7.20.0 and later, measurementId is optional
  const firebaseConfig = {
    apiKey: 'your api key',
    authDomain: 'your settings ',
    databaseURL: 'xxx',
    projectId: 'xxx',
    storageBucket: 'xxx',
    messagingSenderId: 'xxx',
    appId: 'xxx',
    measurementId: 'xxx',
  };

  // Initialize Firebase
  const firebase = initializeApp(firebaseConfig);
  const db = getDatabase(firebase, firebaseConfig.databaseURL);
  const oriUrl = window.location.host;
  const curUrl = oriUrl + window.location.pathname;

  const readVisits = async (url, selector) => {
    const dbKey = decodeURI(url.replace(/\/|\./g, '_'));
    let count = 1;
    const res = await get(child(ref(db), dbKey));
    if (res.exists()) {
      count = parseInt(res.val() || 0, 10) + 1;
    }
    await await set(ref(db, dbKey), count);
    if (selector.length > 0) {
      // eslint-disable-next-line no-param-reassign
      selector[0].innerText = count;
    }
  };

  readVisits(oriUrl, document.querySelectorAll('.post-meta #site-visits .count'));
  if (curUrl && curUrl !== '_') {
    readVisits(`page/${curUrl}`, document.querySelectorAll('.post-meta #page-views .count'));
  }
};

viewCounter();

• After the above steps, the view counter is ready to go :)

Extra issue: is it okay to expose Firebase apiKey?

When you see that I put apiKey etc. in a static .js file, you must have a question in mind: is that a security issue here? I had the same question. Then I found the following article: Is it safe to expose Firebase apiKey to the public?

The answer is a bit long, so I will sum up here: for a small size and not a critical site, it is okay to do thing like this. The apiKey here in Firebase is not like the traditional api keys we know, it is okay to expose it. All you need to do is to set up whitelist site for Firebase (that is why I mentioend earlier in the article that you could set the Firebase database rules for opening read/write public). Here is how you can do it in details:

• Go to Firebase Console
• Check Authentication Menu -> Sign-in method tab
• Scroll down to Authorized domains
• Click “Add Domain” button and add your domains (localhost is added as default, and I do not delete as it is needed for local testing)

The above should be okay for a small site. I am not going to over-engineering it, but you can always tailor the security rules accoriding to your business requirements :)

You are more then welcome to cite the content of this blog, but please do indicate the original reference link.

My way of achieving it is based on x-hexo-app-connect and some of the steps describe blow are based on the instruction provided by x-hexo-app-connect.

Getting Started

1.1 Make a sub-directory under the express project directory. e.g. we make it under experss_proj/blog, use this command under the project directory:

1

$ mkdir blog

1.2 Enter the command to install hexo-cli globally

1

$ sudo npm i hexo-cli -g

1.3 Go to the blog’s directory (created in step 1.1) and enter the following command in terminal to init Hexo:

1

$ hexo init

1.4 In the blog directory, enter the command in terminal to install x-hexo-app-connect package:

1

$ npm i x-hexo-app-connect --save

1.5 Create a index.js file in the blog’s root directory. e.g. if you want to make it using a bash terminal (Mac platform), enter this command below:

1

$ touch index.js

1.6 Fill in the index.js with the following code:

• for the return part, it is to config the Hexo blog, the route is the most important, that is where you will configure how the Hexo blog should be visited from the original Express server. In my case, the blog will be visited via route express_site/blog

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

  const Hexo = require('hexo'); module.exports = (app) => { const blogDirPath = __dirname; const hexo = new Hexo(blogDirPath, {}); // eslint-disable-next-line global-require return require('x-hexo-app-connect')(app, hexo, { // The Configs/Options log: false, compress: false, header: true, serveStatic: false, route: "/blog" }); };

  1.7 In the app.js (the home js file of Express), add the following to use the x-hexo-app-connect in Express

  1 2 3 4 5 6 7

  const app = express(); // Put it after app is initialized, as the app will be used as parameter const blogRouter = require('./blog/index')(app); ... app.use(blogRouter);

  1.8 This step is very important, it is to set “root” parameter in the _config.yml file under the root directory of Hexo. And set the root as /blog. If not set this root, the Hexo blog home page can be visited via express_site/blog, but when click the link of articles, categories, archives etc. the express will report 404 as the routes cannot be found

  1 2 3 4

  # URL ## Set your site url here. For example, if you use GitHub Page, set url as 'https://username.github.io/project' url: http://localhost:3000/ root: /blog # for locating themes static files (.css etc)

• If do not set root as /blog, the following url will be generated as localhost:3000/all-categories which will lead to 404 error.

  

1.9 Apply themes. You can just following the set up instruction of each specific theme, then it should be okay.
1.10 After all the above steps, run the Express server, if the terminal showes the following, then it means you are good:)

1
2
3

INFO  Validating config
INFO  Start processing
INFO  [Hexo-App-Connect] is running on route /blog

Hope this blog helps you in some way :)

You are more then welcome to cite the content of this blog, but please do indicate the original reference link.

It is Jingjie here, it is my first blog on my own blogging site.I have always been wanting to have a place where I can write and post my own blogs. instead of writing things here and there.

The reason of writing blog

I am a passionate software engineer and by nature of this area, I learned lots of thing online (i.e. I received help from others) and wrote down a great number of notes (mainly on my own OneNote) regarding the knowledge I acquired: for strengthening my understanding of thing learned and picking them up in the future.

And for all these days working on academia and industry, I have my own understanding of software, technology etc. I would like to share them with others, and hoping they will be kind of helpful in some way (i.e. I would like to return the favor).

For this blog site, I will post my understanding of computer and life as well. Things start :)