Skip to main content

DOCKER

  1. Containerization

Containerization is like putting each part of your software (your application code, all its libraries, dependencies, and even a miniature operating system) into its own standardized, self-contained box โ€“ a container.

Key Idea: It packages an application and all its dependencies into a single, isolated unit. This unit can then be run consistently across different computing environments, regardless of the underlying infrastructure.

Problem it Solves:

"It works on my machine!" This common developer lament arises because differences in operating systems, libraries, and configurations between development, testing, and production environments can lead to unexpected bugs and deployment issues. Containerization aims to eliminate this.

  1. Virtual Machines (VMs) vs. Containers: A Crucial Distinction

Before containers, Virtual Machines (VMs) were the primary way to achieve isolated environments. Understanding the difference is key.

FeatureVirtual Machine (VM)Container
Isolation LevelHardware-level virtualization (hypervisor)Operating System (OS)-level virtualization
OS FootprintEach VM has its own full Guest OS (e.g., Windows, Linux)Shares the Host OS kernel
Resource UsageHigh (CPU, RAM) due to full OS overheadLow (lightweight, efficient)
PortabilityPortable, but larger imagesHighly portable, smaller images
Use CaseRunning multiple different OSes on one machineRunning multiple isolated applications on one OS
AnalogySeparate apartments in a building, each with its own utilitiesSeparate rooms in a house, sharing the same foundation and plumbing

Key Takeaway: Containers are much more lightweight and efficient than VMs because they don't carry the overhead of a full operating system for each application. They share the host OS kernel, leading to faster startup times and better resource utilization.

What is Docker?โ€‹

If containerization is the concept, Docker is the most popular platform/tool that implements containerization. It's an open-source platform that enables developers to build, ship, and run applications inside containers.

Think of Docker as:

  • The Containerization Engine: It provides the tools to create, run, and manage containers.
  • A Standard for Packaging: It defines a standardized format for packaging applications and their dependencies into "Docker Images."
  • A Registry: It offers Docker Hub, a cloud-based registry where you can store and share Docker images.

Image vs Container:

  • A Docker image is a static, read-only template that contains everything needed to run an application, such as code, libraries, and dependencies.
  • A Docker container is a running instance of that image, which is mutable and can perform operations, retain changes (like logs or data), and interact with other containers or the host system.

Key Docker Conceptsโ€‹

  • Docker Engine: The core of Docker. It's a client-server application with:

    • Docker Daemon (dockerd): The server, a persistent background process that manages Docker objects (images, containers, networks, volumes).
    • Docker CLI (docker): The command-line client that allows users to interact with the Docker Daemon.
    • REST API: An interface for programs to communicate with the Daemon.

  • Docker Image:

    • A lightweight, standalone, executable package that includes everything needed to run a piece of software, including the code, a runtime, libraries, environment variables, and config files.
    • Nature: It's a read-only template used to create Docker containers.
    • Layers: Images are built from a series of layers. Each instruction in a Dockerfile creates a new layer. This layering makes images efficient as common layers can be shared between multiple images, and updates only require rebuilding changed layers.
    • Example: An image could contain an Ubuntu base OS, Node.js runtime, and your web application code.

  • Docker Container:

    • A runnable instance of a Docker image.
    • Nature: It's the actual running environment where your application executes.
    • Ephemeral (เค•เฅเคทเคฃเคฟเค•): Containers are designed to be ephemeral. You can start, stop, move, or delete them. If a container crashes, you can quickly spin up a new one from its image.
    • Isolation: Each container runs in isolation from other containers and the host system, ensuring consistent behavior.

  • Dockerfile:

    • A simple text file that contains a set of instructions (commands) for building a Docker image.
    • Purpose: It's like a recipe. You define the base image, copy your application files, install dependencies, expose ports, and define the command to run when the container starts.
# Use an official Node.js runtime as a parent image
FROM node:18-alpine

# Set the working directory in the container
WORKDIR /app

# Copy package.json and package-lock.json
COPY package*.json ./

# Install app dependencies
RUN npm install

# Copy the rest of the application code
COPY . .

# Expose port 3000 to the outside world
EXPOSE 3000

# Define the command to run your app
CMD ["node", "server.js"]

  • Docker Hub (Registry):

    • A cloud-based public registry service provided by Docker for finding and sharing Docker images.
    • Functionality: We can push our custom images to Docker Hub (or a private registry) and pull images created by others. This acts as a centralized repository for container images.

  • Docker Compose:

    • A tool for defining and running multi-container Docker applications.
    • Purpose: When your application consists of multiple services (e.g., a web app, a database, a cache), Docker Compose allows you to define all these services in a single docker-compose.yml file and manage them together with a single command (docker-compose up).
version: '3.8'
services:
  web:
    build: .
    ports:
      - "3000:3000"
    depends_on:
      - db
  db:
    image: postgres:13
    environment:
      POSTGRES_DB: mydb
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password

  • Docker Volumes:

    • The preferred mechanism for persisting data generated by and used by Docker containers.
    • Purpose: By default, data inside a container is lost when the container is removed. Volumes allow you to store data outside the container on the host machine, ensuring data persistence even if the container is deleted or recreated.
    • Types: Anonymous volumes, named volumes, bind mounts.

  • Docker Networks:

    • Enable communication between Docker containers and between containers and the host machine.
    • Purpose: Docker provides various networking drivers (bridge, host, overlay, etc.) to allow containers to communicate securely and efficiently, even across different hosts (in the case of overlay networks).

5. The Docker Workflow (Simplified)โ€‹

  1. Write a Dockerfile: Define how your application image should be built.
  2. Build the Image: Use docker build to create a Docker image from your Dockerfile. This creates layers.
  3. Run the Container: Use docker run to create and start a container from your image.
  4. Manage: Use docker ps, docker stop, docker rm to manage running containers.
  5. Push/Pull (Optional): Push your image to Docker Hub (docker push) or pull existing images (docker pull).
  6. Orchestrate (for multi-service apps): Use docker compose to manage multiple interconnected containers.

6. Benefits of Containerization (and Docker)โ€‹

  • Consistency/Portability: "Write once, run anywhere." Ensures applications run identically across development, testing, and production environments, eliminating "it works on my machine" issues.
  • Isolation: Each container is isolated, preventing conflicts between applications and ensuring that one application's issues don't affect others on the same host.
  • Efficiency: Lightweight nature compared to VMs means better resource utilization and faster startup times.
  • Scalability: Easy to scale applications up or down by launching more or fewer containers. Orchestration tools like Kubernetes take this to the next level.
  • Faster Development Cycles: Developers can quickly set up consistent development environments.
  • Simplified Deployment: Streamlines the deployment process, making it more reliable and repeatable.
  • Version Control: Docker images can be versioned, allowing easy rollbacks to previous stable versions.
  • Modularity: Encourages microservices architecture, where applications are broken down into smaller, independent services, each running in its own container.

7. Real-World Applications in Web Developmentโ€‹

  • Consistent Development Environments: Every developer on a team uses the exact same setup, reducing "works on my machine" issues.
  • Microservices Architecture: Breaking down large monolithic web applications into smaller, independent services (e.g., user service, product service, order service), each running in its own container.
  • CI/CD Pipelines: Integrating Docker into Continuous Integration/Continuous Delivery pipelines for automated building, testing, and deployment of applications.
  • Database Management: Running databases (PostgreSQL, MongoDB, Redis) in containers for development and testing, making it easy to spin up and tear down instances.
  • Local Testing: Easily test different versions of dependencies or an application against different environments without cluttering your local machine.
  • Serverless Functions (often powered by containers): While not directly Docker, concepts of containerization underpin many serverless platforms.

8. Conclusion and Next Stepsโ€‹

Containerization, with Docker at its forefront, has revolutionized how software is developed, delivered, and deployed. It's a cornerstone of modern DevOps practices and cloud-native application development.

For your notes, remember these core ideas:

  • Containerization = Isolated, Portable Packages.
  • Docker = The Tool to Implement It.
  • Image = Blueprint (read-only).
  • Container = Running Instance (read-write layer on top).
  • Dockerfile = Recipe for building an Image.
  • Benefits: Consistency, Efficiency, Scalability, Portability.

Commandsโ€‹

A. Containersโ€‹

CommandDescription
docker run -d -p <host_port>:<container_port> <image_name>:<tag>Run a container in detached mode from a specified image, and map container port to host port.
Example: docker run -d -p 3000:3000 my-mern-app
docker psList all running containers.
docker ps -aList all containers, including stopped ones.
docker stop <container_id or name>Stop a running container. You can get the container ID from docker ps.
Example: docker stop my-container
docker start <container_id or name>Start a stopped container.
Example: docker start my-container
docker restart <container_id or name>Restart a running or stopped container.
Example: docker restart my-container
docker rm <container_id or name>Remove a stopped container. You can remove multiple containers by specifying multiple IDs.
Example: docker rm my-container
docker logs <container_id or name>View logs from a container (helpful for debugging or monitoring).
Example: docker logs my-container
docker exec -it <container_id or name> /bin/bashOpen an interactive terminal (bash) inside a running container to inspect or interact with it.
Example: docker exec -it my-container /bin/bash
docker attach <container_id or name>Attach to a running container's main process (e.g., view output or send input).
Example: docker attach my-container

B. Imagesโ€‹

CommandDescription
docker build -t <image_name>:<tag> <path_to_dockerfile>Build a Docker image from a Dockerfile.
Example: docker build -t my-mern-app .
docker imagesList all Docker images available on your local machine.
docker rmi <image_name>:<tag>Remove a Docker image from your local system.
Example: docker rmi my-mern-app
docker tag <image_id> <new_image_name>:<new_tag>Tag an existing image with a new name or version.
Example: docker tag abc123 my-mern-app:v2
docker pull <image_name>:<tag>Download a Docker image from Docker Hub (or another registry).
Example: docker pull node:14
docker push <image_name>:<tag>Upload a Docker image to a registry like Docker Hub.
Example: docker push my-mern-app:v1
docker history <image_name>:<tag>View the history of an image, including layers and commands used to create it.
docker inspect <image_name>:<tag>Get detailed information (JSON format) about a Docker image, including configuration and layers.

What is Port Mapping?โ€‹

When a container runs a service (e.g., a Node.js server on port 3000), it listens inside the container. But if you want to access it from your host machine (like your browser or Postman), you need to map the container's internal port to a port on your host.

docker run -p <host_port>:<container_port> <image_name>
docker run -It -p 3000:3000 my-node-app
# with environment values
docker run -it -p 3000:3500 -e key=value -e stripe_key=value my-node-app

Docker Compose: Orchestrating Multi-Container Applications

What happens when your application isn't just one container, but a collection of interconnected services? For example:

  • A web application (Node.js, Python Flask, React frontend served by Nginx)
  • A database (PostgreSQL, MongoDB, MySQL)
  • A caching service (Redis)
  • A message queue (RabbitMQ, Kafka)

Manually starting each container with docker run, ensuring they are on the same network, mapping ports, and setting environment variables, would quickly become cumbersome, error-prone, and difficult to reproduce. This is where Docker Compose shines.

1. What is Docker Compose?โ€‹

Docker Compose is a tool for defining and running multi-container Docker applications. It allows you to define all the services that make up your application in a single YAML file, and then spin up, tear down, and manage all of them with a single command.

Key Idea: It simplifies the process of configuring, running, and linking multiple interdependent Docker containers.

2. The docker-compose.yml File: The Heart of Composeโ€‹

The core of Docker Compose is the docker-compose.yml (or compose.yaml, the preferred modern name) file. This YAML file is a declarative configuration that describes your entire application stack.

Why YAML? YAML (YAML Ain't Markup Language) is chosen because it's human-readable and easy to write, making it ideal for configuration files.

Structure of a docker-compose.yml Fileโ€‹

The file typically has a few top-level keys:

  • version: Specifies the Compose file format version (e.g., 3.8). This is important for compatibility and feature sets.
  • services: This is the most important section. It defines the individual containers (services) that make up your application. Each service corresponds to a Docker container.
  • volumes (Optional): Defines named volumes for persistent data storage. These volumes can be shared between services or persist data even if containers are recreated.
  • networks (Optional): Defines custom networks for your services to communicate over. By default, Compose creates a default network for all services in the file, but you can define custom ones for more control or isolation.

Example docker-compose.yml (Web App with Database)โ€‹

Let's consider a common web development scenario: a Node.js API, a PostgreSQL database, and a React frontend served by Nginx.

# docker-compose.yml
version: '3.8' # Specify the Compose file format version

services:
  # 1. Backend Service (Node.js API)
  backend:
    build: 
      context: ./backend # Path to the directory containing the Dockerfile for the backend
      dockerfile: Dockerfile # Name of the Dockerfile (optional if named Dockerfile)
    ports:
      - "3001:3001" # Map host port 3001 to container port 3001
    environment:
      # Environment variables for the backend service
      DATABASE_URL: postgres://user:password@db:5432/mydatabase 
      NODE_ENV: development
    depends_on:
      - db # Ensure 'db' service starts before 'backend'
    volumes:
      - ./backend:/app # Mount the local backend code into the container for live updates (dev)
      - /app/node_modules # Anonymous volume to prevent host's node_modules overwriting container's

  # 2. Database Service (PostgreSQL)
  db:
    image: postgres:15-alpine # Use a pre-built PostgreSQL image from Docker Hub
    environment:
      POSTGRES_DB: mydatabase # Database name
      POSTGRES_USER: user     # Database user
      POSTGRES_PASSWORD: password # Database password
    volumes:
      - db_data:/var/lib/postgresql/data # Persist database data using a named volume
    ports:
      - "5432:5432" # Expose database port (optional, good for local access)

  # 3. Frontend Service (React App served by Nginx)
  frontend:
    build:
      context: ./frontend # Path to the directory containing the Dockerfile for the frontend
      dockerfile: Dockerfile # Name of the Dockerfile (optional if named Dockerfile)
    ports:
      - "80:80" # Map host port 80 to container port 80 (Nginx default)
    depends_on:
      - backend # Ensure 'backend' service starts before 'frontend' (if frontend calls backend on startup)
    volumes:
      - ./frontend:/app # Mount the local frontend code into the container (dev)
      - /app/node_modules # Anonymous volume for frontend node_modules
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro # Mount custom Nginx config

# Define named volumes for data persistence
volumes:
  db_data: # This volume will persist PostgreSQL data

Explanation of Key Elements in the Example:โ€‹

  • services:
    • backend, db, frontend: These are the names of our services. Docker Compose uses these names as hostnames within the default network, allowing services to communicate by name (e.g., backend can reach the database at db:5432).
    • build:
      • context: Specifies the path to the build context (directory containing the Dockerfile and application code).
      • dockerfile: (Optional) Specifies the name of the Dockerfile if it's not Dockerfile.
      • When build is used, Docker Compose will build an image for this service using the specified Dockerfile.
    • image: Used when you want to pull an existing image from Docker Hub or a private registry (like postgres:15-alpine). If both build and image are specified, build usually takes precedence.
    • ports: Maps ports from the host machine to the container. Format: "HOST_PORT:CONTAINER_PORT".
      • "3001:3001" for backend: The backend service inside its container listens on port 3001. We expose it on host port 3001 so we can access it from our browser or other tools on the host.
      • "80:80" for frontend: Frontend Nginx listens on port 80. We map it to host port 80, so the app is accessible via http://localhost.
    • environment: Sets environment variables inside the container. This is crucial for configuration (e.g., database connection strings, API keys).
    • depends_on: Declares dependencies between services. This ensures services are started in the correct order. backend will not start until db is running. frontend will not start until backend is running. Important Note: depends_on only waits for the dependent container to start, not for the application inside the container to be fully ready (e.g., database accepting connections). For more robust readiness checks, you might use healthcheck or external scripts in production.
    • volumes: Mounts paths.
      • ./backend:/app: This is a bind mount. It syncs the local backend directory (where your Node.js code is) to the /app directory inside the container. This is invaluable for development, as code changes on your host machine are immediately reflected in the running container without rebuilding the image.
      • /app/node_modules: This is an anonymous volume. When you bind mount your source code, you don't want the node_modules directory from your host (which might have different binaries or versions) to overwrite the node_modules generated inside the container during npm install. By defining an anonymous volume on top of /app/node_modules, Docker essentially "masks" the host's node_modules and uses the container's generated ones.
      • db_data:/var/lib/postgresql/data: This mounts the named volume db_data (defined at the bottom) to the PostgreSQL data directory inside the container. This ensures that your database data persists even if you stop and remove the db container.
      • ./nginx.conf:/etc/nginx/conf.d/default.conf:ro: Mounts a local Nginx configuration file into the frontend container in read-only mode (:ro).
  • volumes (top-level):
    • db_data:: Declares a named volume called db_data. Docker will manage this volume, and its data will persist independently of containers.

3. Key Docker Compose Commandsโ€‹

Once you have your docker-compose.yml file, you interact with your application stack using the docker compose command (note: older versions used docker-compose with a hyphen).

  • docker compose up:
    • The most frequently used command.
    • Reads the docker-compose.yml file.
    • If images are not specified or build is used, it builds the necessary images.
    • Creates networks and volumes if they don't exist.
    • Starts all defined services as containers.
    • By default, it runs in the foreground, showing logs from all services.
    • Use docker compose up -d to run in detached mode (in the background).
  • docker compose down:
    • Stops and removes all containers, networks, and default volumes created by docker compose up.
    • Use docker compose down --volumes to also remove named volumes (useful for a clean slate, but be careful with production data!).
  • docker compose ps:
    • Lists all running services (containers) defined in your docker-compose.yml file, along with their status and ports.
  • docker compose logs [service_name]:
    • Displays the logs from all services.
    • You can specify a service_name (e.g., docker compose logs backend) to view logs for a specific service.
    • Use -f or --follow to stream logs in real-time.
  • docker compose build [service_name]:
    • Builds or re-builds images for services that have a build instruction.
    • You can specify a service_name to build only that service's image.
  • docker compose exec [service_name] [command]:
    • Runs a command inside a running service container.
    • Example: docker compose exec db psql -U user mydatabase (to open a PostgreSQL client inside the db container).
  • docker compose restart [service_name]:
    • Restarts one or more services.
  • docker compose stop [service_name]:
    • Stops one or more running services without removing them.

4. How Docker Compose Works (Under the Hood)โ€‹

When you run docker compose up:

  1. Parsing the YAML: Docker Compose reads your docker-compose.yml file.
  2. Project Name: It determines a "project name" (usually based on the directory name where the docker-compose.yml file resides). This name is used to prefix resources (containers, networks, volumes) to isolate them from other Compose projects.
  3. Network Creation: It creates a default network (e.g., myproject_default) for all your services unless you define custom networks. This network allows services to communicate with each other using their service names as hostnames.
  4. Image Building/Pulling: For each service:
    • If a build instruction is present, it executes docker build to create the image.
    • If an image is specified, it pulls the image from Docker Hub (or configured registry).
  5. Volume Creation: It creates any named volumes defined in the volumes section.
  6. Container Creation & Start: It then creates and starts containers for each service, respecting depends_on order, and applying all the configurations (ports, environment variables, volumes, etc.) specified in the YAML file.
  7. Resource Grouping: All containers, networks, and volumes created by a single docker compose up command are grouped under the project name, making it easy to manage them as a single unit with docker compose down.

5. Benefits of Using Docker Composeโ€‹

  • Simplified Development Workflows: Spin up an entire complex application stack with one command (docker compose up), rather than multiple docker run commands with complex flags.
  • Reproducibility: Ensures that every developer on a team, or any environment (dev, testing), runs the exact same application stack with the same configurations.
  • Version Control for Infrastructure: The docker-compose.yml file can be version-controlled alongside your application code, treating your infrastructure configuration as code.
  • Isolation: Each Compose project runs in its own isolated network, preventing port conflicts or service name clashes with other projects on the same host.
  • Easy Scaling (for Development/Testing): While not a production-grade orchestrator like Kubernetes, you can easily scale individual services for testing purposes (e.g., docker compose up --scale web=3).
  • Efficient Resource Management: Docker Compose intelligently recreates only the containers that have changed, saving time and resources during development iterations.

6. Where Docker Compose Excels (and Where it Doesn't)โ€‹

Docker Compose is excellent for:

  • Local Development Environments: Its primary and most common use case.
  • Testing Environments: Setting up isolated integration or end-to-end testing environments.
  • CI/CD Pipelines: Creating temporary environments for automated builds and tests.
  • Simple, Single-Host Deployments: For smaller applications that don't require high availability or complex scaling across multiple servers.

Docker Compose is generally NOT suitable for:

  • Production Orchestration of Large-Scale Applications: It's designed for a single host. For multi-host, high-availability, auto-scaling, and advanced deployment strategies in production, you need a full-fledged container orchestration platform like Kubernetes or Docker Swarm.