Burned by Docker and it was my fault (Postgres issue)

About this video

- **Introduction**: Hussein shares an interesting problem he encountered with PostgreSQL databases, specifically related to indexes and shared memory. - **Index-Only Scan Concept**: - Explains the concept of "Index-Only Scan," where PostgreSQL retrieves all necessary data from the index itself without accessing the heap table (main data storage). - He created an index with non-key columns to enable fetching values directly from the index. - **Problem Encountered**: - Despite executing an Index-Only Scan, the query was unexpectedly slow (~3 seconds). - The execution plan showed 10,000 heap fetches, which should not occur in a proper Index-Only Scan. - Ran a full `VACUUM` on the table to ensure all rows were visible to transactions, but the issue persisted. - **Root Cause Discovery**: - Found online that someone faced a similar issue due to insufficient shared memory. - Shared memory is a common memory space accessible by multiple processes, crucial for PostgreSQL operations like caching disk pages. - His setup was running inside a Docker container with a default shared memory size of 64 MB, which was insufficient for his large dataset (~50 million rows). - **Solution**: - Increased the shared memory size to 1,000 MB. - Re-ran the full `VACUUM` with verbose mode, successfully cleaning up transaction rows and making them visible to queries. - After the fix, the Index-Only Scan executed in just 3 milliseconds without accessing the heap. - **Key Takeaways**: - Emphasizes the importance of debugging and understanding underlying issues to become better software engineers. - Encourages persistence in problem-solving and highlights the satisfaction of resolving complex issues. - **Personal Note**: - Mentions being in Julian, a mountainous area, to try delicious pies. - Ends with well-wishes and a farewell.

Course: Docker

### Course Description: Docker This comprehensive course on Docker is designed to equip students with the knowledge and skills necessary to create, manage, and deploy containerized applications effectively. The course begins with an introduction to Docker, focusing on its importance in modern software development, particularly in continuous integration and continuous deployment (CI/CD) pipelines, Jenkins tasks, and Kubernetes clusters. Students will learn how to create lightweight containers that encapsulate their applications in an isolated environment, allowing for consistent execution across different platforms. This isolation ensures that applications run seamlessly regardless of the underlying infrastructure, making Docker a critical tool for developers. The course delves into the practical aspects of Docker by guiding students through the process of creating a Docker image and running a container. Starting with setting up a Dockerfile, participants will learn how to define the environment and dependencies required for their application. Through hands-on examples using Node.js and Express, students will build a simple web application and containerize it using Docker. The course also covers essential commands such as `docker build` and `docker run`, demonstrating how to expose ports, install dependencies, and execute applications within containers. Additionally, students will explore how to scale their applications by running multiple containers and load-balancing them using tools like Nginx or HAProxy. By the end of this section, learners will have a solid understanding of how to leverage Docker for deploying stateless, self-contained applications. Beyond the basics, the course introduces advanced topics such as microservices architecture and orchestration. Students will gain insights into how Docker facilitates the development of distributed systems by enabling the creation of modular, scalable services. The course includes practical demonstrations of running multiple containers simultaneously, simulating real-world scenarios where applications are deployed across various environments. Furthermore, learners will be introduced to the integration of Docker with Kafka, a distributed streaming platform, to build robust data processing pipelines. By combining Docker with Kafka, students will understand how to handle high-throughput, fault-tolerant systems that are essential for modern applications. Overall, this course provides a thorough grounding in Docker, empowering students to harness its full potential in both development and production environments.