They made Kafka 80% faster by switching file systems

About this video

### Final Comprehensive Summary Algro achieved a remarkable 82% improvement in CFA producer rights by optimizing file system performance, as detailed in their technical blog titled *"Unlocking CFA Potential by Addressing Latency Using EBPF."* Instead of modifying Kafka's source code, they leveraged ready-made tools to analyze Kafka and its underlying TCP protocols. By tracing kernel interactions and analyzing file system calls, they identified bottlenecks tied to the ext4 file system, specifically metadata locking issues that caused delays during data commits. To address this, Algro replaced ext4 with the XFS file system, which resolved the locking bottleneck and significantly improved response times. The blog does not delve into why XFS outperformed ext4, emphasizing that the choice of file system depends on specific use cases. Kafka, a publish-subscribe system enabling asynchronous data consumption via brokers, was central to their analysis. They mapped slow producer requests to specific file system operations, using tools like TCB and Wireshark to measure latencies by matching Kafka requests to responses. TLS encryption was temporarily disabled to allow unencrypted data inspection during analysis. With eBPF (Extended Berkeley Packet Filter), they dynamically traced file system calls, linking them to Kafka threads and topics for precise debugging. Slow requests were traced back to ext4's metadata locking mechanism, which blocked thread access and caused significant delays. Switching to XFS eliminated this bottleneck, demonstrating the critical role of file systems in high-performance applications. The case study underscores the importance of understanding file system mechanisms, such as journaling, which ensures data consistency by logging changes before writing them to disk. Systems like ext4 and XFS employ journaling to prevent data loss during crashes. However, optimizations like disabling journaling or reducing commit intervals can improve performance but may risk partial data corruption. Techniques such as "reverse mapping" and XFS's "Fast Commit" feature were highlighted for their ability to reduce latency and enhance performance. The analysis also emphasized the value of profiling tools like async profiler and eBPF in diagnosing performance issues and identifying bottlenecks. Contributions to improving systems like Kafka and advancing file system technologies were praised, reflecting the ongoing need for innovation as workloads grow. The summary concludes by acknowledging efforts in delivering high-quality technical content, particularly through projects like operating system courses, reinforcing the importance of continuous learning and development in the field of system optimization. **Key Takeaways:** 1. File system choice significantly impacts application performance. 2. Tools like eBPF, TCB, and Wireshark are invaluable for diagnosing latency issues. 3. Journaling mechanisms, while ensuring data integrity, can introduce performance bottlenecks. 4. Optimizations must balance performance gains with potential risks, such as data corruption. 5. Continuous improvement and innovation are essential for addressing evolving system challenges. **Boxed Final Answer:** {Algro improved CFA producer rights by 82% by replacing ext4 with XFS, resolving metadata locking bottlenecks. Their analysis highlights the importance of file systems, journaling mechanisms, and diagnostic tools like eBPF in optimizing high-performance applications.}

Course: OS Fundamentals

### Course Description: OS Fundamentals The **OS Fundamentals** course provides a comprehensive exploration of core operating system concepts, focusing on process management, scheduling, and resource allocation in Linux-based systems. Students will gain hands-on knowledge of how processes are prioritized and managed within the Linux environment, including an in-depth understanding of "niceness" values and their impact on CPU resource distribution. The course begins with foundational topics such as assigning priority levels to processes, where values range from -20 (highest priority) to 19 (lowest priority). Through practical demonstrations using tools like `top` and `renice`, students will learn how to monitor and adjust process priorities dynamically, ensuring optimal system performance. Additionally, the course delves into advanced concepts such as real-time processes and their dominance over standard processes, equipping learners with the skills to manage complex workloads effectively. A significant portion of the course is dedicated to understanding workload types and their implications for system scalability. Students will explore two primary categories of workloads: I/O-bound and CPU-bound tasks. Using real-world examples, such as PostgreSQL for I/O-bound applications and custom C programs for CPU-intensive tasks, learners will analyze how different workloads affect system resources. The course emphasizes the importance of vertical scaling (adding more resources to a single machine) versus horizontal scaling (distributing workloads across multiple machines) and provides strategies for achieving cost-effective scalability. By leveraging Linux commands like `top`, students will gain insights into CPU metrics, memory usage, and system-level operations, enabling them to diagnose and optimize performance bottlenecks. Throughout the course, students will engage in interactive experiments using Raspberry Pi devices, simulating multi-core environments to observe process behavior under varying conditions. These hands-on exercises will reinforce theoretical concepts and encourage creative problem-solving. By the end of the course, participants will have a solid grasp of Linux process management, workload optimization, and system monitoring techniques. Whether you're a beginner looking to understand the basics of operating systems or an experienced developer aiming to enhance your system administration skills, this course offers valuable insights and practical tools to help you succeed in managing modern computing environments.