~/

How do pipes and FIFOs work on Linux?

— October 9, 2018

In my earlier post, I mentioned briefly that I’m working on a shell in Go. I’ve completed a working prototype and I’m looking into implementing pipes. After doing some readings, I think I understand how anonymous pipes and named pipes (FIFOs) work now.

Motivation

Here’s an example on executing a set of bash commands sequentially. We will first list all files in the current directory line by line, and then sort them in ascending order.

ls -la | sort

This is done through the pipe character (|), which is also known as the vertical bar. The idea to using pipes is very simple: the output of one process is used as the input to another process. This method of communication allows processes to transfer data between them without the need of temporary files. Note that pipes are always half-duplex (i.e. data flows in one direction).

How it works?

Pipe, as the name suggests, contains two ends: one for reading, and one for writing. We can create pipes by using the pipe2() system call. This will create two file descriptors referring to the ends of the pipe. Data written to the write end of the pipe is buffered by the kernel in memory until it is read from the other end of the pipe.

I’m going to dive a little bit into the Linux kernel source code: linux/fs/pipe.c.

Let’s first look into writing to a pipe. Pipes in Linux have limited capacity. In recent Linux versions, the max pipe size is 16 pages (i.e., 65,536 bytes in a system with a page size of 4096 bytes).

// L32-L36
/*
 * The max size that a non-root user is allowed to grow the pipe. Can
 * be set by root in /proc/sys/fs/pipe-max-size
 */
unsigned int pipe_max_size = 1048576;

Attempting to write to the pipe when it is full will block (or fail with EAGAIN if the O_NONBLOCK flag is set when creating a new pipe).

// L455-L472
if (filp->f_flags & O_NONBLOCK) {
  if (!ret)
    ret = -EAGAIN; // Fail the non-blocking I/O
  break;
}
...
pipe_wait(pipe); // Block and wait
...

We can also see the following snippet of code in the pipe_write() function. When a process attempts to write to a pipe in which all readers on the other end of the pipe have been closed, a SIGPIPE signal will be generated. This is actually the “Broken pipe” error!

// L368-L372
if (!pipe->readers) {
  send_sig(SIGPIPE, current, 0); // If signal ignored, we'll get a EPIPE error.
  ret = -EPIPE;
  goto out;
}

We’ll now look into reading from a pipe. When a process attempts to read from a pipe that has no buffered data, blocking occurs until we have a new event (see pipe_wait(pipe)). Now, if all of the writers on the write end of the pipe have been closed, reading from the pipe will return EOF. This makes sense because there won’t be any more input in the future.

Implementation

We will attempt to implement the following set of bash commands in C and Go.

ls -la | sort

General Idea:

  • With n processes, we will need n - 1 pipes.
  • The parent process will make n - 1 system calls to create the respective pipes (This might be inefficient, but we’ll discuss this later.)
  • The parent process then forks itself n times using the fork() system call.
  • For every single forked process:
    • we will use dup2() to duplicate the file descriptors that we have from pipe2() to the respective stdin or stdout file descriptors.
    • close the unnecessary pipes.
    • call execve() to replace the current process with a new one.
  • The parent process will wait for all the forked process to finish.

In the example above, the parent process handles all the pipe creation. This set of file descriptors gets duplicated across all forked processes, which will then need to be closed. We can potentially avoid this overhead by creating just one pipe in every single forked process (except the last one) to connect the current and next process, but there will be more complexity in the implementation.

Here’s an example on how we would implement it in C: https://gist.github.com/imjching/61b4f07abaf3a079d704a24c4a7406b8

Here’s a similar example in Go. In this case, instead of forking processes, we will use exec.Command to spawn a new process. Go does not offer a classic Unix fork function since goroutines, spawning processes, and exec’ing processes covers most use cases for fork
(see https://gobyexample.com/execing-processes).

package main

import (
  "bytes"
  "fmt"
  "io"
  "os"
  "os/exec"
)

func main() {
  pr, pw := io.Pipe()

  lsCmd := exec.Command("ls", "-la")
  lsCmd.Stdout = pw

  var buffer bytes.Buffer
  sortCmd := exec.Command("sort")
  sortCmd.Stdin = pr
  sortCmd.Stdout = &buffer

  fmt.Println("parent: running ls")
  lsCmd.Start()
  fmt.Println("parent: running sort")
  sortCmd.Start()
  lsCmd.Wait() // Wait until program is done
  fmt.Println("parent: ls completed")
  pw.Close()
  sortCmd.Wait()
  fmt.Println("parent: sort completed")
  io.Copy(os.Stdout, &buffer)
}

Note that io.Pipe() is making a pipe2() system call under the hood (see src/os/pipe_linux.go). It returns two File types, which represent open file descriptors.

// Pipe returns a connected pair of Files; reads from r return bytes written to w.
// It returns the files and an error, if any.
func Pipe() (r *File, w *File, err error) {
  var p [2]int

  e := syscall.Pipe2(p[0:], syscall.O_CLOEXEC)
  // pipe2 was added in 2.6.27 and our minimum requirement is 2.6.23, so it
  // might not be implemented.
  if e == syscall.ENOSYS {
    ...
  } else if e != nil {
    return nil, nil, NewSyscallError("pipe2", e)
  }

  return newFile(uintptr(p[0]), "|0", kindPipe), newFile(uintptr(p[1]), "|1", kindPipe), nil
}


Named pipes (FIFOs)

Unlike anonymous pipes above, named pipes are created as special files in the file system. Any process with sufficient access privileges can open that file for reading or writing. Data written to these pipes are not persisted in the file system. However, the named pipe file itself needs to be removed manually.

Named pipes can be created using mkfifo command or system call.

mkfifo mypipe

The above command creates the corresponding device on the file system.

$ ls -la | grep mypipe
prw-r--r--  1 imjching  staff  0 Oct 10 16:46 mypipe

$ file mypipe
mypipe: fifo (named pipe)

Since the named pipe is just like a normal file, we can delete the pipe that we have just created by calling rm mypipe.

Takeaways

  • EAGAIN is an error type that is often raised when performing a non-blocking I/O.
man 2 intro | less -Ip EAGAIN
> EAGAIN Resource temporarily unavailable. This is a temporary condition and later calls to the same routine may complete normally.

  • The Linux source code is MASSIVE. There’s a lot that I don’t know about it.

  • The file command allows you to determine the file type of a specific file.

Readings!

Here are some useful resources that I have encountered while reading about pipes and FIFOs:

  • Chapter 44 (pages 889 - 920) of TLPI (The Linux Programming Interface)! :smile:

  • It seems like pipes are implemented using virtual file systems (e.g. pipefs for Linux) based on what I have read so far. I don’t understand file systems well enough to explain it, so I would suggest reading this Quora post if you want to know how Unix pipes are implemented under the hood.

  • Notes on fork() and pipe()

  • How do I use execve in C?

Fun fact: It seems like Firefox uses pipes under the hood for small messages when communicating between two actors (See IPC Protocols)!

Related posts