TIL: Find shared lines with comm

diff is a well-known tool to see what lines are different between files; comm is a less-known tool to see the common lines.

I found comm when I was looking for a way to identify lines that are shared (or not) between files. Specifically, I had two lists: one with all the APIs in my codebase, and another with all the tested APIs. From this I wanted to get a list of untested APIs.

As an aside, my go-to in this situation is to drop in a Ruby one-ish-liner:

ruby -e "puts (File.readlines('all') - File.readlines('tested'))"

In Ruby, Array.- (minus) returns a new array after removing the elements from the right-hand side. IMO, this is pretty nice. Ruby has a wide variety of methods for combining arrays (+ to merge, - for difference, & for intersection, | for union). The only caveat is that this is trickier if you haven’t written the lines to a file. That’s pretty common in pipelines and was, in fact, exactly the situation in which I found myself.

Let’s say we have these two lists: mammals.txt and pets.txt:

$ cat mammals.txt
cat
dog
lion
rabbit
whale
$ cat pets.txt
cat
dog
fish
rabbit
snake

We can compare the files with comm:

$ comm mammals.txt pets.txt
		cat
		dog
	fish
lion
		rabbit
	snake
whale

The first time I saw this output I was pretty confused. Is this working? It kind of looks like I just dumped all the lines with weird indentation? comm produces output in three columns:

  1. lines that are only in the first file
  2. lines that are only in the second file
  3. lines that are in both files

Here’s a labeled version:

$ comm mammals.txt pets.txt
just          | just       |
mammals.txt   | pets.txt   | both
--------------+------------+-------
              |            | cat
              |            | dog
              | fish       |
lion          |            |
              |            | rabbit
              | snake      |
whale         |            |

You can use -1 to hide the first column, -2 to hide the second column, and, yes, -3 to hide the third column. In this case, if you want just animals that are mammals and pets: comm -12 mammals.txt pets.txt:

$ comm -12 mammals.txt pets.txt
cat
dog
rabbit

If you want pets that aren’t mammals, that would be comm -13 mammals.txt pets.txt.

To get back to my original use case, if I want a list of untested APIs, you write:

comm -23 all_apis.txt tested_apis.txt

comm works with process substitution so you can also write:

comm -23 <(command to find all APIs) <(command to find tested APIs)

One caveat: the input lines need to be sorted, so you may need to add | sort.

Text