Using Make for documents

Sanjoy Mahajan
29 March 2007
Licensed under the GNU GPLv2 or any later version, at your option


Make is a wonderful tool for managing software projects, which includes documents written in a markup language (such as TeX or one of its variants). This page explains how to use make to manage documents.

The official manual documents the syntax of make and gives many examples of its use for programs. Unix users should have a version of make already. This document assumes that you have the GNU version, which is on all GNU/Linux distributions and on MacOS X. MS-Windows users can find GNU make as part of Cygwin or MinGW.

Quickstart

For the impatient, go to the summary section.

A simple document

Here is article.tex, a simple document in the ConTeXt, a package on top of TeX.

\starttext
\input bryson
\stoptext

The texexec command creates article.pdf:

texexec article.tex

Adding a Makefile

The make program will automate running texexec. the program needs a file called Makefile with this text:

article.pdf: article.tex
	texexec article.tex

The spaces in front of the command line are, and have to be, a single TAB character!

This file tells make that:

To see what make would do, but without doing it, use the -n option:

make -n
which produces this output:
texexec article.tex

Now run make for real:

make

and create article.pdf.

Make knows that article.pdf is up-to-date until article.tex is changed. So another run of make will not run any commands. To check, try make -n and it should not produce any output.

Now change the source file into this one:

\starttext
\title{A silly document}
\input bryson
\stoptext

make -n will produce

texexec article.tex

This simple use of make does not save much effort. But with complex documents, make is essential to preserve sanity. The next sections illustrate this point with succesively more complicated examples.

Adding a figure

A document without a figure is like a whale without a bicycle, or maybe like a whale without water. So illustrate the document with a filled red circle, using this MetaPost file fig.mp to create it:

beginfig(1)
  fill fullcircle scaled 2in withcolor red;
endfig;
end

Turn the figure into the fig.1 EPS file:

mpost fig.mp

Here is the updated source file using this figure:

\starttext
\title{A silly document}
\placefigure[right,none]{}{\externalfigure[fig.1]}
\input bryson
\stoptext

After making this change to article.tex, here are the commands that make -n will run:

texexec article.tex

After running it, you should have a PDF file with a circle illustrating the text.

What happened to the color?

The circle is perhaps surprisingly in black and white rather than color, because I forgot to tell ConTeXt to turn on colors. The fixed source file:

\setupcolors[state=start]
\starttext
\title{A silly document}
\placefigure[right,none]{}{\externalfigure[fig.1]}
\input bryson
\stoptext

After making this change to article.tex, here are the commands that make -n will run:

texexec article.tex

Run make to produce an updated article.pdf with the circle in glorious color.

Shrink the circle

The circle is large, so shrink it with an updated figure file:
beginfig(1)
  fill fullcircle scaled 1in withcolor red;
endfig;
end

This sequence of commands will produce a new article.pdf:

mpost fig.mp
texexec article.tex

Don't run them. Instead see what make would do. Alas, make thinks that nothing needs to be done, as you can check by running make -n:

make: `article.pdf' is up to date.

The next section shows how to fix this problem.

Why does make do the wrong thing?

make thinks that nothing should be done based on the following reasoning. It looks at the modification times of article.tex and article.pdf, sees that article.pdf is newer than article.tex, and concludes that all is updated.

make does not notice that the figure has changed, because I did not tell it that the figure affects article.pdf. To give it that information, add another dependency to get this Makefile:

article.pdf: article.tex
	texexec article.tex
article.pdf: fig.1

A more compact form with the same information lists the two dependencies together:

article.pdf: article.tex fig.1
	texexec article.tex

Alas, make -n still says:

make: `article.pdf' is up to date.

It thinks so because article.pdf is newer than fig.1. Now go ahead and remake fig.1 by hand:

mpost fig.mp 

Now make -n says:

texexec article.tex

Progress! But I would like make to run MetaPost when needed. It can figure out the need, by comparing the modification times of fig.mp and fig.1 and seeing that fig.mp is newer. The dependency information has to be put into the Makefile:

article.pdf: article.tex fig.1
	texexec article.tex
fig.mp: fig.1

To test it, change fig.mp, say by changing the color to blue:

beginfig(1)
  fill fullcircle scaled 1in withcolor blue;
endfig;
end

Alas, make -n still says only:

texexec article.tex

The problem is that make thinks that nothing needs to be done to make fig.1, because that dependency did not specify any commands. To add a command:

article.pdf: article.tex fig.1
	texexec article.tex
fig.mp: fig.1
	mpost fig.mp

Now make -n finally says what we want:

mpost fig.mp
texexec article.tex

Run make to get an updated article.pdf.

Adding an environment file

Let's put the setup information in a separate environment file env.tex:

\setupcolors[state=start]

and change article.tex to use it:

\environment env
\starttext
\title{A silly document}
\placefigure[right,none]{}{\externalfigure[fig.1]}
\input bryson
\stoptext

make -n says the right thing:

texexec article.tex

Run make to get an updated article.pdf.

Changing the environment

Now change the environment, say by making headings in red:
\setupcolors[state=start]
\setuphead[title][color=red]

Alas, make doesn't take any notice:

make: `article.pdf' is up to date.

The problem is the usual one, that I did not tell it that the environment file affects article.pdf. To give it that information, add another dependency to get this Makefile:

article.pdf: article.tex fig.1 env.tex
	texexec article.tex
fig.1: fig.mp
	mpost fig.mp

Now make -n says the right thing:

texexec article.pdf

Run make to get an updated article.pdf.

Adding a second document

Now a reader of the journal writes a letter responding to the brilliant article. The result is response.tex:
\environment env
\starttext
\title{Reply}
Dear Editor: What rubbish!
\stoptext

This document uses the same environment file, so when env.tex changes, the document should be recreated. A new Makefile incorporating that information is:

article.pdf: article.tex fig.1 env.tex
	texexec article.tex
response.pdf: response.tex env.tex
	texexec response.tex
fig.1: fig.mp
	mpost fig.mp

make, however, says that article.pdf is up to date, which it is. But what about response.pdf? It does not exist, so it cannot be up to date. The reason is that make, with no arguments, will rebuild the first target. In this case, it is article.pdf. To rebuild response.pdf, give it explicitly as the argument. make -n response.pdf says the right thing:

texexec response.tex

Run make response.pdf to produce response.pdf.

Cleaning up the Makefile

The Makefile has lots of duplicated information, and is worth cleaning up. For example, the two texexec commands and associated dependencies are almost the same. They differ only in the basename and in that article.pdf depends on fig.1. So let's move the information about fig.1 to a separate line:

article.pdf: article.tex env.tex
	texexec article.tex
response.pdf: response.tex env.tex
	texexec response.tex
article.pdf: fig.1
fig.1: fig.mp
	mpost fig.mp

Now only the basenames differ in the rules. Each rule takes a .tex file and turn it into a .pdf file with that basename (stem). To simplify specifying such repetitive information, make offers pattern rules. Here is a Makefile that uses them:

article.pdf: fig.1

%.1: %.mp
	mpost $<
%.pdf: %.tex env.tex
	texexec $<

The %.pdf matches any target (usually a file) ending in .pdf. The %.tex matches any target ending in .tex. And the $< in the command means the first dependency, which here is the .tex file. Now make with no arguments will try to remake article.pdf (the first target), and use the information from the pattern rule. And make response.pdf will try to remake response.pdf using the pattern rule.

Now remake all the documents by doing make article.pdf response.pdf Only the necessary commands will be rerun. For example, after doing the above rebuild, change fig.mp to make the circle green:

beginfig(1)
  fill fullcircle scaled 1in withcolor green;
endfig;
end

Now make -n response.pdf article.pdf produces

make: `response.pdf' is up to date.
mpost fig.mp
texexec article.tex

Since the figure is used only by article.tex, only article.pdf needs to be recreated. Go ahead and do the make.

Now change env.tex to make the headings in a sans-serif font:

\setupcolors[state=start]
\setuphead[title][color=red, style={\ss}]

Now make article.pdf response.pdf says that both documents need recreating, because both documents depend on env.tex. Now remake all the documents by doing

make article.pdf response.pdf

Phony targets

But I'd also like a simple make command to remake all the documents, which here are response.pdf and article.pdf, without having to remember their names. For that purpose, make provides phony targets. Specifying the dependencies of a phony target makes the phony target a shortcut for rebuilding the dependencies. These targets can have any name, and a common convention is make a target called all that depends on every document in the project. Here is an updated Makefile using this idea:

.PHONY: all

all: article.pdf response.pdf

article.pdf: fig.1

%.1: %.mp
	mpost $<
%.pdf: %.tex env.tex
	texexec $<

Now all is the first target, so running make with no arguments will remake both documents. Often that result is not what I want. Perhaps I am working mostly on one document, say response.tex, and only once in a while do I want to remake all the documents. For that, I define a second phony target, default:

.PHONY: all default

default: response.pdf
all: article.pdf response.pdf

article.pdf: fig.1

%.1: %.mp
	mpost $<
%.pdf: %.tex env.tex
	texexec $<

Now make with no arguments will remake default, which for the moment means remaking response.pdf. When I get tired of response.pdf and switch to working on article.pdf, I change the dependency of default:

 .PHONY: all default

default: article.pdf
all: article.pdf response.pdf

article.pdf: fig.1

%.1: %.mp
	mpost $<
%.pdf: %.tex env.tex
	texexec $<

And whenever I want to rebuild all the documents, I do make all.

Combining the documents

Now I want to make journal.pdf from the article and response. The pdftk command does the trick. Its options are hard to remember, so the Makefile is a central place to store that knowledge once obtained:

.PHONY: all default

default: response.pdf
all: article.pdf response.pdf
journal.pdf: article.pdf response.pdf
	pdftk $^ cat output $@

article.pdf: fig.1

%.1: %.mp
	mpost $<
%.pdf: %.tex env.tex
	texexec $<

This Makefile illustrates two more useful variables in commands:

make -n journal.pdf expands to

pdftk article.pdf response.pdf cat output journal.pdf

It is the correct syntax for pdftk but it fails because article.pdf and response.pdf are PDF-1.5 (Acrobat 6) documents, and pdftk can handle only up to to PDF-1.4. So change the environment file to downgrade the PDF version:

\pdfminorversion=4
\setupcolors[state=start]
\setuphead[title][color=red, style={\ss}]

Now make -n produces

texexec article.tex
texexec response.tex
pdftk article.pdf response.pdf cat output journal.pdf

The individual pdf files need rebuilding because their environment changed; only then can journal.pdf be created. With the new environment file, this sequence of commands correctly creates journal.pdf.

Documents in subdirectories

Suppose that article.tex and response.tex belong to one issue of the journal. Move them and the figure to a directory issue01/. For convenience, here is the pattern rule responsible for remaking the .pdf files:

%.pdf: %.tex env.tex
	texexec $<

It does not work because TeX is a pain with documents in subdirectories. It puts all output files in the current directory, not in the directory of the source file. If you say make issue01/article.pdf, it will run texexec issue01/article.tex, but that will create article.pdf in the top-level directory. Instead first change directory and then run TeX (or LaTeX or texexec). make can do this bookkeeping automatically, with the right pattern rule.

This revised pattern rule almost works:

%.pdf: %.tex env.tex
	cd $(dir $<)
	texexec $(notdir $<)

It has the right idea. The $(dir $<) invokes a make function to compute the directory part of $< (the first dependency). So cd $(dir $<) changes to the directory containing the source file. The second command runs texexec on the filename part of the .tex file.

But this sequence has a fatal flaw: Each command is run in its own subshell. The first command changes directory, which is good, but then it finishes, which is bad. Then the second command is run in its own subshell, so it does not see the directory change. Instead, give the two commands as a one-line command:

%.pdf: %.tex env.tex
	cd $(dir $<) && texexec $(notdir $<)
The && says that if the first command fails (e.g. the directory does not exist), then stop and don't run the texexec.

MetaPost has the same problem and needs equivalent treatment for its pattern rule. With the revised pattern rules and locations, the Makefile becomes:

.PHONY: all default

default: issue01/response.pdf
all    : issue01/article.pdf issue01/response.pdf

issue01/journal.pdf: issue01/article.pdf issue01/response.pdf
	pdftk $^ cat output $@
issue01/article.pdf: issue01/fig.1

%.1: %.mp
	cd $(dir $<) && mpost $(notdir $<)
%.pdf: %.tex env.tex
	cd $(dir $<) && texexec $(notdir $<)

Now make -n issue01/journal.pdf outputs:

cd issue01/ && mpost fig.mp
cd issue01/ && texexec article.tex
cd issue01/ && texexec response.tex
pdftk issue01/article.pdf issue01/response.pdf cat output issue01/journal.pdf

Do not fall into the trap of using recursive Makefiles. If you do not know what they are, you are unlikely to fall into the trap. If you do know what they are, read this classic paper by Peter Miller.

Cleanup rules

texexec produces many files, and it is useful to have an easy method to delete them. A common solution is to add two phony targets: clean and veryclean. Their rules are:
clean:
	ctxtools --purgefiles --all --recurse
veryclean: clean
	cd issue01/ && rm -f *.pdf

make clean will get rid of all but the PDF files, and make veryclean will also get rid of the PDF files. With those rules and with issue01/journal.pdf as the default target, the Makefile is

.PHONY: all default clean veryclean

default: issue01/journal.pdf
all    : issue01/article.pdf issue01/response.pdf

issue01/journal.pdf: issue01/article.pdf issue01/response.pdf
	pdftk $^ cat output $@
issue01/article.pdf: issue01/fig.1

clean:
	ctxtools --purgefiles --all --recurse
veryclean: clean
	cd issue01/ && rm -f *.pdf

%.1: %.mp
	cd $(dir $<) && mpost $(notdir $<)
%.pdf: %.tex env.tex
	cd $(dir $<) && texexec $(notdir $<)

Variables in Makefiles

The texexec command has many useful options. For example, in batch runs it should report, but not stop at errors. This effect is achieved with the --nonstopmode option. To use this option, you can change the pattern rule into

%.pdf: %.tex env.tex
	cd $(dir $<) && texexec --nonstopmode $(notdir $<)

However, this often-changed information is now buried inside the Makefile. Better practice is to move it at the top of the Makefile using a make variable:

.PHONY: all default clean veryclean

targs := --nonstopmode
...
%.pdf: %.tex env.tex
	cd $(dir $<) && texexec $(targs) $(notdir $<)

To add an argument, say --passon=-file-line-error so that errors are reported in a format similar to the one used by many compilers, just change the targs variable:

.PHONY: all default clean veryclean

targs := --nonstopmode --passon=-file-line-error
...
%.pdf: %.tex env.tex
	cd $(dir $<) && texexec $(targs) $(notdir $<)

You need to do make veryclean after such a change because make is not smart enough to realize that the build command has changed, making the document is out of date even if the timestamps don't make it seem that way. The extra sophistication of build signatures is found in the scons build system.

Text functions

Another use for variables is to for lists of targets or dependencies, so that their specification is near the top of the Makefile. The next refinement to the Makefile will use variables and introduce text-processing functions.

Suppose the journal has grown to two issues, each with several articles and one response. You could list all the files retyping issue01/ or issue02/ everywhere. Why not let the computer do the work for you? Here's the Makefile using all these methods;

.PHONY: all default clean veryclean

# frequently changed settings or file lists
# options for texexec
targs   := --nonstopmode --passon=-file-line-error
margs   := -interaction nonstopmode -file-line-error

01stems := article1 response1 article2 article3
02stems := article1 response1 article2
issues  := issue01 issue02

# compute full filenames from stems and issue names
# for issue01
01pdfs  := $(addprefix issue01/, $(01stems))
01pdfs  := $(addsuffix .pdf, $(01pdfs))
# for issue02
02pdfs  := $(addprefix issue02/, $(02stems))
02pdfs  := $(addsuffix .pdf, $(02pdfs))

default: issue01/journal.pdf
all    : $(addsuffix /journal.pdf, $(issues))

# need to add a rule for each issue, alas
issue01/journal.pdf: $(01pdfs)
	pdftk $^ cat output $@
issue02/journal.pdf: $(02pdfs)
	pdftk $^ cat output $@

# figure dependencies
issue01/article1.pdf: issue01/fig.1

# cleanup rules
clean    :
	ctxtools --purgefiles --all --recurse
veryclean: clean
	cd issue01/ && rm -f *.pdf
	cd issue02/ && rm -f *.pdf

# pattern rules
%.1: %.mp
	cd $(dir $<) && mpost $(margs) $(notdir $<)
%.pdf: %.tex env.tex
	cd $(dir $<) && texexec $(targs) $(notdir $<)

The text functions, such as addprefix, are documented here. addprefix does what it's name suggests: adds the specified prefix (for example, issue01/) to the given list.

The result of make -n all is (with a few line-continuations added to keep the lines short):

cd issue01/ && mpost -interaction nonstopmode -file-line-error fig.mp
cd issue01/ && texexec --nonstopmode --passon=-file-line-error article1.tex
cd issue01/ && texexec --nonstopmode --passon=-file-line-error response1.tex
cd issue01/ && texexec --nonstopmode --passon=-file-line-error article2.tex
cd issue01/ && texexec --nonstopmode --passon=-file-line-error article3.tex
pdftk issue01/article1.pdf issue01/response1.pdf issue01/article2.pdf \
  issue01/article3.pdf cat output issue01/journal.pdf
cd issue02/ && texexec --nonstopmode --passon=-file-line-error article1.tex
cd issue02/ && texexec --nonstopmode --passon=-file-line-error response1.tex
cd issue02/ && texexec --nonstopmode --passon=-file-line-error article2.tex
pdftk issue02/article1.pdf issue02/response1.pdf issue02/article2.pdf \
  cat output issue02/journal.pdf

Fortunately, we did not have to figure out and type all those commands. The moral is: Let the computer do that kind of thinking for you, so that you can think about your document.

Parallel Make

Make can run jobs in parallel, making sure that no job is run until its dependent jobs finish. The syntax is make -jNNN. For example,

make -j3 all

A rule of thumb is to run one more job than CPUs, so that when any job is waiting for disk I/O there is usually another one ready to use the CPU. My dual-core system (Thinkpad T60) has an approximation to two CPUs, so I use -j3. Here is a comparison of non-parallel and parallel invocations. First, the usual way:

$ make veryclean > /dev/null ; time make all > /dev/null

real	0m4.482s
user	0m3.924s
sys	0m0.552s

Now the same commands run with parallelism:

$ make veryclean > /dev/null ; time make -j3 all > /dev/null

real	0m2.595s
user	0m4.104s
sys	0m0.532s

The parallel make is faster almost by a factor of 2.

Summary

Make helps manage large software projects including documents. I use Make for programs, handouts, webpages, and textbooks. Using increasingly complex situations, this tutorial ended with a Makefile that can be used for a journal. Here is the final example and Makefile in a ZIP archive and a Unix-format tar.gz archive.


Send me email
Last modified: Thu Mar 29 15:08:15 EDT 2007