How to define Makefile target(s) when rule calls script?

Question

I am trying for the first time to use make to run a series of scripts. I have a tree dir structure like this:

project
├── data
│   └── run1
│       └── pass
│            ├── 0
│            ├── 1
│            └── 2
├── include
│   └── variables.mk
├── Makefile
└── scripts
    └── operations.sh

I have one dataset, run1, which has multiple dirs in pass, all of which hold txt files. The operations.sh script uses a method that recursively searches a named dir (here pass). I expect more data (run2, run3 etc), and would like to be able to process them in the same way when the data is available. To this end I use include on variables.mk (not sure if this is appropriate but it works fine), defining $(INPUT_RUNS), which I will simply update as new runs arrive.

I have written a test Makefile

include $(CURDIR)/include/variables.mk
DATA_DIRS := $(addprefix $(CURDIR)/data/, $(foreach r, $(INPUT_RUNS), $(r))/pass)
OUT_DIRS := $(addprefix $(CURDIR)/analysis/, $(foreach r, $(INPUT_RUNS), $(r)))

##targets 
all: operations_run 

##operations
operations_run: $(OUT_DIRS) $(DATA_DIRS)
  mkdir -p $</operations
  sh scripts/operations.sh $</operations $(DATA_DIRS)

This specifies a set of dirs (data, analysis per run). I can then make a target with which to run operations.sh. This works fine. But it doesn't actually use make properly to my mind. I want to make the output, and then if rerunning make, not regenerate the output if no part of the data or analysis has changed.

My question therefore: generally, a target is a file. The operations.sh script runs a method not developed by me, and has particular rules about input and output (both are dirs as seen). I would like to make the target a set of files produced by operations.sh. I would like it to work something like

%.output.txt: $(DATA_DIRS)
    sh operations.sh $< > $@

I think I understand how to use % to name the dependencies, though haven't tested this. Can I give $(DATA_DIRS) as dependency, while makeing the target files? Conceptually I have no idea where to start on that aspect.

Any help is very much appreciated.

Answer 1

I think the rule should be like:

$(CURDIR)/analysis/%: $(CURDIR)/data/%/pass
   sh operations.sh $< $@

Basically having a rule for individual outdirs. Supposing operations takes an input and output directory.

Though that may "think" the output directory was altered for whatever reasons (think temporary files from viewer or so). Personally I like to manually "stamp" a completion.

$(CURDIR)/analysis/%/.done: $(CURDIR)/data/%/pass
   sh operations.sh $< $(@:/.done=)
   touch $@

This will put an empty .done file in the outdir with the timestamp of last successful creation.

and

operations_run: $(addsuffix /.done,$(OUT_DIRS))

to run the whole set.

Answer 2

If you do not know what the real inputs of your script are, it will be difficult to tell make whether a particular target must be re-done or not. Make compares last modification times of target files and prerequisite files. Directories are more difficult to use for this analysis because a directory's last modification time has a different meaning: it changes when files are added or deleted, not when the content of a file changes. You should first understand what the real inputs of your script are and then express dependencies between output files and input files.

Note: you can tell make to build the list of directories to process with something like:

DATA_DIR := $(shell find $(CURDIR)/data -type d -name pass)