4coder»Forums
Jesse
64 posts
Programmer at NASA GSFC
Build function for projects with multiple Translation Units
Edited by Jesse on Reason: Clarity
Like with Handmade Hero, I've split my project into a few TU that compile linearly as part of the batch build process, but I want these to occur concurrently.

Has anyone accomplished this before without having to resolve to multiple batch files spawning from the build.bat? Tips?
511 posts
Build function for projects with multiple Translation Units
One solution is to write a Build application that uses CreateProcess to launch multiple compiles and then uses WaitForMultipleObjects to wait until all compiles are finished.

The next solution I found on stackoverflow:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
@echo off
setlocal
set "lock=%temp%\wait%random%.lock"

:: Launch processes asynchronously, with stream 9 redirected to a lock file.
:: The lock file will remain locked until the script ends.
start "" 9>"%lock%1" cl tu1.cpp
start "" 9>"%lock%2" cl tu2.cpp

:Wait for both processes to finish (wait until lock files are no longer locked)
1>nul 2>nul ping /n 2 ::1
for %%N in (1 2) do (
  (call ) 9>"%lock%%%N" || goto :Wait
) 2>nul

::delete the lock files
del "%lock%*"

:: Finish up
echo Done - ready to continue processing
Mārtiņš Možeiko
2559 posts / 2 projects
Build function for projects with multiple Translation Units
Edited by Mārtiņš Možeiko on
If you pass /MP for cl.exe then it will spawn multiple cl.exe (by default how many cores you have) to compile TUs in parallel.
Like this:
1
cl.exe /MP a.cpp b.cpp c.cpp


You can adjust how many child cl.exe's to spawn with number after /MP like this: /MP2
Jesse
64 posts
Programmer at NASA GSFC
Build function for projects with multiple Translation Units
Edited by Jesse on
mmozeiko
If you pass /MP for cl.exe then it will spawn multiple cl.exe (by default how many cores you have) to compile TUs in parallel.
Like this:
1
cl.exe /MP a.cpp b.cpp c.cpp


You can adjust how many child cl.exe's to spawn with number after /MP like this: /MP2


How does this work when the different TUs have unique compiler / linker options set?

@ratchetfreak, thanks! If I can't figure out a simpler method I'll give that a try.
Mārtiņš Možeiko
2559 posts / 2 projects
Build function for projects with multiple Translation Units
It doesn't work for different cl flags. For that you need to invoke new cl.exe instances. Same with linker.
Jesse
64 posts
Programmer at NASA GSFC
Build function for projects with multiple Translation Units
Then the question becomes, are you aware of a method that does that in a batch file? AFAIK a batch file will not continue to execute until the current line has returned (has finished). But I want to spawn all cl instances regardless if they've returned or not.
511 posts
Build function for projects with multiple Translation Units
the start command will let batch continue execution of subsequent commands immediately however it doesn't really provide a way to wait on the that command again. It's either wait now or fire and forget.

That's why the SO solution messes around with lock files
39 posts / 1 project
Build function for projects with multiple Translation Units
On OSX/Linux this is easy with bash --you can use & and `wait`--although it occurs to me that I've never had to do anything to keep the error output correctly ordered. But I think rather than a build.bat that induces headaches you could just move to powershell or perl even (which, if you have git, is in your path). as long as it ticks the boxes--low overhead, no weird build tool dependency, easy to read off--who cares if its a bat or not? hell. maybe vbscript'll be able to do it. having a quick look the easiest non-batch method looks to be a powershell script although i've no idea how long it takes powershell to set itself up and begin executing a script

Alternatively since this is the 4coder forum, you could just have a very simple C function as your build--like, swoop into the working dir and hit every build_*.bat in there in parallel, dump stdout to a buffer, done
Jesse
64 posts
Programmer at NASA GSFC
Build function for projects with multiple Translation Units
graeme
Alternatively since this is the 4coder forum, you could just have a very simple C function as your build--like, swoop into the working dir and hit every build_*.bat in there in parallel, dump stdout to a buffer, done


I don't have the super! edition, but now I have a reason for it.
Mārtiņš Možeiko
2559 posts / 2 projects
Build function for projects with multiple Translation Units
Edited by Mārtiņš Možeiko on
If you really need different flags for cl.exe for different TUs, then I really think it is much easier to start use regular buildsystem like makefiles than hack up bat files. Result will be more readable and more maintainable.

Here's example with simple makefile that builds one executable with automatic dependency tracking. I use this template for most of my Linux stuff.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
CC     := clang

CFLAGS := -pipe -fPIE -fpie -fvisibility=hidden \
          -std=c99 -ffast-math \
          -Wall -Wextra -Werror \
          -MMD
LDFLAGS :=

# debug
CFLAGS += -O0 -g
LDFLAGS +=

# release
#CFLAGS += -O3 -fdata-sections -ffunction-sections
#LDFLAGS += -s -Wl,-z,relro,-z,now,-gc-sections

SRC := file1.c    \
       file2.c    \
       other.c    \
       whatever.c \

OBJ := ${SRC:.c=.o}
DEP := ${SRC:.c=.d}

TARGET := target.exe

.PHONY: all clean

all: ${TARGET}

clean:
		rm -f ${OBJ} ${DEP}

${TARGET}: ${OBJ}
		@echo [L] ${TARGET}
		@${CC} ${LDFLAGS} -o $@ $^

%.o: %.c Makefile
		@echo [C] $<
		@${CC} ${CFLAGS} -c -o $@ $<

-include ${DEP}

This is written for Linux and uses clang, but you can easily adjust it to use cl.exe and run on Windows.
Save it as "Makefile" and invoke it "make -j4" and it will compile everything in 4 codes as much as possible.
It should be pretty obvious how to extend it to support multiple target executables and different flags for different set of C files.
Jesse
64 posts
Programmer at NASA GSFC
Build function for projects with multiple Translation Units
Thanks for sharing mmozeiko! That may well be the path I take here.

It's a bit frustrating that the differences between my translation units are, what I believe, trivial. Each TU only exports a unique function, and one in particular links to an outside library, but everything else is identical. So I was hopeful for a simple flag switch or some kind of grouping mechanism that could keep things wrapped up in a simple bat file.
Mārtiņš Možeiko
2559 posts / 2 projects
Build function for projects with multiple Translation Units
Edited by Mārtiņš Možeiko on
Exporting symbol from translation unit doesn't require any change in compiler flags. Just don't put "static" before symbol you want to export.
And linking to library also doesn't require separate flags to translation unit. Just add library to link flags.
Jesse
64 posts
Programmer at NASA GSFC
Build function for projects with multiple Translation Units
I should have started the thread with my current current build system. I wasn't able to successfully merge them into a cl /MP line. Are my TU as trivial as I thought them?

1
2
3
4
5
cl %CommonCompilerFlags% ..\code\jesse.cpp /LD %CommonLinkerFlags% -PDB:jesse_%random%.pdb /EXPORT:GameUpdate

cl %CommonCompilerFlags% ..\code\jesse_opengl45.cpp /LD %CommonLinkerFlags% opengl32.lib -PDB:jesse_opengl45_%random%.pdb /EXPORT:RenderUpdate

cl %CommonCompilerFlags% ..\code\win32_jesse.cpp %CommonLinkerFlags% user32.lib gdi32.lib opengl32.lib
Mārtiņš Možeiko
2559 posts / 2 projects
Build function for projects with multiple Translation Units
Edited by Mārtiņš Možeiko on
Ah I see, you are building each translation unit to separate dll/exe file. That's different then, those cannot be joined with /MP. /MP works only for compiler not linker.

Why do you want to split jesse.cpp and jesse_opengl45.cpp in separate dll's?
If you would build them to same dll file then those two files could be built in parallel:
1
cl %CommonCompilerFlags% /MP ..\code\jesse.cpp ..\code\jesse_opengl45.cpp /LD %CommonLinkerFlags% opengl32.lib -PDB:jesse_%random%.pdb /EXPORT:GameUpdate /EXPORT:RenderUpdate


Another option would be compile all three cpp files to obj files with /MP and then link each one separately. Linking usually is much faster process that compilation:
1
2
3
4
5
cl %CommonCompilerFlags% /c /MP ..\code\jesse.cpp ..\code\jesse_opengl45.cpp ..\code\win32_jesse.cpp

cl jesse.obj /LD %CommonLinkerFlags% -PDB:jesse_%random%.pdb /EXPORT:GameUpdate
cl jesse_opengl45.obj /LD %CommonLinkerFlags% opengl32.lib -PDB:jesse_opengl45_%random%.pdb /EXPORT:RenderUpdate
cl win32_jesse.obj %CommonLinkerFlags% user32.lib gdi32.lib opengl32.lib

Jesse
64 posts
Programmer at NASA GSFC
Build function for projects with multiple Translation Units
I build primary components of my engine, with the exception of the platform layer, as dlls that I hot reload dynamically using Casey's technique. So it looks at last /MP won't do the trick then. It's so interesting how the build process works ... Thanks for the assistance!