4coder » Forums » Build function for projects with multiple Translation Units
JesseMeyer
Jesse
50 posts

None

#8054 Build function for projects with multiple Translation Units
1 year, 10 months ago Edited by Jesse on Aug. 14, 2016, 6:16 a.m. Reason: Clarity

Like with Handmade Hero, I've split my project into a few TU that compile linearly as part of the batch build process, but I want these to occur concurrently.

Has anyone accomplished this before without having to resolve to multiple batch files spawning from the build.bat? Tips?

The meaning of life is to dance and sing while the music is being played.
ratchetfreak
364 posts
#8055 Build function for projects with multiple Translation Units
1 year, 10 months ago

One solution is to write a Build application that uses CreateProcess to launch multiple compiles and then uses WaitForMultipleObjects to wait until all compiles are finished.

The next solution I found on stackoverflow:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
@echo off
setlocal
set "lock=%temp%\wait%random%.lock"

:: Launch processes asynchronously, with stream 9 redirected to a lock file.
:: The lock file will remain locked until the script ends.
start "" 9>"%lock%1" cl tu1.cpp
start "" 9>"%lock%2" cl tu2.cpp

:Wait for both processes to finish (wait until lock files are no longer locked)
1>nul 2>nul ping /n 2 ::1
for %%N in (1 2) do (
  (call ) 9>"%lock%%%N" || goto :Wait
) 2>nul

::delete the lock files
del "%lock%*"

:: Finish up
echo Done - ready to continue processing
mmozeiko
Mārtiņš Možeiko
1690 posts
1 project
#8056 Build function for projects with multiple Translation Units
1 year, 10 months ago Edited by Mārtiņš Možeiko on Aug. 14, 2016, 12:17 p.m.

If you pass /MP for cl.exe then it will spawn multiple cl.exe (by default how many cores you have) to compile TUs in parallel.
Like this:
1
cl.exe /MP a.cpp b.cpp c.cpp


You can adjust how many child cl.exe's to spawn with number after /MP like this: /MP2
JesseMeyer
Jesse
50 posts

None

#8058 Build function for projects with multiple Translation Units
1 year, 10 months ago Edited by Jesse on Aug. 14, 2016, 1:45 p.m.

mmozeiko
If you pass /MP for cl.exe then it will spawn multiple cl.exe (by default how many cores you have) to compile TUs in parallel.
Like this:
1
cl.exe /MP a.cpp b.cpp c.cpp


You can adjust how many child cl.exe's to spawn with number after /MP like this: /MP2


How does this work when the different TUs have unique compiler / linker options set?

@ratchetfreak, thanks! If I can't figure out a simpler method I'll give that a try.

The meaning of life is to dance and sing while the music is being played.
mmozeiko
Mārtiņš Možeiko
1690 posts
1 project
#8069 Build function for projects with multiple Translation Units
1 year, 10 months ago

It doesn't work for different cl flags. For that you need to invoke new cl.exe instances. Same with linker.
JesseMeyer
Jesse
50 posts

None

#8074 Build function for projects with multiple Translation Units
1 year, 10 months ago

Then the question becomes, are you aware of a method that does that in a batch file? AFAIK a batch file will not continue to execute until the current line has returned (has finished). But I want to spawn all cl instances regardless if they've returned or not.

The meaning of life is to dance and sing while the music is being played.
ratchetfreak
364 posts
#8075 Build function for projects with multiple Translation Units
1 year, 10 months ago

the start command will let batch continue execution of subsequent commands immediately however it doesn't really provide a way to wait on the that command again. It's either wait now or fire and forget.

That's why the SO solution messes around with lock files
graeme
33 posts
#8079 Build function for projects with multiple Translation Units
1 year, 10 months ago

On OSX/Linux this is easy with bash --you can use & and `wait`--although it occurs to me that I've never had to do anything to keep the error output correctly ordered. But I think rather than a build.bat that induces headaches you could just move to powershell or perl even (which, if you have git, is in your path). as long as it ticks the boxes--low overhead, no weird build tool dependency, easy to read off--who cares if its a bat or not? hell. maybe vbscript'll be able to do it. having a quick look the easiest non-batch method looks to be a powershell script although i've no idea how long it takes powershell to set itself up and begin executing a script

Alternatively since this is the 4coder forum, you could just have a very simple C function as your build--like, swoop into the working dir and hit every build_*.bat in there in parallel, dump stdout to a buffer, done
JesseMeyer
Jesse
50 posts

None

#8081 Build function for projects with multiple Translation Units
1 year, 10 months ago

graeme
Alternatively since this is the 4coder forum, you could just have a very simple C function as your build--like, swoop into the working dir and hit every build_*.bat in there in parallel, dump stdout to a buffer, done


I don't have the super! edition, but now I have a reason for it.

The meaning of life is to dance and sing while the music is being played.
mmozeiko
Mārtiņš Možeiko
1690 posts
1 project
#8084 Build function for projects with multiple Translation Units
1 year, 10 months ago Edited by Mārtiņš Možeiko on Aug. 16, 2016, 7:02 a.m.

If you really need different flags for cl.exe for different TUs, then I really think it is much easier to start use regular buildsystem like makefiles than hack up bat files. Result will be more readable and more maintainable.

Here's example with simple makefile that builds one executable with automatic dependency tracking. I use this template for most of my Linux stuff.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
CC     := clang

CFLAGS := -pipe -fPIE -fpie -fvisibility=hidden \
          -std=c99 -ffast-math \
          -Wall -Wextra -Werror \
          -MMD
LDFLAGS :=

# debug
CFLAGS += -O0 -g
LDFLAGS +=

# release
#CFLAGS += -O3 -fdata-sections -ffunction-sections
#LDFLAGS += -s -Wl,-z,relro,-z,now,-gc-sections

SRC := file1.c    \
       file2.c    \
       other.c    \
       whatever.c \

OBJ := ${SRC:.c=.o}
DEP := ${SRC:.c=.d}

TARGET := target.exe

.PHONY: all clean

all: ${TARGET}

clean:
		rm -f ${OBJ} ${DEP}

${TARGET}: ${OBJ}
		@echo [L] ${TARGET}
		@${CC} ${LDFLAGS} -o [email protected] $^

%.o: %.c Makefile
		@echo [C] $<
		@${CC} ${CFLAGS} -c -o [email protected] $<

-include ${DEP}

This is written for Linux and uses clang, but you can easily adjust it to use cl.exe and run on Windows.
Save it as "Makefile" and invoke it "make -j4" and it will compile everything in 4 codes as much as possible.
It should be pretty obvious how to extend it to support multiple target executables and different flags for different set of C files.
JesseMeyer
Jesse
50 posts

None

#8087 Build function for projects with multiple Translation Units
1 year, 10 months ago

Thanks for sharing mmozeiko! That may well be the path I take here.

It's a bit frustrating that the differences between my translation units are, what I believe, trivial. Each TU only exports a unique function, and one in particular links to an outside library, but everything else is identical. So I was hopeful for a simple flag switch or some kind of grouping mechanism that could keep things wrapped up in a simple bat file.

The meaning of life is to dance and sing while the music is being played.
mmozeiko
Mārtiņš Možeiko
1690 posts
1 project
#8095 Build function for projects with multiple Translation Units
1 year, 10 months ago Edited by Mārtiņš Možeiko on Aug. 17, 2016, 2:34 a.m.

Exporting symbol from translation unit doesn't require any change in compiler flags. Just don't put "static" before symbol you want to export.
And linking to library also doesn't require separate flags to translation unit. Just add library to link flags.
JesseMeyer
Jesse
50 posts

None

#8097 Build function for projects with multiple Translation Units
1 year, 10 months ago

I should have started the thread with my current current build system. I wasn't able to successfully merge them into a cl /MP line. Are my TU as trivial as I thought them?

1
2
3
4
5
cl %CommonCompilerFlags% ..\code\jesse.cpp /LD %CommonLinkerFlags% -PDB:jesse_%random%.pdb /EXPORT:GameUpdate

cl %CommonCompilerFlags% ..\code\jesse_opengl45.cpp /LD %CommonLinkerFlags% opengl32.lib -PDB:jesse_opengl45_%random%.pdb /EXPORT:RenderUpdate

cl %CommonCompilerFlags% ..\code\win32_jesse.cpp %CommonLinkerFlags% user32.lib gdi32.lib opengl32.lib

The meaning of life is to dance and sing while the music is being played.
mmozeiko
Mārtiņš Možeiko
1690 posts
1 project
#8098 Build function for projects with multiple Translation Units
1 year, 10 months ago Edited by Mārtiņš Možeiko on Aug. 17, 2016, 4:31 a.m.

Ah I see, you are building each translation unit to separate dll/exe file. That's different then, those cannot be joined with /MP. /MP works only for compiler not linker.

Why do you want to split jesse.cpp and jesse_opengl45.cpp in separate dll's?
If you would build them to same dll file then those two files could be built in parallel:
1
cl %CommonCompilerFlags% /MP ..\code\jesse.cpp ..\code\jesse_opengl45.cpp /LD %CommonLinkerFlags% opengl32.lib -PDB:jesse_%random%.pdb /EXPORT:GameUpdate /EXPORT:RenderUpdate


Another option would be compile all three cpp files to obj files with /MP and then link each one separately. Linking usually is much faster process that compilation:
1
2
3
4
5
cl %CommonCompilerFlags% /c /MP ..\code\jesse.cpp ..\code\jesse_opengl45.cpp ..\code\win32_jesse.cpp

cl jesse.obj /LD %CommonLinkerFlags% -PDB:jesse_%random%.pdb /EXPORT:GameUpdate
cl jesse_opengl45.obj /LD %CommonLinkerFlags% opengl32.lib -PDB:jesse_opengl45_%random%.pdb /EXPORT:RenderUpdate
cl win32_jesse.obj %CommonLinkerFlags% user32.lib gdi32.lib opengl32.lib

JesseMeyer
Jesse
50 posts

None

#8099 Build function for projects with multiple Translation Units
1 year, 10 months ago

I build primary components of my engine, with the exception of the platform layer, as dlls that I hot reload dynamically using Casey's technique. So it looks at last /MP won't do the trick then. It's so interesting how the build process works ... Thanks for the assistance!

The meaning of life is to dance and sing while the music is being played.