Like with Handmade Hero, I've split my project into a few TU that compile linearly as part of the batch build process, but I want these to occur concurrently.
Has anyone accomplished this before without having to resolve to multiple batch files spawning from the build.bat? Tips?
Has anyone accomplished this before without having to resolve to multiple batch files spawning from the build.bat? Tips?
Edited by Jesse
on
Reason: Clarity
One solution is to write a Build application that uses CreateProcess to launch multiple compiles and then uses WaitForMultipleObjects to wait until all compiles are finished.
The next solution I found on stackoverflow:
The next solution I found on stackoverflow:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | @echo off setlocal set "lock=%temp%\wait%random%.lock" :: Launch processes asynchronously, with stream 9 redirected to a lock file. :: The lock file will remain locked until the script ends. start "" 9>"%lock%1" cl tu1.cpp start "" 9>"%lock%2" cl tu2.cpp :Wait for both processes to finish (wait until lock files are no longer locked) 1>nul 2>nul ping /n 2 ::1 for %%N in (1 2) do ( (call ) 9>"%lock%%%N" || goto :Wait ) 2>nul ::delete the lock files del "%lock%*" :: Finish up echo Done - ready to continue processing |
If you pass /MP for cl.exe then it will spawn multiple cl.exe (by default how many cores you have) to compile TUs in parallel.
Like this:
You can adjust how many child cl.exe's to spawn with number after /MP like this: /MP2
Like this:
1 | cl.exe /MP a.cpp b.cpp c.cpp |
You can adjust how many child cl.exe's to spawn with number after /MP like this: /MP2
Edited by Mārtiņš Možeiko
on
mmozeiko
If you pass /MP for cl.exe then it will spawn multiple cl.exe (by default how many cores you have) to compile TUs in parallel.
Like this:
You can adjust how many child cl.exe's to spawn with number after /MP like this: /MP2
How does this work when the different TUs have unique compiler / linker options set?
@ratchetfreak, thanks! If I can't figure out a simpler method I'll give that a try.
Edited by Jesse
on
It doesn't work for different cl flags. For that you need to invoke new cl.exe instances. Same with linker.
the start command will let batch continue execution of subsequent commands immediately however it doesn't really provide a way to wait on the that command again. It's either wait now or fire and forget.
That's why the SO solution messes around with lock files
That's why the SO solution messes around with lock files
On OSX/Linux this is easy with bash --you can use & and `wait`--although it occurs to me that I've never had to do anything to keep the error output correctly ordered. But I think rather than a build.bat that induces headaches you could just move to powershell or perl even (which, if you have git, is in your path). as long as it ticks the boxes--low overhead, no weird build tool dependency, easy to read off--who cares if its a bat or not? hell. maybe vbscript'll be able to do it. having a quick look the easiest non-batch method looks to be a powershell script although i've no idea how long it takes powershell to set itself up and begin executing a script
Alternatively since this is the 4coder forum, you could just have a very simple C function as your build--like, swoop into the working dir and hit every build_*.bat in there in parallel, dump stdout to a buffer, done
Alternatively since this is the 4coder forum, you could just have a very simple C function as your build--like, swoop into the working dir and hit every build_*.bat in there in parallel, dump stdout to a buffer, done
graeme
Alternatively since this is the 4coder forum, you could just have a very simple C function as your build--like, swoop into the working dir and hit every build_*.bat in there in parallel, dump stdout to a buffer, done
I don't have the super! edition, but now I have a reason for it.
If you really need different flags for cl.exe for different TUs, then I really think it is much easier to start use regular buildsystem like makefiles than hack up bat files. Result will be more readable and more maintainable.
Here's example with simple makefile that builds one executable with automatic dependency tracking. I use this template for most of my Linux stuff.
This is written for Linux and uses clang, but you can easily adjust it to use cl.exe and run on Windows.
Save it as "Makefile" and invoke it "make -j4" and it will compile everything in 4 codes as much as possible.
It should be pretty obvious how to extend it to support multiple target executables and different flags for different set of C files.
Here's example with simple makefile that builds one executable with automatic dependency tracking. I use this template for most of my Linux stuff.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 | CC := clang
CFLAGS := -pipe -fPIE -fpie -fvisibility=hidden \
-std=c99 -ffast-math \
-Wall -Wextra -Werror \
-MMD
LDFLAGS :=
# debug
CFLAGS += -O0 -g
LDFLAGS +=
# release
#CFLAGS += -O3 -fdata-sections -ffunction-sections
#LDFLAGS += -s -Wl,-z,relro,-z,now,-gc-sections
SRC := file1.c \
file2.c \
other.c \
whatever.c \
OBJ := ${SRC:.c=.o}
DEP := ${SRC:.c=.d}
TARGET := target.exe
.PHONY: all clean
all: ${TARGET}
clean:
rm -f ${OBJ} ${DEP}
${TARGET}: ${OBJ}
@echo [L] ${TARGET}
@${CC} ${LDFLAGS} -o $@ $^
%.o: %.c Makefile
@echo [C] $<
@${CC} ${CFLAGS} -c -o $@ $<
-include ${DEP}
|
This is written for Linux and uses clang, but you can easily adjust it to use cl.exe and run on Windows.
Save it as "Makefile" and invoke it "make -j4" and it will compile everything in 4 codes as much as possible.
It should be pretty obvious how to extend it to support multiple target executables and different flags for different set of C files.
Edited by Mārtiņš Možeiko
on
Thanks for sharing mmozeiko! That may well be the path I take here.
It's a bit frustrating that the differences between my translation units are, what I believe, trivial. Each TU only exports a unique function, and one in particular links to an outside library, but everything else is identical. So I was hopeful for a simple flag switch or some kind of grouping mechanism that could keep things wrapped up in a simple bat file.
It's a bit frustrating that the differences between my translation units are, what I believe, trivial. Each TU only exports a unique function, and one in particular links to an outside library, but everything else is identical. So I was hopeful for a simple flag switch or some kind of grouping mechanism that could keep things wrapped up in a simple bat file.
Exporting symbol from translation unit doesn't require any change in compiler flags. Just don't put "static" before symbol you want to export.
And linking to library also doesn't require separate flags to translation unit. Just add library to link flags.
And linking to library also doesn't require separate flags to translation unit. Just add library to link flags.
Edited by Mārtiņš Možeiko
on
I should have started the thread with my current current build system. I wasn't able to successfully merge them into a cl /MP line. Are my TU as trivial as I thought them?
1 2 3 4 5 | cl %CommonCompilerFlags% ..\code\jesse.cpp /LD %CommonLinkerFlags% -PDB:jesse_%random%.pdb /EXPORT:GameUpdate cl %CommonCompilerFlags% ..\code\jesse_opengl45.cpp /LD %CommonLinkerFlags% opengl32.lib -PDB:jesse_opengl45_%random%.pdb /EXPORT:RenderUpdate cl %CommonCompilerFlags% ..\code\win32_jesse.cpp %CommonLinkerFlags% user32.lib gdi32.lib opengl32.lib |
Ah I see, you are building each translation unit to separate dll/exe file. That's different then, those cannot be joined with /MP. /MP works only for compiler not linker.
Why do you want to split jesse.cpp and jesse_opengl45.cpp in separate dll's?
If you would build them to same dll file then those two files could be built in parallel:
Another option would be compile all three cpp files to obj files with /MP and then link each one separately. Linking usually is much faster process that compilation:
Why do you want to split jesse.cpp and jesse_opengl45.cpp in separate dll's?
If you would build them to same dll file then those two files could be built in parallel:
1 | cl %CommonCompilerFlags% /MP ..\code\jesse.cpp ..\code\jesse_opengl45.cpp /LD %CommonLinkerFlags% opengl32.lib -PDB:jesse_%random%.pdb /EXPORT:GameUpdate /EXPORT:RenderUpdate |
Another option would be compile all three cpp files to obj files with /MP and then link each one separately. Linking usually is much faster process that compilation:
1 2 3 4 5 | cl %CommonCompilerFlags% /c /MP ..\code\jesse.cpp ..\code\jesse_opengl45.cpp ..\code\win32_jesse.cpp cl jesse.obj /LD %CommonLinkerFlags% -PDB:jesse_%random%.pdb /EXPORT:GameUpdate cl jesse_opengl45.obj /LD %CommonLinkerFlags% opengl32.lib -PDB:jesse_opengl45_%random%.pdb /EXPORT:RenderUpdate cl win32_jesse.obj %CommonLinkerFlags% user32.lib gdi32.lib opengl32.lib |
Edited by Mārtiņš Možeiko
on