A lot has been written about improving Swift compile times, but the compiler and linker are just part of the equation that slows down our development cycle.
Most projects leverage great 3rd party tools like SwiftLint, Sourcery, SwiftGen, SwiftFormat and many more. Leveraging those tools is the right thing to do but we have to be aware that all of those tools come with some time delay for our build -> run development cycle.
We often set those tools to run as build-phases which means they run each time you attempt a build but none of those tools need to be run each time we build.
Even tools that generate code we need for our projects like Sourcery or SwiftGen don't need to be re-run unless we made changes in very specific parts of the application.
Case Study - NYT
As an example New York Times main application leverages a lot of 3rd party and internal tooling, the total time all the tools take is 6s on my (powerful) machine.
Only 6 seconds or as much as 6 seconds?
Let's put this into context:
- I build a project between 200-400 times on an average workday.
- Let's assume 90% of the time we don't need to run those tools with each build.
- We have 30 iOS developers working on the main app
Lower limit: 200 * 6s * 30 * 90* => 9 hours wasted per day
We are wasting 45 hours per week, if we can improve that it's almost like hiring a new full-time developer, except it's free.
Let's look at how we can improve this with a process change a dash of bash shell programming.
Establishing a new process
If we are not going to run those tools each build we need to guarantee a few things:
- We need a way to run all our tooling manually and it needs to be consistent since we have a lot of different tools
- Our repository should never be in a bad state, so no code can exist at any point that is not processed by our tooling
- Since we should never trust humans when a machine can be used, we are going to make sure CI enforces the repository state
If we try commiting and forget to run the tool we'll get an error like (here using GitUp IDE):
Implementation
We are going to create a shell script that can be run manually by calling ./Scripts/process.sh. It will also be a part of our pre-commit hook
First, let's set the script up with 2 modes:
- fail-on-errors -> any error will cause the script to exit with failure code, used by CI / pre-commit
- local -> manual developer runs, we don't fail script and we can use this mode to add some console coloring & extra debug information if need be
#!/bin/zsh
if [[ -n "$CI" ]] || [[ $1 == "--fail-on-errors" ]] ; then
FAIL_ON_ERRORS=true
echo "Running in --fail-on-errors mode"
ERROR_START=""
COLOR_END=""
INFO_START=""
else
echo "Running in local mode"
ERROR_START="\e[31m"
COLOR_END="\e[0m"
INFO_START="\e[34m"
fi
Next, let's configure some required variables for our project and the most common tooling:
final_status=0
# :- is like swift's ?: so default fallback value
PODS_ROOT=${PODS_ROOT:-"Pods"}
PROJECT_DIR=${PROJECT_DIR:-$(pwd)}
# Needed for SwiftGen
export PRODUCT_MODULE_NAME=${PRODUCT_MODULE_NAME:-"OurProjectModuleName"}
if [[ `xcode-select -p` =~ CommandLineTools ]] ; then
echo "${ERROR_START}Your toolchain won't run Swiftlint or Sourcery. Use\n sudo xcode-select -s /path/to/Xcode.app\nto fix this.${COLOR_END}"
fi
Checking if an external tool has done anything to our project
Now we are ready to start running our tools but the issue we'll immediately run into is the fact that each tool might be doing its job in many different ways:
- Files might be added/changed e.g. SwiftGen, Sourcery
- Messages might be printed on stdout e.g. SwiftLint
- Only the status code will be returned
How can we account for all of that?
File changes
We can leverage git diff to verify if anything was changed in our repo:
# execute like this `process name command`
function process()
local initial_git_diff=`git diff --no-color`
eval "$2"
if [ "$FAIL_ON_ERRORS" = "true" ] && [[ "$initial_git_diff" != `git diff --no-color` ]]
then
echo "${ERROR_START}$1 generates git changes, run './Scripts/process.sh' and review the changes${COLOR_END}"
final_status=1
fi
}
process "Sourcery" "${PODS_ROOT}/Sourcery/bin/sourcery --prune --quiet"
Standard output
Our processing function simply needs to see if output was generated:
local output=$(eval "$2")
if [[ ! -z "$output" ]]
then ...
process_output "SwiftLint" "${PODS_ROOT}/SwiftLint/swiftlint lint --quiet"
Status code
Simply checking the status code after executing the command should suffice:
eval "$2"
local return_value=$?
process_return_code "MyCustomRubyScript" "ruby Scripts/myScript.rb MyProject.xcodeproj"
Final tweaks
Colors and timing
Let's improve information printed by the script by adding timing and coloring to our process functions:
function process() {
echo "\n${INFO_START}# Running $1 #${COLOR_END}"
local initial_git_diff=`git diff --no-color`
local start=`date +%s`
eval "$2"
if [ "$FAIL_ON_ERRORS" = "true" ] && [[ "$initial_git_diff" != `git diff --no-color` ]]
then
echo "${ERROR_START}$1 generates git changes, run './Scripts/process.sh' and review the changes${COLOR_END}"
final_status=1
fi
local end=`date +%s`
echo Execution time was `expr $end - $start` seconds.
}
Final status check
if [[ $final_status -gt 0 ]]
then
echo "\n${ERROR_START}Changes required. Run './Scripts/process.sh' and review the changes${COLOR_END}"
fi
exit $final_status
Connecting to pre-commit and CI
In our pre-commit and CI scripts we simply run it in failure mode:
./Scripts/process.sh --fail-on-errors
failed=$?
exit $failed
Summary
Final script you can use for your workflow
We have been using this approach for a few years now and it has saved us thousands of work-hours
During normal development, we don't run those processing tools, when we change code that requires them we simply run it from the command line (or desktop shortcut if you are into that).
We can sleep peacefully because when we try to commit code our pre-commit hook runs it for us and lets us know if there were any changes that we need to review before proceeding. Even if someone messes up their pre-commit hook setup there is always CI ensuring they fix it.
