One major difference between these CPUs is that the F3 has a hardware floating point unit, while the F1 must emulate its floating point support using some very large software routines. Floating point support is used in the IMU (which is responsible for estimating the craft’s attitude) and also as part of some PID controllers such as the new Harakiri controller. It’s not heavily used in any other parts of the code.

But how much time does the Naze32 spend doing floating point operations anyway? What is the total possible speedup available from just adding a hardware floating point unit? Could I speed my Naze32 up some other way?

In order to find out, I built a Sampling Profiler feature for Cleanflight. A sampling profiler works by periodically interrupting the program’s execution to check what code is currently running. The profiler makes 1000 of these checks per second and sends the results out to a serial port to be logged.

The checks can be used to measure what percentage of the time is spent on each line of code in the program. If one piece of code accounts for 30% of the execution time of the program, then we should expect 30% of our random samples to end up seeing that line of code being run. In this way we can construct an accurate picture of the program with a nice constant overhead.

If the profiler slows down the CPU too much, we only have to reduce our sampling rate and sample for longer. Our results will be the same, and CPU sampling overhead will decrease.

Limits

This profiler is somewhat limited because it can only see the top function that was executing when it takes a sample. It doesn’t currently have the ability to look up the stack and see which functions were responsible for calling the function that was interrupted.

This can make it harder to work out which parts of the code are the culprits for slowdowns, because some routines are shared by many parts of the program. It also means that functions which mostly just call other functions to do their work for them will have very low execution times as measured by the profiler.

The very highest priority interrupt handlers (I2C EV and ER interrupt handlers) are not measured by the profiler. This is because the profiler is not high priority enough to interrupt them. I configured it this way because of the I2C documentation’s warning that the interrupt handlers should not themselves be interrupted (I assume they’re too timing sensitive).

Results

You can see the raw output from the profile log decoder here. The results are pretty interesting. I was running a Naze32 Full with a PPM receiver, PID controller 0, looptime set to 0 and all other features disabled, so pretty much a barebones system.

You can see that the majority of runtime is spent on floating point routines (48%) plus trigonometry routines (5%), which are the likely callers of most of the floating point routines. This is great news for the SPRacingF3 because these are precisely the routines that will be sped up by its hardware floating point unit!

Of the remaining time, 36% is spent just waiting to receive data from the gyroscope over the I2C bus. I haven’t examined this code before, but from a quick glance it appears that the CPU performs a busy-wait while it waits for a response to be received from the gyroscope. If this task could be handed off to the DMA controller, which would perform the read in the background, it could potentially free up to 36% of the CPU time for other tasks.

It’s clear that it’s not worth speeding up any other part of the code, because the remainder accounts for such a small percentage of the total runtime. This information is a win for developers because we don’t need to waste our time on something that isn’t going to make a difference to looptimes!

Speeding up the Naze32

So, the SPRacingF3 is likely faster than the Naze32 because its floating point operations are faster. But can I squeeze more speed out the Naze32? There are two main approaches here.

One option is to speed up the floating point routines themselves. One way of doing this is to use routines which are slightly less precise than the default implementation, which can bring large speedups. The default implementation is extremely pedantic about being precise and handling error conditions in order to conform to IEEE floating point standards, but in a noisy system like a quadcopter that precision isn’t noticeable. To this end, some developers have been discussing Taylor approximations of our trigonometry routines over on the Cleanflight issue tracker.

The other option, which is even simpler, is to avoid the calls that require floating point in the first place. Since I’m flying PID controller 0, the main place in the code that is actually using floating point is the IMU’s attitude estimator. This is responsible for figuring out the roll/pitch angles and heading of the quadcopter. That information is required for Angle and Horizon flight modes, as well as the flight modes based on sensors like the magnetometer and barometer. But good old Acro flight mode doesn’t use the IMU at all, because it only uses rotation rates read directly from the gyroscope.

Luckily, I happen to be an acro-only pilot. So in my case, the only thing the IMU is being used for is the pretty display of the 3D model in the Configurator’s welcome page. Let’s see what happens when I turn off my accelerometer, which causes the IMU to never run:

Entering CLI Mode, type 'exit' to return, or 'help'

# set acc_hardware
acc_hardware = 0

# set acc_hardware=1
acc_hardware set to 1
# save

$ blackbox_decode LOG-ACRO.TXT
Decoding log 'LOG-ACRO.TXT' to 'LOG-ACRO.01.csv'...

Log 1 of 1, start 00:18.056, end 00:28.247, duration 00:10.190

Statistics
Looptime            325 avg           10.3 std dev (3.2%)
I frames     978   44.1 bytes avg    43094 bytes total
P frames     978   24.5 bytes avg    23994 bytes total
E frames       2    9.0 bytes avg       18 bytes total
S frames       8    4.0 bytes avg       32 bytes total
Frames      1956   34.3 bytes avg    67088 bytes total
Data rate  191Hz   6696 bytes/s      67000 baud

29325 loop iterations weren't logged because of your blackbox_rate settings (9552ms, 93.75%)

Yep, that’s a looptime of 325 with standard deviation of 10.3 microseconds! It looks like 1/8 logging rate is about as much as the Blackbox can keep up with at 250,000 baud. It’s now a lean, mean, acro machine! Here’s what the distribution of execution time now looks like according to the profiler:

i2cRead didn’t get slower, it probably stayed the same, but now it accounts for a larger portion of the total runtime because the floating point routines have shrunk so much. Now we’re seeing the actual flight code taking up a noticeable portion of the runtime, like mixTable() which is responsible for deciding what strength each motor should be driven at, and the PID controller itself. Here’s the raw profiler report for my acro-only configuration.

Sourcecode

The version of Cleanflight with (extremely experimental, only useful for developers) profiling support can be found here:

https://github.com/sherlockflight/cleanflight-dev/tree/profiler

You must read the documentation first, which is here:

https://github.com/sherlockflight/cleanflight-dev/blob/profiler/docs/development/Profiler.md

The decoder tool which reads profile logs is here (it’s also my first application written in Go, so it’s probably horrible):

https://github.com/sherlockflight/cleanflight-profiler

One way of solving this problem is to use the CPU’s DMA controller to send the write buffer out to the flash in the background. This way the buffer can be slowly written to the flash chip all the time while other tasks are executing, instead of the whole CPU pausing to make one big slow write every now and then. That’s still something I want to implement in the future, but in the meantime I looked into other ways to reduce the variance.

Since I want every Blackbox logging iteration to take a similar amount of time, I need to perform similar tasks in every iteration. So flushing the buffer to the flash in only 1 in 4 iterations is a big no-no. Instead I decided to flush the buffer out to the flash in every iteration, even if the buffer wasn’t full yet. This increases the mean execution time a little bit, because each flush operation itself has some overhead, but it causes a dramatic improvement in the standard deviation. Instead of two widely-spaced peaks in execution time, there is now one large, tightly-grouped peak:

Here’s the new statistics:

Flushing every iteration only causes a modest 9% increase in mean logging overhead, while it reduces the standard deviation by 81%!

This fix will be part of the next release of Cleanflight (the current release as of this writing is 1.9).

One great feature for tuning your craft’s performance is the Blackbox flight log. However, the choice between logging to an OpenLog device or to an onboard flash chip brings with it quite different performance impacts, and you may need to factor this in when you’re choosing your logging device.

Logging to an OpenLog logging device

When logging to an OpenLog, the Blackbox writes the log to the serial port’s transmit buffer in memory, then returns control to the flight loop immediately. It never waits for the OpenLog to catch up, or for the buffer to be ready. Cleanflight’s DMA support slowly streams the transmit buffer to the serial port in the background, allowing the rest of the system to continue execution with little to no overhead.

When logging to the serial port, the extra overhead that Blackbox adds to each flight loop iteration (in microseconds) looks like this:

93 60 62 62 63 62 63 62 62 62 61 63 63 64 66 65 62 65 61 63 62 63 66 69 63 65 61 62 61 62 65 62 90

Every 32 loop iterations Blackbox logs an “intraframe”, which allows Blackbox to resync the log when it encounters missing or damaged log iterations. Because these frames are larger than normal, they take longer to write to the buffer, and so appear as an increase in overhead of ~30μs over the baseline (up to 90μs). So when logging to an OpenLog, Blackbox adds a mean of 64μs of overhead to the looptime, with a tight standard deviation of 5μs. That overhead is illustrated with this density plot:

Logging to onboard flash (Naze32 full)

When the Blackbox is set up to log to an onboard flash chip, it writes the flight log slightly differently. When there is space available in the flash’s write buffer in memory, Blackbox can complete its write in a similar time to writing to the serial port’s transmit buffer, and so it can quickly continue with the rest of the flight loop. But unlike the serial port, the flash chip is not written to in the background by the DMA engine. So when the buffer fills up, the Blackbox must briefly pause to write it through to the flash chip over the SPI bus. The looptime overhead from Blackbox logging (in microseconds) looks like this:

209 56 54 189 56 196 56 55 190 56 55 189 56 55 189 60 55 193 56 55 188 58 55 189 56 55 190 55 190 56 55 188

In the iterations where Blackbox only writes to the write buffer, it finishes its work quickly in about 55μs. But when it must flush the buffer through to the flash chip itself, the overhead is 190μs! The mean Blackbox logging overhead is 105μs, a reasonable 1.6x that of writing to the serial port, but the standard deviation is 65μs, which is 13 times the deviation of logging to the serial port! This is visible as a strong twin-peak pattern in the density plot:

Why looptime deviation matters, and why it doesn’t

The problem with variance in looptime is that there are several algorithms in Cleanflight whose results depend on the delay between flight control loops, but don’t have any built-in correction to deal with looptime variation. With a fairly speedy looptime of 1500μs, a standard deviation of 65μs represents a potential for a 4.3% error in calculations which (depending on PID controller) could translate to noticeable noise in signals sent to motors and flight instability.

However, this variance only comes in to play if you set the “looptime” setting to a time faster than the slowest loop that the flight controller encounters. As long as you set the looptime to a longer value than the slowest possible loop (when the variance in execution time is at its worst), then Cleanflight can make sure that every flight control loop starts with a similar spacing, minimising error.

The worst case results when looptime is set to zero, because Cleanflight cannot add any delay between loops in order to smooth out variations in execution time.

Here’s what the whole-system looptime variation looks like when looptime is set to zero and Blackbox is logging to the serial port. The mean looptime achieved is 1230μs, and the standard deviation is 82μs (6.7%).

Note that this shows the sum of all looptime variation present in Cleanflight, not just the 5μs that’s due to Blackbox logging.

Here’s the system looptime variation (with looptime = 0) when Blackbox is logging to flash:

The mean was 1292μs and the standard deviation 102μs (7.9%).

We can use these graphs to decide on the best looptime to choose in order to minimise looptime variation. In the case of logging to the serial port, it looks like most loops can complete in under 1300μs, so let’s have Cleanflight aim to space its loops at 1300μs intervals by setting looptime to 1300:

The new mean is 1328μs and the standard deviation 68μs (5.1%). There is still a strange tri-peak distribution which would be nice to remove, but the centre peak is now much taller than its surroundings than it was before.

I think the three peaks are likely due to variations in the time taken to read sensors like the gyroscope and accelerometer and update the craft’s attitude estimate. Cleanflight attempts to begin the execution of its imuUpdate() routine at regular intervals, but Blackbox records its timestamp at the end of this routine, which is used to derive the variation shown above.

When logging to flash, our looptime will have to be set slower in order to encompass the secondary peaks in activity that we saw in the 1300-1400 looptime range. So let’s set looptime to 1400μs:

The mean is 1419μs and the standard deviation 56μs (4.0%), about half the standard deviation we had at looptime 0!

In the future

In the future I would like to investigate adding support for DMA to the Blackbox’s flash engine. This would allow the Blackbox to log to flash with a similar delay and variation as writing to the serial port, eliminating the 135μs flash-writing penalty observed above.