Sunday, November 23, 2014

Flir Lepton Thermal Imaging Sensor + Gameduino 2

I recently got my hands on a pair of Flir Lepton thermal imaging sensors and have spent the last week bringing them online in my spare time. These are absolutely incredible devices that I believe will pave the way to consumer devices incorporating thermal imaging cameras. The footprint of the camera module (and optical assembly) is about the size of a dime. The resolution is 80x60 at 14bpp which is remarkable despite sounding low.

Thermal Andrew :]
I have successfully implemented a driver for the Lepton module and displayed frames on an LCD. This is all running on an STM32F4 processor on a Nucleo board. Attached to it is a Gameduino 2 which incorporates the FT800 graphics processor. I have implemented my own colorization and min/max scaling before uploading the frames to the GPU.

Front System Overview
I have used some simple jumper wires to interface with this camera. This setup is running at 21MHz with no issues. I am using a breakout board provided by Pure Engineering. You can pick up one from Tindie if you are interested. The Lepton module can be ripped out the Flir One iPhone accessory for now.

Rear System Overview and Second Camera (future project ;])
I wrote my own driver for the Lepton core and the FT800 graphics processor. Continue reading for more details!

Hardware Overview

As I mentioned, I am using the Flir Lepton sensor and a breakout board designed by Pure Engineering.

Lepton seated in the Pure Engineering breakout board
Breakout Board Rear

Protocol and Driver Implementation

This module incorporates two interfaces. One is i2c and the other is SPI. The i2c interface is used to configure the module in a number of ways. This is something that I have not explored yet. In the default configuration of this camera, it will export frame data over SPI. Each packet is 164 bytes long. The first two bytes are line number, the second two bytes are a CRC and the final 160 bytes are the pixel data for that line. Each pixel is sent as two bytes to accommodate the 14-bit range. Data is sent in big-endian format (high byte first).

VoSPI Packet
Occasionally the camera will send a synchronization packet. In this case, line number is sent as xFxx. The x bits are "don't cares" so this can be tested for by applying a bitmask to the first byte of 0x0F and testing that it is equal to 0x0F.

I defined some symbols to make parsing the protocol easier.


#define LEPTON_WIDTH 80
#define LEPTON_HEIGHT 60
I defined a Lepton_t struct to maintain a reference to its' SPI module and chip select line. I also have a LeptonColor_t struct to store a simple RGB565 color that the FT800 can use.
typedef struct Lepton_t {
    volatile SystemSpiModule_t *Spi;
    volatile SystemGpioModule_t *CsPort;
    volatile uint32_t CsPin;
} Lepton_t;

typedef struct LeptonColor_t {
    struct {
        uint8_t red : 5;
        uint8_t green : 6;
        uint8_t blue : 5;
} LeptonColor_t;
The API for this driver is very simple.
 * Initializes the Lepton module
void LeptonInit(Lepton_t *lepton);

 * Reads a frame into the Lepton_t framebuffer.
void LeptonReadFrame(Lepton_t *lepton);

 * Returns a pixel from the frame buffer at a given x, y
uint16_t LeptonReadPixel(Lepton_t *lepton, uint8_t x, uint8_t y);
Allocating a FrameBuffer as part of the Lepton_t struct was a decision I made to keep the driver self-contained. It is certainly possible to refactor this code and not buffer the entire frame.

The implementation of this driver is simple. There are a few private utility functions to start/end transfers, reset the camera and read various amounts of data.
/* Lepton Private Functions ***************************************************/

inline void LeptonBeginTransfer(Lepton_t *lepton) {
    lepton->CsPort->Output.Port &= ~(1 << lepton->CsPin);

inline void LeptonEndTransfer(Lepton_t *lepton) {
    lepton->CsPort->Output.Port |= (1 << lepton->CsPin);

inline void LeptonReset(Lepton_t *lepton) {
    for(volatile int i = 0; i < 100000; i++);

uint8_t LeptonReadByte(Lepton_t *lepton) {
    lepton->Spi->Data = 0x00;
    return lepton->Spi->Data;

inline bool LeptonReadLine(Lepton_t *lepton, uint8_t line, uint8_t *buffer) {
    bool success = true;

    for(int i = 0; i < LEPTON_PACKET_HEADER_LENGTH; i++) {
        buffer[i] = LeptonReadByte(lepton);

    if((buffer[0] & 0x0F) == 0x0F) {
        success = false;
    } else if(buffer[1] != line) {
        success = false;

    for(int i = 0; i < LEPTON_PACKET_CONTENT_LENGTH; i++) {
        buffer[i] = LeptonReadByte(lepton);

    return success;
These utility functions are assembled to create the Lepton driver that is exposed in Lepton.h.
/* Lepton.h Implementations ***************************************************/

void LeptonInit(Lepton_t *lepton) {
    // Setup the SPI Module
    lepton->Spi->Config.SlaveManageEnable = true;
    lepton->Spi->Config.InternalSelect = true;
    lepton->Spi->Config.DeviceMode = SystemSpiDeviceMode_Master;
    lepton->Spi->Config.Prescaler = SystemSpiPrescaler_2;
    lepton->Spi->Config.ClockPhase = SystemSpiClockPhase_Second;
    lepton->Spi->Config.ClockIdle = SystemSpiClockIdle_High;
    lepton->Spi->Config.Enabled = true;


void LeptonReadFrame(Lepton_t *lepton) {
    for(int i = 0; i < LEPTON_HEIGHT; i++) {
        if(!LeptonReadLine(lepton, i,
            &lepton->FrameBuffer[i * LEPTON_PACKET_CONTENT_LENGTH])) {

uint16_t LeptonReadPixel(Lepton_t *lepton, uint8_t x, uint8_t y) {
    uint16_t pixelIndex = (y * LEPTON_PACKET_CONTENT_LENGTH) + (2 * x);

    uint8_t high_byte = lepton->FrameBuffer[pixelIndex];
    uint8_t low_byte = lepton->FrameBuffer[pixelIndex + 1];

    return (high_byte << 8) | low_byte;
This camera uses SPI mode 3 which means that you must sample on the second edge of an idle-high clock. I am clocking my camera at 21MHz which is just a slight overclock.

Generating Pseudocolor Images

In the image below you can see a few fascinating details about boiling water in a kettle. You can see the hot steam that is flowing out of the spout. You can also see that the power cord is quite warm (relatively speaking).

Boiling Kettle
As you can see, the pseudocolor image exposes some interesting information about the scene. Generating a pseudocolor involves selecting colors from a gradient. For example, you could map black to the coolest object in your scene, red to the hottest and ignore the green and blue channels.

Red Black Gradient

There are basically two ways to display thermal image data. The first is as a grayscale image. This has the benefit of introducing no ambiguity into the image that we view. This is quite common in military applications. The second way is to apply a colorization algorithm to generate a pseudocolor image. This uses the bits in our display more cleverly to display a wider range of temperatures.

Consider a grayscale image on an 24-bit RGB display. In this context you would only be able to see 256 levels of temperature. If you apply false-colorization and display the image on a 24-bit RGB display you have the potential to display a much greater number of levels of temperature. As an example, I am using a rainbow to colorize my image which provides 1792 levels.
Black - Red    : 256 Levels - Coldest
Red - Yellow   : 256 Levels
Yellow - Green : 256 Levels
Green - Cyan   : 256 Levels
Cyan - Blue    : 256 Levels
Blue - Violet  : 256 Levels
Violet - White : 256 Levels - Hottest
You can see that the advantage to using a pseudocolor image is very clear for most applications.

Rainbow Gradient
To generate a colorized image, I first find the minimum and maximum values for a given image.

// Compute the min/max values of this frame.
int min = 16384;
int max = 0;

for(int i = 0; i < LEPTON_HEIGHT - 1; i++) {
    for(int j = 0; j < LEPTON_WIDTH; j++) {
        uint16_t value = LeptonReadPixel(&camera, j, i);

        if(value < min) {
            min = value;

        if(value > max) {
            max = value;
I generated an RGB565 lookup table of colors in C#. This table has 320 entries and is loaded into flash memory.
const LeptonColor_t colors[] = {
    { .red = 0, .green = 0, .blue = 0 },
    { .red = 1, .green = 0, .blue = 0 },
    { .red = 2, .green = 0, .blue = 0 },
    { .red = 3, .green = 0, .blue = 0 },
    { .red = 4, .green = 0, .blue = 0 },
    { .red = 5, .green = 0, .blue = 0 },
    { .red = 6, .green = 0, .blue = 0 },
    { .red = 7, .green = 0, .blue = 0 },
    { .red = 8, .green = 0, .blue = 0 },
    { .red = 9, .green = 0, .blue = 0 },

    // ...

    { .red = 31, .green = 50, .blue = 31 },
    { .red = 31, .green = 51, .blue = 31 },
    { .red = 31, .green = 52, .blue = 31 },
    { .red = 31, .green = 53, .blue = 31 },
    { .red = 31, .green = 54, .blue = 31 },
    { .red = 31, .green = 55, .blue = 31 },
    { .red = 31, .green = 56, .blue = 31 },
    { .red = 31, .green = 57, .blue = 31 },
    { .red = 31, .green = 58, .blue = 31 },
    { .red = 31, .green = 59, .blue = 31 },
    { .red = 31, .green = 60, .blue = 31 },
    { .red = 31, .green = 61, .blue = 31 },
    { .red = 31, .green = 62, .blue = 31 },
    { .red = 31, .green = 63, .blue = 31 },
This allows me to lookup the corresponding color for a given normalized temperature value very quickly. I scale the value of each pixel to be within range of the color lookup table.
// Upload the frame to the GPU.
for(int i = 0; i < LEPTON_HEIGHT; i++) {
    for(int j = 0; j < LEPTON_WIDTH; j++) {
        uint16_t value = LeptonReadPixel(&camera, j, i);

        int scaled_value = (320 * (value - min)) / (max - min);
        scaled_value = scaled_value < 0 ? 0 : scaled_value;
        scaled_value = scaled_value > 319 ? 319 : scaled_value;

        LeptonColor_t color = colors[scaled_value];

        // ...
Using this technique I can utilize my LCD to display a wider range of temperature values.
A "hot" apartment building against the cold NYC sky :]

Caveats of Interpolating Pseudocolor Images

If you are planning to use interpolation to scale up the images captured by the Lepton sensor, pay careful attention to how you interpolate the data.

Nearest Neighbour Interpolation

If you take the naive approach and use nearest neighbour to scale up the images, you do not need to be concerned. The result will accurately reflect the captured images in pseudocolor but will be larger in size.

Bilinear/Linear/Sinc Interpolation

If you decide to use a better interpolation algorithm like bilinear, linear or sinc you must generate the final image by interpolating from the 80x60x14bpp sensor data directly. If you generate a pseudocolor image at 80x60 and scale it up, the result will not be accurate.

Imagine that you have two objects next to one another. One object is red and the other is blue. The natural interpolation between these colors is to transition through violet. This is incorrect! If you interpolate the data as temperature values they should interpolate through yellow, green, cyan and finally blue. You are displaying the region between these two objects as hotter than both of them.

Interpolation Error!
This has been a fun experiment and I am looking forward to working on more projects with this sensor.

Flir Lepton + Gameduino + Nucleo running on a portable USB power supply :]
Thanks for reading!


  1. Great work Andrew and thanks for sharing. The Lepton can also be acquired in single quantities here:

  2. Very cool (sic) project! Have you given thought to using both modules for stereo vision? If you used two half-mirrors, could you not use regular cameras overlaying the FOV of each module, and get true interpolation based on the physical geometry of the environment?

    Also, one project that works amazingly well to rescale images is SmillaEnlarger. As it is open source, you could add the routine to your software to avoid the blurriness inherent in most rescaling algorithms. You can find an example pic at my blog.

  3. Thanks for this right up. It was very helpful. I don't have a sensor yet. I am considering using one in an industrial application to monitor the temperature of machinery. Does the 14bit pixel value scale to an actual temperature? I don't need super accurate, +/-5 degrees would be fine. Thanks!

    1. Hey Stevek!

      The 14-bit values can be mapped to temperature but you need to calibrate the scale first. Commercial cameras do this with a mechanical shutter with a temperature sensor attached to it.

      The shutter swings in front of the sensor for a brief moment, the known temperature of the shutter is mapped to one of the 14-bit values and then you can determine the absolute temperature of an object in the scene.

    2. Hey Andrew ,

      Good job man! This page was really helpful! I have been working on the Lepton Camera in my currently University. Right now, I am trying to measure the temperature using the Lepton Image. Could you please give more information about calibrate the scale?

      Thank you!

  4. Hey Andrew!

    Great post! I'm planning to get one and experiment with it. Can you tell me what is the maximum distance this camera module "can see"? Is it possible to recognize a human body shape from aprox. 50m with this?


  5. Hello Andrew, this post is really cool!
    I was particularly fascinated by pseudo color palette generation: I would like to know more about that, so I will be really glad if you could share the code for rgb565 palette generation, or if you could share some useful links.


  6. Very nice. Any chance of putting your entire code up on your GitHub please?

  7. Hi, Thank you for posting.

    I am planning to buy the device (lepton) for my graduate research to detect temperature range between 40℃ to 100℃ of solar panel.
    I am asking if the device can detect temperature range I want, by making some change on the example code? or If you have any advice?

    So, if I change the code and order the camera to detect temperature within that range.

    Any help would be much appreciated.

  8. This comment has been removed by the author.

  9. Hi Ghonaim.a, you could try commenting out the for loop pair below these lines:

    int min = 16384;
    int max = 0;

    Then change these lines too:
    int min = 15284;
    int max = 12820;

    Though these are just a guess, going by the lepton module datasheet. So check their accuracy by pointing the lepton module at a known-temperature object, in the 40-100 degree range and checking the colour you get.

    By the way, the colour to temperature ratio changes depending on the cameras temperature, so try to keep the camera the same temperature when using it.

    I'd also love to see the authors code hosted on github or even attached to this article as .cpp and .h files :)