Friday, November 27, 2020

nerfnet: Wireless Networking over nRF24L01 2.4GHz Radios

I recently picked up a set of nRF 2.4GHz radio transceivers. These are low-cost radios with a SPI interface that allow exchanging 32 byte packets across a radio link that can run at up to 2MBit on-air data rates. They are popular among hobbyists who want to introduce wireless to their Arduino-flavored projects. I was able to buy ten of these radios with trace antennas for just $11 and three more with SMA-connected antennas for $18.

NRF24L01 Radios

My first inclination is to try something a bit more extreme with this hardware. There is a GitHub project named RF24Audio that allows transmitting audio data over these radios. I wondered if video could be possible and started brainstorming about how a video transport over this link would look. The further I got into the specifics of streaming video the more convinced I was that an abstract link that could carry any form of data would be more fun.

This led me to build nerfnet: a simple application that allows sending network frames over NRF24L01 radios. This is implemented by exploiting the TUN/TAP virtual network device API under Linux on a Raspberry Pi. The code is available on GitHub for you to review and use.

I was able to demonstrate nearly 90kBit throughput as measured by iperf. I suspect this is the first time that iperf has been used to characterize a link composed of these radios.

andrew@andrew-pi:~/Projects/nerfnet $ iperf -c 192.168.10.2
------------------------------------------------------------
Client connecting to 192.168.10.2, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.1 port 34490 connected with 192.168.10.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec   110 KBytes  89.4 Kbits/sec

Continue reading or watch the video to learn more about how I pulled this off.

Virtual Networking

In order to create a network link over these radios I needed a way to create a virtual network device under Linux. This is done using the popular TUN/TAP API. The API is incredibly simple in that you open a device file and perform a couple of ioctl calls to configure a new tunnel interface. Here is the code to create the virtual network device:
// Opens the tunnel interface to listen on. Always returns a valid file
// descriptor or quits and logs the error.
int OpenTunnel(const std::string_view& device_name) {
  int fd = open("/dev/net/tun", O_RDWR);
  CHECK(fd >= 0, "Failed to open tunnel file: %s (%d)", strerror(errno), errno);

  struct ifreq ifr = {};
  ifr.ifr_flags = IFF_TUN | IFF_NO_PI;
  strncpy(ifr.ifr_name, std::string(device_name).c_str(), IFNAMSIZ);

  int status = ioctl(fd, TUNSETIFF, &ifr);
  CHECK(status >= 0, "Failed to set tunnel interface: %s (%d)",
      strerror(errno), errno);
  return fd;
}
I decided to also bring the interface up programmatically so that the user would not need to bring the link up after each use of nerfnet.
// Sets flags for a given interface. Quits and logs the error on failure.
void SetInterfaceFlags(const std::string_view& device_name, int flags) {
  int fd = socket(AF_INET, SOCK_DGRAM, 0);
  CHECK(fd >= 0, "Failed to open socket: %s (%d)", strerror(errno), errno);

  struct ifreq ifr = {};
  ifr.ifr_flags = flags;
  strncpy(ifr.ifr_name, std::string(device_name).c_str(), IFNAMSIZ);
  int status = ioctl(fd, SIOCSIFFLAGS, &ifr);
  CHECK(status >= 0, "Failed to set tunnel interface: %s (%d)",
      strerror(errno), errno);
  close(fd);
}
Putting this all together, just a few lines are required to create the interface.
  // Setup tunnel.
  int tunnel_fd = OpenTunnel(interface_name_arg.getValue());
  LOGI("tunnel '%s' opened", interface_name_arg.getValue().c_str());
  SetInterfaceFlags(interface_name_arg.getValue(), IFF_UP);
  LOGI("tunnel '%s' up", interface_name_arg.getValue().c_str());
This approach is the same used by popular VPN software such as OpenVPN. They create an interface called tun0 and use that to establish a connection into a virtual network.

The next step was to read frames from the virtual network to send across the nRF24L01 radios. This is done in a background thread. Each network frame is pushed into a queue. When the radios are ready to exchange frames, they are popped from this queue.
void RadioInterface::TunnelThread() {
  // The maximum number of network frames to buffer here.
  constexpr size_t kMaxBufferedFrames = 3;

  running_ = true;
  uint8_t buffer[3200];
  while (running_) {
    int bytes_read = read(tunnel_fd_, buffer, sizeof(buffer));
    if (bytes_read < 0) {
      LOGE("Failed to read: %s (%d)", strerror(errno), errno);
      continue;
    }

    {
      std::lock_guard<std::mutex> lock(read_buffer_mutex_);
      read_buffer_.emplace_back(&buffer[0], &buffer[bytes_read]);
    }

    while (GetReadBufferSize() > kMaxBufferedFrames && running_) {
      SleepUs(10000);
    }
  }
}
With virtual networking sorted out, it was time to move onto hardware.

Hardware

Before moving into the details behind the protocol used to exchange network frames, let's first take a look at the hardware involved. Each node in this network consists of a Raspberry Pi and nRF24L01 radio.

The RF24 library documentation has some notes about hardware connections and other aspects of the driver library. Here is a table of the connections to make between the radio and Raspberry Pi.

Raspberry Pi/nRF24L01 Wiring
It is important to keep the wires short. The SPI interface between the radio and Raspberry Pi is clocked at 10MHz. If the wires between the Raspberry Pi and radio are long enough or make poor contact, the link will be prone to bit flips and corruption.

Once the connections have been made, the Raspberry Pi computers will be ready to communicate. The nerfnet binary will log errors when there are issues interacting with the radios over SPI. This can be used to identify issues with the hardware connections.

Protocol

The nRF24L01 radios exchange 32-byte frames. In order to communicate over this transport, a simple protocol is used. This protocol supports two commands: Ping and NetworkTunnelTxRx.

This protocol is built using Google's Protocol Buffers. Protocol Buffers utilize an IDL (Interface Definition Language) to define the data format for messages. A code generator then produces code to allow assembling and encoding/decoding these messages into a wire format that can easily be sent across networks. This saves much of the hassle associated with designing networking protocols by using a well-tested, existing solution.

The Ping command is intended for testing a link. It allows round-tripping an integer value to ensure that the transmit and receive functions are working for both ends of the radio link. 
  // A simple ping request.
  message Ping {
    // A value to round-trip from secondary to primary.
    optional uint32 value = 1;
  }
  
  // The response to a ping request.
  message Ping {
    // The value round-tripped back from secondary to primary.
    optional uint32 value = 1;
  }
The NetworkTunnelTxRx command allows exchanging partial network frames. It encodes a bytes field for the payload and how many bytes are remaining in the frame. Once the remaining byte count reaches zero, nerfnet writes the frame into the virtual network tunnel.
  // A TxRx request for the network tunnel.
  message NetworkTunnelTxRx {
    // The bytes to send to the secondary radio.
    optional bytes payload = 1;

    // The number of remaining bytes for this packet. Once zero, write out
    // to the tunnel interface.
    optional uint32 remaining_bytes = 2;
  }
  
  // A TxRx response for the network tunnel.
  message NetworkTunnelTxRx {
    // The bytes to send back to the primary radio.
    optional bytes payload = 1;

    // The number of remaining bytes for this packet. Once zero, write out
    // to the tunnel interface.
    optional uint32 remaining_bytes = 2;
  }
These messages are assembled together with a oneof. The secondary radio responds to requests with responses and the primary radio implements a timeout for receiving a response.
  oneof request {
    Ping ping = 1;
    NetworkTunnelTxRx network_tunnel_txrx = 2;
  }
  
  oneof response {
    Ping ping = 1;
    NetworkTunnelTxRx network_tunnel_txrx = 2;
  }
Here is the code for the secondary radio to handle requests from the primary radio.
void SecondaryRadioInterface::HandleRequest(const Request& request) {
  switch (request.request_case()) {
    case Request::kPing:
      HandlePing(request.ping());
      break;
    case Request::kNetworkTunnelTxrx:
      HandleNetworkTunnelTxRx(request.network_tunnel_txrx());
      break;
    default:
      LOGE("Received unknown request");
      break;
  }
}
The protocol is quite simple. It could likely be improved to handle breaks in connectivity but proves the concept well for this hobby project.

Usage

The nerfnet binary is simple to use. As mentioned above, nerfnet relies on a concept of primary and secondary radios. It is arbitrary which radio performs which function, so just pick one. Here are the commands needed to start the tool and assign IP addresses to the links.
# Primary radio.
sudo ./nerfnet/net/nerfnet --primary
sudo ip addr add 192.168.10.1/24 dev nerf0
# Secondary radio.
sudo ./nerfnet/net/nerfnet --secondary
sudo ip addr add 192.168.10.2/24 dev nerf0

Performance Benchmarking: iperf, ping

Once the link is up, it behaves like any other computer network. This is nice because you can use any existing network programs. This includes performance benchmarking applications such as iperf. Here is a quick example of running iperf a couple of times to characterize performance.

Here is performance using the default 5000us RF delay.
andrew@andrew-sensor-pi:~/Projects/nerfnet $ iperf -c 192.168.10.1
------------------------------------------------------------
Client connecting to 192.168.10.1, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.2 port 48348 connected with 192.168.10.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec  62.2 KBytes  50.5 Kbits/sec
Here is performance using a 1000us RF delay.
andrew@andrew-sensor-pi:~/Projects/nerfnet $ iperf -c 192.168.10.1
------------------------------------------------------------
Client connecting to 192.168.10.1, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.2 port 48350 connected with 192.168.10.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec   110 KBytes  89.6 Kbits/sec
The nerfnet tool has a command line argument `--rf_delay_us` that determines how much idle time to leave between transmit/receive operations. This can be tuned to trade stability for throughput. I have found that 1000us works well, but could go as low as 500us before packets begin to drop.

The latency is high, I suspect due to artificial delays introduced by RF24 driver library used here. It is functional though.
andrew@andrew-sensor-pi:~/Projects/nerfnet $ ping 192.168.10.1
PING 192.168.10.1 (192.168.10.1) 56(84) bytes of data.
64 bytes from 192.168.10.1: icmp_seq=1 ttl=64 time=40.9 ms
64 bytes from 192.168.10.1: icmp_seq=2 ttl=64 time=41.6 ms
64 bytes from 192.168.10.1: icmp_seq=3 ttl=64 time=41.10 ms
64 bytes from 192.168.10.1: icmp_seq=4 ttl=64 time=38.3 ms
64 bytes from 192.168.10.1: icmp_seq=5 ttl=64 time=39.9 ms
The `--rf_delay_us` flag would likely not be required with a higher quality SPI driver for the NRF24L01 device. More on that in the closing remarks.

Use Case: SSH

Once nerfnet has been configured to run, any networking application can be used. This includes common tools such as SSH. Here is a quick video of me connecting to a system over this NRF-based network link. I used htop to view system performance and tree to list the contents of my home directory.



Performance over this link is definitely slow, but not terrible. Applications based on curses, a text-mode windowing system, tend to run slower. This is shown by the performance of loading the htop system monitor. Performance is comparable to using SSH over a dial-up internet connection.

Use Case: Streaming Audio

Another interesting use case is trying to stream multimedia over the network. Since we have a full network connection, JPEGs, MP3s and even MP4 movie files could be transferred. This is all pretty boring through, because on a long-enough time scale, any sufficiently large file can be transferred.

More interesting would be to try streaming compressed audio real-time to have it playback with the popular VLC application. Here is how the server is started:
cvlc -vvv --no-sout-video \
    --sout '#transcode{acodec=opus,ab=16,channels=1,samplerate=32000}:rtp{dst=localhost,port=9000,sdp=rtsp://:8080/test.sdp}' \
    --sout-rtp-caching=5000 \
    Brenticus\ -\ Progressive\ House.mp3
While the server is running on one of the Raspberry Pi's, I setup a client on the other. The client is a simpler configuration. The only tuning I performed here was to increase the caching buffer to 2 seconds to avoid any dropouts due to swings in latency caused by other users of the link.
cvlc rtsp://192.168.10.1:8080/test.sdp \
    --aout alsa --alsa-audio-device plughw:1,0 \
    --network-caching=2000
I did some testing with various codecs and transports to see what would give the best performance over this limited bandwidth link. The first attempt I made was AAC over an HTTP transport at 32kbps. This was far too resource intensive for the link. I lowered the bitrate down to 8kbps before the audio was not experiencing dropouts. Clearly the HTTP transport has too much overhead.

The next transport attempt was AAC over RTSP. This worked far better. The primary radio which hosts this server was emitting packets at a high rate with no return packets. I was able to run the server at 24kbps AAC. This was a big improvement but audio quality left something to be desired.

The next step was to try another codec. I decided to give Opus a try which is specifically designed for low-bandwidth links. This worked out exceptionally well. At a low bitrate of just 16kbps, the audio quality exceeds that of the 24kbps AAC stream. I decided this was good enough.

The YouTube video for this project above has footage showing this streaming audio setup in action. Pretty cool.

Closing Remarks

This has been a fun project to put together. This relatively crude solution only took a couple of days to put together. It could be vastly improved. One obvious issue is that this is not encrypted. It will also handle breaks in connectivity poorly. This could be improved by adding sequence numbers to the protocol to detect breaks in the stream.

One potential next step would be to write a driver for the nRF24L01 radio from scratch. The RF24 library is nice in that you can get started easily, but leaves a bit to be desired in the area of error propagation/handling. I suspect performance could be improved with a better driver.

I hope you enjoyed the video and blog. If you want to see more from me, I encourage you to subscribe to my YouTube channel.

Thanks for stopping by!

1 comment :