Over 10 years we help companies reach their financial and branding goals. Engitech is a values-driven technology agency dedicated.

Gallery

Contacts

411 University St, Seattle, USA

engitech@oceanthemes.net

+1 -800-456-478-23

English Technology

Don’t use /dev/ttyS0!

Last summer I started working on an Open Source Robotic Lawn Mower project. In my prototype, I connected the Ardusimple RTK GPS board to the first hardware serial port of the Raspberry Pi 4 and everything seemed to work fine until it didn’t.

I noticed that the GPS suddenly seemed to send invalid NMEA sentences. This caused the robot to lose its position and therefore it just stopped.

First Thought: Load

Since the robot worked for some time and then out of nowhere just stopped, I thought that there might be a performance problem with my software which could cause the receiving errors. Therefore I checked the CPU load and indeed one core was running at 100% load at the time. I then disabled lots of modules until I was basically just streaming the GPS data to ROS. After some driving around, the errors still persisted. Then I thought that the baud rate might just be too high. I reduced it to 9600 baud and the issue still persisted. So load was not the issue here.

Second Thought: Heat

I noticed that it took longer for the problems to start after decreasing the CPU load. Therefore I concluded that the heat generated by the CPU might have to do something with the problem. In order to test this hypothesis, I used a hair dryer to heat the Raspberry Pi and the issues started immediately! Cooling the Pi again with a fan caused the issues to stop. This was strange, but repeatable. Since nothing but the temperature changed, I concluded that heat indeed is the issue! The temporary fix was to add a cooling fan to the Raspberry Pi, but the issue haunted me further. I really don’t like temporary fixes and I wanted to understand this.

The Explanation

Today I had the time to understand the issue here. It turns out that the Raspberry Pi has two different hardware serial implementations. The miniUART and the PL011. By default, if you just enable the serial port of the Raspberry Pi, the miniUART is used for serial communication on the GPIO header and the PL011 is used for Bluetooth. I now found out that the miniUART is lacking some features and is bound to the CPU core frequency. Suddenly, the issue made sense: as soon as the CPU gets throttled, the miniUART’s clock is reduced as well and the timing is off. I could not believe that the timing difference here was so significant. At 9600 baud, a little CPU throttling should not break the serial port.

The Proof

In order to test my theory, I’m programming a Raspberry Pi Pico to send serial data to a Raspberry Pi 4. The Raspberry Pi 4 then echos the data back to the Pico. If the Pico does not get a response in time or gets a different response, it blinks an LED. We can then hook up an oscilloscope to check the timing at different CPU temperatures.

The Hardware Connections

The hardware setup is very simple: Just connect the serial port (TXD to RXD and vice versa). Additionally, we add an LED to PICO’s GPIO 15. Connect the 5V header from the Raspberry Pi 4 to the Pico and everything should be running as expected.

Hardware Connections
Software: The Raspberry Pi 4 part

The software on the Raspberry Pi 4 is this simple python script:

#!/usr/bin/env python

import sys
import serial

ser = serial.Serial(sys.argv[1], 9600)
while True:
        data = ser.read()
        ser.write(data)

The code just opens a serial connection with 9600 baud, waits for data and returns it back to the serial port.

Software: Pico Firmware

The software on the Pico is very simple as well. We first open the serial port with 9600 baud and set the GPIO 15 as output. In the loop function we transmit a byte and wait for the reply with a timeout of 100 ms. If the result matches, we do nothing. If the result differs or if there was a timeout, we flash the LED for 100 ms.

void setup() {
  Serial1.begin(9600);
  pinMode(15, OUTPUT);
}

byte b = 0;
void loop() {
  digitalWrite(15, LOW);

  // Clear the serial read buffer
  while (Serial1.available())
    Serial1.read();

  // Write one byte
  b++;
  Serial1.write(b);
  unsigned long start = millis();
  while (!Serial1.available() && millis() - start < 100);
  bool error = false;
  if (Serial1.available()) {
    byte r = Serial1.read();
    error = r != b;
  } else {
    error = true;
  }
  if (error) {
    digitalWrite(15, HIGH);
  }
  delay(100);
}
Results

When using /dev/ttyAMA0 we observe that the LED stays off constantly. This indicates that all data transmission is correct. Even when heating the Raspberry Pi to throttling temperature using a hair dryer (in order not to change the CPU load), there is no change visible. The serial waveform measured by the oscilloscope is very stable as well. There is no observable change in timing between a throttled CPU and the normal operation mode. Here is a screenshot of my oscilloscope when using /dev/ttyAMA0:

As you can see, for this test I’m transmitting 0b01010101 since it shows an alternating pattern and therefore timing issues tend to be visible quite easily. As expected, every transmitted bit has a length of 104us.

Let’s now compare this to the /dev/ttyS0 serial interface (a.k.a. miniUART). I have set the previous waveform as reference and changed the Raspberry Pi 4 to use /dev/ttyS0 instead of /dev/AMA0. Everything else is the same!

With a cool CPU, the timing looks as expected:

As you can see, the timing resembles the one in the reference. But as soon as I heat up the CPU using a hair dryer until it throttles (>80 degrees), the waveform looks as follows:

You can clearly see that as soon as the CPU throttles, the serial baud rate drops as well. At first I thought that the problem might only occur on high baud rates, but as you can see it is significant and present at all baud rates. So if you want a reliable communication even at high temperatures don’t use /dev/ttyS0!

Author

Clemens

Entrepreneur, Loves Software Engineering, Hardware Design, Robots and Open Source.

Leave a comment

Your email address will not be published.