Occasionally it may be useful to visualize a continuous stream of data arriving through a serial connection as a web page in a browser. But how do you go about it? May we suggest a bit of scripting?


Basically, the problem boils down to periodically creating a webpage containing the latest serial data. A web browser then can display the page, be it on the same computer or over a network. Therefore, what is needed is a small program that continuously converts the serial data into for instance HTML or PHP files. Aha! Isn’t PHP a programming language for Internet related applications? Yes, it is. So, can we use PHP to get the job done? Yes, we can. But there are other ways too, as we will see.

Use automatic page refresh

It all starts by remembering that HTML allows the use of a meta tag that instructs the browser to reload a page periodically:
 
<meta http-equiv="refresh" content="10">
 
This tag tells the browser to reload the page containing the tag every ten seconds. (In case your browser does not support this tag, it can be replaced by a piece of JavaScript. See the free download below this article. If we create a web page that includes this tag and point the browser to it, the latter will keep reloading it every ten seconds (the timeout can, of course, be another value). If we rewrite the page every ten seconds and put fresh serial data into it, then the browser will show it too.

Split the task in two

It is also possible to put the refresh tag in a PHP file instead of an HTML file and the browser will do the same thing. The PHP file then could also contain a script to read the data from the serial port. This is the point where things become complicated because PHP doesn’t natively support serial ports. And even if it did, it would mean that every time the browser requests the (latest version of the) page, the script would have to open the serial port, get some data and close the port again. Data received outside this window is lost. Furthermore, some Arduino-like systems may reset when the serial port is opened, making the setup useless. A solution to this is to split the process in two subprocesses:

- Process 1: A script to continuously read the serial port and update a data file that is imported by the PHP webpage, see Listing 1.

Listing 1: A PHP script that reads data from the serial port and writes it to a file named "data.txt".  
<?php

// Linux $comPort = "/dev/ttyACM0";
$comPort = "COM15";

include "php_serial.class2.php";
$serial = new phpSerial;
$serial->deviceSet($comPort);

// On Windows (10 only?) all mode settings must be done in one go.
$cmd = "mode " . $comPort . " baud=115200 parity=n data=8 stop=1 to=off xon=off";
$serial->_exec($cmd);
$serial->deviceOpen();

echo "Waiting for data...\n";
sleep(2); // Wait for Arduino to finish booting.
$serial->serialflush();

while(1)
{
  $read = $serial->readPort();

  if (strlen($read)!=0)
  {
    $fp = fopen("data.txt","w");
    if ($fp!=false)
    {
      fwrite($fp,trim($read));
      fclose($fp);
    }
  }
}

?>

- Process 2: A browser that periodically reloads the PHP webpage so that it can refresh the data, see Listing 2 and Figure 1.
 
Figure 1: The dynamic PHP file generated by the PHP
script is served by a WAMP web server to our browser
(see its address bar). It presents the comma-separated
values read from the serial port data in a basic table.
Values below 500 are printed in red, the others are green.
The title of the Command Prompt window shows the
command to execute the script.

Listing 2: This PHP webpage formats the contents of a file named "data.txt" as a table.
<?php

$page_title = "Arduino on PHP";

// Start of HTML page.
echo "<!DOCTYPE HTML PUBLIC '-//W3C//DTD HTML 4.01//EN' 'http://www.w3.org/TR/html4/strict.dtd'>";
echo "<html>"; // Page begin.
echo "<head><title>",$page_title,"</title>"; // Head begin.
echo "<meta http-equiv='refresh' content='1'>";
echo "<meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>";
echo "<link rel='shortcut icon' href='favicon.ico' />";
echo "<link rel='icon' type='image/x-icon' href='favicon.ico' />";
echo "<link rel='icon' type='image/png' href='favicon.png' />";
echo "</head>"; // Head end.
echo "<body><center>"; // Body begin.

echo "<p>",$page_title,"</p>"; // Page title.

// Create a table from data file.
$handle = fopen("data.txt","r");
if ($handle!=NULL) 
{
  // Read one line from the file, then close it.
  $data = fgets($handle);
  fclose($handle);
  
  // Synchronise to the data.
  if ($data[0]=='$')
  {
    // Remove whitespace.
    str_replace(' ','',$data);
    // Split data in fields separated by ','.
    // Expected format: "$,id1,value1,id2,value2,CRLF"
    list($startchar,$id1,$value1,$id2,$value2,$newline) = explode(",",$data);
    // Create array from list.
    $numbers = array($id1=>$value1,$id2=>$value2);
    // Sort array in ascending key order.
    ksort($numbers);
    
    // Table begin.
    echo "<table border='1' border-spacing='5' style='text-align:center;'>";
    echo "<tr><th>ID</th><th>Value</th></tr>";
    foreach ($numbers as $x => $x_value) 
    {
      echo "<tr>"; // Table row begin.
      echo "<td>", $x, "</td>"; // Table column 1.
      echo "<td>"; // Table column 2 begin.
      if ($x_value>=500) echo "<font color='green'>";
      else echo "<font color='red'>";
      echo $x_value;
      echo "</font></td>"; // Table column 2 end.
      echo "</tr>"; // Table row end.
    }
    // Table end.
    echo "</table>";
  }
}
echo "</body>"; // Body end.
echo "</html>"; // Page end.
?>
 
 

 
Figure 2: This time a Python script produces a dynamic HTML file (as can be seen from the browser’s address bar).

Oops, web server needed…

Doing this solves the serial port opening and closing problem and the data loss resulting from it, but it requires a script to run in the background. If this is a PHP script, then the computer must be able to execute PHP scripts. Also, you need a web server to feed the PHP webpage to a browser. If not, the browser will simply show the PHP code making up the page instead of the page itself. The traditional way of getting this done is by installing a so-called “AMP” or “WAMP” package. AMP stands for Apache-MySQL-PHP, the ‘W’ is for Windows, and together they form an all-singing, all-dancing web server.

Don’t use PHP…

We have tried this method and managed to get it to work, but not without problems. Besides difficulties in setting up the web server, the main problem we encountered was getting PHP to reliably open a serial port to receive the data. Searching the Internet, there seems to be only one PHP library for serial communications, PHP Serial. All others appear to have been derived from it. As the author mentions on the GitHub page “Windows: it seems to be working for some people, not working for some others.” We clearly were in the second group. To get serial communications to work with PHP, we first had to open and immediately close the port with a serial terminal program (Tera Term for instance); impossible to get it going otherwise. We therefore ditched the PHP method and went for Python instead.

Subscribe
Tag alert: Subscribe to the tag python and you will receive an e-mail as soon as a new item about it is published on our website!

… use Python instead

Python 3 with pySerial turned out to work perfectly fine on our Windows 10 test computer so we wrote a script to read data from the serial port and write the webpage containing the data. Now that there is no longer a need for PHP, the Python script might as well produce a plain HTML file Listing 3. 

Listing 3: A Python script that reads data from the serial port and generates an HTML file for it.
import serial
import time

file_name = "serial.html" # Once created, open this file in a browser.

# Adapt serial port nr. & baud rate to your system.
serial_port = 'COM15'
baudrate = 115200

page_title = "Arduino on Python";

def write_page(data_list):
    fo = open(file_name,"w+")
    # Start of HTML page.
    fo.write("<!DOCTYPE HTML PUBLIC '-//W3C//DTD HTML 4.01//EN' 'http://www.w3.org/TR/html4/strict.dtd'>")
    fo.write("<html><head><title>"+page_title+"</title>") # Page & Head begin.
    fo.write("<meta http-equiv='refresh' content='1'>")
    fo.write("<meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>")
    fo.write("<link rel='shortcut icon' href='favicon.ico' />")
    fo.write("<link rel='icon' type='image/x-icon' href='favicon.ico' />")
    fo.write("<link rel='icon' type='image/png' href='favicon.png' />")
    fo.write("</head><body><center><p>"+page_title+"</p>") # Head end, body begin.

    # Table begin.
    fo.write("<table border='1' border-spacing='5' style='text-align:center;'>")
    fo.write("<tr><th>ID</th><th>Value</th></tr>")
    for i in range(0,len(data_list),2):
        fo.write("<tr>") # Table row begin.
        fo.write("<td>"+data_list[i]+"</td>") # Table column 1.
        fo.write("<td>") # Table column 2 begin.
        fo.write("<font color='")
        # Values >= 500 will be printed in green, smaller values will be red.
        if (int(data_list[i+1])>=500): fo.write("green")
        else: fo.write("red")
        fo.write("'>")
        fo.write(data_list[i+1])
        fo.write("</font></td>") # Table column 2 end.
        fo.write("</tr>") # Table row end.
    fo.write("</table>") # Table end.
    fo.write("</body>") # Body end.
    fo.write("</html>") # Page end.
    # Done, close file.
    fo.close()

s = serial.Serial(serial_port,baudrate) # Open serial port.
s.dtr = 0 # Reset Arduino.
s.dtr = 1
print("Waiting for data...");
time.sleep(2) # Wait for Arduino to finish booting.
s.reset_input_buffer() # Delete any stale data.

while 1:
    data_str = s.readline().decode() # Read data & convert bytes to string type.
    # Clean up input data.
    # Expected format: "$,id1,value1,id2,value2,...,CRLF"
    data_str = data_str.replace(' ','') # Remove whitespace.
    data_str = data_str.replace('\r','') # Remove return.
    data_str = data_str.replace('\n','') # Remove new line.
    data_str += '123,65,1,999,cpv,236' # Add some more data
    print(data_str)
    # Split data in fields separated by ','.
    data_list = data_str.split(",")
    del data_list[0] # Remove '$'
    # Write HTML page.
    write_page(data_list)

All the data formatting done by the PHP web page can be done directly in Python. Everything is now in one place. The web page may still be served up by a web server (pun intended), but a browser can also display and refresh the page without one. Therefore, there is no longer a need for a (W)AMP package either, making everything much simpler.
Figure 3: This Arduino sketch generates a serial data stream that
can be used for script development, debugging, and testing.

Sounding off

In this article we presented a method of displaying serial data in a web browser. The method is neither new, exclusive, or "the best". If you know of another way — simpler, more elegant, whatever — please share it with us using the Add Comment button below. And, of course, you don’t need Python to write the script, feel free to use any programming language capable of doing serial communications and writing files. The advantage of using Python with pySerial is that it will run on Windows, macOS, and Linux machines (and more).
The code developed for this article in the shape of PHP and Python scripts, and an Arduino sketch can be downloaded below.
(170111)
Do you want to read more ElektorLabs articles? Become an Elektor member now!