Skip to content

GPS and time

What time is it? and Where am I? – two seemingly simple questions but just look what happens when you start to explore the depths of the questions.

Mario Corchero is doing a talk on It’s time for datetime at PyCon coming up in May. “We will also speak about different standards of time, time zones, Daylight Saving Times, leap seconds, serialization and datetime arithmetics.”

Working with time is not a trivial challenge. Python includes a native module in the standard library to work with it but datetime keeps being together with unicode a common source of errors. This often leads to the widespread of many other libraries in the attempt of easing the work of working with datetime. Datetime is one of those API that looks easy to use but given the many concepts around time, is it easy to get backfired if the developer has not solid knowledge about the them.

Those are the issues at the bottom of an NTP stratum server based on GPS time. Fortunately most of the shenanigans can be left to the client so the server part can be simplified.

Another case where these issues got interesting is trying to put together a standalone clock with a Picaxe board and a clock chip. Setting the clock was a challenge for human interface ideas. Automatic DST adjustments were another can of worms entirely.

On the ‘where am I’ front is Chris Rizos And Donald Grant with How GPS keeps up with a continent in constant motion. How can you keep GPS and the map copacetic?

The Australian continent, perched on the planet’s fastest moving tectonic plate, is drifting at about seven centimetres a year to the northeast. This is taking features marked on our maps out of line with the global navigation satellite systems (GNSS) such as GPS.

These reference points and their coordinate system of latitudes, longitudes and heights are called a geodetic datum. Every country has its own datum, and the one Australia has used to date is called the Geocentric Datum of Australia 1994, or GDA94.

Satellite positioning systems – such as GPS, the Russian GLONASS, the European Union’s Galileo and China’s BeiDou – give coordinates based on a datum that is not fixed to any continent, but rather the average of all continents. So the coordinates of fixed features on the Earth’s surface, such as the Australian continent, are always changing, like slow-moving ships at sea.

Historically, coordinate differences of a metre or so have not been an issue, because positioning systems have not been accurate enough for users to notice. When GDA94 was first introduced, the GPS locations were only accurate to around 100 metres and sometimes much worse.

But two important things have happened since then. Australia has moved about 1.6 metres northeast, effectively moving the location of mapped features and their associated GDA94 coordinates.

At the same time, positioning technology has evolved considerably.

There are really two issues here. One is a model for the shape of the earth for an accurate location coordinate and the other is a map of features on that earth. Most GPS systems use a fairly simple sea level datum but allow you to choose from among the most common.See Wikipedia on Geodetic Datum. The math can be rather interesting at it is trying to model complex shapes in three dimensions and those shapes have a lot of interesting important features.

Chasing Australia as it moves northeast presents another challenge. Rather than update maps on a regular basis, the GDA94 datum might be updated instead. See Esri Australia Technical Blog Australia is on the move GDA2020.

Fasten your seatbelts!

The persistence of technological creativity explored

What makes Western Culture to nest for the wealth of modern society and the path for getting out of the mire of human misery? Joel Mokyr has some ideas: The persistence of technological creativity and the Great Enrichment: Reflections on the ‘Rise of Europe’ [Editor’s note: This column first appeared as a chapter in the Vox eBook, The Long Economic and Political Shadow of History, Volume 1, available to download here.]

This difficulty was resolved in late medieval and early modern Europe. What emerged and turned out to be of great importance, is that political fragmentation was coupled with an intellectual and cultural unity, a more or less integrated market for ideas, that allowed Europe to benefit from the increasing return associated with intellectual activity. This unity was rooted in Europe’s classical heritage (with the widespread use of Latin as the lingua franca of intellectuals), and the structure of the Christian Church. While, for much of the Middle Ages, the intensity of intellectual activity (in terms of both the number of participants and the intensity of the debates) was light compared to what it was to become after 1500, it was transnational. By 1500 or so, national boundaries mattered little in the thin but lively community of intellectuals in Europe. Many of its leaders moved back and forth within Europe, despite the slow and uncomfortable nature of travel. Two of the most prominent leaders of 16th century humanism, the Valencia-born Juan Luis Vives and Desiderius Erasmus embodied this footlooseness: Vives studied in Paris, lived most of his life in Flanders, but was also a member of Corpus Christi College in Oxford and served for a while as tutor to Henry VIII’s daughter Mary; Erasmus moved back and forth between Leuven, England, and Basel but also spent time in Turin and Venice. In the 17th century such mobility among intellectuals became even more pronounced.

Moreover, through the printing press and the much improved postal system, written knowledge spread faster than ever.

The book might be a slog but maybe worth registering with the site to download

Buckaroo Banzai weird?

It’s cited as one of 366 Weird Movies.

“Would a watermelon in the midst of a chase sequence not be, in its own organic way, emblematic of our entire misunderstood enterprise? At once totally logical and perfectly irrational?”–W.D. Richter, explaining why there is a watermelon inside the Banzai Institute

Any movie that successfully incorporates a band of rock and roll scientists, an invasion by aliens uniformly named “John,” the Eighth Dimension, inexplicable watermelons, and Jeff Goldblum as a New Jersey neurosurgeon who dresses like a cowboy—while working inside the Hollywood system, with a $12 million dollar budget—has worked hard enough to deserve a space on the List of the Best Weird Movies ever made.

By the way, the reason that they put a watermelon in the vice is because a grape wouldn’t have shown up on camera.

IMDB link: The Adventures of Buckaroo Banzai Across the Eighth Dimension (1984)

What’s weird about all that?

Young men from A to Z

TXRed posts From “Animal House” to “Zulu”?.

“So, if I, a female-type person, want to sum up the best and worst of undiluted guy-ness, Animal House and Zulu would be it?”

But as an outsider, I think Dr. McD was right. From everything I’ve seen and heard, and hidden my eyes from, you had to sum up young men on their own, without the leavening (or encouraging) presence of ladies, those two films might make a good start.

vive la difference! no matter how un-PC or out of style it may be people know and some are even self aware.

gpsd and finding the GPS via USB

gpsd has a goal of finding the GPS receiver when it is attached to your system invisibly and providing data via a network port so an application doesn’t have to worry about what kind of receiver, its particular protocol, or how it’s connected. Doing that is a can of worms.

The current most common method for connecting a GPS receiver to a computer is via a serial port that runs through a serial to USB converter. That converter is usually a single chip and it shows up as a USB TTY device. When UDEV sees such a device, it looks in /lib/udev/rules/60-gpsd.rules to see if gpsd should be launched to check it out. That rules file is installed with gpsd. What is interesting is that both the Prolific Technology, Inc. PL2303 and the FTDI 8U232AM / FT232 are “disabled in Debian as it matches too many other devices

That’s why that cable didn’t work to get the serial output from the GPS module. It would work if I was using the USB port on the module itself as that’s U-Blox but doing the serial to USB off the module pins with a Prolific based patch cable didn’t. So I removed the comment indicator at the beginning of the rule and, whatdoyouknow?, it works. On ttyUSB1 no less.

The lesson out of this is that gpsd is most likely to find a GPS receiver on a USB port if you use the converter chip that comes with the receiver. Using a couple of the most common serial to USB converter chips is going to need some rules adjusting.

I bought a couple of simple converters that are intended for microcontroller access. While the ESP8266 boards I have, a NodeMCU and a Wemos D1, both have onboard USB and the handshake lines that can be used to put the ESP8266 into firmware download mode, I still needed the simple converts to examine the debug output from the second, output only, ESP8266 UART. I am also using one of them so I can monitor the GPS receiver output via the RY835AI pins to see what I2C configuration from the ESP8266 is doing. I don’t want to use the USB port on the RY835AI because it would supply power to the module and I want the module to get its power from the microcontroller board.

The next question is why all the fuss about gpsd? The NTP stratum 1 server will be able to do a lot more than just dish out time based on GPS clocks. One option is to serve up the data on a web page. Another option is to stream it out a network port compatible with the gpsd schema. So I need to understand gpsd. Next on that process is going to be some Python gpsd client code experiments. 

NTP via GPS on an ESP8266 using MicroPython: design issues

It’s been done before, lots of times. Sort of. But maybe not just this way. The collection of parts and pieces provides for a broad curriculum of learning possibilities. Here are some of the issues that have come up so far.

SNTP vs NTP. NTP has a long history and it is intended for always on computers where the clock needs tweaking over time to keep it ‘close enough’ without any hiccups (the principle of least surprise for any software using the clock). The simple version doesn’t have the sophisticated algorithms to correct errors gently. It is designed for computers that are turned on and off and it just bangs the clock to the correct time and doesn’t worry about what anybody using the clock might think.

The first issue is about the nature of keeping time. There is general agreement on the length of a second based on the TAI (International Atomic Time).  In the 70’s gravitational time dilation became a concern so that getting clocks all over the world in sync meant making adjustments to a chosen altitude on a chosen geoid. Then there’s the the issue raised by tidal and other forces that influence orbital and rotational periods. Navigators need time to sub-second resolution that accommodates these astronomic variations so there’s a difference between whether adjustments are made to the atomic seconds in sub-second sizes for navigation or in leap-second sizes for most everybody else.

This is the first step in timekeeping: counting seconds and trying to fit the count to the earth’s current rotational period. Next up is when  to start counting. They call this the epoch. NTC uses 1900, POSIX uses 1970, MicroPython on the ESP8266 uses 2000, GPS uses 1980. The reason behind this is about how high a computer can count easily. Converting between these counts is complicated because some use leap seconds and others do not.  After you’ve got the seconds since some known time and date and the leap seconds settled, you can then figure out the current date and time. Of course, at that point, you have to start looking at time zones and daylight savings times rules. 

At least, with a GPS based NTP server, the GPS receiver does the conversion between GPS time and UTC and the difference between UTC and NTP is fixed (2,208,988,800). NTP is a bit more concerned about leap seconds than UTC because NTP likes smooth transitions but my server isn’t going to worry about that. Both SNTP and NTP use the same UDP packet containing several timestamps for measuring transmission lag and other critical parameters. As with most computer clocks, the timestamp counts seconds.It has a 32 bit part for seconds and another 32 bit part for the fraction of a second. That means it can count for 136 years with a precision of 232 picoseconds [NTP Era and Numbering]. The rollover in 2036 is farther out than the expected lifespan of the hardware so I don’t worry about that. The fractional seconds is another problem as the ESP8266 doesn’t clock time in binary fractions.

The idea is to use the high precision pulse per second (PPS) line from the GPS receiver to mark the start of each second and then calculate the NTP fraction of a second from this using the time functions in MicroPython. The PPS triggers an interrupt, the interrupt service routine reads the ESP8266 clock and stores it in a global variable. When NTP needs a timestamp, it reads the ESP8266 clock, finds the difference to the Top of Second value in milli or micro seconds, converts that value to modulo 2^32 and posts the result into the NTP timestamp. That conversion may be obvious to many but it took me a while to figure out what people were talking about.

To convert a decimal fraction of a second like milliseconds to the NTP fraction is an equal ratios problem. How many milliseconds is to 1000 milliseconds in a full second has to be the same as how many NTP fractional parts there are in 2^32 fractional parts in a full second. That becomes a simple A/B = C/D algebra problem where B is given as 1000 for milliseconds or 1,000,000 for microseconds and D is 2^32 or 4294967296. If you are converting from NTP timestamp to decimal fractional seconds, then you know C and can calculate A. If you are going from decimal to NTP (like in our NTP server) then you know A and solve for C.

After all this, we get into computer word sizes and binary value representations. MicroPython has some facilities for this but it is a bit of shoehorning to make it fit, I’ll have to use Python arrays to get unsigned 32 bit integers and the struct class to handle bytearray packing for the NTP packet. Nifty compsci stuff that is.

So what’s wrong with 1975 programming?

Notes from the Architect picks up Varnish as his application example to highlight that things have changed since then. You can extrapolate from his examples to, say, Python or Java versus BASIC or C.

I have spent many years working on the FreeBSD kernel, and only rarely did I venture into userland programming, but when I had occation to do so, I invariably found that people programmed like it was still 1975.

So what’s wrong with 1975 programming? The really short answer is that computers do not have two kinds of storage any more.

And people program this way.

They have variables in “memory” and move data to and from “disk”.

Take Squid for instance, a 1975 program if I ever saw one: You tell it how much RAM it can use and how much disk it can use. It will then spend inordinate amounts of time keeping track of what HTTP objects are in RAM and which are on disk and it will move them forth and back depending on traffic patterns.

Well, today computers really only have one kind of storage, and it is usually some sort of disk, the operating system and the virtual memory management hardware has converted the RAM to a cache for the disk storage.

In other words, a critical part of 1975 programming was about memory management and temporary data storage on the machine. In modern applications programming, this is best left to the operating system. This is why garbage collection is such a big deal in modern programming languages. Application memory management in applications is a major source of bugs and can provide an access point for malicious code injection.

Of course, 1975 programming isn’t dead. A microcontroller is much like a 1975 computer in terms of memory management needs and there isn’t an OS to handle everything as the program itself directly deals with hardware. So there’s nothing wrong with 1975 programming. Like all software efforts and any other craft, tools and methods all have their place and a good craftsman chooses the right tools and methods for the task at hand.

Tech directions: IkaScope

They call it the IkaScope. “Robust and reliable analog front end, Measuring signals with IkaScope is quick, easy and intuitive. Sampling rate 200 MSPS; Bandwidth 25 MHz; Memory 4000 pts; Input range ± 40V; Coupling AC / DC; Refresh rate 200 FPS*.”

At this time, it’s a gleam in the designer’s eye so who knows what will eventually show up. The concept as described is feasible and illustrates a number of ideas about where instrumentation is going.

Wireless is more than just convenience. It provides electrical and functional isolation. This wireless uses a standard Wifi network and provides an access point if need be. Now, if the display and control used standard HTML – i.e. the device served up a standard web page rather than requiring some proprietary protocol, the only custom software would be bundled with the custom hardware.

The portability expression is also telling. This is an oscilloscope in a probe form factor. The level of integration not only provides capabilities in small form factors, it can also provide cost reductions.

For speculation, consider how the IkaScope idea might be expanded to multiple channels. How might a collection of independent probes coordinate their measurements? Then there’s signal processing, data analysis, other measuring needs, and … just where can it go?

Doors open. Ideas flood in. Interesting times.

1700’s tech: A piano story

It may not have been crafting nanocircuits on silicon but innovation and craftsmanship have legs in history. Here’s a good story of that: What Does the World Oldest Surviving Piano Sound Like?: Watch Pianist Give a Performance on a 1720 Cristofori Piano.

Imagine your favorite works for the piano—the delicate and haunting, the thundering and powerful. … Now imagine all of it never existing. A giant hole opens up in world culture. … The piano seems inevitable when we look back into music history. Its immediate predecessors, the clavichord and harpsichord, so resemble the modern piano that they must have evolved in just such a way, we think. But it needn’t have been so.

Other makers tried different mechanisms, but “Cristofori was an artful inventor,” the Met remarks, “creating such a sophisticated action for his pianos that, at the instrument’s inception, he solved many of the technical problems that continued to puzzle other piano designers for the next seventy-five years of its evolution.” These designers made shortcuts, since Cristofori’s “action was highly complex and thus expensive.” But nothing matched his design, and those features were “gradually reinvented and reincorporated in later decades.”

Though Baroque composers at the time, including Johann Sebastian Bach, “were aware of it,” most, like Bach, harbored doubts. “It was only with the compositions of Haydn and Mozart” decades later “that the piano found a firm place in music.” A place so firm, it’s nearly impossible to imagine the last 250 years of music without it.

The problem with earlier keyboard instruments was musician control of the loudness of notes played. That is the problem that the invention of the piano solved.

Besides dynamic range, there is a lot of other invention and innovation in the piano from soundboards and concert amplification capabilities to the user interface. A history of music technology teaches many lessons.

Seek and ye shall find – that it’s a harder journey than you ever imagined.

Luboš Motl takes off on Nautilus’ disillusioned ex-physicist. “Bob Henderson wrote an autobiography for Nautil.Us” and that provided the ammunition for a rant on Life’s Journey that any college freshman with ambition should read carefully.

This is a sketch of the “path towards the deep laws of the Universe” that I already had in mind when I was 4 years old or so – and I think that other physicists who don’t relate to Henderson’s complaints would tell you something similar. Henderson is telling us that he was gradually discovering some of these things during his grad school years. One actually has to work hard at some moment, be materially modest, be confused much of the time, and try many paths that don’t lead to interesting outcomes, while the greatest discovery in a century arrives relatively rarely (approximately once a century, if you want to know).

Those are shocking facts!

You should have known it before you entered the graduate school.

In the college but also in the later years, I was talking to lots of people who begged for recommendations like that. What should I do not to get lost? My answer was never so direct but yes, my current answer would be: If you need this leadership repeatedly, just quit it. If you don’t know what you’re doing, why you’re doing it, and where you are going, and how you may roughly get there, then it’s a bad idea to start or continue the journey. People who are picking an occupation should feel some “internal drive” and they should have at least a vague idea what they’re doing, why, and how. Again, I don’t think that this common sense only holds in theoretical physics. Theoretical physics only differs by the deeper caves in which one may get lost – because deeper caves are being discovered or built by theoretical physicists, too.

There are more “shocking facts” from a ‘been there, done that’ source. When Motl gets on a rant, he can spin off some very useful information and ideas and fodder for introspection and analysis. Heave to and take a gander.