Y2K bug story

How Caltech is preparing for the millennium bug
This story is adapted from a series of three articles that originally ran in the April, May, and June 1999 issues of On Campus, the employee newspaper at Caltech.

Now that the year is coming to a close, local, national, and world
media sources have stepped up their news coverage of the problem that is facing many of the world’s computer systems. In brief, all of the nicknames refer to the technological confusion that can result when computers are unable to read the year 2000, and therefore malfunction. Since many of the world’s day-to-day operations are dependent upon date-sensitive computers and computer chips, the potential effects are unknown to any specific degree, even by industry experts.

In order to understand the potential scope of Y2K, it is necessary to first
examine how the problem originated and developed into a world issue.

When people hear the word computer, they often think of the lightning-fast desktop personal computers that are widely available today and have become essential in many homes and offices. In the 1950s, however, a computer was a machine bigger than a desk, which was used to compute numerical calculations quickly and accurately. In this system, the data was stored on a collection of cardboard punch cards.

Because space on these cards was limited, and since dates were used repeatedly in many computer programs, programmers made the decision to limit the year date to two digits, with the first two digits of the year implied (1957 was recorded as 57, etc.). This practice continued into the 1990s, long after computer storage space, or "memory," became cheap.

The problems started to surface in the early 1990s, when some computers
began trying to process year 2000 dates. In some cases, the computers interpreted the date as 1900 and used this faulty information in their calculations. In others, the computers malfunctioned or shut down.

Today, the problems associated with Y2K can be divided into two categories–computer code and embedded chips. The computer code problems refer to the situations described above, where the two-digit year is woven into the computer code of personal computers, mainframes, and any other computer system that relies on dates in the programming code to perform its operations. (Any system that has the potential to be affected by the year 2000 rollover is referred to as "noncompliant.")

The second problem is with embedded chips. Once the power and possibilities of the silicon computer chip were realized, computers started shrinking in size and increasing in power at a miraculous rate. Today, chips the size of a fingernail are able to process data many times faster than the room-sized mainframes from the early days of computing. And these chips have been utilized in virtually every area of human life. From medical devices to automobiles, and telecommunications systems to the personal computer, embedded chips have become an essential part of modern life.

The good news is that not every chip contains date-sensitive programming. The bad news is that the date-sensitive programs that do exist on embedded chips cannot be changed; they are permanently inscribed onto the chips themselves. An extensive assessment process is usually the only way to identify what chips need replacing, and the time and cost involved in hunting down and replacing noncompliant chips is considerable. Estimates for fixing the problem of noncompliant chips and computer code run into the hundreds of billions of dollars in the United States alone.

Caltech is not immune. Many of the Institute’s operations rely on either computer code or embedded chips for daily functions. But through an early
assessment of potential Y2K problems, as well as an organized campuswide effort, Caltech is working to ensure that it will be ready to conduct business as usual when the year 2000 arrives.

The Institute’s Y2K preparations are organized into several areas, the first of which is Caltech’s Physical Plant.

When people start thinking about the potential effects that the year 2000 bug could have on their working lives, one of the first questions asked is usually, "Will we have power?" Most people have experienced a power outage in their lives, or know somebody who has gone without power, and therefore know how much of an inconvenience this can be. In modern society, most people rely on power in one way or another to perform their daily jobs, and for everything from plumbing and heating to microwaving lunches.

So, it is reassuring to know that the organization that takes care of power on Caltech’s campus, Physical Plant, is preparing to deal with any problems that may arise.

But ironically, although Physical Plant (in conjunction with the city of Pasadena) provides the foundation of power that runs the entire campus, out of all organizations on campus they are among the least likely to be affected by the year 2000 rollover. "About 90 percent of our equipment does not use embedded chips," says Bill Irwin, director of Physical Plant. And for those machines that do, the problems are likely to be annoyances, not disasters. "The likelihood is that anything that goes wrong with an embedded chip will be a minor problem–a stray warning light or something that doesn’t affect the operation of the machine," says Irwin.

For the time being, Physical Plant is concentrating on what it sees as the bigger issues. Because, even though embedded chips are not a problem for the majority of Physical Plant’s operations, those machines that rely on date-sensitive computer programming are. So, more than two years ago, Physical Plant started a comprehensive assessment of its operations, identifying which systems needed to be upgraded, replaced, or removed. Physical Plant employees have educated themselves on compliance issues, both through research and by attending Y2K preparation seminars, and have been keeping in contact with individual manufacturers regarding the compliance status of their equipment on campus.

Staffers are also being careful when getting new equipment. "Anything we have purchased since 1998 has had to be Y2K compliant," says Reza Ohadi, manager of campus operations at Physical Plant, and the person responsible for overseeing the day-to-day Y2K preparations.

Although Physical Plant encompasses a number of major areas on campus–everything from utilities and custodial services to transportation vehicles and a specialized machine shop–its Y2K preparations can be organized into two main systems, the plant system and the campus system.

About 60 percent of the campus energy for consumption is produced by a cogeneration system in Physical Plant, consisting of a gas turbine and a steam turbine, both of which are compliant.

The plant also features a new digital control system, which was built and installed this summer. As the name suggests, the control system coordinates much of the plant’s operations. At the same time, Physical Plant is training its operators in the use of the new system.

Every other piece of equipment in the plant system has been checked and been given the Y2K-compliant stamp of approval, although Physical Plant isn’t taking any chances and has a couple of contingency plans in place for the plant system. For example, in the event of an emergency, all plant equipment can be run manually, without computer assistance.

Also, since about 40 percent of the power on campus is provided by the city of Pasadena, Physical Plant has a backup plan in case Pasadena loses power temporarily. Why would this happen, you may ask? Although Pasadena has assured both its residents and Caltech’s Physical Plant that they will have power, the issue is one of interconnectedness. Pasadena is just one link in the national electrical power "grid." And, like most connected systems, if one part of the grid goes down, it could have a temporary effect on other parts of the grid.

So, in the event that Pasadena is temporarily not able to provide Caltech with power, the Institute has the capability to disconnect from the city and run on its own at a reduced power level. "If natural gas is available from the Southern California Gas Company’s pipeline, we can run for approximately one month," says Irwin. "Without natural gas, our liquid fuel reserves would keep us going for about six days." In either case, Physical Plant would use the technique of "load-shedding," which means conserving electricity by allocating power only to the vital systems needed to run the campus.

Once the power from the plant system is generated, Physical Plant relies on an energy management system to distribute the correct amount of power to each building on campus. As part of this system, a Y2K-compliant mainframe computer communicates with individual distributed-processing control panels in each building. Called "smart" panels, because they are not date-dependent, these devices run on a seven-day cycle–since their operations are controlled by the day of the week rather than by calendar date, they will not be affected when the clock strikes 2000. On Saturday, January 1, 2000, the panels will simply regulate the power for each building based on the preprogrammed specifications for that Saturday.

The energy management system also has a built-in backup plan–for both general and Y2K-related problems. The system is structured so that the individual control panels in the buildings are not dependent on the mainframe computer. The panels, when cut off from the mainframe, will continue to function according to their most recent program specifications, and can be changed manually by hooking up a laptop computer directly to each panel and programming new instructions (Physical Plant will buy a couple of Y2K-compliant laptops for this purpose).

The other two areas covered by the campus system–elevators and the fire alarm system–are both compliant. For the elevators, the newer elevator control systems came with factory-installed compliant software, while the older elevators are not processor-based and so are compliant by default. And as far as the fire alarm system goes, it was completely replaced with a new in the spring.

When it comes to Caltech’s administrative systems, it is impossible to talk about year 2000 preparations without discussing the APE (Administrative Process Engineering) initiative. APE was launched in the summer of 1996, and was designed to solve many administrative computing systems problems, not the least of which was the potential year 2000 problem. As part of APE, administrative computing systems were totally replaced with new Y2K-compliant systems (Oracle, Exeter, and other third-party software systems).

In 1997, administrative computing at the Institute underwent its own restructuring, independent of the APE initiative. With the addition of Robert Neary as Caltech’s chief administrative information officer (CAIO), administrative computing was restructured to include telecommunications and information security (both of which have been deemed Y2K compliant) under the AIS umbrella. In addition, AIS provides computing support to traditional administrative organizations such as human resources and payroll, finance, the controller’s office, and physical plant. Now that the APE initiative has completed its tasks and all of the administrative computing systems have been tested and implemented, AIS is taking control of many of these computing systems. An additional activity currently underway is Y2K-specific testing of the Oracle applications. This testing was started after applying more than 100 Y2K 'patches' supplied by the vendor, and is expected to complete and be implemented in November, on Thanksgiving weekend. And as part of Caltech's Watson Lecture Series, Neary will present "The Y2K Problem: Solved?" on November 17, at 8 p.m. in Beckman Auditorium. The lecture will review the current status of worldwide corrections, share the good news, and remind people how to prepare for those things not likely to be fixed in time.

The biggest potential Y2K problem for AIS is with outside vendors and other partners with whom the Institute conducts business. Mindful of the fact that not every outside organization with whom it does business may have fixed their own Y2K problems, AIS is taking a proactive approach to ensuring that outside vendors will be compliant in time. "We have sent letters requesting information on their compliance," says Neary. "And we’re tracking that information so that as we get closer to the year 2000, if any of them appear to be having problems we can make intelligent decisions about what to do."

An important type of outside vendor that Caltech deals with is banks, which play an important part in the daily transactions at the Institute, especially in the areas of payroll and insurance. "This applies to issues like direct-deposit checking," says Neary. "That’s something that’s very difficult for us to test in our Oracle implementations, because we can test to see if our interfaces work, but we can’t test the banks’ Y2K compliance." Instead, Caltech is relying on compliance information supplied by the banks themselves.

AIS is involved in one other very important area for Y2K preparation–the spread of information. Through a comprehensive Web site, AIS gives an overview of the Y2K issues at Caltech, including an up-to-date assessment of APE implementations on campus, and provides links to other sources of information about Y2K. The Web site is located at http://www.acs.caltech.edu/y2k/.

When it comes to faculty research, one problem becomes immediately apparent–the diversity of research equipment makes it impossible to provide a quick universal Y2K fix.

"From the faculty point of view, research efforts are independent and heterogeneous," says Vice President and Provost Steve Koonin, who is also the main faculty representative in administration. Koonin has informed the faculty that it is their responsibility to look into their own research facilities to see if they need fixing.

Caltech’s Seismo Lab performs a number of functions, not the least of which is providing accurate, up-to-date earthquake information for residents of Southern California. In addition, many faculty conduct research at the Seismo Lab. So it is good to know that in general the Seismo Lab has not had many Y2K problems, and those that they have had–updating a few computer programs–have been or are on their way to being solved. Seismo also uses many instruments that rely on highly sensitive timing, which are calibrated with the Global Positioning System (GPS). GPS timing involves using one of a number of government satellites (all of which have been deemed compliant) and satellite ground-support systems (which are currently not compliant), in conjunction with a control segment consisting of monitoring stations and ground antennas (all compliant), to pinpoint a location. Seismo also has some GPS field equipment–such as handheld units–which has been or is being upgraded to take care of the Y2K problem.

The following information is adapted from an article, originally published in the CIS News, by Linda Davis, chief technical writer for JPL’s Institutional Computing and Information Services (ICIS) office.

When JPLers arrive at work on Monday, January 3, 2000, they can expect to find no interruptions due to the Y2K problem. As an arm of NASA, administered by Caltech, JPL received a federal government directive to identify potential problems and implement solutions several years in advance of 2000. In
response, JPL formed a Y2K project office with overall responsibility for ensuring that all JPL software, hardware, and firmware is fully compliant with the
requirements of both NASA and JPL.

JPL created an extensive Year 2000 software inventory, and screened
approximately 28 million lines of computer code with a JPL-developed search tool to pinpoint Y2K problems. Almost all required repair or replacements, testing, and installation have been completed. Final testing and certification efforts are still under way.

In addition, nearly all of JPL’s desktop computers are compliant, and a contingency plan has been developed for the Laboratory as a whole, to provide a systematic approach to any unexpected occurrences.

Finally, JPL has launched a public Y2K Web site at http://www.jpl.nasa.gov/y2k/ that contains information about financial, health care, insurance and utility Y2K concerns. There's a FEMA Handbook and links to software vendors' Web sites as well as information specific to JPL, NASA, and Caltech.

Caltech’s other off-campus facilities are also in good shape. At the Owens Valley Radio Observatory (OVRO), the custom software has been written to be Y2K-compliant. In addition, older workstations have been upgraded with newer Y2K-compliant operating systems, and noncompliant equipment has been
replaced as necessary.

At Palomar, the staff has also been examining which computer programs and operating systems need to be replaced or upgraded, and has made nearly all of the necessary modifications.

Although the LIGO project reaches halfway across the country in both
directions (with off-campus sites in Louisiana and the state of Washington), both sites are linked to a series of on-campus sites through an extensive networking infrastructure. The hardware and software that run these networks are fully compliant and up to date, and several old machines will be taken off of the network before the end of this year.

At the Keck Observatory in Mauna Kea, Hawaii, the telescopes and mirror control systems themselves have been tested, as well as all of the telescope instruments. The last major operational subsystem, used for alignment and phasing of the 10-meter primary mirror, was tested in early September. Testing of non-operational systems is now in progress.

Finally, staff at the Caltech Submillimeter Observatory, also at Mauna Kea, do not have much to worry about. A new Y2K-compliant telescope control computer and software system was installed this summer.

As part of the APE project, an entirely new Y2K-compliant network, CITNET 2000, is in the process of being installed throughout the campus. ITS is working with the APE project during CITNET’s installation and will be responsible for maintaining the network when it’s completed (by the end of this year). At that point all other older networks maintained by ITS will be discontinued. Since the network itself is entirely new, there are very few pieces of related equipment that are noncompliant, and those that are have been or will be replaced. As an added precaution, ITS has verified that APE software can run over the old network if it must.

Although ITS is responsible for maintaining the network, when it comes to individual desktop computers, departments and divisions that don’t use ITS system administrators need to check their own systems. So what does this mean when it comes to your desktop computer? Like JPL employees, most people on campus will find that it means checking out their computer systems to ensure that all hardware and software are up to date. In order to make that process easier, ITS is providing an extensive Web site (see "Caltech Web Sites" section, below) that includes compliance information for most major computer and software manufacturers. In addition, people may contact ITS at the help desk (help@its.caltech.edu) for further assistance.

A report released in February by the United States Senate Special Committee on the Year 2000 Technology Problem stated the following: "The Committee is among the most broad-based, best-informed bodies in existence, yet it cannot predict what will occur on January 1, 2000. The data simply does not exist."

Because of this, the committee is encouraging citizens to take precautionary measures and treat any potential Y2K-related effects as if they were the result of "a winter storm, with the implication that similar preparation is appropriate. Americans should prepare for Y2K based on facts and reasonable predictions about the problem’s effects on vital services."

For Southern California residents, this advice has an added dimension, because many of the suggested Y2K preparations closely mirror those that residents should already be taking to prepare for earthquakes (extra food, water, candles, money, etc).

The Y2K Web site of AIS (Administrative Information Services), this offers a clearinghouse of information on Caltech’s plans and progress in Y2K preparations, including links to other Y2K sources.

The Y2K Web site of ITS, this offers a comprehensive guide to assessing your own computer’s Y2K needs, for both hardware and software.

The official Web site of the Senate committee, it includes the full 164-page report referenced above, news releases, and other miscellaneous Y2K info, including links to Y2K pages for each of the major United States federal agencies.

Created by Peter de Jager, a computer programmer whose seminal "Doomsday 2000" article published in the September 3, 1996, issue of Computer World brought the problem to the world’s attention, the site provides a wealth of Y2K information such as articles (including a reassuring update from de Jager entitled "Doomsday Avoided"), links to vendors of Y2K diagnostic products, a Y2K legal reference, and links to Y2K user groups that discuss potential Y2K problems and solutions.

You and the Year 2000: A Practical Guide for Things That Matter, by Jeffrey Shepard.

Written by a psychologist who first started researching Y2K as a preparation for his business systems, and who branched out into examining all of the potential areas that people may want to look at, this book is thoroughly researched and fun to read, a good reference for anybody looking to avoid the hype and get a level-headed treatment of the subject.

The Complete Y2K Home Preparation Guide, by Edward Yourdon and Robert Roskind.

Provides an in-depth look at potential problems and solutions, and includes a 137-page reference section on other Y2K resources (books, Web sites, community groups, checklists, and sample letters to utilities and government sources, to name several).

by Ryan Poquette