VX Heaven

Library Collection Sources Engines Constructors Simulators Utilities Links Forum

What do you Feed a Trojan Horse?

Cliff Stoll
Proceedings of the 10th National Computer Security Conference

[Back to index] [Comments]

Techniques to Solve Hacking Problems

Cliff Stoll
Lawrence Berkeley Laboratory
Berkeley, CA 94720
Revised 29 May 1987


Computer security is sometimes best served by corking up known holes in a system, and sometimes by tracking an intruder to the source. Techniques used to pursue the latter course include high speed network traces, operating system alarms, off-line monitoring, and traffic analysis. But technical methods are not enough. It's just as important to coordinate efforts with law enforcement agencies and other professional organizations, and to understand the constraints set on each organization. Persistent sleuthing can ultimately locate the source, but it may require considerable time and effort.


When you know a computer system is under attack, you're presented with a choice: should the drawbridge be raised and outside access cut off, or should the source of the attacks be determined?

This paper addresses what to do when you choose to track the penetration. Related topics, such as how to detect an attack, or how to protect a system, are largely ignored here, although all of these topics are intimately intertwined.

Once you decide to find the origin of the attacks, you must start a traceback effort. Occasionally, this will be easy, and the suspected intruder collared quickly. Usually, the intruder will have taken steps to conceal the pathway into your system (often using stolen resources to do so), and unwinding the connections may challenge your best efforts.

Tracebacks through digital and analog networks are theoretically straightforward - after all, an outside attacker made a physical connection into your computer. In practice, however, unwinding a complex connection can be quite daunting, especially in a short time.

Making a traceback is essential for the prosecution of an attacker; it also teaches lessons in network connectivity, coordination between law-enforcement agencies, and the strengths and weaknesses of our interlinked digital networks.

Our problem, then, is to unwind the connections made by an unknown intruder, and ultimately determine who's in the system. This effort requires a thorough understanding of operating systems, networks, telephony, and digital communications. Familiarity with legal issues and law enforcement protocols will be helpful. Fitting together diverse clues, some of which are misleading, eventually may lead to an answer, although a prosecution may not necessarily follow.

Should you ignore the attack?

There are good reasons not to try to catch an attacker. You may subject your system to danger - the intruder may gain sufficient privileges to delete or modify important files * [* Perhaps more damaging than a massive deletion of files is the slight modification of files - this may go undetected]. You risk the disclosure of sensitive information. It may prove to be a wild goose chase. The attacker might be illusory - a figment of your operating system. As we shall see, unwinding a complex connectivity can become expensive, requiring coordinated efforts of several technicians. Prosecution may prove impossible, due to legal problems, or infeasible, due to political or economic factors. You may embarrass your organization by admitting that an outsider has access to your computers.

If you decide not to trace the source of an attack, there are several alternatives available. You may ignore the intrusion completely. You may close your doors to the attack, by changing passwords, tightening modem access, or strengthening your operating system. Or you may simply legitimize the activity.

There are also many valid reasons to chase down an attacker. The intruder may be out to injure your organization, possibly for personal benefit. Lost resources can be recovered only by locating whoever took them. A criminal may be caught and prosecuted by means of a traceback. As a community service, tracebacks of illegal activities help make networks safer for everyone.

Network tracebacks can be a rewarding area of academic research. Trying to catch someone within a computer can lead you through problems in operating systems, networking and network topology, as well as digital and analog telecommunications. Such work also touches on technology law and the ways in which various organizations respond to novel problems.

Organizing your efforts

Once you've decided to trace an attacker, you'll need to organize your efforts. Early on, designate one person to serve as a single point of contact until the problem is solved. Since your staff will want to know what's happened, stop rumors by holding a meeting.

You'll need to warn staff members to be quiet about the investigation. If the word leaks out, not only will your work have been wasted, but a malicious intruder might damage your system. Law enforcement organizations won't help you if they believe you may leak investigative information. Unless your staff realizes the sensitivity of this matter, they may mention it on electronic bulletin boards, at conferences, or to colleagues.

Start a logbook, collecting and analyzing your evidence within it. Record all suspicious activities, along with their dates and times. Maintain a clear distinction between conclusions based on firm evidence and suspicions based on indirect evidence or assumptions. Summarize telephone calls and messages from other sites. Keep your logbook out of any computer accessible from the system under attack - assume that the intruder is searching for it.

You will repeatedly explain what happened to a variety of people, many of whom won't understand computer jargon. For this purpose, prepare a summary of what happened, using layman's terms. Describe exactly what damage has been done; include loss of services, disclosure of information, and the costs to rebuild lost files; quantify this damage in dollars, if you can.

Keep records of the costs of the break-in, and your expenses in repairing it. This is needed to determine the level of the offense (felony vs. misdemeanor); it also indicates which agency will have jurisdiction over the case (local/state/federal). Since some expenses in tracking and solving the problem can be recovered through lawsuit, these records can be essential should the case go to trial.

Many legal issues come up in trying to track and prosecute a computer intruder. What privacy rights exist? Can you bring suit for damages against someone breaking into your computer? Can you recover for the costs of tracking? What constitutes a breakin? The laws and interpretations have changed in the past year, so visit a law library to learn about current statutes involving computer crime. This is especially helpful in communicating with law-enforcement people, who may not be aware of recent codest.1

Early on, determine how deeply you are threatened. Does the attacker have system privileges2? Have Trojan horses, logic bombs, or virus programs been created? Is there a danger to your system if you try to catch the attacker? Have you the resources to chase down the attacker? Set limits on how much time and effort you will commit to the task.

Who should you tell?

Soon after detecting an attack, you should spread the news to people who can help solve the problem and to people running other systems at risk. But limit the spread of this news! Certainly, inform your management and your funding agency. If you have evidence of attacks via other systems, inform trusted system managers of those sites by telephone (never send computer security messages by electronic mail!).

Several external organizations may be able to help you. Your local police are charged with the enforcement of local and state laws - you should be in dose contact with them. Federal investigation and enforcement efforts are coordinated through the FBI and the Secret Service; the US Department of Justice will handle the prosecution in these cases. Problems which involve the Milnet, Arpanet, or military computers can be referred to the Defense Communications Agency and the Air Force Office of Special Investigations.

You should contact the National Computer Security Center in Ft. Meade; while they perform the difficult task of trying to prevent these problems, they also keep track of attacks, and can coordinate efforts to understand and solve such problems. Additionally, be aware of the Institute for Computer Sciences within the National Bureau of Standards. Both of these agencies can advise you on security holes which your intruder has used.

When traces must be performed over digital networks, it's important to be in close touch with the appropriate network operations center. It's important to know beforehand whom to call: when confronted with an attack, it's difficult to reach the correct person quickly.

Throughout your contacts with these organizations, keep records of who you've spoken to, and what response you've received. In addition to reinforcing your own memory, these records are helpful in prodding agencies to take appropriate action.

Operating System Accounting

Fundamental to the detection and tracking of any unauthorized computer user is adequate resource accounting. Modern operating systems typically record resource usage, task names, times of login, and connection port. Often, this is the only information available to determine the· extent of an intrusion3.

The quality of auditing data varies with operating system, and with the system manager's needs. With good accounting data, and reasonable summaries of activity, audit trails are easily constructed. Spotty accounting, with inaccurate clock times and missing records, prevent the detection of even the most gross violations.

Even if a computer does not recharge for usage, accounting records should be kept. Without these audit records, it's impossible to reconstruct what has happened in the past - an essential part of tracing an attacker. From this, it follows that no individual should be able to disable accounting. The accounting records should, at minimum, include port number used to access the computer, task name executed, flags for attempted access to protected files, and start/end times.

Accounting records can also be used to identify the incoming port and speed. If network connection (e.g. Milnet or Arpanet) is indicated, the originating host name may be included in the accounting records. If the accounting records show a serial port, then determine the baud rate of this port: generally, high speed connections are from on-site, and low speed connections are from off-site, usually through modems.

The login/logout times are used as timestamps to control searches into other accounting records. These times are compared to local area network connection times or compared with telephone company billing records. To simplify record keeping, save these dates and times in GMT, and keep your clocks accurate to a second (non-synchronized clocks confuse traces across several systems).

If no obvious damage is done, (perhaps only files have been read), a successful invasion of a computer may go undetected for a long time. Thus, detailed accounting records should be saved for at least a year. These records can be used to show how an invader succeeded in entering your system, and can point out accounts which have been poisoned by the attacker.

In some circumstances, standard accounting records may not be trustworthy. An invader may have disabled accounting or modified accounting records. Accounting may be incomplete for some nodes on your network. In any case, ambiguous clocks and sloppy record keeping will confuse the interpretation of audit trails.

Local Area Nets

Almost every large computer system, and many smaller ones, use local area networks to interconnect terminals, modems, computers, and other networks. Often these are referred to by the manufacturer's name (Micom, Develcon, Sytek, etc.). They usually introduce significant holes into the security of a system - an ideal place to plant Trojan horses. Seldom are these LANs programmed by the systems staff, or considered as security problems4; they are usually set up by a communications group, and then little more than routine maintenance is performed. Few systems people pay attention to the connectivity they provide** [** For example, some local area networks allow outside dial-in users to immediately dial out from a common modem pool, forcing the host to pay for long distance calls, and providing an excellent hiding place for hackers]. A connection through a LAN may be impressively difficult to trace.

When an attack originates from the LAN, it's necessary to find the originating port. Usually, accounting data from the operating system will tell which computer port the activity came in from. Historical accounting data from the LAN controller is needed to determine from which port or network the attack originated. Some LAN controllers simply cannot provide this information - avoid using such systems! When this audit information is available, it's important to collect and save the data for later analysis. Make certain that the connections are recorded along with related housekeeping information, such as baud rates, dates, and times. Since there will likely be hundreds or thousands of connections recorded every hour, an accurate time stamp is essential - periodically calibrate this clock.

When a local ethernet is suspected of being in the line of the problem, it may be necessary to audit all of the connections on the ethernet. Likely, this will require time-domain reflectometry to physically locate all of the drops on the cable. Because of the potential of promiscuous-mode listening, ethernet problems must be taken seriously5.

Telephone Traces

Eventually, a telephone trace may be needed. For many reasons, telephone companies may appear to drag their heels before tracing a call. It's often technically difficult, requiring skilled technicians, and phone companies may say that they are worried about the risk of lawsuits. Such statements appear unfounded + [+ Such a backward telephone trace, (called a "trap and trace") has been held not to violate privacy rights, and does not require a court order when performed on your own telephone line. See 18 U.S.C.A 3121 (b) (1) and (b) (3). Indeed, some telephone companies are now offering residential service that displays the originating telephone number while the dialed phone is ringing]; there is no liability when acting under a search warrant. Most telephone companies have departments which are expert in tracing telephone lines, and telephone traces are common procedures.

A telephone trace can be obtained through the appropriate police agency, who will contact the District Attorney for a search warrant. Affidavits and relevant evidence will be reviewed before the courts, and a warrant issued. The telephone company will probably need some advance warning before installing the necessary equipment, or placing technicians on standby. This calls for advance coordination between the computer site, the police, and the telephone tedmicians. Advance dry-runs will help iron out problems6.

Depending on the mechanism used, telephone traces may take place in real-time or after the fact. In either case, it may be necessary to know the exact start time of the incoming call to the second. When tracing a call through multiple exchanges or through long distance exchanges, simultaneous coordination by several groups may be required, since traceback equipment is seldom integrated with the long distance billing system. For real-time traces, telephone technicians can recognize digital traffic carried by its characteristic sound on a line, and it's usually straightforward to be certain that a particular connection is the valid one.

Because of deregulation, interstate and crosscarrier telephone traces can be difficult. Despite this, many complex telephone traces can be done within a matter of minutes, provided that all organizations along the line are forewarned. Local and long distance telephone systems need automatic traces to pinpoint troubles in lines and switching gear, so the equipment and techniques exist.

After a successful telephone trace, a law-enforcement agency may set a pen-register on the telephone line. Such a device records the phone numbers of all out-dialed calls ++ [++ The installation of a pen-register and related recorders requires a court-order], and helps determine the extent of an individual's telephone contacts.

Digital Network Traces

Packet switched networks, such as the Internet, have information on the originating node written within the packet header block7. When the network links directly to the host computer, packet information may be recorded by the accounting program. Some networks convert the packets into serial data streams (such as RS-232), and send this stream to the host; in such cases the packet header information is unavailable to the host, but may be available at the X.25 interface.

Packet header information may be counterfeit, garbled, or missing. When this is suspected, a call should immediately be made to the network operations center. Such organizations can quickly unwind the linkages within their systems and trace the path of a connection. Such unwinding can only be done while the connection is active; Internet does not record connections for later analysis. Other networks, such as Telenet, Datapac, and Tymnet, do save records for billing purposes, and historical connections can be reconstructed, provided that accurate times are available for comparison.

Digital networks are worldwide, and some information may be hidden when network boundaries. are crossed. For this reason, international traces require close cooperation between the network operations people. Find out in advance who is responsible for these traces, and exchange telephone numbers. Know your network location and port number as referenced by the network - it can save several minutes in performing a real-time network trace.

Digital networks which use datagram ack/nak flow control can be timed to determine round trip travel time. On the Milnet/Arpanet, many seconds may elapse before a datagram is acknowledged on a coast to coast connection. After accurately timing these packet receipts, a statistical average will indicate a nominal distance to the originating site. A similar method can be used when Kermit or Xmodem protocols are used over serial lines.

Some intruders will use each successive computer as a jumping off point to get into another system. After stringing together several computers, modems, and a variety of networks, such connections may become frustratingly complex. Leapfrogging between computers and networks can allow an intruder essentially toll-free access to many systems, with intermediate sites paying for the connections. Fortunately, interactive response time suffers, and these users eventually simplify their connectivity.

Alarms and Monitors

When an attack is suspected, it's important to determine exactly what information is being sent out of the system, and what files are being accessed by the intruder. Accounting information provides a pointer, but there is no substitute for a complete printout of the bidirectional chatter. This can be recorded by the operating system, but off-line monitoring is easier, better hidden and entails less overhead. Recording equipment (such as a PC with serial lines and a hard disk) can easily be daisychained + [+ RS-232 data lines can be multi-dropped to drive several receivers] to modems, providing continual monitoring of all serial traffic. Network software (such as the TCP/IP daemons) are easily modified to save traffic, as well; this can be done on-line or off-line.

The monitoring software and equipment can be programmed to alarm whenever particular character sequences are detected. Thus, an alarm can be set off whenever a particular account is accessed or whenever a certain password is entered. You may wish to build more sophisticated alarms, using expert-systems techniques.

When immediate response is needed, alarms can be connected to an auto-dialer ++ [++ Hayes-compatible modems make excellent auto-dialers, and can be programmed to send information to pocket pagers], to ring a telephone or pocket pager. Since many intrusions last for only a few minutes and may occur at any hour, a pocket pager is essential to quick tracing of these calls.

Operating system alarms usually warn the system manager when false logins have been detected, or when protected files have been read. These are very useful for the detection of intruders, but cannot be fully trusted. When an invader acquires system privileges, it's likely that he will disable accounting, turn off the alarms, and modify any files that record his presence. These on-line alarms, then, aren't very dependable once an attack has succeeded.

Traffic analysis

After monitors have recorded the intruder's traffic, analyze what has happened. Annotate and save all printouts; keep a detailed record of what happened in your logbook. Using the printouts, try to determine what the intruder was looking for, what he tried to access, and what keywords were important. Did he try to link to another computer? What passwords did he try? Did he modify any of your files? How long did each session last? What attracted his interest?

Of course, each intrusion will be different; detailed traffic analysis is crucial to the solution of the problem, partly because it describes the intruder's interests, and in part because it provides law-enforcement agencies with evidence useful in the prosecution of the intruder. Keep all of these records off line - never allow your security related records to be readable from any computer network or dial-up line.

Communications with other sites

When an attack on your site has been detected, everyone wants to know about it. It's necessary to strictly limit the spread of this information if you wish to trace the problem. To prevent rumors from spreading, hold meetings to discuss progress, warning all members that the information should not be spread. Talk openly with trusted site managers, but do not leak information to the press, or to various bulletin boards.

It's essential that all communications be kept out of electronic mail, and no files be kept anywhere of this activity. Intruders will naturally scan file systems, searching for keywords that might indicate that they have been detected. Electronic mail is an especially fruitful section to search.

When coordinating work with another site, communicate by telephone. Keep any files related to this activity encrypted. Assume that all of your files are regularly read by the intruders. Do not keep files with obvious names like "security" or "hacking". When building monitoring programs, do give away their function with titles like "monitor" or "watchdog".

Relationships with Law Enforcement Agencies

Presently, the federal government and several states have tight computer security laws. These are enforced through the FBI, the Secret Service, and various state and local police agencies. Police training and awareness has been recently increased, although most of it seems to be directed towards computer crimes with direct, measurable economic implications (e.g. theft via computer, accessing bank records).

For a poorly researched case, there isn't much hope for enforcement due to the novelty of the laws, the lack of judicial caselaw, and the need for highly trained specialists. A well documented case, including a detailed analysis of the losses, will increase the chances of police support.

Close cooperation with all levels of law enforcement agencies is essential. You'll need to carefully explain the nature of the problem, what your losses have been, and how your problem relates to existing laws. Policies by law enforcement agencies aren't established for these offenses and the police. likely will be hesitant to commit the resources to open a full investigation. This can sometimes be overcome by persistently explaining the need for support, and by doing as much of the work as possible yourself.

It's important to communicate with the law enforcement community early. Determining what agency to contact may be difficult; within any organization, probably only one or two people can appreciate the nature of the crime being committed. For this reason, you may find it fruitful to explain your problem to all possible agencies, and allow them to refer you to the correct organization. Each law enforcement organization has its own specialties; don't assume that you'll necessarily get more support from a national agency - indeed, you may find that your local police are far more interested and supportive.

When telephone traces are needed, advance arrangements with your police contacts will prove invaluable: the telephone companies generally require court-orders, and the police know how to work with the appropriate district attorney to obtain these. The phone companies, in turn, are comfortable reporting to the police, but do not wish to report to injured parties, for fear of lawsuits.

Conclusion: Locking the Barn Door

You can trace connections back to the source in most circumstances. You'll need to keep detailed records, analyze audit trails, set monitors, traps, and alarms, and closely watch your operating system. Actual line traces may be in real-time or historical. Ironically, there are relatively few technical challenges; the main problems are in coordinating the efforts of many organizations.

Once you've tracked the rascal, finding out just who's been giving you such grief, you'll still have to close all the doors, whether or not he's been prosecuted (or even arrested). You'll need to simultaneously delete the accounts which have been compromised, eliminate the security holes, and change all passwords on your system +++ [+++ With hundreds of users, a complete password change is distasteful. Alas, but no other method (password aging, account expiration, account requalification) can assure you of a clean system, without compromised accounts]. Pulling the plug may be quite involved, and calls for advance preparation

Our networks provide rich connectivity; alas, but few people are paying attention to the risks which are created. When confronted with attacks and intrusions, system managers often talk about tracing the connections, but seldom actually initiate traces. With a little perseverance, it's possible to unwind connections, and find out who's at the other end. After tracing an intruder, tell your tale to others - let the rest of the world learn from your sorrows.


I am grateful to the U.S. Dept. of Energy for supporting this work through contract DE-AC03-76SF00098. For suggestions, comments, and critical reviews, I am indebted to Marv Atchley, Bruce Bauer, Rick Carr, Dave Farnham Mike Gibbons, Roy Kerth, Steve Kougoures, Dave Jones, Martha Matthews, Sandy Merola, Bob Morris, Ken Sebrell, Phil Sibert, Dave Stevens, Steve White, Regina Wiggins, and Hellmuth Wolf. For the best enchilada in the West, I'm grateful to Garcia's Restaurant in Alburquerque.


1 Relevant codes include 18 U.S.C.A. 1030 (The Federal Computer Crime Statute); 18 U.S.C.A. 1343 (Wirefraud Codes); 18 U.S.C.A. 2518 (Interception of Electronic Communications). For an example of a modem state statute, see CA Penal Code S. 502 (1986)

2 Anecdotal stories abound of intruders obtaining system privileges. See, for example, 2600 Magazine Vol 4, January 1987, page 6-7.

3 More complete auditing tools are being proposed, especially for Unix. viz: J. Picciotto, The Design of an Effective Auditing Subsystem, 1987 IEEE Symposium on Security and Privacy, April, 1987.

4 Secure networks are receiving increasing attention, and a lively debate has grown as to how to address these knotty problems. See D. Nessett, Adding Central Authentication to DECNET, DOE 10th Computer Security Group Conference, May 1987, page 56.

5 For a general review of Local Area Networks, see R. Stallings, Editor: IEEE Tutorial, Local Network Technology, 1985.

6 The need to prepare in advance for computer break-ins, and to create routine responses to them is noted by B. Reid, Reflections on Some Recent Widespread Computer Breakins, Comm. ACM, Feb. 1987.

7Milnet protocols are amply documented in the DDN Protocol Handbook series, NIC-50004, 50005, and 50006, available from the DDN Network Information Center, SRI International, 333 Ravenswood Ave, Menlo Park, CA 94025.

[Back to index] [Comments]
By accessing, viewing, downloading or otherwise using this content you agree to be bound by the Terms of Use! aka