sBurpasceid (gf, YSt7) 89ual2u09 uoneNc:upY swarshg many VINIO


9661 49qUW2IIT



Co-sponsored by The USENIX Association and SAGE, the System Administrators Guild

® Ba | Ss OAs

The Advanced Computing Systems Assoolation

The Twelfth Systems Administration Conference

(LISA 98) Proceedings

Boston, Massachusetts December 6-11, 1998

For additional copies of these proceedings contact

USENIX Association 2560 Ninth Street, Suite 215 Berkeley, CA 94710 USA Telephone: 510-528-8649

The price is $32 for members and $40 for nonmembers.

Past USENIX Large Installation Systems Administration Workshop and Conference Proceedings (price: member/nonmember)

Large Installation Systems Admin. I Workshop 1987 Phildelphia, PA $4/$4 Large Installation Systems Admin. II] Workshop 1988 Monterey, CA $8/$8 Large Installation Systems Admin. III Workshop 1989 = Austin, TX $13/$13 Large Installation Systems Admin. IV Conference 1990 Colorado Spgs,CO $15/$18 Large Installation Systems Admin. V Conference 1991 San Diego, CA $20/$23 Systems Administration VI Conference 1992 Long Beach, CA $23/$30 Systems Administration VII Conference 1993 Monterey, CA $25/$33 Systems Administration VIII Conference 1994 San Diego, CA $22/$29 Systems Administration [IX Conference 1995 Monterey, CA $30/$38 Systems Administration X Conference 1996 Chicago, IL $30/$38 Systems Administration XI Conference 1997 San Diego, CA $30/$38

Outside the U.S.A. and Canada, please add $12 per copy for postage (via air printed matter). Copyright © 1998 by The USENIX Association. All rights reserved. This volume is published as a collective work. Rights to individual papers remain with the author or the author’s employer. Permission is granted for the noncommercial reproduction of the complete work for educational or research purposes.

ISBN 1-880446-40-5

AIX is a trademark of IBM.

Adaptive Server is a trademark of Sybase, Inc. Backup Server is a trademark of Sybase, Inc. Cisco is a trademark of Cisco Systems, Inc. Enterprise is a trademark of Sun Microsystems, Inc. IBM RS/6000 is a trademark of IBM.

Java is a trademark of Sun Microsystems. ORACLE PL/SQL is a trademark of Oracle, Inc. Oracle is a trademark of Oracle, Inc. Oracle8 is a trademark of Oracle, Inc. PostScript is a trademark of Adobe Corporation. SGI is a trademark of Silicon Graphics, Inc. SQL Server is a trademark of Sybase, Inc. SQL*Net is a trademark of Oracle Corporation. Sniffer is a trademark of Network General. Solaris is a trademark of Sun Microsystems, Inc. Sun is a trademark of Sun Microsystems, Inc. Sybase is a trademark of Sybase, Inc.

UNIX is a registered trademark in the United States and other countries licensed exclusively through X/Open Co. Ltd. VERITAS File System is a trademark of VERITAS Software Corporation. VERITAS FirstWatch is a registered trademark of VERITAS Software Corporation. VERITAS Volume Manager is a trademark of VERITAS Software Corporation. VERITAS is a registered trademars of VERITAS Software Corporation.

USENIX acknowledges all trademarks appearing herein. & Printed in the United States of America on 50% recycled paper, 10-15% post consumer waste.

USENIX Association

Proceedings of the Twelfth Systems Administration Conference (LISA XII)

December 6-11, 1998 Boston, MA, USA


ACKNGWIEG BMIENTS .. 5: svaeccsnniesasasscearnasanedsnradandatesessuacscansansigbedaccsheessaceneiatosasadenonssee sossdiadsheusonesbtesendsecsveviesnsssgcbenssanseres iv PLEA CEs ss.cs scsreseesenensceacwecestsnass eeuhteatonsi Ri vbaucetakdeasilewtain eastind cena siade savvostaulaide avis cdetbedsinstawnsteeonsunauabavadeisaianats caleranieiniaveds Vv PUNO TDG oc ssececceicecscteeseneaeceesickoaceescacaaas Sestah cca tc ks gas Naas c aa cae SEs ease Secu TAN Nas BN a Saas oe Sena ce Ta gaNEA ES LATTES vi Opening Remarks Wednesday (9:00-10:30 am) Chairs: Xev Gittler and Rob Kolstad Security Wednesday (11:00am-12:30pm) Chair: Phil Cox TOADAIN: coses cecal soniscastaupecansznasc omienuanaeetyansesrue negro somerset etaaa se een ectnores oe esteem rea talent Eater 1 Dan Farmer, Earthlink Network; Brad Powell, Sun Microsystems, Inc.; Matthew Archibald, KLA- Tencor Infrastructure:.A.Prerequisite:for Effective Security sciccssiccscasssssnssstssvssstevsescececessseaecenssstssisneaieniecsssscesenssssscusssseeenses 11 Bill Fithen, Steve Kalinowski, Jeff Carpenter, and Jed Pickel, CERT Coordination Center SSU; Extending SSH for Secure Root: Administration ssscicsscctesssesscsevescscsscsessoateaccssevnsscacensecssteesstasuacsacescesesdevscestucses 21

Christopher Thorpe, Yahoo!, Inc.

Pushing Users and Scripts Around

Wednesday (2:00pm-3:30pm) Chair: Ozan S. Yigit

SystemiManagement/ With NetScript siiciccsccsrvcievecesvorsaavaacevussaververvisessactsvedvarcvenssecvensnssucsenstvecassns vasensdedivavanieosAaveccesss 37 Apratim Purakayastha and Ajay Mohindra, IBM T. J. Watson Research Center

Accountworks: Users Create Accounts on SQL, Notes, NT, and UNIX ou....eecceecesscssessceseesseeseeenseeceseeeaeeseeesseeeenees 49

Bob Arnold, Sybase, Inc.

Single Sign-On and the System Administrator ...........cccccssssssescsssseeeccecesceeseceeeccsceccsesstecseeecsesaceesacacseseeasecsaeeaeaeeeeaes 63 Michael Fleming Grubb and Rob Carter, Duke University

Storage Performance Wednesday (4:00pm-5:30pm) Chair: Marc Staveley

Using: Gigabit Ethernet to Backup Six Terabyte: scsscecssscrscauscanecesssssencssezsccsbecssssacesosaaseitbeasecessensevuata niatertersaseecusseerins 87 W. Curtis Preston, Hughes Space and Communications

COMMBUTINS Data DasesS V Stems oi. sisacexass ieandesesaceancoaasonyecandeyysansdsdundassnsyed cantedasncesdecieossauepesaas Ghar deansane attececensbaustectbanize 97 Christopher R. Page, Millennium Pharmaceuticals

1998 LISA XII December 6-11, 1998 Boston, MA

Distributed Computing Thursday (9:00am-10:30am) Chair: Phil Cox

A Configuration Distribution System for Heterogeneous Networks .........c.ccssscsssssssesesesescscscscscscsesescseseseacavsacarenceees 109 Glédson Elias da Silveira, Federal University of Rio Grande do Norte; Fabio Q. B. da Silva, Federal University of Pernambuco

An NFS Configuration Management System and its Underlying Object-Oriented Model ........c.c.cccccssssesssessseseeeeees 121 Fabio Q. B. da Silva, Juliana Silva da Cunha, Danielle M. Franklin, Luciana S. Varejao, and Rosalie Belian, Federal University of Pernambuco

Design and Implementation of an Administration System for Distributed Web Server ..........cccccccsssssssssseseseesseseseeee 131 C. S. Yang and M. Y. Luo, National Sun Yat-Sen University, Taiwan, R.O.C.

Networking Thursday (11:00am-12:30pm) Chair: Eric Anderson MRIG— The: MultiRouter dT raltio Grapnett sscvscssiscsssnsccevai dense cavenravsvervadiobvesactistichcesesbecyen caceolves dusvaseah opieavonea ven oisees: 141 Tobias Oetiker, Swiss Federal Institute of Technology, Zurich Wide: Area Network: Ecology’ sisasconssseesaworreceiecanernaussemaausiiesnst cacestcaieensre rRNA 149

Jon T. Meek, Edwin S. Eichert, Kim Takayama, Cyanamid Agricultural Research Center/American Home Products Corporation

Automatically Selecting a Close Mirror Based on Network Topology ..........c:csscssssccsseseesseseescsscsseseescescesessscsseecsceese 159 Giray Pultar,

Infrastructure Thursday (2:00pm-3:30pm) Chair: John Orthoefer

What to Do When the Lease Expires: A Moving Experience ........ceecsssesssessesesssscseesssseseseescseceaeeecsesscseseeseasarseeaes 167 Lloyd Cha, Chris Motta, Syed Babar, and Mukul Agarwal, Advanced Micro Devices, Inc.; Jack Ma and Waseem Shaikh, Taos Mountain, Inc.; Istvan Marko, Volt Services Group

Anatomy: ofan Athena’ Workstatons: .asscncsvscsssscinivnsessscovser rorveseatansveaasstrsrneeracetseatarntennutsinansonesionmas teats 175 Thomas Bushnell, BSG and Karl Ramm, MIT Information Systems

Bootstrappingian Intrastruchire: sssccesesersteessasassyeasricvenstecepins parreeomaneranstimminsaianes ann ERE 181 Steve Traugott, Sterling Software and NASA Ames Research Center; Joel Huddleston, Level 3 Communications

Printing and Configuring Files Thursday (4:00pm-5:30pm) Chair: David Kensiski

Ganymede: An Extensible and Customizable Directory Management Framework .............c:csccssesseeeseecessessseseesseees 197 Jonathan Abbey and Michael Mulvaney, The University of Texas at Austin

Burlditie An. Entetprise; Printing Sy Stein suscesszcscspeeserssseescesesssss cerca revceraas secctutacs yates seca caces asteeesb send sestacsnnenbasi leonisones 219 Ben Woodard, Cisco Systems

Large Scale: Brint Spool Senvice \éscnccssisesstsrwnsipheessarncnsnaeovcracssosceusscasenavaesnetcdeccsbesssveesasdessiteastart aavesstedscssssastionedeatiees 229 Ignacio Reguero, David Foster, and Ivan Deloose, CERN

ii 1998 LISA XII December 6-11, 1998 Boston, MA

Distributing Software Packages Friday (9:00am-11:00am) Chair: E. Scott Menter

mkpkesAvsoftware packaging tool j.i2:: cf sscossocceasesnatled edecsssdenessssnvssncdessonstenpuenedenseqonstondtssonteosesvessernsesnnigeanasacniwestor 243 Carl Staelin, Hewlett-Packard Laboratories

SEPP.— Software Installation arid Sharing:System <c:scscsssvsssscrcscessevscascscsessztececsvesvaestvensedeveavnsstosstesutensseusysdovensasvettene 253 Tobias Oetiker, Swiss Federal Institute of Technology, Zurich

Synctree for Single Point Installation, Upgrades, and OS Patches .............sccscseesssessesssescssseessseeessssescesseseeesesecesseeeses 261 John Lockard, University of Michigan; Jason Larke, ANS Communications, Inc.

New Thoughts and Evolution Friday (11:00am-12:30 pm) Chair: Melissa D. Binde

The Evolution of the CMD Computing Environment: A Case Study in Rapid Growth 0... eeseeseseseeseeeeecsseseseeees 271 Lloyd Cha, Chris Motta, Syed Babar, and Mukul Agarwal, Advanced Micro Devices, Inc.; Jack Ma and Waseem Shaikh, Taos Mountain, Inc.; Istvan Marko, Volt Services Group

Computer lmmunGlogy sssiscscscscsscscisccecdsceriavassasaevavesssssivstsvans ave tosaxcuanseseevetateaetibucsbacvacectosnavtesdbeosessstecestetdbeletenszecseens 283 Mark Burgess, Oslo College

A Visual Approach for Monitoring LOeS siiscsssccssts sacstcasessesctovsaseszexsaesssecdostsnscausseetescasavdacsevecsaeessensenssotansbsascbssenseaganes 299 Luc Girardin and Dominique Brodbeck, UBS, Ubilab

Mailing Lists Friday (2:00pm-3:30pm) Chair: Tim Hunter Mailman: The'GNU Marling, List!Manager jasissicsscssssodssseasosvcassdpicsnivsrsoces svachesncancascencaatenissadtncteveasteascesistiscestscbniuestvas 309

John Viega, Reliable Software Technologies; Barry Warsaw and Ken Manheimer, Corporation for National Research Initiatives

Drinking from the Fire(walls) Hose: Another Approach to Very Large Mailing Lists 2.00.0... cesssseeseeseeeseeeeeeeeeeeees 317 Strata Rose Chalup, Christine Hogan, Greg Kulosa, Bryan McDonald, and Bryan Stansell, Global Networking and Computing, Inc.

Request: v3::A: Modular, Extensible Task Tracking:To0). ssscscssssgisenssesacvsssvesvevaxsscseascadersiteasssavescavsatesascstecsneseaccorerteees 327 Joe Rhett, Navigist

1998 LISA XII December 6-11, 1998 Boston, MA iii


PROGRAM CO-CHAIRS Xev Gittler, Goldman, Sachs & Co. | Rob Kolstad, The SANS Institute

PROGRAM COMMITTEE Eric Anderson, Univ. of California, Berkeley Melissa D. Binde,

Phil Cox, NTS, Inc.

Tim Hunter, KLA-Tencor Corp.

David Kensiski, Digital Island, Inc.

Kurt J. Lidl, VUNET Technologies, Inc.

E. Scott Menter, ESM Services, Inc.

John Orthoefer, GTE Internetworking

John Sellens, VUNET Canada, Inc.

Marc Staveley, Sun Microsystems, Inc.

Ozan S. Yigit, Sun Microsystems, Inc.

READERS Strata Rose Chalup, VirtualNet Consulting

Rik Farrow, Independent Consultant

David Nochlin, UBS, Inc.

David Parter, University of Wisconsin

Gretchen Phillips, University of Buffalo

Phil Scarr, Global Networking and Computing

INVITED TALKS COORDINATORS Pat Wilson, Dartmouth College

WORK-IN-PROGRESS COORDINATOR Peg Schafer, Harvard University

GLOBAL LISA re ae el Joel Avery, Nortel, Inc.

Rob Kolstad, The SANS Institute


ADVANCED TOPICS Adam Moskovitz, Genome Therapeutics Corp. |


David Parter, University of Wisconsin

Strata Rose Chalup, VirtualNet Consulting

Rob Kolstad, The SANS Institute

Gretchen Phillips, University of Buffalo

Adam Moskovitz, Genome Therapeutics Corp.

TERMINAL ROOM COORDINATOR Lynda McGinley, University of Chicago |


Data Reproductions

USENIX ASSOCIATION Ellie Young, Executive Director

Judith F. DesHarnais, Conference Coordinator Daniel V. Klein, Tutorial Coordinator

Cynthia Deno, Marketing and Exhibitions

USENIX SUPPORT STAFF Linda Barnett, USENIX Association Cami Edwards, USENLX Association Lana Erlanson, USENLX Association

Vanessa Fonseca, USENIX Association Toni Veglia, USENIX Association

1998 LISA XII December 6-11, 1998 Boston, MA


Welcome to Boston! We’ve planned the biggest and best LISA ever. We’ve assembled a program that includes almost 30 of the best papers, the now- traditional invited talks track, and the new Practicum track with outstanding speakers and timely topics.

Special thanks to Judy DesHarnais, the USENIX conference coordinator, who has organized the entire week of facilities and amenities. Special thanks also to Dan Klein, the USENIX tutorial coordinator, who has again assembled a super set of courses with terrific instructors.

The Conference Proceedings, lovingly chosen by the Program Committee and submitted by the various speakers, dwarfs its predecessors in size. Thanks to all who participated: those who submitted abstracts, those who reviewed them, and those who wrote final papers and created presentations.

The Invited Talks were marshaled this year by Phil Scarr and Pat Wilson who threaded their way through the minefield of conflicting topics and schedules to create a great track.

The new Practicum committee invited a slightly different style of speakers in the very pragmatic mode for a third track. We can’t help but believe that this track will succeed stupendously.

Of course, this conference would be happening under a tent in a field without the incredible efforts of the USENIX Office. We probably wouldn’t even have the tent if it were not for the efforts of folks from both the Conference and Administrative offices. Thanks to them!

Thanks for coming to Boston! See you in the hallways!

Xev Gittler Rob Kolstad

1998 LISA XII December 6-11, 1998 Boston, MA



DORAL A AUS os vcicacsoncrtvxncnccetvsatonceasionssns 197 Mukul Agarwal) cucnestsusaneeuncest 167 Mukul. Agarwal ssscsscssecsccssctecesaseiacsersscezees 271 Matthew Archibald oo... ccc eeeeeseereeeees l Bob Amol? sisosmennecotatecencennnd 49 BS Gi Sievwsreevies PscsisinisiestheateteaserretereMenes 175 BL PREY shots Sesensnctivanciaatpaptecieciastataans 167 Syed Babar) st vivotettrtaisascsseesise 8s 271 Rosalie Belian ..........ccccceeeesseeseeeseeseeeees 121 Dominique Brogbeck: scicsssesssscsssspscsevssasvas 299 Mark UrPesS iciccsescvedecevessesexcsenceuseasceveeven 283 Thomas: Bushell scccnssspepceseectsennsceaes 175 NOE Carpentets, .secsssicesscvcatsicispleanpctounarencancns 11 ROD Camtel os: csacss ocescasncssnnsccadsnspacasncssedesssacs 63 Lloyd: Chad ssssissessccsatscesevemsssceraonsiscenesetees 167 WEGV.G GIA wi ccssurioerdenayoooannnnniasgadebngentedanonses 271 Strata Rose Chalup .......:csccsisesssccsssesesesees 317 Juliana Silva da Cunha ....... eee eeeeeeeees 121 Fabio:Q..B; daiSilva>sisccstecaenitesisneteesess 109 Fabio Q. B. da Silva .......cceseeseesseeeeeseeees 121 EVan Deloose ues, athe ssccssttacteeescteses eee? 229 EdwinS, Hichett: ccc scssstoseeetnseatscasecescars 149 Dat Pariniehy cicsissscisesseas caesisacsatysseronadvscapatseents 1 Bil PHRGn: cancnsmcnnciauenmaan 11 DaVIGFOSUEE ciccacscsivesiseciavescntsessseensiddonsoeas 229 Danielle M. Franklin. .........:cscssce0ceesseseeees 121 Duie Girardin ccccsccesncticsowuniesavsvcnnsenwetess 299 Michael Fleming Grubb ............. es eeeeeeeees 63 Christine Hogan scciscsssaxesinssansssmsensccacovees a7 Joel Huddleston 0... ceeeeseeseeseeeeeeeeeees 181 Stéve KalinowSsky sci.cssseses cosateavexcevezecescesesz 11 GOS KGS easseekeiecoveicnuitanstecsasesaceenienes 317 Jason Lark? ssccmscvcnnvecninnncaura 261 John Locka vecsccvssvasesaccenscssaevenavsavevssaxceee 261 IMEX, EMO sn acsescacsanscenssnnndubvanCeonvanieurauranies 131

MAC KSINI AN Scabescs eeveoxises actesenscwcrubseeeestatan eens 167 MACK INIA, sects sotiices oh c5..55 casi sectisivedesdessecadesades 271 Ken: Manheime rf wsiccsevsccsssssecsrerrsevaseceeers 309 UStVaN: Mark .......;..0creecsnsennsncosvsnsesterecesees 167 IstvaniMarko: ssserissciasnscaiesarsviss 271 Bryan McDonald .iscdscssitetscvrasdcrvecoseve 317 VO Ti MCC ci N te oh she coveceecdarenlesaccatoord 149 Ajay Mohindra, 2 ccsccrnnssescummtinnrenenss 37 Chris Motta Bi a.. cisicdeskescseeeseiothsthacedtvcetes 167 ChiisiMotta. 2. xcncnsremeeueeanae 271 Michael Mulvaney ...........ccccscesseseseeeseeees 197 TOBIAS! Oetiker: exissFccen semen ks 141 LObIS OcHker. .ccivcscexcevescrivernereiassscswctvees 253 CUMSTO PHOT RR: PARE scnitesassecenccnnsteaicnsconnd 97 DOUPickel] sitsrsscstevcesnsessvsereuevsseeatuescastetecaertys 1] Brad POWEll)<......c:-sscsseaceuesensesensSyesensaeceonnabens ] W. Guttis: Preston cacsscesmernncdiieccsnchics 87 GITAY. PUNAE: cwisccosvssnasecasenteussscteloedvaceenseads 159 Apratim Purakayastha ............cscsseeseeteeees 37 eat) Ra iisvccscccaseserteecdiloanscoMrtreseetods 175 lonacio: Repuer®) ssscsdecuasscattemeriess 229 NOIRE acornccttenss doccdensaescgelt eva os enone 327 Waseeiin SHAN’ ......0:-scccsecesessnsesescedasesteaee 167 Waseem Shaikh ccssscsssesseccscesssszessssevenves 271 Glédson Elias da Silveira ........... cece 109 Garl'Stae lit ccrsnesseescsesseviasanssecnzeni 243 ASE VAD SIRGBELS sicescrsmarctneninceccncnsthincinnias 317 Kitty TaRAyaie exssesccnavsecnnearonunans 149 Christopher Thorpe? cccceccssessssseecessestsivessveoses 27 SLEVE TEAM BORE, ...:.<-ccesecsdsnessecaseessnsesisotetiene 181 Luciana'Ss Varejao secs ceases 121 WORM; VIE BA sass nistnssrsnaslenosonsiuncescoanssausbaatnets 309 Batry: Warsaw sssscsnwascivssaccncseceess 309 Ben W000 a8 inccsssncviccssascscnccsvesvscsntvancerdoves 219 OcS. AWS acacsunscteaecarnain 131

1998 LISA XII December 6-11, 1998 Boston, MA


Dan Farmer Earthlink Network Brad Powell Sun Microsystems, Inc. Matthew Archibald KLA-Tencor


Titan is a freely available host-based security tool that can be used to improve or audit the security of a UNIX system. It was written almost completely in Bourne shell, with a master script controlling the execution of many smaller programs. Each of the programs either fixes or detects potential security problem, and its simple and extremely modular design also makes it useful to help check or enforce the adherence of a system against. its security policy. Finally, anyone who can write a shell script or program can easily create their own Titan modules.

Titan does not replace other security tools, nor does it fix or patch security bugs; its primary purpose is to improve the security of the system it runs on by codifying as many security tricks to secure an OS that the authors could think of. And when used in combination with other security tools it can help make the transformation of an “out of the box” system into a firewall or security conscious system a significantly easier task.

NOTE: Due to time, resource, and expertise limitations, the first release of Titan is only known to run on Solaris Operating Systems, versions Solaris 2.x and Solaris 1.x. However, many of the small sub-programs within Titan work well with other UNIX’s, and other than taking the time to create Titan modules for them, there is nothing Sun specific about Titan that would prevent it working on other UNIX systems.


UNIX is often, and justifiably, criticized for being a difficult system to administer because it is not only complex and cantankerous but hard to secure. Its enormous configurability, the fact that vendors don’t ship secure systems, and that it requires significant amounts of time, resources, and expertise to safeguard a host are only some of the reasons that so many UNIX systems are insecure on the Internet. To com- pound the problem, like all modern operating systems it not only becomes less secure as time goes on (sim- ply due to usage), but with the rapidly changing secu- rity field, it also requires considerable effort to stay abreast of the latest information time that most sys- tem administrators simply don’t have.

Titan tries to provide at least a partial solution to all these problems by trying to locate and fix many of the more common procedural problems that crop up, as well as put into one place all those damn OS tweaks that can assist in securing your system. Titan improves the security of a system by:

¢ Cutting off entry points into the system.

° Mitigating or preventing the effects of various denial of service (DOS) attacks.

¢ Turning on or improving the level of logging and auditing features.

¢ Improving network and local (e.g., host level) defenses.

e Assisting in programmatically defining and enforcing a system security policy.

1998 LISA XII December 6-11, 1998 Boston, MA

It is important to note that Titan’s focus is the correction of procedural problems. While it can be used as an adjunct to other auditing tools, whether host or network based, it does not attempt to find problems that it cannot correct. An automated tool that changed weak passwords, unpatched or insecure sys- tem binaries, and unrestricted filesystem mounts, for instance, could break or disrupt operations to an unac- ceptable level. Like most other security tools, Titan is not meant to be used only once: to achieve effective security requires an ongoing concern and continued attention to good security practices. Any competent system administrator should have considered, if not resolved or repaired, nearly all of the problems that Titan addresses on their security critical systems.

Anyone working in security or systems adminis- tration who has been around the Internet for any length of time has done it —- making the same changes, over and over again, to secure a system. Worse yet, each new OS release brings tiny, seemingly arbitrary changes that can invalidate prior work. And forget it when a major new release comes out, or you have to work with another operating system altogether! Just among the authors of Titan we’ve ftp’d Crack, COPS, and other security programs from the net thousands of times and we’re sure we’re not alone.

The analysis of the security of a system is depressing the same sets of problems always come up. But what’s worse is that these problems can almost always be easily fixed so why aren’t they? And the final sling of indignity is that vendors keep changing


the damn commands to do the same things, even within the same major version of the OS (what are the arguments to ndd(1M) that change that TCP behav- ior?) And it’s certainly not only Sun it’s DEC, it’s HP, it’s IBM, it’s everyone that has even a mildly complex system. Yes, even Microsoft.

So why do these same problems show up over and over, regardless of the supplier? Good question! We don’t know the answer, but what we do know is that having a tool to help ensure your system’s consis- tency is a very positive step in the right direction. Hence Titan.

Titan’s main design goals are:

e Security comes first. We can mandate that for this tool, Security comes first. After Titan has been run on a system it should be more secure than before and there will be no remaining sig- nificant host level security problems that we know of, other than those involving vendor OS and independent daemon security issues and patches (e.g., if the system is a WWW server, CGI or CGI-like interfaces could be problem- atic). The system will not be 100% secure none are but it will be pretty darn secure, especially after applying security patches. Due to customer pressure vendors can’t take the chance with their patches and system releases that things will break but we can. In our test- ing and use over the years we haven’t run into a single thing that Titan has irreparably broken, but it certainly could happen.

e Easy to use. Titan should be simple to install

and run. While knowledge about the system

will always help, you can trust Titan to do the right thing in most cases.

Policy based. Titan can assist in the creation of

a programmatically defined technical system or

site security policy. Classes or types of security

(such as firewall, desktop, etc.) are simple to

define and apply to the appropriate system, and

help produce a consistently secure system in ways that are readily comprehensible.

e Freely available source code. In security it is imperative to have complete source code avail- ability. Having full control over what is run and possessing the potential for total understanding of exactly what actions it performs is manda- tory for a truly effective security tool.

e Modular. Titan is non-monolithic and easily

extended. Shell scripts or other programs can

either be taken out of or added into Titan’s framework. Doing so will not affect the other programs.

Useful. Quite simply, we’ve found Titan to be

enormously handy and something that can be

used quite frequently if security is a significant concern for you or your systems.

We’re often asked how to tighten down the OS when a firewall product gets installed. There is a

Farmer, Powell, and Archibald

reasonable expectation from the customer that after the firewall is installed, the system will not be com- promised by an attack that is outside the scope of the firewall product. After all, aren’t firewalls supposed to protect you? You wouldn’t say it was safe to run a business on the Internet unless you could protect it, would you? Unfortunately most people don’t know what security is, and the firewall sales people are not going to help.

And it really is unreasonable to expect the user, a customer, to understand all, or even most of the secu- rity issues of running a system on the Internet. Why should they? However, this does place both the spe- cific firewall vendor and security people in general in a rather awkward situation. Indeed, what probably scares firewall vendors more than anything else is the fact that firewalls are failing because users or adminis- trators don’t fix, remove, or upgrade old versions of a potentially vulnerable network service.

Titan Does Not...

It is important to note that there are several pro- cedural security problems that Titan does not attempt to fix or seal off. CGI programs, often deployed on WWW servers, are becoming one of the more widespread security problems on the Internet, and are nearly impossible to programmatically detect or pre- vent. Titan also does not render a system impervious to breakins not only are inside attacks common, but new bugs and holes are constantly being found. Remember there is an arms race going on out there.

In addition, Titan does not address the problems of secure software distribution or updates. This means that Titan is probably not a viable tool to secure all systems on a large network due to the administrative costs involved in setting up and maintaining all the copies of Titan. While NFS, rdist, or other methods of software replication and deployment could be used, we would warn that the inherent security risks of such methods, as well as the myriad dangers of controlling an organization from a central location probably out- weigh the benefits in most situations.

Using Titan

Using Titan is fairly simple, but we want to reit- erate it cannot secure a system, nor can it fix prob- lems that will inevitably arise unless you continue to run it. We suggest that you employ the following sequence when first utilizing Titan:

¢ Read the Titan documentation and look at the programs. Does it do what you want? Does it fit into your security policy?

e Examine or secure your system using your nor- mal set of tools and procedures. Note any flaws.

e Back up your system.

e Run Titan.

e Examine your system again. Are the flaws

1998 LISA XII - December 6-11, 1998 Boston, MA

Farmer, Powell, and Archibald

gone? Are new ones there? Does everything still work?

Install strong authentication on your system, at least for remote logins. Anyone using the same reusable password in cleartext over the Internet except in an emergency is a fool.

Continually monitor your system, running Titan and other security tools as well as applying security patches as necessary or as your secu- rity policy changes over time.

Report any problems found with Titan to us so we can fix them in subsequent releases!

After the initial use of Titan we suggest running it in the verify mode at least once a week. You can run Titan from cron in the fix mode, but since it can affect your system drastically we would suggest being cau- tious about doing so (in addition, if you run the Trip- wire integrity checking tool, it will complain vigor- ously about all the files Titan changes). In addition, we highly recommend that stronger authentication (such as with the Logdaemon package or hardware methods) be installed and utilized on the system. When data is traversing the network, strong encryp- tion should be used, if possible, in addition to the extra authentication.

If Titan does the majority of things that you per- sonally do to secure your systems but misses a few points, you might consider writing some additional Titan modules to perform the tasks. We would also ask you to send us email about this if security related, we would certainly consider including your module in the general Titan release or rewriting it ourselves.

A Case Study

Lucinda Williams is a ten year veteran of system administration and security on the Internet, and has been recently appointed chief security officer for the medical center of her alma mater, Evil University (Evil U, aka After modifying Evil U’s gen- eral security policy to fit in with the needs of her con- stituents, she has started to implement the technical aspects of the program. Since the university is strapped for cash, the firewall (an Ultra running Solaris 2.6) must also serve as a WWW server. Here are the steps she takes to create and secure the medical center’s firewall:

e A new system is installed with the absolute minimum number of options required to run the system, which lives on its own subnet to pre- vent local packet sniffing. Immediately, inetd is (temporarily) disabled to help ward off intrud- ers attacking the system before it has been properly secured. As soon as she downloads all the security tools and files needed to install the system, it is physically disconnected from the network. GCC (the GNU C compiler) is also installed so that various security tools can be compiled. (She could alternately compile them on another system that is known to be

1998 LISA XII December 6-11, 1998 Boston, MA


uncompromised and ftp the binaries over, but

it’s safer to compile them locally to help ensure

that they have not been tampered with.) The

Apache httpd server is installed because of its

good security and source code availability. The

most recent version of sendmail is then put into place.

The packet screen is the first defensive tool set

up. Almost without exception, any critical sys-

tem that is not protected by a screening router, proxy firewall, or both (or, less ideally, a pro- gram such as screend or IP Filter that duplicates this function) has not been adequately pro- tected. While not sufficient to secure a system or network by itself, it is a necessary part of any security solution. Under most circum- stances the router shouldn’t have to allow more than a half-dozen ports or so from the outside world. DNS, SMTP, http, telnet, and some

ICMP is all Lucinda allows, although she is

forced by Evil U’s policy to allow NFS, Net-

BIOS, finger, and ftp to the rest of the Univer-


At the time of this writing, ftp://sunsolvel. contains all the Sun OS

and program patches (including many security

fixes). Sun also provides a description of and a

tar file containing all their recommended

patches for their released OS’s, which Lucinda ftp’s and installs. This is done before running

Titan, since the patches might undo some of the

system modifications Titan performs.

e She has previously checked Titan against her firewall security policy, and has had to make a few small changes:

e The issues file needed revision to reflect

Evil U’s administrative policies.

e She needs to allow root ftp access (a

truly abysmal idea, but one required by university policy), so in the Titan module, she removes root from the $DEFFTPUSERS variable. The university has a customized (and mandatory) compiled C program that all systems must run via the /etc/aliases file for inventory purposes. She has several choices here she can modify the Titan module ( that doesn’t like mail aliases that point to a program, ignore the warnings, or not run the alias module at all. The last can be accomplished either by creating a Titan policy file with all the modules that she does want to run, or by simply moving the shell script out of the modules directory.

e She runs Titan with the -f flag to fix all the problems it detects, and then installs Titan in cron to run with the -v flag once per week.

e Logdaemon despite being vulnerable to


session hijacking and eavesdropping is used to improve the authentication of all users instead of the popular but much more complex and potentially dangerous ssh. All accounts have their normal UNIX reusable password dis- abled.

Next she runs Tiger and/or COPS, fixes any problems found, and creates a cron job to run the tool once per day, mailing the results to her. Logging tools are then set in place next. The TCP and portmapper wrappers and swatch are all installed, with syslog sending information both locally and to a central server, and further- more any critical events are mailed to Lucinda’s pager.

As the final step in the setup process, she removes GCC and makes a full backup of the system, storing it off-site.

The system is now ready to run. She’ll test the initial security with a remote security scanner that is run on the outside of her domain (and hopefully out- side the university). Any of the widely available pro- grams such as SATAN, ISS, SAINT, or CyberGuard would work, depending on her familiarity with these and her budget. Initially, the port scanner is the most important thing to run. She also subscribes to the Bug- Traq, Sun security advisory, and CERT mailing lists, and will keep a close eye on the system logs and activ- ity. She has also created an addendum to her local security policy that will require any and all CGI pro- grams to be audited and personally approved by her as well as being placed in sbox (a CGI safety wrapper). If all this is done, the system shouldn’t take too much time to set up and continue to run, and should be a very secure system. The rest of Evil U is perhaps her largest security concern, since they have significant access to the rest of the network she maintains, but there is little she can do but use the TCP and other wrappers and auditing tools to watch the traffic.

NFR, tcpdump, or other packet watching tools can be potentially marvelous tools, but do require a significant time investiture to run effectively. The widespread availability of very inexpensive large high speed disks (to save the voluminous audit data) does make the process more viable, however.

Titan Features

Although Titan has been a dynamic system, con- tinually adding features and additional fixes or tests, we feel it important to cover some of the more inter- esting tests or features. Code fragments will frequently be given, but with any of the problems listed below, looking at the source code of the corresponding Titan sub-program can be illuminating.

The following sections discuss some of the changes that Titan makes to a system. However, any list we could create will be out of date fairly soon the complete and up to date list can be found at the

Farmer, Powell, and Archibald

online Titan documentation. Since Solaris 2.x contains several ways to improve a host’s security that earlier versions of Solaris did not, we naturally have more Titan modules for it and generally recommend run- ning it or a similar system if security is a concern.

Kernel Level Configuration

Why is it that almost every proxy firewall we see has ip_forward_src_routed enabled? Source routing and other such options may have their uses, and at one time were fine ideas, but they do not belong in the world of Internet security, unless you’re trying to cir- cumnavigate it. Tools to abuse and bypass systems that aren’t sewn up tightly proliferate on the Internet.

Most modern UNIX’s allow a great deal of ker- nel tuning from the command line. Solaris, for instance, has ndd(1M), which can get and set configu- ration parameters in TCP kernel drivers. Putting them in the /etc/system file makes the change take effect at boot time. Titan closes various kernel and TCP/IP pro- tocol holes that we’re aware of, including:

¢ Fixing the stack. Ever since Alephl’s pivotal paper detailing how to exploit buffer overflows, stack smashing programs have perhaps become the most common type of exploited coding error. Solaris allows the kernel stack to be non executable; it adds the following entry into /etc/system so that zero-fill-on-demand pages are marked rw- instead of rwx: ¢ Don’t allow executing code on the stack

set noexec_user_stack et e And log it when it happens. set noexec_user_stack_log = 1

NFS bind. Titan sets the privileged port defini- tion to all ports above 2050. NFS, which uses UDP port 2049, has been historically set in an unsafe port range. If you want to protect other services above this range, simply change this parameter. ndd -set /dev/udp \ udp_smallest_nonpriv_port 2050 ndd -set /dev/tcp \ tep_smallest_nonpriv_port 2050

SYN time-out. A good example of Titan’s use as a short-term workaround until a security patch has been disseminated by the vendor. This script was produced from a CERT advi- sory and placed into a Titan module to run on all local systems to reset system parameters to a safer level until the vendor was able to produce a permanent fix (still useful on older systems!): ndd -set \ /dev/tcp tcep_ip_abort_cinterval \ 10000 echo "tcp_param_arr+14/W 0t10240" | \ adb -kw /dev/ksyms /dev/mem ndd -set /dev/tcp tep_conn_req_max 8192

1998 LISA XII December 6-11, 1998 Boston, MA

Farmer, Powell, and Archibald

e Ping echo. Titan can set up your system so that it does not respond to broadcast ping requests. Why is this important? Attackers often use a ping flood as a DOS attack. In addition, by turning off response to broadcast echo it makes it more difficult for potential attackers to probe our system by sending a ping -s to the broad- cast network address:

ndd -set /dev/ip \ ip_respond_to_echo_broadcast 0

Startup files

/etc/rc.* , /etc/rc?.d/*, etc. The re shell scripts are full of services which startup at boot time that you may not be aware of. Titan will disable services that can potentially be used to gather system information remotely or aid a potential intruder in an attack this includes the auto- mounter, the dmi, Ipsched, snmpdx, and other daemons. Titan disables these services by either commenting out the services or by sim- ply moving the files from the rc* directories.

Configuration files Titan enables the privacy flags that were introduced in sendmail version 8 with the ““goaway”’ option (among other things this dis- ables VRFY and EXPN), as well as setting the sendmail logging to a reasonable level:

Opgoaway O LogLevel=5

inetd.conf. Titan tears out many of the default services in the Internet services daemon’s con- figuration file. Most of the daemons installed by default are too chatty, historical sources of system vulnerabilities, and operationally unnec- essary. Any inetd service that isn’t protected by using tcp_wrappers or otherwise restricted and logged (and/or encrypted) is inherently inse- cure. You should view any program that talks to the network with grave suspicion.

ftpusers. If the file /etc/ftpusers exists and has users names listed in it, then those users are not allowed to use ftp. Titan adds in system users such as “bin”, “Ip”, “root’’, and others. nsswitch.conf. The contents of this file can be as dangerous as having a “+” in your /etc/hosts.equiv. Having an “nis”

nis” or “nis+” entry in this file gives control of crucial files that your system trusts to a remote system. Titan takes the approach that if the local host doesn’t know about a remote system, then that remote system can be a threat. Titan errs on the side of safety and simply builds a minimal /etc/nsswitch.conf file using the /etc/nss- witch.files sample file as a starting point for you to build upon, if necessary.

e syslog.conf. Titan modifies /etc/syslog.conf so

1998 LISA XII December 6-11, 1998 Boston, MA


that console auth notice messages also get logged to a file.

File and Directory Permissions

System umask. Titan forces the system (root) to use a default file creation mask (022) that is more secure than the default. This forces all the system daemons to create files with saner file permissions.

System files and directories. One of the oldest (and still one of the easiest) ways of bypassing system security is to find a directory or a binary file that a privileged user (root most often) is going to access or execute. If that file or the directory that the file/binary lives in is modifi- able by another user, then that user can gain additional privileges. Because of the great num- ber of potential problems and different security models for different types of systems (firewalls, servers, desktops, etc.), Titan has three modules that each repair or change one or more aspects of this. All of them can be run for maximum security.

General System Configuration

eeprom. One of the few things that Titan simply checks and doesn’t fix is if you have “‘security- mode” set in your EEPROM (we don’t fix this because you have to choose a password your- self). Don’t let us say “we told you so”! If you don’t set your EEPROM password, someone else may set it for you and halt your system. Then you’ll need a new EEPROM (or many of them, if they break into multiple systems!), which can take a significant amount of time, especially if you’re running an older or discon- tinued architecture.

vold. Vold(1M) may seem innocuous, but let- ting users mount file systems without being root doesn’t sound like a good thing to us, even if the Sun vold doesn’t allow SUID root shells on the file system being mounted. You might consider allowing this on desktops, but cer- tainly not on servers or firewalls.

Passwords and Authentication

/etc/passwd. Titan deletes or disables system accounts that are never (or should never be) logged into. Any user or system accounts left on the system have passwords or are disabled, and system accounts other than root and sys (which starts accounting) also have a special non-interactive shell put in place.

Network Information Services. Titan disables NIS, NIS+, and DNS for name resolution from /etc/nsswitch.conf. While all of these network naming services are insecure, you might have to enable DNS, although we suggest keeping it off whenever possible. Never run NIS or NIS+


on a secure system or it won’t be.

¢ loginlog. Titan creates /var/adm/loginlog so that the system will log more than five failed login attempts. (This doesn’t work with all services (telnet for example).

Titan and Your Security Policy

Titan creates an /etc/issue file with a dire warn- ing to stay away from your system. The contents of this file appear in front of the login prompt. By default it contains:


This system is for the use of

# authorized users only. Individuals i # using this computer system without # i authority, or in excess of their i # authority, are subject to having # # all of their activities on this i # system monitored and recorded by # | system personnel. it # In the course of monitoring individuals i # improperly using this system, or in # ## the course of system maintenance, # # the activities of authorized users i# ## may also be monitored. i# # # # Anyone using this system expressly i f consents to such monitoring and is # ## advised that if such monitoring #

## reveals possible evidence of criminal } # activity, system personnel may

## provide the evidence ao monitoring # # to law enforcement of


Although we strongly suggest running all (or nearly all) the modules in Titan, we realize that not everyone can afford such strident security measures. Titan thus allows you to run different sets of modules as desired by using a simple configuration file. This configuration, or policy file, is a standard UNIX-style configuration file that uses pound signs (“#”) for comments and contains one Titan module (with any arguments desired) per line. We include two sample files, for potential use on desktop and server systems.

Note that the desktop policy disables send- mail(8). Since firewalls often deliver or forward mail, this is an optional Titan module. Desktops and most servers have no business running sendmail, however.


As previously mentioned, Titan is a master script that runs a collection of Bourne shell scripts. How- ever, before getting any results or fixes from Titan, you must first run the Configure