Go to Database Directory || Go to Bibliography


Published in 91 Law Library Journal (1999) 9-27. Reproduced with permission of the author.

Evaluation Criteria and Quality Control for Legal
Knowledge Systems on the Internet: A Case Study
[*]

Marie Stefanini Newman [**]

Professor Newman discusses evaluation criteria for law-oriented Internet sites and how to use these criteria when launching new sites or improving existing sites. She also discusses the use of quality control procedures to ensure accuracy and reliability in Internet sites, and concludes with a case study of the Pace University School of Law's Web site on the United Nations Convention on Contracts for the International Sale of Goods (CISG).

Publishing on the Internet
-    Scholarly Internet Publishing
-    Media that Review Internet Sites
Criteria for Evaluating and Creating Internet Sites
-    Content
-    Accuracy of the Site
-    Authorship of the Site
-    Currency of the Site
-    Uniqueness of the Site
-    Design of the Site
-    Searching the Site
Resources Necessary to Create and Maintain an Internet Site
An Illustrative Application - The Pace CISG Site
-    Introduction to the CISG Site
-    Practicing the Principles of Good Web Design

1. Start with quality content
2. Make sure the content is accurate
3. Make sure authors are known and accountable
4. Keep your cite up-to-date and publicize its currency
5. Make your database unique
6. Organize your material
7. Make the database design user-friendly
8. Provide powerful search tools
Companion Internet Databases
Conclusion

Publishing on the Internet

Scholarly Internet Publishing

The last few years have seen an explosion of information available on the Internet. Anyone with access to a server and some knowledge of HTML can create an Internet site and become a "publisher."[1] Colleges, universities, other scholarly institutions, and individual scholars are taking advantage of the Internet to publish original materials in a cost-effective and timely manner. Universities have set up some of the highest quality sites available on the Internet.[2] Universities bring significant assets to Internet publishing, including faculty authors with expertise [page 9] in their fields, computer specialists, and librarians with the skills and knowledge to organize and index information.

Many law schools have become involved in publishing materials on the Internet. Examples of important primary sources accessible at sites maintained by law schools are the decisions of the United States Supreme Court [3] and all the Circuit Courts of Appeals;[4] the Code of Federal Regulations and the United States Code;[5]and analogous materials on the state level.[6]

Coexisting with such valuable resources on the Internet are low-quality sites of questionable provenance and uncertain authority. The information they contain is raw and unfiltered and has not been subjected to peer review or editorial evaluation.[7] Some sites include actual misinformation.[8] What makes these sites particularly dangerous is their potential to mislead many users, especially unsophisticated researchers who think that information found online is necessarily accurate. "Novice network users -- like many television viewers -- may easily fall into the trap of taking everything they read, see, or retrieve from the Net as THE TRUTH."[9]

Media That Review Internet Sites

Professional journals, which have traditionally published reviews of print but more recently have also covered electronic materials, now publish formal reviews of Web sites. For example, Library Journal at now has a monthly column called "Web Watch." There are also a growing number of Web sites that purport to evaluate and grade other Web sites. [page 10] Some of the best known are the Argus Clearinghouse;[10] the Lycos Top 5%;[11] the Encyclopaedia Britannica Internet Guide;[12] and the Magellan Internet Guide.[13] Unfortunately, the reviews appearing in these and other sites rarely adhere to articulated standards, such as those that have evolved over time to evaluate print media. "[M]ost current 'web reviewing' sources ... do not really provide reviews at all. Instead, they describe and rate web sites according to widely varying and idiosyncratic criteria which seem to be mostly ingenious variations on the concept of 'cool.' "[14] Such reviews tend not to be very useful for the researcher who needs legal materials from the Internet because a legal site's contents are more important than its appearance, and most Internet "reviews" do not actually evaluate contents.

There is another limitation of these reviews that is unique to the Internet. It is impossible to evaluate more than a small percentage of the hundreds of thousands of sites already mounted on the World Wide Web. It is equally impossible to monitor and review the thousands of new Internet sites that come online every week.[15]

Given the dearth of truly worthwhile reviews of Internet resources, the researcher is left to make his or her own judgment about the usefulness of a site.[16] As with traditional research using print resources, an important component of the Internet research process is weighing the merits of the sources you have found and deciding which are reliable based on your own knowledge of the subject.

Criteria for Evaluating and Creating Legal Internet Sites

Because researchers should regard Internet sites skeptically, those who would create new sites or redesign existing ones should design them to pass muster by the most skeptical researcher.[17] [page 11] The criteria you should use in designing and managing your legal information Web site are the same criteria you would apply if you were evaluating its usefulness and reliability as a researcher encountering the site for the first time.[18] A recent article recommends "evaluating Web sites on the basis of user-centered principles," because sites that "violate principles of sound design ... waste the time of Web surfers."[19] Principles of sound design include: creating a focused site that has a clear purpose and audience in mind; designing an initial screen with complete information about the site's purpose and content; making it easy for the researcher to tell who is responsible for the site and when it was last updated; being careful that all design features are justified by their capacity to make the site easier to navigate; choosing to link to a few authoritative sites rather than to all sites that are potentially relevant; and providing high-quality content that is appropriate for the Web as a medium.[20]

Content

Although many different criteria are applied to evaluate Internet sites, the most important is substantive content. It is the first thing a researcher examines when evaluating a site, and is the single most important component. Therefore, creating content that will meet your users' needs should be the first issue addressed when creating an Internet site. The designers of the most successful sites "understand the kinds of information the users typically use, perhaps some specific sources, the form of these sources, the strategies users employ to locate them, and the problems they experience in looking for information they need."[21]

A site may contain original information or it may be a set of links to other sites. It may be a combination of original information and links to other sites. It may contain primary or secondary sources, or both. The information may be factual, or it may represent the opinion of a personal or institutional author. Regardless of the type of information you provide, make sure that it is of the highest quality. This can be difficult, but no less important, when the information at the end of your link is on a site beyond your control.

Accuracy of the Site

Researchers who use your site will assess its accuracy by comparing it to other reference works; they will also evaluate it based on their own accumulated knowledge about the subject. [page 12] Any organization that publishes substantive material on the Internet should employ a subject specialist not only to write original material for the database, but also to check the work of other contributors and guarantee its accuracy and quality. It should also employ an editor or database manager to implement quality control procedures to ensure that there are no obvious mistakes such as typographical, grammatical, or spelling errors. The writing should be of the same quality that a researcher would expect to find in a scholarly treatise or law review article. Sloppy grammar, spelling, and writing may indicate that staff responsible for those errors are casual about the content too,[22] and that quality control is not considered important. All documents, especially scanned documents that have gone through an OCR translation process, should be subjected to a rigorous review before they are mounted on the Internet.[23]

Researchers evaluating your site for accuracy will also look for apparent biases. If a site seems to be promoting a particular political or social agenda,[24] the information it contains may be of questionable utility to those who do not share that agenda.

Authorship of the Site

Because researchers who access your site need to know who is responsible for the information, make sure that the names of both personal and institutional authors and other Web site personnel are provided on the home page. Authors' credentials and information about how to contact them should also be easily accessible, perhaps on a page linked to the home page. If outside contributors provide materials for the Internet site, their names should be listed and their credentials provided, as in a law review article. Make this information easily accessible so that researchers can judge for themselves if the author can be relied upon. Concealment of this information about yourself or other contributors may lead cynical evaluators to suspect a hidden bias.

It is also useful to provide the names and contact information of the staff members who maintain the site, including the Web master. These are the individuals researchers will need to contact for technical questions or to report problems such as stale links. [page 13]

Currency of the Site

Researchers will look to see if the information on the Web site is up to date.[25] Ensure that researchers can easily determine how current each component of your site is. Make it clear that the site is being maintained on a consistent basis, and be sure to check for stale links [26] at least once a month and correct any problems immediately.

Uniqueness of the Site

Legal publishers market products that may be duplicative of their competitors' products to a greater or lesser extent.[27] Likewise, institutions sometimes choose to duplicate materials that are already available at another site.[28] However, there is no point in creating a new Internet site if there are already other sites that cover the same topic well. On the other hand, the more difficult materials are to locate, the more useful your site will be if you gather them together and make them available over the Internet.[29]

"Since so many other organizations are providing good Internet collections to the world's library patrons free of charge, libraries must ask why they should duplicate these efforts. ... An academic library that wants to provide an Internet collection should consider picking a single subject area of expertise and focus on making that subject-specific collection the best one available."[30]

The most successful legal sites are more than electronic versions of paper texts and "incorporate features unique to the basic properties of the web,"[31] namely hypertext links to cases, statutes, and other legal materials.[32] [page 14] The use of hypertext links improves on the text [33] and makes it significantly more useful to researchers.

Design of the Site

Because Internet users and evaluators give high marks to sites that have a clean, attractive design and are easy to navigate, make the arrangement of information logical and intuitive and as simple as possible.[34] Consider using a technophobe to evaluate the navigational links and user-friendliness of your site.

Graphics should be chosen not so much to make a site more attractive, but so as to play a more important role by "deliver[ing] essential parts of the site's content, or act[ing] as visual cues to orient the user within a site that is large or subtly organized."[35] Some sites are distinguished more by a gratuitous use of graphics than by high-quality content. Overreliance on purposeless graphics is particularly distracting in legal sites because the law is a highly text-oriented discipline [36] and legal publishing has historically made limited use of graphics. "[T]he textual nature of the material means that very little of practical utility is added by the use of graphical or animated material."[37]

There is an additional reason to design Internet sites to be sparing of graphics. Because a graphics-intensive site takes more time to load than a site with fewer graphics, you should consider providing a text-only alternative for researchers using a slow connection. One manual on Web design suggests designing for the "lowest common denominator," so that pages will be functional for those without the technical capability to view graphics efficiently.[38] Finally, if you use graphics, be considerate of persons with visual disabilities who use screen readers to glean information from the Internet. For each graphic, provide a caption that pops up only when the cursor lands on it, briefly describing the graphic in question.[39]

Before making their sites available to the public, Web designers should view them using both the Netscape and Microsoft Explorer browsers to get a feel for what users will experience when they access the sites. [page 15] Make sure that the substantive content of your site is accessible within three mouse clicks of the home page.[40] Admittedly, it becomes more difficult to adhere to that principle as the content of databases becomes richer.

Another way to make the site attractive and easy to use is to keep formats consistent. Just as a printed book is usually printed in one typeface, a Web site should settle on one font for a polished, professional look. It is very helpful to the researcher if the links to the home page always look the same and always appear in the same spot on all pages. This type of consistency helps to orient the researcher by providing visual cues, which, in this case, might appropriately be graphical. If a Web site will include a large volume of a particular category of materials, such as the full text of cases, it will be helpful to create a template at the very beginning of the process; the template will make it easier for the HTML technicians to process the material and ensure that the presentations are consistent in format.

Searching the Site

Some sites have fairly rudimentary searching ability, while the more sophisticated sites offer full-text searching, field searching, or both in order to make retrieval more precise. Proximity searching, use of Boolean commands, and phrase searching are highly desirable for sites offering legal information because so many researchers who access legal sites are experienced LEXIS-NEXIS and Westlaw users and have grown to expect such search capabilities everywhere. Such researchers will also expect search results to be displayed logically and to have retrieved materials only one mouse click away, so these features should also be standard at legal Web sites.[41] Researchers should be offered online help such as detailed instructions on search protocols and an e-mail address or phone number of a person who can be contacted in the event of difficulty.

Resources Necessary to Create and Maintain an Internet Site

Before deciding what the focus of a site should be, consider the resources of your institution and set realistic goals. Faculty involvement with and contributions to the substantive content of the site are essential. A curricular focus on the same area of law as your site will help ensure a pool of faculty specialists as well as a group of students who may be highly motivated to work on the site. [page 16] Creating a database in an area where there is a pre-existing library collection will make it easier to verify the accuracy of materials written for the site.[42]

Personnel and financial resources must be considered too. The issue of who is going to be responsible for an Internet site usually arises early in discussions about whether a law school should undertake such a project. In some schools, the law library assumes responsibility for creating and maintaining that school's Internet presence. In other schools, the computing department may have the responsibility. It is probably fair to say, however, that few schools are so well endowed with personnel that they can afford to take one or two people away from their other tasks to concentrate exclusively on Internet site creation and maintenance. This means that the people assigned to work on the Internet site either do so in addition to their regular duties or have some of their duties transferred to other individuals; neither is an acceptable long-term solution. If finances permit, it is obviously preferable to have one or several staff members who devote themselves full-time to work on the Internet site. An institution must have the resources not only to mount the original site but also to maintain it once it is made available over the Internet; grants are more likely to be available to defray some or all of the start-up expenses of an Internet site than to cover the costs of ongoing maintenance.

An Illustrative Application -- The Pace CISG Site

Introduction to the CISG Site

The United Nations Convention on Contracts for the International Sale of Goods (CISG)[43] is a uniform international sales law. The CISG database is a joint product of the Pace University School of Law Library and the Institute of International Commercial Law, one of the scholarly centers at the law school. The database was created to satisfy the needs of both scholarly researchers and practitioners. From the founding of the database, two principles have guided our efforts: (1) the database should provide complete coverage of the CISG so that all researchers' needs would be met; and (2) the documents included in the database should be as error-free as possible.

As a joint product, the database has drawn on the expertise of a number of different individuals. [page 17] Primary responsibility for the substantive content of the database rests with Professor Albert H. Kritzer,[44] who has written much of the original material available at the site.[45] Librarians with legal backgrounds are involved in the database, with one serving as Web master [46] and others managing [47] part-time HTML technicians [48] who scan and key documents and prepare them for inclusion in the database. We employ one part-time proofreader for English and French materials who works off-site; the number of hours she works each week is determined by the flow of materials from the HTML technicians. For other languages, we either ask the authors to proofread their manuscripts once HTML markup is completed, or we contract with other freelance proofreaders for discrete proofreading projects.[49] In addition to the staff members who work at Pace, there are scholarly contributors to the database from all parts of the world whose primary contact is Professor Kritzer.

Practicing the Principles of Good Web Design

Let us turn to the principles of user-centered Web design listed earlier [50] in order to see the steps taken at the Pace CISG site to follow those principles.

1. Start with quality content. Illustration 1 shows how the database directory, which is also the home page, identifies the scope of the project. [For Illustration 1, go to <http://www.cisg.law.pace.edu>.] Every effort has been made to provide complete coverage of the law addressed. Because the CISG has been adopted by both civil and common law countries,[51] it was important to make available the sources that scholars and practitioners from both traditions would require when doing research on the CISG.[52] [page 18] The database includes the full text of the uniform law itself in English and in other languages.[53] The treaty is presented in two different formats: the first, in a variety of official and unofficial languages, as a full-text document, and the second, in English, with each article presented as a separate file with many links to available related materials.[54] The database also includes the full text of antecedent uniform sales laws,[55] useful for comparison with the CISG. Other primary sources include cases construing the CISG by common and civil law jurisdictions as well as by international and domestic arbitral tribunals. Secondary sources have also been loaded, including articles and commentaries on the CISG written by scholars from many different countries,[56] and aids to researching the legislative history of the CISG. There is a bibliography on the CISG containing thousands of citations -- more than any other bibliography on the subject. Other materials include links to companion international trade databases produced elsewhere. The database directory is designed to be easy to use; the bulleted items are only one click away from the researcher, making it easy to access the material and determine its suitability for the intended purpose.

2. Make sure the content is accurate. From the founding of the database, we have had a commitment to making it as error-free as possible. Quality control procedures are in place at many levels. The small number of errors and the speed with which they are corrected attest to this. Professor Kritzer constantly reads new cases and articles about the CISG, and if he finds information that contradicts what we have online, he determines which information is correct and submits the corrections to be made as soon as possible. If Professor Kritzer finds incorrect information in a document being readied for the Internet, he makes the changes to the document before it is loaded.[57] [page 20] As part of this commitment to quality, proofreaders read each and every document at least three times after it has been put into HTML. We have found that errors and corruption can occur during the conversion process, and that even running spellcheck programs is not entirely foolproof, especially with foreign words and phrases. Each document is checked one last time before it is put online, and we spot-check documents after they are online to make sure their appearance and format are consistent with the rest of the database. This is a labor-intensive process; automation is of only limited utility.

3. Make sure authors are known and accountable. The CISG home page has a statement of authorship with brief information about the individuals involved in creating the site. Each individual listed may be reached by e-mail. The address and telephone number for the institute are supplied. For those who are unfamiliar with the Institute of International Commercial Law, there is a link to information about the Institute. In addition, the names of authors of segments of the database are provided, along with their credentials, so users can judge for themselves whether to rely on the information. Illustration 2 is an example of the type of biographical information that is typically made available to users of the CISG database. [For Illustration 2, go to <http://www.cisg.law.pace.edu/cisg/text/cross/cross-90.html>, click the asterisk beside the author's name, and jump to an example of the type of biographical information provided.]

4. Keep your site up-to-date and publicize its currency. When you access the CISG database, it is easy to tell when information was last updated. In the fall of 1997, we began using the phrase "Last updated" with a date when pages were changed. Other indicia of currency appear in the materials themselves.[58]

5. Make your database unique. In the case of the CISG, there are sites in France,[59] Germany,[60] Italy,[61] Spain,[62] and Finland [63] that make cases available over the Internet. However, they are presented primarily in their original languages, not particularly helpful for the researcher who does not read these languages. The intent in creating our database was to present materials in their original languages and to offer translations into English to help the many people for whom English is a first or second language. The importance of this approach was demonstrated by the 11th Circuit Court of Appeals when it cited the Pace database as "a promising source [for] persuasive authority from the courts of other States Party to the CISG."[64] [page 21]

Internet databases are increasingly going beyond duplicating what is available in paper and publishing original works that are available only in electronic format. That is true of the CISG database. For instance, if one goes to the section of the database that includes primers on the CISG, one finds not only materials that would be difficult to locate in most libraries, but also one text that was written specifically for the database by two of South America's foremost scholars on the CISG.

6. Organize your material. The skills of the information specialist are especially important.[65] Article 7 of this treaty, presented in Illustration 3, exemplifies our approach to organizing the presentation of the articles of the CISG. [For Illustration 3, go to <http://www.cisg.law.pace.edu/cisg/text/e-text-07.html>.]

Following the text of Article 7, relevant materials are grouped by type of material available: legislative history, case law, and scholarly writings, with all materials available by hypertext link. One of the guides to Article 7 was written specifically for the database and is available nowhere else. Other linked materials were written by scholars from Croatia, Germany, India, Italy, Spain, and Sweden.

7. Make the database design user-friendly. The CISG site was purposely designed [66] to be sparing of graphics. Minimal use has been made of color.

An even more important design issue is navigability of the site. This is an area in which the participation of librarians makes a difference, for we understand how to organize resources in such a way that the organization will be intelligible to laypeople as well as to subject specialists. The CISG home page has a simple design; it groups materials by category and subcategory and is easy to navigate without help. Most CISG pages have links to the database directory (i.e., the home page) at both the bottom and the top of the page.

8. Provide powerful search tools. The CISG database offers several search engines.[67] It is possible to do a full-text search of the over four hundred case presentations and of the bibliography, using Boolean connectors or proximity searches. One can also search the case presentations using a fielded search similar to field searching on Westlaw or segment searching on LEXIS. It is also possible to do an automatic search for cases that discuss a particular article of the CISG by going directly to the article in which you are interested; an automatic search is launched by pressing the "Cases Involving CISG Article" button, as shown in Illustration 4. [In lieu of Illustration 4, for an illustrative case retrieval mechanism, go to the "case button"on the Annotated Text page for Article 78 CISG <http://www.cisg.law.pace.edu/cisg/text/e-text-78.html>.] A list of cases will be retrieved, each case available by hypertext link. [page 23] A recent innovation is the harvest broker that allows researchers to query the entire CISG database using Boolean operators or phrases, and to search several other related international trade databases [68] simultaneously, so that if an item is not available at our site but is available elsewhere, it will be retrieved.

Companion Internet Databases

Because of its success, the CISG database has spawned a number of companion Internet databases. In addition to the coordinated CISG databases produced by two of the leading law schools of France and Germany, law schools of three other countries have recently joined our consortium -- the CISG-Finland [69] database of Turku University, the CISG-Spain and Latin America [70] database of Universidad Carlos III de Madrid, and the CISG-Brasil [71] database of the Universidade Federal Fluminense of Rio. CISG-Finland has adapted approaches of ours as well as approaches devised by other consortium members. The format and search engine improvements devised by the authors of companion databases make them even more valuable to scholars and practitioners in their countries. Further format and search engine improvements are being devised by scholars from Belgium, Canada, and Israel working on CISG databases for their countries.

Each new consortium database serves its own national market; together we serve the world market by linking our materials over the Internet. The team at Pace constantly works to expand the CISG Internet consortium. A CISG-Belgium database will soon be on the Internet thanks to the efforts of the Katholieke Universiteit Leuven. A CISG-Israel database is being prepared at Bar Ilan University of Tel Aviv. We are currently discussing further expansion with scholars from Egypt, Japan, Russia, and Vietnam. The object is a CISG database for each of these countries, and eventually for others, produced by knowledgeable persons at leading law schools of each country.

As new participants join the CISG consortium, we all learn from each other. We benefit from the opportunity to evaluate and, where appropriate, adopt innovative and user-friendly approaches devised by other sites' Web masters. [page 26]

Conclusion

Above all, remember these two principles when designing legal Internet sites. First, as to the content and accuracy of your database, make absolutely certain that your work adheres to the highest standards of scholarship. Second, as to design and user-friendliness, make sure that the site is as easy to use as possible.

Producing a high-quality database that satisfies the criteria I have identified calls for receptivity to Internet innovations and hard work. We must all recognize the need to learn from others by surfing the Internet and by networking with other producers of scholarly databases.[74] There are great benefits to be derived from courting others to join your endeavor, generate ideas, and share the work. [page 27]


FOOTNOTES

* © Marie Stefanini Newman, 1999. This article is an expanded version of an address presented at the program "Twenty-First Century Scholarship: Legal Knowledge Systems on the Internet" during the Annual Meeting of the Association of American Law Schools, San Francisco, California, Jan. 10, 1998. All uniform resource locators (URLs) cited in this article were verified on Sept. 22, 1998, except as otherwise noted, and were correct as of that date.

** Acting Law Library Director and Adjunct Professor of Law, Pace University School of Law, White Plains, New York. The author thanks Professors Nicholas Triffin and Albert H. Kritzer, Pace University School of Law, for their support and guidance. Special thanks go to several members of the staff of the Pace University School of Law Library: Vicky Gannon, Head of Circulation, who patiently and carefully read several drafts and provided many insightful suggestions; Heide-Marie Bliss, Head of Public Services; and David Williams, Reference Librarian and Webmaster, who provided indispensable research assistance. Finally the author thanks Marianne Davy and Jeff Wahl of the Department of Computer Services at Pace for their help with the footnotes and illustrations for this article.

1. See Katherine T. Durack, Introducing New Users to the Internet, in TEACHING ELECTRONIC INFORMATION LITERACY: A HOW-TO-DO-IT MANUAL 87 (Donald A. Barclay ed., 1995). Will Manley has pointed out that "the Internet is the global village's vanity press. It has no editorial board and no editorial principles." Will Manley, The Worldwide Vanity Press, AM. LIBR., Jan. 1998, at 136, 136.

2. See Steven R. Knowlton, How Students Get Lost in Cyberspace, N.Y. TIMES, Nov. 2, 1997, 4A (Education Life), at 18, 18.

3. See, e.g., U.S. House of Representatives, Internet Law Library (visited Sept. 22, 1998) <http://www.lawguru.com/ilawlib/> (links to eleven sites offering Supreme Court opinions, most of which are maintained by law schools).

4. See, e.g., Ctr. Info. Law & Policy, Villanova Law School, Federal Court Locator (visited Oct. 13, 1998) <http://vls.law.vill.edu/Locator/fedcourt.html> (links to decisions from all the Circuit Courts of Appeals; with the exception of the Fifth Circuit, all of these sites are maintained by law schools).

5. See, e.g., Cornell Law School, Legal Information Institute (visited Sept. 22, 1998) <http://www.law.cornell.edu> (provides searchable versions of the Code of Federal Regulations and the United States Code).

6. See, e.g., Ctr. Info. Law & Policy, Villanova Law School, State Court Locator (visited Oct. 13, 1998) <http://vls.law.vill.edu/Locator/statecourt/> ("intended to be the home page for state court systems on the Internet, ... provides links to home pages and opinions by state judiciaries that can be viewed and downloaded by users).

7. D. Scott Brandt points out: "There are three traditional ways in which information gets filtered. First, if it is written and/or issued by an authoritative source, such as the federal government or a reliable organization, it is generally accepted at face value as having validity. Second, if it is authenticated as part of an editorial or peer review process by a publisher, it is generally accepted as reliable. Third, if it is evaluated by experts, reviewers, or subject specialists/librarians as part of collection development, it is generally accepted as authoritative." D. Scott Brandt, Evaluating Information on the Internet, COMPUTERS LIBR., May 1996, at 44, 44.

8. The Internet has been characterized as being "one of the biggest wellsprings of misinformation ever developed." Manley, supra note 1, at 136. The Wolfgram Memorial Library of Widener University has compiled a useful collection of links to sites of dubious accuracy. Janet E. Alexander & Marsha A. Tate, Evaluating Web Pages: Links to Examples of Various Concepts (last modified April 29, 1998) <http://www.science.widener.edu/~withers/examples.htm>.

9. Durack, supra note 1, at 87.

10. Argus Clearinghouse, Ratings System (visited Oct. 13, 1998) <http://www.clearinghouse.net/ratings.html>.

11. Lycos, Best of the Web Top 5% (visited Oct 13,1998) <http://point.lycos.com/categories/>.

12. Encyclopaedia Britannica's Internet Guide (visited Oct 13, 1998) <http://www.ebig.com>.

13. Magellan Internet Guide, Net Reviews (visited Oct 13, 1998) <http://www.mckinley.com>.

14. Boyd R. Collins, Beyond Cruising: Reviewing, LIBR. J., Feb. 15, 1996, at 122, 124 (referring to remarks made by James Rettig, Assistant Dean for Reference and Information Services, College of William and Mary, and Wilson Web reviewer, at the 15th Annual Charleston Conference, 1995).

15. "[T]he number of Web sites doubl[es] every six months." Jeffrey Selingo, A New Archive and Internet Search Engine May Change the Nature of On-Line Research, CHRON. HIGHER EDUC., Mar. 6,1998, at A27, A27.

16. There are a number of sites on the Internet that provide information about evaluating other sites and list criteria that the researcher should keep in mind. Two relate specifically to legal sites. The Information Technology and Implementation Working Group of the American Association of Law Libraries lists criteria for evaluation and gives examples of valuable sites (http://www.bc.edu/aallwg). New York University School of Law Library has posted a sample evaluation form for foreign and international legal databases (http://www.law.nyu.edu/library/evaluate.html). Some useful generic sites are: Robert Harris, Evaluating Internet Research Sources (last modified Nov. 17, 1997) <http://www.virtualsalt.com/evalu8it.htm>; Hope N. Tillman, Evaluating Quality on the Net (last modified Nov. 13, 1997) <http://www.hopetillman.com/findqual.html>; T. Matthew Ciolek, The Six Quests for The Electronic Grail: Current Approaches to Information Quality in WWW Resources (last modified June 20, 1996) <http://www.ciolek.com/PAPERS/six-quests1996.html>.

17. Two useful guides for building Web pages are available over the Internet, offering guidance for Web page and site design. See Builder.Com (visited Sept. 22, 1998) <http://builder.cnet.com>; Yale C/AIM Web Style Guide (visited Sept. 22, 1998) <http://info.med.yale.edu/caim/manual/contents.html>.

18. See supra note 16 and accompanying text.

19. Steven W. Sowards, Save the Time of the Surfer: Evaluating Web Sites for Users, 15 LIBR. HI TECH 155, 155 (1997).

20. Id. at 155-57.

21. Eileen G. Abels et al., A User-Based Design Process for Web Sites, 8 INTERNET RES.: ELECTRONIC NETWORKING APPLICATIONS & POL'Y 39, 41 (1998).

22. Discussing indicators of lack of credibility for Internet sites, Robert Harris points out that "[a]n occasional split infinitive or comma in the wrong place is not unusual, but more than two or three spelling or grammar errors is cause for caution, at least. Whether the errors come from carelessness or ignorance, neither puts the information or the writer in a favorable light." Harris, supra note 16.

23. "Creating an electronic text can be a very time-consuming process. Keyboarding is the best method to reach the levels of accuracy that humanities scholars would expect to find. ... Even if the letters in the text are recognized accurately by the OCR program, work still needs to be done to create an accurate and usable text for processing." Susan Hockey, Electronic Texts: The Promise and the Reality, ACLS NEWSL., Feb. 1997, at 3, 4. The specific focus of Hockey's remarks is electronic texts in the humanities; given the importance of law, legal texts should be held to an even higher standard.

24. "When evaluating information found on the Internet, it is important to examine who is providing the 'information' you are viewing, and what might be their point of view or bias. The popularity of the Internet makes it the perfect venue for commercial and sociopolitical publishing. These areas in particular are open to highly 'interpretative' uses of data." Elizabeth Kirk, Evaluating Information Found on the Internet (last modified Mar. 5, 1998) <http://www.library.jhu.edu/elp/useit/evaluate/>.

25. See id. Kirk points out that currency is not an issue for some types of information, using an example from the humanities to illustrate her point. For other types of data, however, currency is important, as is the regularity with which the data are updated. Legal databases fall into the category of materials that require constant updating. When undertaking a legal database, one should make sure that the resources will be available to maintain it on a regular basis. See infra text accompanying note 42.

26. Checking for stale links is no longer an onerous process. It can now be automated using Web diagnostic software whose primary purpose is to check the validity of hypertext links. For a good general review of link checking software, see Neil Randall, Diagnose and Repair Your Web Site, PC MAG., Feb. 10, 1998, at 237.

27. For example, the United States Code Annotated (U.S.C.A.) and the United States Code Service (U.S.C.S.) offer competing versions of the text of federal statutes currently in force, but provide different aids to the user. Another difference between the two sets is that the "U.S.C.A. uses the text as it appears in the official version, the U.S.C." J. MYRON JACOBSTEIN ET AL., FUNDAMENTALS OF LEGAL RESEARCH 163 (7th ed. 1998). The U.S.C.S. "follows the text of the public laws as they appear in the United States Statutes at Large." Id. at 164.

28. See supra note 3.

29. In addition, it is probably easier to get grant money and other funding to create a unique site rather than a duplicative site.

30. Stephen E. Toub, Adding Value to Internet Collections, 15 LIBR. HI TECH 148, 150 (1997).

31. Mark Gould, Three Generations of Legal Websites (last modified Mar. 27, 1998) <http://web.archive.org/web/20000619064016/http://aranea.law.bris.ac.uk/Bileta/frame1.html>.

32. The Australasian Legal Information Institute (<http://www.austlii.edu.au/>) is a good example of a legal Web site rich with hypertext links.

33. See Gould, supra note 31.

34. For a good discussion of designing for the Web, see RAY E. METZ & GAIL JUNION-METZ, USING THE WORLD WIDE WEB AND CREATING HOME PAGES 102 (1996).

35. Sowards, supra note 19, at 156.

36. Probably for this reason, legal publishing has historically made limited use of graphics, the best known use being the ubiquitous key employed by the West Publishing Company.

37. Gould, supra note 31.

38. See JENNIFER NIEDERST, DESIGNING FOR THE WEB 49 (1996).

39. For a good discussion of the production of electronic documents for individuals with disabilities, see Mark Paciello, People With Disabilities Can't Access the Web! (visited Sept. 22, 1998) <http://www.w3j.com/5/s3.paciello.html>. The paper includes information on designing Web pages for researchers who do not access the Internet in a visual manner, specifically dyslexics and the visually impaired. It also addresses the issue of providing text transcriptions of audio clips to assist hearing-impaired researchers.

40. See PC Magazine's Guide to Web Site Design, PC MAG., Jan. 20, 1998, at l69, 169 (recommending that "none of your content [be] more than three clicks away from the home page").

41. Richard Haigh recommends that database developers devise a "vector-based retrieval system that provides the user with options to weight retrieved cases by age, frequency of reference, level of court, and jurisdiction. If cases were ordered according to some specified options, it would not matter if the searcher retrieved an unmanageable number of cases." Richard Haigh, What Shall I Wear to the Computer Revolution? Some Thoughts on Electronic Researching in Law, 89 L. LIBR. J. 245, 263 (1997) (footnote omitted).

42. For example, Pace University School of Law's Virtual Environmental Law Library (VELL) (<http://www.law.pace.edu/env/vell6.html> relies on the expertise of the faculty of the school's Center for Environmental Legal Studies and of the librarian who is the liaison to the environmental program, the work of J.D. and LL.M. candidates who gather and evaluate resources for inclusion in the site and perform HTML markup, and the specialized library collections in environmental and international law. Thus the VELL database both builds on and adds to Pace's reputation for excellence in environmental law studies.

43. United Nations Convention on Contracts for the International Sale of Goods. Apr. 11, 1980, U.N. Doc. A/CONF.97/18, reprinted in S. TREATY DOC. No. 98-9 (1983) and 19 I.L.M. 668.

44. Professor Kritzer, an adjunct faculty member at Pace, is the Executive Secretary of the Institute of International Commercial Law, the editor of the INTERNATIONAL CONTRACT MANUAL (1990), and the author of GUIDE TO PRACTICAL APPLICATIONS OF THE UNITED NATIONS CONVENTION ON CONTRACTS FOR THE INTERNATIONAL SALE OF GOODS (1990).

45. It is probably impossible to create a scholarly database without the participation and guidance of a faculty member who has deep knowledge of the subject area and is willing to be involved in the project. See also supra text accompanying note 42.

46. During the start-up phase of the database, the webmaster's work occupied approximately three-quarters of his time. This meant that his library responsibilities had to be assigned to other members of the staff. As the database has matured, however, his involvement has been reduced considerably, so that he has been able to resume some of his former duties in the library.

47. The same phenomenon occurred with my duties as database manager. At the beginning, I spent close to half my time outside the library setting up procedures for the CISG database, hiring technicians, working on database design, instituting quality control procedures, and editing and proofreading documents. For over a year, the law library had to do without two key people for significant amounts of time and was adversely affected at least for the short term.

48. The project has employed anywhere from one to four HTML technicians at anyone time. The usual work schedule is between twenty-five and thirty-five hours per week. Several of the HTML technicians were Pace law students, who rarely were able to work even ten hours per week. I have found that most law students are generally not very reliable as part-time employees for database projects. Most have part-time jobs in law firms that they view as being more important to their future career goals. Undergraduates and graduate students in other disciplines have proven much more reliable.

49. See infra text accompanying note 57 for further discussion of our proofing procedures.

50. See supra text accompanying note 20.

51. See UNITED NATIONS, MULTILATERAL TREATIES DEPOSITED WITH THE SECRETARY-GENERAL, at 394, U.N. Doc. ST/LEG/SER.E/14, U.N. Sales No. E.96.V5 (1996) and CISG: List of Contracting States (last updated Sept. 16, 1998) <http://www.cisg.law.pace.edu/cisg/countries/cntries.html>.

52. "Perhaps the most important feature of the civil law tradition is the centrality of written laws that ... the courts simply interpret and apply. ... Another distinctive feature of the civilian tradition is the importance and influence of legal scholarship (la doctrine). ... Prior judicial decisions are not formally binding sources of law in the civilian tradition. ... [P]rior cases are sometimes a source of law, but never a source of legal rules. ... The function and nature of the judicial decision in the common law tradition is markedly different from its civilian counterpart. ... [J]udges make law and ... prior cases are the fundamental sources of law." Jeffrey L. Friesen, When Common Law Courts Interpret Civil Codes, 15 WIS. INT'L L.J. 1, 7-9 11-12 (1996) (footnotes omitted). See also JOHN HENRY MERRYMAN, THE CIVIL LAW TRADITION (2d ed. 1985) and RENE DAVID & JOHN E.C. BRIERLEY, MAJOR LEGAL SYSTEMS IN THE WORLD TODAY (3d ed. 1985) (both contain good discussions of the three prevalent legal systems - civil law, common law, and socialist law -- and the sources they apply).

53. As of June 18, 1998, the text was available in the official English, French, and Spanish texts. It was also available in unofficial Czech, Danish, Dutch, Finnish, German, Japanese, Norwegian, Persian, Polish, Portuguese, and Swedish translations.

54. See infra text accompanying note 65.

55. The predecessors to the CISG are the Convention Relating to a Uniform Law on the International Sale of Goods, July 1, 1964, 834 U.N.T.S. 107, and the Convention Relating to a Uniform Law on the Formation of Contracts for the International Sale of Goods, July 1, 1964, 834 U.N.T.S. 169.

56. We have loaded the full text of dozens of articles on the CISG written by scholars from around the world. In response to our request for authors to submit their dissertations on the CISG for inclusion in the database, we recently received the first dissertation. We are particularly glad to include dissertations because they are notoriously difficult to access.

57. Substantive changes are made only with the permission of the author.

58. In the country case schedule, 1998 cases are listed, and in the bibliography, 1998 articles are cited.

59. Claude Witz, CISG France Database (visited Sept. 22, 1998) <http://witz.jura.uni-sb.de/CISG/>.

60. Univ. of Freiburg, Inst. of Foreign and Int'l Private Law (Dept. I), CISG Online Project (visited Sept. 22, 1998) <http://www.cisg-online.ch/>.

61. Michael Joachim Bonell, Database of the Centre for Comparative and Foreign Law Studies -- Rome (visited Sept. 22, 1998) <http://soi.cnr.it/~crdcs/crdcs/index.htm>.

62. Pilar Perales Viscasillas, CISG Spain and Latin America Database (visited Sept. 22, 1998) <http://www.uc3m.es/cisg> [hereinafter Spain].

63. Tuula Ammala, CISG Finland Database (visited Sept. 22, 1998) <http://www.utu.fi/oik/tdk/xcisg/cisg.htm> [hereinafter Finland].

64. MCC-Marble Ceramic Center, Inc. v. Ceramica Nuova D'Agostino, S.P.A., 144 F.3d 1384,1389 n.14 (1lth Cir. 1998).

65. "[T]here is a professional group that we can use far more effectively than it has yet been used in the development of the Web: we need to put the design and implementation of the Web much more in the hands of information specialists, not in the hands of programmers ... we need people who are able to do what librarians, cataloguers, archivists, editors, publishers, and book reviewers have always done. They know a lot, and their knowledge is germane to the development of the Web." Richard C. Rockwell, The World Wide Web as a Resource for Scholars and Students, ACLS NEWSL., Feb. 1997, at 9, 10.

66. Most of the credit for the navigability of the CISG site goes to Professor Nicholas Triffin, who organized the site and created its navigational tools.

67. Webmaster David Williams is responsible for the variety and utility of search engines available to researchers in the CISG database.

68. Besides the Pace CISG site, the other databases searched at this time are the CISG France Database (<http://witz.jura.uni-sb.de/CISG/>); the CISG Online Project (<http://www.cisg-online.ch/>); the International Trade Law Monitor at the University of Tromso (<http://www.lexmercatoria.org/>); the United Nations Commission on International Trade Law (<http://www.un.or.at/uncitral>); the International Institute for the Unification of Private Law (<http://www.unidroit.org>); CISG-Brasil (<http://www.uff.br/cisgbrasil/>); CISG-Finland (<http://www.utu.fi/oik/tdk/xcisg/cisg.htm>). This number will continue to grow as related databases are made available over the Internet. See infra notes 69-11 and accompanying text.

69. Finland, supra note 63.

70. Spain, supra note 62.

71. Universidade Federal Fluminense, CISG-Brasil (visited Oct. 13, 1998) <http://www.uff.br/cisgbrasil/>.

72. Several staff members of the Pace University School of Law who are involved in Internet site design and maintenance attended and made presentations at the program "Twenty-First Century Scholarship: Legal Knowledge Systems on the Internet" during the 1998 Annual Meeting of the Association of American Law Schools in San Francisco, California. Listening to the presentations made by scholars from Australia, Norway, and several American law schools, we learned about different approaches that others have adopted and have begun investigating whether they would work in our environment.


Pace Law School Institute of International Commercial Law - Last updated June 19, 2003
Go to Database Directory || Go to Bibliography
Comments/Contributions