Abstract

The Comma Separated Values (CSV) file format is the most common import and export format for spreadsheets and databases. Although many CSV files are simple to parse, the format is not formally defined by a stable specification and is subtle enough that parsing lines of a CSV file with something like line.split(",") is eventually bound to fail. This PEP defines an API for reading and writing CSV files. It is accompanied by a corresponding module which implements the API.

To Do (Notes for the Interested and Ambitious)

• Better motivation for the choice of passing a file object to the constructors. See https://mail.python.org/pipermail/csv/2003-January/000179.html
• Unicode. ugh.

Application Domain

This PEP is about doing one thing well: parsing tabular data which may use a variety of field separators, quoting characters, quote escape mechanisms and line endings. The authors intend the proposed module to solve this one parsing problem efficiently. The authors do not intend to address any of these related topics:

• data interpretation (is a field containing the string “10” supposed to be a string, a float or an int? is it a number in base 10, base 16 or base 2? is a number in quotes a number or a string?)
• locale-specific data representation (should the number 1.23 be written as “1.23” or “1,23” or “1 23”?) – this may eventually be addressed.
• fixed width tabular data - can already be parsed reliably.

Rationale

Often, CSV files are formatted simply enough that you can get by reading them line-by-line and splitting on the commas which delimit the fields. This is especially true if all the data being read is numeric. This approach may work for a while, then come back to bite you in the butt when somebody puts something unexpected in the data like a comma. As you dig into the problem you may eventually come to the conclusion that you can solve the problem using regular expressions. This will work for a while, then break mysteriously one day. The problem grows, so you dig deeper and eventually realize that you need a purpose-built parser for the format.

CSV formats are not well-defined and different implementations have a number of subtle corner cases. It has been suggested that the “V” in the acronym stands for “Vague” instead of “Values”. Different delimiters and quoting characters are just the start. Some programs generate whitespace after each delimiter which is not part of the following field. Others quote embedded quoting characters by doubling them, others by prefixing them with an escape character. The list of weird ways to do things can seem endless.

All this variability means it is difficult for programmers to reliably parse CSV files from many sources or generate CSV files designed to be fed to specific external programs without a thorough understanding of those sources and programs. This PEP and the software which accompany it attempt to make the process less fragile.

Existing Modules

This problem has been tackled before. At least three modules currently available in the Python community enable programmers to read and write CSV files:

• Object Craft’s CSV module [2]
• Cliff Wells’ Python-DSV module [3]
• Laurence Tratt’s ASV module [4]

Each has a different API, making it somewhat difficult for programmers to switch between them. More of a problem may be that they interpret some of the CSV corner cases differently, so even after surmounting the differences between the different module APIs, the programmer has to also deal with semantic differences between the packages.

Module Interface

This PEP supports three basic APIs, one to read and parse CSV files, one to write them, and one to identify different CSV dialects to the readers and writers.

obj = reader(iterable [, dialect='excel']
             [optional keyword args])

csvreader = csv.reader(file("some.csv"))
for row in csvreader:
    process(row)

obj = writer(fileobj [, dialect='excel'],
             [optional keyword args])

csvwriter = csv.writer(file("some.csv", "w"))
for row in someiterable:
    csvwriter.writerow(row)

csvwriter = csv.writer(file("some.csv", "w"), fieldnames=names)
csvwriter.write(names)
for row in someiterable:
    csvwriter.write(row)

class Dialect:
    # placeholders
    delimiter = None
    quotechar = None
    escapechar = None
    doublequote = None
    skipinitialspace = None
    lineterminator = None
    quoting = None

class excel(Dialect):
    delimiter = ','
    quotechar = '"'
    doublequote = True
    skipinitialspace = False
    lineterminator = '\r\n'
    quoting = QUOTE_MINIMAL

class exceltsv(excel):
    delimiter = '\t'

dialect = get_dialect(name)
names = list_dialects()
register_dialect(name, dialect)
unregister_dialect(name)

csvreader = csv.reader(file("some.csv"), dialect="excel",
                       quotechar="'", delimiter=':')

Implementation

There is a sample implementation available. [1] The goal is for it to efficiently implement the API described in the PEP. It is heavily based on the Object Craft csv module. [2]

Testing

The sample implementation [1] includes a set of test cases.

Issues

1. Should a parameter control how consecutive delimiters are interpreted? Our thought is “no”. Consecutive delimiters should always denote an empty field.
2. What about Unicode? Is it sufficient to pass a file object gotten from codecs.open()? For example:

   csvreader = csv.reader(codecs.open("some.csv", "r", "cp1252"))
   
   csvwriter = csv.writer(codecs.open("some.csv", "w", "utf-8"))
   

   In the first example, text would be assumed to be encoded as cp1252. Should the system be aggressive in converting to Unicode or should Unicode strings only be returned if necessary?

   In the second example, the file will take care of automatically encoding Unicode strings as utf-8 before writing to disk.

   Note: As of this writing, the csv module doesn’t handle Unicode data.

3. What about alternate escape conventions? If the dialect in use includes an escapechar parameter which is not None and the quoting parameter is set to QUOTE_NONE, delimiters appearing within fields will be prefixed by the escape character when writing and are expected to be prefixed by the escape character when reading.
4. Should there be a “fully quoted” mode for writing? What about “fully quoted except for numeric values”? Both are implemented (QUOTE_ALL and QUOTE_NONNUMERIC, respectively).
5. What about end-of-line? If I generate a CSV file on a Unix system, will Excel properly recognize the LF-only line terminators? Files must be opened for reading or writing as appropriate using binary mode. Specify the lineterminator sequence as '\r\n'. The resulting file will be written correctly.
6. What about an option to generate dicts from the reader and accept dicts by the writer? See the DictReader and DictWriter classes in csv.py.
7. Are quote character and delimiters limited to single characters? For the time being, yes.
8. How should rows of different lengths be handled? Interpretation of the data is the application’s job. There is no such thing as a “short row” or a “long row” at this level.

References

There are many references to other CSV-related projects on the Web. A few are included here.

Copyright

This document has been placed in the public domain.