Comments on: CSV Parsing: Haskell versus Python

By: Josh

Josh — Tue, 25 Nov 2008 19:42:16 +0000

Did I really forget to carry the zeroes? D’oh!

Thanks for the correction…

By: Greg

Greg — Wed, 19 Nov 2008 00:37:55 +0000

Josh,

Well, the file had 157562 lines and is 9779275 bytes in size. So yeah, that’s about 62 bytes per line. But processing that number of lines in 1.7 seconds is only 5.5MBps. So, by that math, it’s megas, not gigas.

Greg

By: Josh

Josh — Wed, 19 Nov 2008 00:22:53 +0000

160,000 records, well over over 50-bytes a peice, in 1.7 seconds?

That’s almost 5GB/sec.

Either I’m misunderstanding something, or the MacBook’s hard drive is apparently over 5 times faster than $50,000.00 RAID arrays currently on the market. That’s amazing!

By: Kelila

Kelila — Tue, 28 Oct 2008 17:46:33 +0000

Good words.

By: Christine Fenton

Christine Fenton — Fri, 25 Jul 2008 16:22:10 +0000

Never, *ever*, **ever** unpack ByteStrings to Strings. That’s the whole point of ByteStrings! If you unpack, you may as well just use `[Char]`. Stick to ByteStrings only and you get [this kind of result](http://shootout.alioth.debian.org/gp4/benchmark.php?test=sumcol&lang=all).

By: Don Stewart

Don Stewart — Sun, 20 Jul 2008 00:33:30 +0000

Note there’s a ByteString-based CSV parser on hackage now,

http://hackage.haskell.org/cgi-bin/hackage-scripts/package/bytestring-csv

which is significantly faster than Text.CSV. That, along with the bytestring-lex package for parsing floating point values from ByteStrings should mean your program is able to be both concise and fast.

By: Stan Domeshok

Stan Domeshok — Tue, 15 Jul 2008 17:16:54 +0000

Real World Haskell wrote a full CSV parser with Parsec, whole thing ended up being about 20 lines of code, you might want to take a look at it.
http://book.realworldhaskell.org/beta/parsec.html

By: Greg

Greg — Tue, 15 Jul 2008 16:32:13 +0000

@author: Ok, as a temporary workaround, you may add timestamps before each “class” of action in order to find what is taking to much time. I think it’s important to find and focus on the code that mainly slow all your program.

By: Reid

Reid — Tue, 15 Jul 2008 15:28:02 +0000

You realize that Python has a csv module as part of the standard library. All you really need is (I think roughly) this:

import csv

for row in csv.reader(‘myfile.csv’):
# do stuff with row.

To make it even better, you can go so far as to use a DictReader which lets you use the row like a dictionary with real string names. And then if you wanted to go further, you could wrap that in a class that lets you use regular attribute lookup. But if you want it to go really fast, you name you indices like so:

_ticker = 0
_bid = 1
…

And your code would still be readable.

By: Erhan Hosca

Erhan Hosca — Tue, 15 Jul 2008 12:08:00 +0000

i don’t think Haskell is designed for parsing CSV files … its not a language with emphasis on I/O …

i’m not sure why you’re using it for that purpose … in the financial space even :)