Comments on: Followup: CSV Parsing in Haskell and Python

By: GalIvory

GalIvory — Thu, 24 Jul 2008 21:45:45 +0000

The party’s over now. I reworked the original with all the `String`s, but got something that was even slower (~33%). (Though maybe the HOFs are part of this?) My last shot is an accumulating parameter and then I am out of ideas. import qualified Data.ByteString.Char8 as B import Control.Arrow import Data.Maybe import Data.Ord import Data.List import System.Environment import System.IO data Row = Row { symbol :: B.ByteString , avg :: Double , day :: B.ByteString , time :: B.ByteString } deriving (Read, Show) main :: IO () main = B.interact $ B.unlines . map recomma . analyze . rowify . B.lines recomma = B.intercalate (B.pack “, “) analyze = map analyze’ . groupBy g . sortBy c where g r0 r1 = symbol r0 == symbol r1 c = comparing symbol analyze’ rows = [symbol', open, low, high, close, day'] where (day’, symbol’) = (day &&& symbol) . head $ rows (open, close) = clipper . sortBy (comparing time) $ rows (low, high) = clipper . sortBy (comparing avg) $ rows clipper = (avgB *** avgB) . (head &&& last) where avgB = B.pack . show . avg rowify = catMaybes . map processRow processRow bytes = case (maybeNum l’, maybeNum h’) of (Just n’, Just m’) -> Just $ Row symbol ((n’ + m’) / 2) day time _ -> Nothing where symbol:l’:h’:_:day:time:_ = B.split ‘,’ bytes — match or program dies! maybeNum b = case reads $ B.unpack b of [(n, "")] -> Just n _ -> Nothing

By: Slarba

Slarba — Thu, 17 Jul 2008 07:34:51 +0000

Well then, the default Haskell mergesort suits your data better :)

By: Top Posts « WordPress.com

Top Posts « WordPress.com — Thu, 17 Jul 2008 00:04:44 +0000

[...] Followup: CSV Parsing in Haskell and Python Tonight I tried improving the Haskell version of my CSV data analysis program. None of the changes made the Haskell [...] [...]

By: Greg

Greg — Wed, 16 Jul 2008 18:41:18 +0000

Slarba,

Okay, assume we only have around 10 tickers over a span of 5 years of data. So each day we have 10s of quotes for each of the 10 tickers.

I installed the bytestring profiling library, and the profile report told me that it was spending 80% of its time in qsort. So I replaced ‘qsort’ with ‘sort’ and the program ran over all 160,000 CSV lines in about 1 second!

I also added a check for a zero return from strtod, as a hacked check for when the string cannot be parsed. I never have zero bids or asks, so that’s sufficient for me. This change didn’t effect running time.

So that leaves open the question why qsort performed so horribly for me. (my intuition is that the default Haskell sort is probably the best thing to use).

I’ve annotated your hpaste code here:

http://hpaste.org/8965#a2

Now the Haskell program is faster than Python (although it certainly took plenty of effort, and using strtod feels like cheating).

Greg

By: Slarba

Slarba — Wed, 16 Jul 2008 17:51:22 +0000

I used quite small amount of distinct tickers… the difference gets smaller when there are more of them.

You need to build the profiling versions of the libraries you intend to use:

runhaskell Setup.hs configure -p
runhaskell Setup.hs build
install…

By: Greg

Greg — Wed, 16 Jul 2008 17:23:58 +0000

Slarba,

Thank you for posting your solution. Unfortunately, it does not perform well for me. For example, when I run it on the first 16,000 lines of my CSV file (not all 157,000 lines), it takes 7.5 seconds. It was running for longer than 30 seconds on the full file, so I stopped it.

I do not know how to profile the code. For example, when I run:

$ ghc –make -O2 -prof Dailyquotes.lhs

Could not find module `Data.ByteString.Char8′:
Perhaps you haven’t installed the profiling libraries for package bytestring-0.9.1.0?

Where do I get the profiling libraries for the bytestring package?

What sort of input file did you run your program on? Perhaps I’m doing something horribly wrong?

Greg

By: Slarba

Slarba — Wed, 16 Jul 2008 16:15:14 +0000

You can make this even run faster with quicksort:

http://hpaste.org/8965

Less than 600 milliseconds on a 2GHz MacBook.

Again, profiling revealed that the sorting is slow.

By: Slarba

Slarba — Wed, 16 Jul 2008 15:21:45 +0000

I don’t believe you: what do you think, was your post helpful for the blogger?

First and only rule of optimization (applies to C, Python, Haskell…): profile to see where the time is REALLY wasted. It was really easy to compile the program to produce time/allocation profile and pinpoint ByteString to String conversion being the problem, and fix it. Another 10 minutes for refactoring a bit.

By: DiD

DiD — Wed, 16 Jul 2008 12:19:03 +0000

“I don’t believe you”: If I could give you an award for that post I would!

By: Slarba

Slarba — Wed, 16 Jul 2008 10:11:52 +0000

Here’s an optimized version.

http://hpaste.org/8959