Lexicographically sorting large files in Linux

Sep 01, 2012

When I hear the word “sort” my first thought is usually “Hadoop”! Yes, sorting is one thing that Hadoop does well, but if you’re working with large files in Linux the built-in sort command is often all you need.

Let’s say you have a large file on a host with 2GB or more of main memory free. The following sort command is a efficient way to lexicographically-order large files.

LC_COLLATE=C sort --buffer-size=1G --temporary-directory=./tmp --unique bigfile.txt

Let’s break this command down and examine each part in detail.

sort image

About the author

Alex Holmes is a senior software engineer with over 15 years of experience developing large scale distributed Java systems. For the last four years he has gained expertise in Hadoop solving Big Data problems across a number of projects. He is the author of "Hadoop in Practice", a book published by Manning Publications. He has presented at JavaOne and Jazoon and is currently a technical lead at VeriSign.

If you want to see what Alex is up to you can check out his work on GitHub, or follow him on Twitter or Google+.

comments powered by Disqus

All posts

» Using the libjars option with Hadoop
» Installing AsciiDoc on OSX
» Java 6 and 7 with the dotted/dotless I
» LZOP decompression - revenge of the useless cat
» Executing variables that contain shell operators
» Using awk and friends with Hadoop
» Configuring and tuning MapReduce's shuffle
» Controlling user logging in Hadoop
» Pipes and useless cats
» Hadoop unit testing with MiniMRCluster and MiniDFSCluster
» How partitioning, collecting and spilling work in MapReduce
» Using sed to perform inline replacements of regex groups
» Sorting text files with MapReduce
» Lexicographically sorting large files in Linux
» Slurper v2
» Bare-metal installation for Nginx and Jekyll
» OSX, Chrome and DNS


Latest Tweets

Status updating sdf ...