I also decided NOT to do a video this post as many people seemed to find my lack of toastmaster skills as an extreme negative, that, and I don’t think I need to be recording my inbox with the half the internet.

Background
Email is a REALLY old protocol, yet as developers we still use it everyday to manage bugs, track revisions, and communicate business ideas and goals (unlike the younger generation who say ‘email is only for old people’).

Being subscribed to even a couple of mailing lists can get out of hand REAL fast — now throw in all your usual email and you’d probably want to give up on this whole ‘internet career’ idea.

Mutt is a shell based email client. This, among other things, allows you to KEEP YOUR FINGERS ON THE KEYBOARD. It also has a whole suite of different programs that guys like me like to interact with it.

My Introduction
I started with pine back in the day, right after scoring my first dial-up account — 94, 95ish…. back then I would “sneak” into one of the computer science labs at the local university to play some bolo, hop on Hotline, and talk all sorts of shit on Carolyn Meinel and other security pundits of the day… ah, the life of a 12-14 year old…anyways I needed an email client better than pine — so began the adventures!

When someone mentions using mutt for their email — they typically leave out all the extra stuff that they use with it — here’s a short list of mine:

Progams We Use

• Procmail — sorting/filtering your mail into different mailboxes
• Fetchmail — downloading your email from different hosts
• Msmtp — sending email
• Gpg — signing/encrypting your email
• Mutt — navigation through your email
• Aspell/Abook — address book and spell checker
• Vim — editor

I currently have 23 mailboxes in my .muttrc (each mailing list gets one, each email account gets one, a few others).

A few of the common key bindings I have setup to navigate and crunch through my email:

Common keys:

• Ctrl-D — mass delete thread — some threads are pointless to read — why are you using your mouse to delete them individually?
• m — create a new email
• s — send currently selected email to a new mailbox
• G — tell fetchmail to download new email (I used to have it be background but not anymore since I don’t want to be reminded everytime you get a new email — you can actually have it backgrounded and not pop up either as well)
• c — switch to a diff. mailbox
• d — delete currently selected message
• v — view current message
• h — toggle headers (see who sent you the email if you suspect bullshit, or just see diff. headers)
• s — save attachement (useful for those daily google analytics pdfs you are sent — you do do this right?)
• a — add new contact to addressbook
• m-tab — select contact from addressbook
• r — reply to sender
• g — reply to group (mailing list)
• p, e/s — (after composing new email) — sign/encrypt with gpg (it’ll prompt you for your passphrase)

Just like screen, vim, emacs, and other cool unix tools you can split your screen — in this case, between folder view and mail view 2 or 3 ways and vertical versus horizontal — however you wish really.

Procmail is where mutt gets handed most of it’s magic I believe. A simple .muttrc macro like the following will suffice:

macro index G "!fetchmail -a -m 'procmail -d %T'\r"
macro pager G "!fetchmail -a -m 'procmail -d %T'\r"

simple .procmailrc
Note: this obviously is nowhere close to a full procmailrc — it just demonstrates it’s awesome ability to sort and filter email

:0
* ^Subject:.*(merb)
$MAILDIR/merb
 
:0
* ^Subject:.*(nodejs)
$MAILDIR/node
 
:0
* ^To:.*(GPSL\@yahoogroups.com)
$MAILDIR/gpsl
 
:0
* ^To:.*(solr-user\@lucene.apache.org)
$MAILDIR/solr

Then, as you can see, we sort through a regex to place incoming mail into their respective folders — that way I can CHOOSE what to read WHEN rather than my inbox choosing me.

Security, or Sending/Receiving Email without Ending up in the Ownage Bin

I’m grouping sending/receiving email into this section since I believe it is overlooked quite a LOT. There is a difference between being a d0x dropping, obsd zealot, master of the universe security wise and just being smart since you are a developer.

To get our email we’d use something like fetchmail — there are a lot of people out there that have hooked this up to a desktop pager — I don’t really do that anymore because I have found it distracting to see my inbox grow over the course of even a few minutes. I just check every couple of hours instead, although it is possible to daemonize fetchmail and not have the pager go off either.

A typical .fetchmailrc might look like the following:

poll mail.host1.com protocol imap user "joe+blows.com" password "pass" ssl
poll mail.booda.net protocol pop3 user "jb" password "pass" ssl
poll mail.bambam.com protocol pop3 user "billy@bob.com" password "pass" ssl

Yes, you are storing your passwords in plaintext but if you set up your permissions right you should be *ok* — you probably have a lot of other problems if someone has owned your home dir. Please notice the ssl entries here — while I don’t care too much about someone pretending they are bob.com when they are really evilbob.com I DO care that some random guy at a computer conference won’t be able to snatch my passwords in plaintext.

To send email we’d use msmtp — although, once again there are a billion and one programs to do this for you.

simple .msmtprc

account default
host smtp.gmail.com
port 587
from blah@gmail.com
tls on
tls_starttls on
tls_certcheck off
auth login
user blah
password *********
logfile ~/.msmtp.log

Notice, once again that we are ensuring tls is ON — after all auth login is no more than base64 encoded…. wireshark and friends could easily pick this up in a netcafe. The certcheck is up to you — if you are paranoid that someone is serving up a bullshit cert you might wish to turn it on, although, what is a cert worth nowadays after seeing something like this?

Better yet, forget the coffee shop! Just simple protection at a developer conference is enough!, Just think of those guys with their beer bellies, ponytails, and wire rimmed glasses who sit all day at a developer conf and collect packets… ALL DAY LONG…. how many systems do you think have been compromised this way?

Anywho, I believe I get a bunch more stuff done, and am able to sift through oceans of email much faster through the use of mutt and it’s friends rather then going to a point-n-click web option. What about you?

So what’s my secret? My tools and my workflow. I did write up a whole article on this but instead I made a short video for you all today — hopefully this is better than reading my one sentence paragraph length rants. Let me know.

So, in short — use vim AND use screen — make your environment comfortable to live in — as you are spending at least 8 hours if not more like 12-13 every day in it.

It’s very much like insertion sort in that it only keeps track of the current item and it’s like bubblesort that uses a series of swaps but in this case it will reverse direction instead of making a pass and coming back around. Hamid Sarbazi-Azad published it in 2000 (this is probably why I’m just now finding out about it).

It turns out this algorithm is incredibly easy to implement — easier than bubblesort.

The Problem

Imagine you are a garden gnome. You typically have your flower pots sorted by the height of the flower so you know where your belladonnas are verus the location of your daturas. Last night, some drunk punk came stumbling through your garden trying to steal your herbs and ended up mixing everything up. You, being a very methodical gnome, have to put everything back in order. The problem is, you have tunnel vision because you ate a bunch of salvia last night and while you were lost exploring the 10th dimension that drunk punk took a plumbers wrench to your head. This of course prevents you from having peripheral vision — you can only see what is in front of you. In this case you can only look at two garden pots at a time to compare them to put them back in order.

Rules
Here are a few rules to get our precious plants put back in order.

You pick up two pots and measure their height.

If the one on the left is shorter than the one on the right move onto the next pot on the right and measure again.

If the one on the left is higher than the one on the right swap their position and move back one pot.

If you have no pots on your left move forward one pot.

Once you have no pots on your right you are done.

I suck at using my words — therefore let me try my best to make a pretty little picture for you (note: this took like 200X longer to make then the succeeding code, and eventually I just ended up adding description/markup to the code to generate this instead):

Seems simple huh? Something that would be easier to explain to a comp. sci 53 freshmen versus bubblesort? Yeh, I think its very tiny codesize (even in c) let’s anyone understand it. Like it’s colleague, bubblesort it shares o(n) for best case and o(n^2) for worst case, so I can’t really recommend you use this for anything but learning/teaching purposes.

Typically you have two popular different groups of sorting algorithms:

• 1) algos using recursion — (nlogn) — ex: quicksort
• 2) algos using 2 nested loops — (n^2) — ex: bubblesort

Yes there are quite a few more but these are the ones mentioned usually.

Gnomesort is interesting though since it only has 1 loop and it’s not recursive — I don’t know where you would find yourself with limitations like these but if you do maybe you can actually use this algorithm.

Code

So, at first I wanted to write this up in ruby as I default to it for quick 5 minute deals like this. Turns out ruby apparently uses quicksort for it’s array.sort. Hrm… well, I figured comparing this to quicksort in ruby wouldn’t show me anything. I’ve also been starting to write more c code lately (as I’ve not done any for closer to a decade), so I figured why not try some ruby inline?

To check this out you’ll need rubyinline (which you probably have if you do any rails dev at all).

require 'rubygems'
gem 'RubyInline'
require 'inline'
require 'benchmark'
 
class Array
 
  inline do |builder|
    builder.c_raw "
      static VALUE gnomesort(int argc, VALUE *argv, VALUE self) {
        long len;
        int i, tmp; 
 
        VALUE *array = RARRAY(self)->ptr;
        len = RARRAY(self)->len;
 
        for(i=1; i<len; ){
          if(array[i-1] <= array[i]) {
            ++i;
          } else {
            tmp = array[i];
            array[i] = array[i-1];
            array[i-1] = tmp;
            --i;
            if(i == 0)
               i = 1;
          }
        }
 
        rb_iv_set(self, \"@unit\", *array);
        return self;
      }
    "
  end
 
end
 
def make_ray
  sizer = 1000
 
  blah = []
  1000.times do |i|
    blah.push rand(1000)
  end
 
  return blah
end
 
a = make_ray
 
Benchmark.bmbm do |x|
  x.report("C")    do for i in 1..10000; a.gnomesort; end end
  x.report("Ruby") do for i in 1..10000; a.sort; end end
end

Now, keep in mind, this is NOT a fair comparison towards the relative speed of these sorts:

• it only sorts ints not strings and so forth
• it’s in c, the other is in ruby
• I only tested relatively small lists — nothing in the millions or billions
• the key size is at max 4 digits — nothing to sneeze at

so take it with a grain of salt — I’m sure pretty much 9 out of 10 of my readers could easily turn that into a quicksort/radix implementation and blow it out of the water.

Anyways, I’m glad Hamid was encouraged to re-name this as it does bring a different perspective to the table and now I know how the music was made!

So if you have been reading hackernews as of late no doubt you have seen straight up holy wars over the use of evented libraries versus threads. Frankly, I think the whole evented versus threads debate is kinda stupid and is only being used by people trying to improve their social equity off of the latest buzzwords — so I guess I’ll go ahead and enhance mine.

So, wait, the audience cries —

In a way, yes. If you choose to use any of numerous evented frameworks out there, chances are at some level your libraries will be utilizing their own threads. Having said that, if you aren’t using evented libraries you are obviously not right in the head. Let me digress.

We all learned in school (or on irc) that there are several different types of i/o models. I’m sure someone will want to say I’m full of shit on this but this is the way I see the world:

I/O Models

• blocking I/O — wait for data and completely block the rest of the program from execution (waste of cpu time)
• non-blocking I/O — continously poll the kernel to see if data is ready (total cpu killer)
• select/poll — traditional multiplexing, blocks on select between file descriptors, waits for N fds to be ready; takes more than one syscall (more expensive)
• signal driven I/O — returns immediately && gives us a handler
• asynchronous I/O — sigio tells us when we can grab data versus asynch tells us when the data has BEEN grabbed

You have synchronous and asynchronous driven I/O.
The first four are considered synchronous because they ALL block in some form or another — the last one obtained through stuff like libaio is considered asynchronous.

Why does this shit matter!?!
This shit matters because if you have one thread you have one execution stack. Multiple threads have multiple execution stacks. When you are using asynchronous I/O you WILL NOT block said execution stack and will be able to continue with things as normal. You see, you can have your cake and eat it too!

However, what a lot pro-thread, fuck-the-evented-model proponents won’t tell you is that if you have a blocking thread you are STILL blocking the rest of your threads completely defeating the purpose. A ton of people miss this point because frankly they have never done any systems programming and have not had to learn about it.

Let’s hop back to IRC for a moment. Pretend your are 14 years old again and you are planning on raiding the pentagon and fucking over your best friends quake2 server. Let’s say we aren’t interested in rooting their servers but more just interested in shitting all over it through a DoS attack so they can’t get their porn on. Our vectors of attack here are pretty easy to identify:

• Look for syscalls that use BLOCKING I/O — if we can queue up enough requests than we are gravy.
• Look for places where they are not using seperate threads or processes on BLOCKING I/O.
• Look for places where they are not setting timeouts on BLOCKING I/O.

get the pattern here? blocking I/O is terrible and we are all guilty of it — I guess it’s a good thing that we have the entire goddamn military to look up to and try to emulate through FPS video games rather than hackers like ryan that want to dominate the world through computers instead. —

rant aside….

but as someone pointed out earlier on — does it come down to the nature of the work involved? I/O or processor (reading/writing data or doing math?) really, your bottlenecks SHOULD NOT be the processor as long as you aren’t trying to re-invent bubblesort from the ground up and you are missing your left side of the brain — it should be the database, file access, and network access as the system time involved in processing that data is orders of magnitude longer than hitting the cpu or the registers — and let’s face it — all of you reading this or probably working on webapps nowadays so you are inherently dealing with databases and reading template files and so forth so this argument of ‘I/O versus processor’ is just fucking retarded

Bradford Cross says “Blocking vs. non-blocking work is related to the issues of return values vs. side effects and being invoked directly vs. polling a work queue” — what a great quote this was! — unfortunately, I’m not quite sure everyone read this article the same way I did

So is my suggestion to abandon threads and just fork everything? No, I hope I made it clear that both are good to have — besides fork is fucking expensive cause you have to copy everything over — even copy-on-write solutions and IPC is not the coolest thing to deal with, that and for our .NET friends — they don’t get to fork.

so what about node?

node.js brings to the table the ability to spawn thousands of connections w/out all the bullshit involved in making it safe and using ONLY ONE PROCESS — I have yet to see any implementations that make it this easy in ruby or python — yes, there are eventmachine implementations, but it’s nowhere even close to comparable, end of story, argument over, period, finale!

alioth shows that v8 beats the shit out of python, ruby, perl, and java6 on a LOT of different cases…. — and when I mean beat the shit out of — I mean making pizza sauce out of it’s face.

Frankly, node could’ve been implemented in ANY language — it would not have mattered — what matters is the fact that it is shifting the paradigm of things towards evented style programming rather than betting the house on threads.

so what’s the problem with ruby threads/et. al.?

practically ALL io, file, networking, database libraries and gems are synchronous and blocking — this is the same for pretty much everything else as well, face it — we’ve been collectively programing like retards for a while and this attitude MUST change otherwise this universe’s space/time continuum will come to an end and Oersted himself will jump in from another dimension yelling at us

The cool thing that node brings to the table is that you can’t make native javascript I/O (synch or asynch) calls cause javascript does not have any I/O!!! This is important to realize because it doesn’t allow a huge repository of synchronous style crap to be built upon (say in the form of the behemoth of elephant crap that rubygems is) and later wake up the next morning with a hangover wondering why you were banging that code all night and didn’t even get pictures.

To sum up — I think the threads vs. event loop argument is completely and utterly retarded.

Birds Eye View of Node.js

I’m sure you’ve all seen/heard stuff about using real-time websockets — this guy used node.js but a lot of ppl don’t know that there is so much more to this framework than meets the eye.

First off, it has no locks, is non-blocking in as many places as possible and uses very little memory. Oh yeh, pretty much everything has a callback, following an evented model. Some POSIX operations return a ‘promise’ which signal a ‘success’ or ‘error’. Node is ridiculously powerful as you can see by reading over the api. There is so much here to do and really allows you to do some ‘system-level’ style of programming with a whole slew of POSIX style (compliant?) system calls

Concurrency without Multi-Threading?
Most of the built in hard-core concurrency comes from the evented nature of the framework. Being able to immediately respond to incoming requests while something is off reading from a database provides much needed clock cycles. Right now it appears there is no straight-forward way of utilizing multiple cores other than putting a load balancer of sorts in front and sending over unix sockets or something but there are plans to including forking in the future so that different processes may emerge taking advantage of the cores and utilize message passing to talk to each other. You can of course fork new processes using the system calls available.

Apparently the developers of node are looking towards using web workers available in HTML5 which will also alleviate some concurrency issues.

So who uses it? Is it production quality yet? Well, you can always ask the good folks over at Plurk.com (twitter for japan) who run it w/over 100k concurrent users — apparently this runs on one box with 8 proccesses…. uhhhh…

… maybe I should go spend some time in japan … I keep finding all sorts of shit I like from there

Hello World in Node.js

var sys = require('sys');
sys.puts('hello!');

so first off we include the sys lib then we do our puts statement .. pretty simple
let’s take a look at a

Simple Node.js HTTP Server

var http = require('http');
http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(8124, "127.0.0.1");
console.log('Server running at http://127.0.0.1:8124/');

yeh… it doesn’t really get too hard but this shows the evented nature of node — shit is retardo fast

Speed Demon Benches

ab -c 100 -n 1000 http://127.0.0.1:8124/
Requests per second:    6792.23 [#/sec] (mean)

… but it’s not just for HTTP

if you have any need for networking this is a badass framework to be using. It uses a microkernel design by including modules that you need rather than a bunch of shit you *might* use — big win in my favor..even down to sys/http modules.

you can find a list of supporting modules here

Binary Data

I’m including this section here because when I was looking for information on sending bytes here and there I found lots of articles on the internet saying this was not easily done with node.js — this IS EASILY DONE now. You CAN make a buffer with binary data now and NOT have to specify ‘utf8′, ‘ascii’, ‘base64′, etc…the ‘net2 branch’ that you might see referenced to on the net is already merged in and the api docs are current though can be rather confusing after reading older documentation found around…so ignore that old shit — so, just to clarify, if you are sending binary data through some random socket you can do that now.

Want to see an example of this mysterious byte buffer?

  Buffer = require('buffer').Buffer;
  buf = new Buffer(8);
  for(i =0; i < 8; i+=1) { buf[i] = 0; }

very interesting that you can do this now since javascript doesn’t like binary data that well

buffer.copy does a memcpy() …. begging to get owned by some hackers — remember to sanitize your data

Downloading/Installing

before doing anything else go ahead and grab the latest (as of this writing):

wget http://nodejs.org/dist/node-v0.1.102.tar.gz
./configure && make && sudo make install

woohoo! simple build process ftw

want to use express? it’s pretty cool, like sinatra and it’ll allow you to get to know the most tubular npm tool.

let’s check our version because all of this code is bleeding edge stuff

feydr@mhu:~$ node -v
v0.1.102

also, let’s grab npm first:

git clone http://github.com/isaacs/npm
cd npm; sudo make install

now we can install express and it’s supporting modules that we want:

sudo npm install jade
sudo npm install less
sudo npm install connect
sudo npm install express

playing with it:

mkdir test/; cd test;
express # generates a new app
node app.js # to start the server
# switch to a diff. window
curl 127.0.0.1:3000

speed?
much slower than native node.js but hey, it takes a lot of pain away
it’s still slightly retarded fast compared to say rails and django

ab -c 100 -n 1000 http://127.0.0.1:3000/
Requests per second:    1200.16 [#/sec] (mean)

Support && References
shit tons of ppl on the #node.js irc channel

Need something to show the boss — some slides

My mind is literally wheeling with the crazy shit that this framework allows you to do… particularly remotely…

Last thing before you leave, watch this.

I grew up learning how to write code using C from man pages managing memory on my own. Malloc and free were my friends and I would pride myself on running valgrind on a paritcular piece of code passing through it’s tests.

The JVM of course uses a managed memory model with it’s garbage collector. This means no more memory leaks right? Wrong.

Example:

Let’s suppose you have an object you instantiated called Blah. Blah gets used but is never de-referenced nor does it have a way to tell the GC that it is a candidate for removing. (You never actually remove an object — you just tell the garbage collector that is is ready to be removed.

” Note the wording: Just because an object is a candidate for collection doesn’t mean it will be immediately collected.” Candidate for Removal

What happens after you instantitate 300k of these Blah objects that use 80 bytes a pop? You just lost 22 meg that you won’t regain until you kill that process!

So to cut to the chase I had a tomcat servlet that kept dying about once a week and I didn’t allocate any time to look at it what the problem was until now as I could just restart it and everything would be peachy. I finally decided to look at it.

This is what I did.

Get a Heap Dump

First, you need to obtain a heap dump — without the heap dump you are having better luck at the casino. Chances are if you are crashing cause of an OOM memory error you won’t be able to get your heap dump from the production site — we don’t really want to screw with that anyways.

So, let’s duplicate our results on our dev box. Like I said before, we were looking at a tomcat servlet that was not working so in theory we should be able to hit our servlet with enough requests to trigger some heap growth.

I wrote a quick little script in ruby to slam our servlet, here it is:

#!/usr/bin/ruby
 
require 'rubygems'
require 'uri'
require 'net/http'
require 'cgi'
require 'timeout'
require 'open-uri'
require 'monitor'
 
File.open('b.txt', 'r') do |f| @filetopost = f.read end
@params = {'text' => CGI::escapeHTML(@filetopost), 'submit' => 'Submit'}
 
def pingsite
  500.times do |i|
    x = Net::HTTP.post_form(URI.parse('http://127.0.0.1:8080/upload/process'), @params)
    puts i
  end
end
 
threads = []
(1..20).each do |i|
  threads << Thread.new do
    pingsite
  end
end
 
threads.each do |t|
  t.join
end

This essentially as you might have guessed posts the contents of a text file through a form to our servlet for processing (text into xml). It makes 10k requests (500 at 20 threads).

BEFORE you run this you’ll want to hop into jconsole. JConsole will allow you to watch the action live as you see your heap grow and be garbage collected. You’ll see that the faster the heap grows the faster the garbage collector will start to run so your graph will start to have it’s lines closer together. What you are wanting to do here is just make the graph curve upward — that’s it.

Once you have a graph that looks like this:

we are ready for the next step. This graph actually grew up into the 400s and was much more pronounced by the time my 10k requests were done — all you really need to do is show proof that there is a growth — the more the merrier as it should help you drill down more easily to determine what is piling up.

Dump the Heap

Find out the process we need to dump:

ps aux | grep tomcat

dump it:

jmap -dump:live,format=b,file=heap.bin 6608

The dump will need to be initiated by the same user owning the tomcat process or someone with perms for it (tomcat6, root, joebob, whatever).

Great! Let’s take a quick look at the size

ls -lh heap.bin

200 meg! WTF!?

Get Relevant Stats

Ok, now let’s hop into jvisualvm (visualvm). I went ahead and changed the perms on the dump file so I could run it as my own user. Once in visualvm we want to go to our classes view in the top right hand corner. From here we can sort by # of instances or % of how much the heap is towards the object. I focused on instance count because that seemed like something that would let me know what was up. I noticed that at least one thing was wrong here. I had something like 300k prepared statement instances being built — what the fuck?!

Fix the Problem

So I went back to my code and did a .close() on them.

        // note this is scala
        pStmt.executeUpdate()
 
        pStmt.close()
 
        if(stmt != null) {
          stmt.close()
        }

Measure Again to Ensure the Problem is Fixed

After this was done we compile our jar, throw it back to our servlet, restart tomcat and run through our bulk POSTing again. The theory here is that we should be able to have a noticeable impact our instance count for prepared statements and have a noticeably different graph as well. I did not know that simply copying your webapp over to the root directory was not enough — you NEED to restart tomcat to get accurate results as items in the PermGen space will be duplicated if you simply copy a new webapp over (and after doing that a couple of times you’ll find a nice little PermGen OOM error in your catalina log files.)

Here’s the new graph:

The new heap size is :20 meg!!

Well, we might not have squashed all of our bugs but I’d bet good money that this servlet doesn’t die again in the next week or so.

Other Profiling Tools I Used but Did not Like:

jhat: I tried this out and it seems like it is supposed to be used on remote servers, however, there was absolute NO thought put into making it a nice UI to use by a human and it didn’t particularly lend itself well to a computer looking at information so frankly I think it’s a piece of shit.

Memory Analyzer: Speaking of piece of shit, this rotten crap came with Eclipse (now, I have my own opinions about how horrible eclipse is to begin with so maybe that’s just my prejudice rubbing off) but this kept crashing every fucking time I tried to load the heap. I tried passing in the arguments -Xms128m -Xmx1024m and all but it just did not like me. Fuck it — I got the job done regardless.

Anyways, I hope this information was somewhat useful for someone else — while documentation exists on the internet for this it wasn’t the most readily available info.

We’ve all been there — we need to parse an xml document so we pull out some xpath expression and get to work on grabbing our little bit of data from our document.

token = REXML::XPath.first(doc, "/books/title/")

This works great and is easy to use no matter your language of choice for MOST things. Now, let’s pretend this particular document is 10 meg — fuck that, let’s pretend it’s 100 meg — actually let’s get real and call it 2.5 gig of xml.

No, it won’t if you are using DOM parsing since that is based upon building a tree. That is if you are willing to wait for the fucker to even load itself into the tree — forget the memory usage.

I ran into this problem earlier today — memory hit a little over what the document size was in htop but the cpu just kept burning while the world was turning. I eventually killed it and contemplated splitting the xml apart — of course … I was going to run into the exact same problem if I continued to utilize the old worn out DOM parsing methods.

This is when I was turned onto SAX parsing from some random mailing list plea for help. All I need to know I learnt from man pages and mailing lists. I had ran across SAX in the past but was not too excited that I had to prepare a language JUST to parse my document rather than pass it some tried and true xpath — basically I was used to parsing html DOM — NOT million line xml documents. I had read that it was faster in the past but my feeble attempts at comparing documents of the 200k size were just an exercise in futility to find any meaning in using SAX.

Now let me address the audience in what happened. That little job I killed that was bitch-smacking one of my cores around for over 10 minutes and using over 10% of 6gig of memory did the EXACT same extraction under 10 seconds utilizing something like 0.1% of memory and only 70%ish CPU. I, to use the phrase slightly, shit a fucking brick!

Want to see an utterly contrived example of pulling 40+k urls from a sitemap which tied up my cores for-fucking-ever using old style DOM parsing but was done in less than 10 seconds with SAX?

require 'rexml/document'
require 'rexml/streamlistener'
require 'mysql'
include REXML
 
MY = Mysql.new("host", "luser", "assword", "dbname")
 
class ParseThatShit
  include REXML::StreamListener
  def tag_start(*args)
  end
 
  def text(data)
    return if data =~ /^\w*$/
    return if data =~ /\d\.\d/
    st = MY.prepare("insert into links (name) values (?)")
    st.execute(data)
    st.close
  end
end
 
pts = ParseThatShit.new
xmlfile = File.new("big-ass-sitemap.xml")
Document.parse_stream(xmlfile, pts)

Remember boys and girls — this Example is just an example to show you the power of SAX — does it take more LOC than a 3 line xpath block? Yeh, but so does your mother.

Use the power of SAX and drink the spice while you slip into the higher dimension of parsing power!

“fuck a bunch of perl monk motherfuckers!”

“perl sucks!”

“perl can suck a dick!”

ok, well that was slightly exagerrated =)

but .. so seems the sentiment sometimes browsing certain portions of the net
(dzone, hackernews, etc.)

Really, imo perl was one of the first dynamic interpreted languages that a LOT of people
got on board with and really opened the door to web development… say what
you want but perl was the “first web language”™

Dancer

A couple of weeks I ran into dancer. I was really quite interested because it’s cousin,
sinatra from rubyland is my favorite framework from that language.

I have to admit that I have not used perl for a while but I had dancer up and tapping it’s toes
within minutes and it is pretty damn fast.

can I sum it up?
yes …. dancer turns perl web development from this…

installing

feydr@mhu:~$ sudo cpan Dancer

cpan will block when it’s dling new modules so make sure to pay attn to any visual beeps if you are in screen or something, or you could always use expect or whatever

Sample Hello World written in Dancer

#!/usr/bin/perl
 
use Dancer;
 
get '/hi' => sub {
  "Hello World!"
};
 
dance;

feydr@mhu:~$ perl ./hi.pl

hit it up at: http://127.0.0.1:3000/hi

Some Slightly Drunken Made Benchmarks:
That last red bar should be at 672 request/second.

need some more example code? check out the dancer website source built using dancer

Deploying Dancer into Production

I’d avoid abusing this ass-rocking software with something as terrible as apache as you just won’t get the same performance and you’ll have to do stupid CGI/FastCGI tricks. However, if you must use apache (because you are already using it for 10 other websites on the same box) then you can still use PSGI.

Dancer runs best under PSGI/PLACK

Wait! What the fuck is that shit? PSGI is a spec akin to Python’s WSGI while PLACK is to ruby’s Rack.

confused? just use Starman!!– Perl Reference

StarMan Features

• preforking web server that reaps dead kids and does auto restarts
• can utilize unix sockets
• very little memory footprint (like 7-12 meg little)
• No Win32 Support, yes this is a feature

It’s also written by Japanese hackers. I don’t think I need to mention why Japanese hackers rock..

back to dancer…

supported session engines by default:

• memory (no persistence — useful for debugging)
• yaml file-based (puke!)
• memcached (distributed production env ftw!)
• cookie (make sure you protect your shit from timing attacks)

other features:

• wilcard routing path matching
• before filters

In comparison to Catalyst? Well, the most obvious is that Dancer is light-weight — like cheerleader on prom night light-weight.

In Closing

For my next ‘small’ project this will probably be written with Dancer rather than Sinatra.
Yes, the terse syntax of perl might not meet the eloquence of ruby but for memory usage
Dancer will win hands down everytime.

]]> http://airodig.com/2010/05/26/dancing-your-way-into-effortless-web-development-with-dancer-and-perl/feed/ 0 http://airodig.com/2010/05/22/getting-oriented-with-orientdb/ http://airodig.com/2010/05/22/getting-oriented-with-orientdb/#comments Sat, 22 May 2010 18:46:47 +0000 feydr http://airodig.com/?p=341 Was on the nosql mailing list the other day and I ran across a message from Cliff Moon trolling it up

“I find the name of your database to be extremely racist.”

To tell you the truth I usually am ctrl-D’ng (mutt mass deleting) all threads from this mailing list since I just do not have time to read them all. This of course caught my eye though and I’ve been waiting for a day to set aside some time to write up an article on it. So that day is today and this is that article.

These guys make a TON of claims and the description page is chockful of buzzwords describing what exactly OrientDB is. Let’s take a quick look.
Features

• Written in Java
• 40k inserts/second on commodity hardware
• Document/Graph Based DataBase
• Supports Schema-less, schema-full, schema-mixed modes
• Security Profiling for users/roles Based Access
• Supports SQL as a query language
• ACID compliant with support for transactions
• Less than 400k Footprint for Full Server (embeddable)

It is truthly very easy to get started unlike other solutions out there. It unfortuantely has a XML config file but the default one that comes with the release I downloaded (0.9.13) clocks in at only 22 lines and is fairly readable. It listens for binary connections on port 2424 by default and a built-in web interface on 2480 by default. The package I downloaded came with jars ready to go so there was no bullshit to configure or anything but if you grab sources ant should work things out for you.

Example:

svn checkout https://orient.googlecode.com/svn/trunk/ orient
export JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.20/
chmod a+x build.sh
./build.sh

Like any good startup script it proudly displays it’s ascii logo.
It really is quite a nice little web interface I have to admit. For those of you who have/want to care about licenses it is under the Apache 2.0 License. There are currently 12 members on it’s google groups mailing list.

Web Interface
The web interface has a RESTful interface that supports GET, PUT, POST, and DELETE operations. This is really cool because then you can do all your requests with javascript and pull down result sets with json. Being able to put this crap behind a load balancer allows you to scale like a motherfucker when it comes to reads, which is one reason why other databases that have RESTful interfaces caught on so quickly. It has a built in profiler that really rocks out and I firmly believe all software should have built in shit like this — it is just common sense for applications like this. There are tabs right next to it that show you configurations, connections and and database pools — cool stuff indeed.

OrientDB Profiler

The schema view is pretty cool as well as you can see all of your classes (tables) , their record count and their cluster count. Physical clusters make for fast IO, whereas logicals save space but are slower. Of course in the day and age of pocket terrabyte storage drives — who the hell needs more space? We need ram and cpu damnit!

Security
I must say that out of all the NoSQL solutions out there OrientDB really fucking shines with it’s security built-in from the get-go — almost every other system out there has added it on as an *optional* feature — as if security should ever be treated that way — it’s like modern day programmers for some reason still do not know that 14 year old kids have scripts that scan the net for open things like this — some mid-20s kids have those exact same scripts.

Apparently the underlying data structure is a “RB+Tree” which as the authors point out is a mix of Red-Black and B+ Tree algos — this they assert give you the best mix of balancing, speed, and 50% less memory usage.

All in all this is a pretty nice offering and I might need to play with again here in the future. I have a couple million xml documents I’m thinking about transforming to json. What are your thoughts on this fellow readers?

]]> http://airodig.com/2010/05/22/getting-oriented-with-orientdb/feed/ 1 http://airodig.com/2010/04/27/play-with-java-and-scala-web-apps/ http://airodig.com/2010/04/27/play-with-java-and-scala-web-apps/#comments Tue, 27 Apr 2010 19:56:59 +0000 feydr http://airodig.com/?p=323 I got wind of the play framework a couple of months ago and have been passively reading their mailing list each day. I have to say — it’s very active and the framework is just top notch. We here at telematter have used a lot of the frameworks in ruby land and our main app is written in merb. Lately though merb has become a ghost-town of support, documentation, and community involvement. Upgrading a server takes an exorbinant amount of time, patience and the willingness NOT to throw you computer through the window.

Dependencies are a big problem here but so are the tests. With the ruby community’s ever so crushing pride about various ways to accomplish a task you usually wind up having to either pick between the options of using a LOT of plugins/dependencies or not using any. Also, you know you have problems when after spending the better part of a morning upgrading your dev environment you can’t run your tests.

So we started looking at different web application frameworks. We have a rather large existing codebase in java already to do some text parsing but xml config files and the ever-growing concern of java developers to deal with encapsulation in the most fucked up manner imaginable leads us having java with a sour taste in our mouths. Scala on the other hand has been what I have deemed “what java SHOULD have been”. Seriously, that language is so elegant compared to it’s predecessor that they should just scrap everything and move on from that point forward. The JVM is really solid shit but java itself — ehh…

Anyways on to the meat of this quick article. Why do I like play?

• class reloading
• easy setup
• fucking awesome test runner
• great documentation && active community
• NO xml config files
• use with scala

Class Reloading

Do I even need to explain this one? Having the ability to reload your classes on the fly without having to go through configuration hell is so ridiculously helpful it’s insane. Your development productivity will gain leaps and bounds and with integrated auto-testing you are assured a winner.

Easy Setup

It does not get easier than a:

wget http://download.playframework.org/releases/play-1.0.2.1.zip
unzip play*
sudo cp -R play* /usr/bin/play
play new my_new_app
play run

Fucking Awesome Test Runner

I need to attach these screenshots to show you the awesomeness of this.

Test Index

Selenium Test Runner

Great Documentation && Active Community

When you first boot up Play (and it boots FAST) you will find that you have all the basics to start hacking up the HelloWorld into whatever you wish. A simple click of the mouse gives you your environment settings (which show you everything from paths to thread info, to monitors). Play also has a cool little route built in http://127.0.0.1:9000/@documentation that has screencasts, sample apps, faqs, and pretty much everything you need to know to start playing around.

Admittedly there were only 12 users online in IRC earlier today when I wrote it but the mailing list has ~30 emails/day with active development/discussion.

NO XML Config Files

Yes, that’s right ladies and gentlemen — no goddamn config files written in fucking XML. WTF! Your main config file, application.conf is just your basic java properties set — no need to get all crazy and declare everything under the sun.

What’s the routes look like?

# Routes
# This file defines all application routes (Higher priority routes first)
# ~~~~
 
# Home page
GET     /                                       Application.index
 
# Map static resources from the /app/public folder to the /public path
GET     /public/                                staticDir:public
 
# Catch all
*       /{controller}/{action}                  {controller}.{action}

yeh….. they are that simple looking

Use with Scala

There is a scala module that I highly recommend you download and use — oh and if you like scala — you can get on the akka bandwagon as well. It’s also highly recommended.

Do yourself a favor and Play Now!

]]> http://airodig.com/2010/04/27/play-with-java-and-scala-web-apps/feed/ 1