The next thing I noticed was the lowered memory footprint and CPU usage under load. I don't have hard numbers yet, but it is significantly lower. Further, and I only say this because I cannot substantiate it yet, it just feels faster. Based on other more technical reviews and comparison, I know this to be the case. Nevertheless, I plan on doing a more scientific evaluation of the situation, mainly regarding the number of requests per second that both can sustain on the same hardware and configuration.

Another piece of software that becomes useful when deploying Django on Nginx is FastCGI, installed conveniently in the python-flup package (sudo apt-get install python-flup). It includes a FastCGI server and requires essentially zero configuration. Although, for convenience, a control script like the one below will be useful.

Here is my shell script for managing the FastCGI server:

#!/bin/bash

CWD=$(cd `dirname $0` && pwd)

MYAPP=test_app
PIDFILE=/tmp/${MYAPP}_fcgi.pid
HOST=127.0.0.1
PORT=8080

# Associate it with the settings file
#SETTINGS=
# Use a socket instead of host/port
#SOCKET=
# Maximum requests for a child to service before expiring

METHOD=prefork
# Maximum number of children to have idle
MAXSPARE=5
# Minimum number of children to have idle
MINSPARE=5
# Maximum number of children to spawn
MAXCHILDREN=10
#MAXREQ=
# Spawning method - prefork or threaded

cd "`dirname $0`"

function failure () {
  STATUS=$?;
  echo; echo "Failed $1 (exit code ${STATUS}).";
  exit ${STATUS};
}

function start_server () {
  ./manage.py runfcgi pidfile=$PIDFILE \
    ${HOST:+host=$HOST} \
    ${PORT:+port=$PORT} \
    ${SOCKET:+socket=$SOCKET} \
    ${SETTINGS:+--settings=$SETTINGS} \
    ${MAXREQ:+maxrequests=$MAXREQ} \
    ${METHOD:+method=$METHOD} \
    ${MAXSPARE:+maxspare=$MAXSPARE} \
    ${MINSPARE:+minspare=$MINSPARE} \
    ${MAXCHILDREN:+maxchildren=$MAXCHILDREN} \
    ${DAEMONISE:+damonize=True}
}

function stop_server () {
  kill `cat $PIDFILE` || failure "stopping fcgi"
  rm $PIDFILE
}

DAEMONISE=$2

case "$1" in
  start)
    echo -n "Starting fcgi: "
    [ -e $PIDFILE ] && { echo "PID file exists."; exit; }
    start_server || failure "starting fcgi"
    echo "Done."
    ;;
  stop)
    echo -n "Stopping fcgi: "
    [ -e $PIDFILE ] || { echo "No PID file found."; exit; }
    stop_server
    echo "Done."
    ;;
  poll)
    [ -e $PIDFILE ] && exit;
    start_server || failure "starting fcgi"
    ;;
  restart)
    echo -n "Restarting fcgi: "
    [ -e $PIDFILE ] || { echo -n "No PID file found..."; }
    stop_server
    start_server || failure "restarting fcgi"
    echo "Done."
    ;;
  *)
    echo "Usage: $0 {start|stop|restart} [--daemonise]"
    ;;
esac

exit 0

And here is a basic Nginx config that hooks into FastCGI (so much more simple than Apache!):

server {
        listen 80 default;
        server_name localhost;

        access_log /var/log/nginx/localhost.access.log;

        location / {
                root /var/www/nginx-default;
                index index.html index.htm;
        }

        location /django {
                fastcgi_pass 127.0.0.1:8080;
                fastcgi_param PATH_INFO $fastcgi_script_name;
                fastcgi_param REQUEST_METHOD $request_method;
                fastcgi_param QUERY_STRING $query_string;
                fastcgi_param CONTENT_TYPE $content_type;
                fastcgi_param CONTENT_LENGTH $content_length;
                fastcgi_param REMOTE_ADDR $remote_addr;
                fastcgi_param SERVER_PORT $server_port;
                fastcgi_param SERVER_PROTOCOL $server_protocol;
                fastcgi_pass_header Authorization;
                fastcgi_intercept_errors off;
        }

        location /site_media/ {
                alias /home/ubuntu/cbot/media/;
                access_log off;
                expires modified +24h;
        }

        location /admin_media/ {
                alias /usr/share/pyshared/django/contrib/admin/media/;
                access_log off;
                expires modified +24h;
        }
}

All of my examples are going to use memcached, as it by far the most efficient option and also one of the easiest to setup. I'm also going to assume that you're running your Django site on Linux (if not, why not!?), specifically Debian or any variant thereof.

First, you need to install memcached:

sudo apt-get install memcached

That will also take care of most of the setup for you. Now let's run it:

memcached -d -m 2048 -u root

The -d option tells it to run in a daemon (background) mode. -m tells it how many MBs of cache to allocate on the heap, and -u for which user to run under. There are many more options than this, and you should probably create a user for it and not run under root, but this is the easiest way to get it up and going.

Now, we need to install a Python library to interface with memcached. The best option that I've found thus far is python-memcache. Time for a little apt-get:

sudo apt-get install python-memcache

...and now you're good to go! Let's get to it.

First thing's first, you've got to add a single entry to your Django settings file (so difficult!):

# Use memcache on the server (much more efficient), local memory caching in dev
CACHE_BACKEND = 'memcached://127.0.0.1:11211/' if SERVER else 'locmem://'

This directive tells Django to use the memcached instance running on localhost (change if you need to) and at the default port, 11211. I have the cache backend set dynamically because I don't use memcahed on my development machine, and instead use basic memory caching which is not nearly as efficient (you shouldn't use it on a server) but it works just the same. Great for testing.

Now that the easy stuff is out of the way, it's time to take a look at how caching in Django actually functions. There are actually four different methods of caching, listed here in terms of increasing complexity:

1. Site-wide
2. Per-view
3. Template fragment
4. Low-level

Let's take a look at the details...

Site-Wide Caching

Side-wide caching will automagically cache EVERY page that doesn't have GET or POST variables. Be careful! This can have adverse effects in many cases, and I find it to be a bit heavy-handed, but I suppose there are some cases where this would be very useful.

In order to enable site-wide caching, you must add a few more entries to your settings:

# This tells memcached how to long hold each entry in memory
CACHE_MIDDLEWARE_SECONDS = 60
# And this sets the cache key prefix, which is useful if you are
# running many things on the same memcached instance
CACHE_MIDDLEWARE_KEY_PREFIX = ''

# You know what this is...
MIDDLEWARE_CLASSES = (
    ...
    'django.middleware.cache.UpdateCacheMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.cache.FetchFromCacheMiddleware',
    ...
)

Doing this also has the added benefit of setting the various HTTP and HTML caching control options automagically using your CACHE_MIDDLEWARE_SECONDS value.

And now your website has caching!

Per-View Caching

This option is quite a bit more flexible than site-wide caching. It will also cache based on the URL, so a call to "/my_super_cached_view/1/" will have a difference cache entry than "/my_super_cached_view/2/".

You can use it as a decorator:

from django.views.decorators.cache import cache_page

@cache_page(5 * 60) # cache for 5 minutes
def my_super_cached_view(request):
    ...

...or you can set it in your URLs file with the same import:

urlpatterns = ('',
    ...
    (r'^my_super_cached_view/(?P\d{4})/$', cache_page(my_super_cached_view, 5 * 50)),
    ...
)

Template Fragment Caching

This is the caching option that I have gotten the least mileage out of. Other people may have more use for it, but my development style doesn't generally favor it.

Within your template, import the cache library:

{% load cache %}
{% cache 120 footer request.user.username %}
    ...
{% endcache %}

In this example, 120 is the number of sections, footer is the name of the cache entry, and the third argument is optional and makes whatever is in footer cache uniquely for each user.

Low-Level Caching

This method is the one that I've used the most, as it is by far the most flexible, but is still incredibly simple and not very "low-level" in my opinion, but I suppose everything is relative! This method allows you to complete specify under which conditions to save and pull from the cache, unlike the previous versions which varied based on the URL or a single variable, like the active user.

Here's an adapted example of how I've used this method to great success:

from django.core.cache import cache

def my_view(request):
    # Cache per user and account
    account = request.session['account']
    user = account.holder.username
    key = 'key_prefix-%s-%s' % (user, account.code)
    # Is it in the cache?
    cached_html = cache.get(key)
    if not cached_html:
        # If not, then let's render the HTML and save it
        cached_html = render_to_string(
            'templates/my_view.html',
            {'foo':expensive_per_user_function(user)}
        )
        cache.set(key, cached_html, time_until_midnight())
    return HttpResponse(cached_html)

In this example, the rendering of a certain section of HTML was very expensive but varied in such a way that using the template variety was inconvenient. As you can see there are only really two methods, cache.get() and cache.set(). Quite simple!

You control the cache by specifying the key, which here is unique for each user viewing a custom a account object. It just so happened that the data that I cached here updated at midnight each night, so I used a utility function called time_until_midnight() that returns the number of seconds until 12am, which you can see here:

def time_until_midnight():
    """ Returns the seconds until midnight """
    today = datetime.date.today()
    tomorrow = today + datetime.timedelta(1)
    midnight = datetime.datetime(tomorrow.year, tomorrow.month, tomorrow.day)
    difference = midnight - datetime.datetime.now()
    return difference.seconds

Done and done!

Caching is your friend.

In the past I made a post about Django, which is my absolute favorite web application (back-end) framework. Coincidentally, Pyjamas (this is for creating the front-end, remember?!) ALSO uses Python. So in case you haven't had an aneurysm from the pure win yet, let me boil it down for you: you can write your entire web application in a single language, and an awesome one at that! For those of you who have created native GUI programs and web apps before, you can easily see the advantages that this type of approach has (hint: web development is... painful).

So enough of my fawning, let's roll!

My example is a modified version of the original GWT sample application used in my previous post. I will be adding persistence to the app. So when you add stocks to your list, they will show up again when you revisit the page, etc. Also, this will not be a global list that is viewable to all users, but is saved on a per-user basis. Also, it will NOT require an account (i.e. no login to tie the stocks lists to), it will be anonymous, based on cookies only. Not terribly complicated but it should make for a good example.

So here is the source as we left it, and here are the rest of the associated files. This is where we'll be starting from.

First off, here is the end product: http://stockwatcher2.derekschaefer.net/

It's the same as before, except it has persistence based on a Django session cookie set to expire in one year.

Try adding a few "stocks" and then refreshing, or even closing the browser and coming back. The same stocks should reappear. The only reason they won't is if you have configured your browser to be especially judicious when it comes to cookies, otherwise it should work fine (tested in Chrome, Firefox, Opera). The only exception is IE 8, as far as I know, which has rather severe problems with complex Javascript and I just can't be asked to make compatibility changes at the moment.

Let's first create the Django model for the database we're going to use to keep track of. It doesn't need to be complex, but we need to track everything based the session.

from django.db import models
from django.contrib.sessions.models import Session

class Stocks(models.Model):
    session = models.ForeignKey(Session)
    symbol = models.CharField(max_length=10)

    def __unicode__(self):
        return '%s | %s' % (self.session.pk, self.symbol)

To tie the stocks to a given session, it's as simple as importing the Session model and creating a foreign key field for it (you must have "django.contrib.sessions" listed under "INSTALLED_APPS" in your settings file, like so:

...
INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    # Uncomment the next line to enable the admin:
    'django.contrib.admin',
    # Uncomment the next line to enable admin documentation:
    # 'django.contrib.admindocs',
    'stockwatcher2.stockwatcher',
)
...

Now let's look at the views. The first thing we need to consider is how Pyjamas and Django are going to communicate. As it turns out, the most convenient way to go about this to use JSON-RPC (a.k.a. JSON remote procedure calls). Pyjamas already has this functionality build it, so we only need to utilize it, but Django has a somewhat harder time. To cut right to it, I've found that the best option is libcommonDjango (which also contains lots of other useful things for Django). It will easily allow us to setup an RPC environment using standard Django views. First, you will have to download libcommonDjango and run "python setup.py install" which will place it in your "site-packages" directory so it will be on your $PYTHONPATH. And that's it! Now let's look at how to use it:

from django_pimentech.network import *
from django.shortcuts import render_to_response
from django.template import RequestContext
from django.contrib.sessions.models import Session

from stockwatcher.models import Stocks

import datetime

service = JSONRPCService()

def index(request):
    request.session.set_expiry(datetime.datetime.now() + datetime.timedelta(365))
    request.session.save()
    return render_to_response('StockWatcher.html')

def getStocks(request):
    try:
        session = Session.objects.get(pk=request.session._session_key)
        stocks = Stocks.objects.filter(session=session)
    except Exception as e:
        return []
    return [(stock.symbol) for stock in stocks]
service.add_method('getStocks', getStocks)

def addStock(request, symbol):
    session = Session.objects.get(pk=request.session._session_key)
    # Don't allow duplicates
    if len(Stocks.objects.filter(session=session, symbol=symbol)) > 0:
        return getStocks(request)
    stock = Stocks()
    stock.session = session
    stock.symbol = symbol
    stock.save()
    return getStocks(request)
service.add_method('addStock', addStock)

def deleteStock(request, symbol):
    try:
        session = Session.objects.get(pk=request.session._session_key)
        Stocks.objects.get(session=session, symbol=symbol).delete()
    except Stocks.DoesNotExist:
        pass
    return getStocks(request)
service.add_method('deleteStock', deleteStock)

As you can see, these are just normal Django view functions. The only difference is we import "from django_pimentech.network import *" and then setup the service variable to be a "JSONRPCService()" object. Finally then adding each view to the service by calling "service.add_method", which can also be done with a decorator but I've found this way to be more compatible.

The other things of note are "request.session.set_expiry(datetime.datetime.now() + datetime.timedelta(365))" and retrieving the session primary key via "request.session._session_key". With the former I am setting the session (and correspondingly the session cookie that will be stored) expiration date to one year (totally arbitrary) from the current time.

Also, you will need to add the following to your URLs file like so:

urlpatterns = patterns('',
    ...
    (r'^services/$', 'stockwatcher.views.service'),
    ...
)

Additionally, you will probably run into a CSRF verification error, in which case you need to disable CSRF verification in your settings file by removing "django.middleware.csrf.CsrfViewMiddleware" from "MIDDLEWARE_CLASSES".

Now the JSON-RPC environment is ready to go, which leaves the Pyjamas front-end.

In terms of organization, it's easiest if you place all the Pyjamas on your Django "MEDIA_PATH", as you can see here: http://downloads.derekschaefer.net/index.php?dir=pyjamas/stockwatcher2/

Once that is done, we just need to modify "StockWatcher.py".

import pyjd

from pyjamas.ui.RootPanel import RootPanel
from pyjamas.ui.Label import Label
from pyjamas.ui.Button import Button
from pyjamas.ui.FlexTable import FlexTable
from pyjamas.ui.HorizontalPanel import HorizontalPanel
from pyjamas.ui.TextBox import TextBox
from pyjamas.ui.VerticalPanel import VerticalPanel
from pyjamas.ui.KeyboardListener import KeyboardHandler, KEY_ENTER
from pyjamas.Timer import Timer
from pyjamas import Window

from pyjamas.JSONService import JSONProxy

from StockPrice import StockPrice

import re
import random
import datetime

class StockWatcher:
    def onModuleLoad(self):
        '''
        This is the main entry point method.
        '''

        # Setup JSON RPC
        self.remote = DataService()

        # Initialize member variables
        self.mainPanel = VerticalPanel()
        self.stocksFlexTable = FlexTable()
        self.addPanel = HorizontalPanel()
        self.newSymbolTextBox = TextBox()
        self.lastUpdatedLabel = Label()
        self.addStockButton = Button('Add', self.addStock)
        self.stocks = []
        self.stocksTableColumns = ['Symbol', 'Price', 'Change', 'Remove']

        # Add styles to elements in the stock list table
        self.stocksFlexTable.getRowFormatter().addStyleName(0, 'watchListHeader')
        self.stocksFlexTable.addStyleName('watchList')
        self.stocksFlexTable.getCellFormatter().addStyleName(0, 1, 'watchListNumericColumn')
        self.stocksFlexTable.getCellFormatter().addStyleName(0, 2, 'watchListNumericColumn')
        self.stocksFlexTable.getCellFormatter().addStyleName(0, 3, 'watchListRemoveColumn')

        # Create table for stock data
        for i in range(len(self.stocksTableColumns)):
            self.stocksFlexTable.setText(0, i, self.stocksTableColumns[i])

        # Assemble Add Stock panel
        self.addPanel.add(self.newSymbolTextBox)
        self.addPanel.add(self.addStockButton)
        self.addPanel.addStyleName('addPanel')

        # Assemble Main panel
        self.mainPanel.add(self.stocksFlexTable)
        self.mainPanel.add(self.addPanel)
        self.mainPanel.add(self.lastUpdatedLabel)

        # Associate the Main panel with the HTML host page
        RootPanel('stocksList').add(self.mainPanel)

        # Move cursor focus to the input box
        self.newSymbolTextBox.setFocus(True)

        # Setup timer to refresh list automatically
        refresh = self.refreshWatchlist
        class MyTimer(Timer):
            def run(self):
                refresh()
        refreshTimer = MyTimer()
        refreshTimer.scheduleRepeating(5000)

        # Listen for keyboard events in the input box
        self_addStock = self.addStock
        class StockTextBox_KeyboardHandler():
            def onKeyPress(self, sender, keycode, modifiers):
                if keycode == KEY_ENTER:
                    self_addStock()
            def onKeyDown(self, sender, keycode, modifiers): return
            def onKeyUp(self, sender, keycode, modifiers): return
        self.newSymbolTextBox.addKeyboardListener(StockTextBox_KeyboardHandler())

        # Load the stocks
        self.remote.getStocks(self)

    def addStock(self, sender, symbol=None):
        '''
        Add stock to FlexTable. Executed when the user clicks the addStockButton
        or presses enter in the newSymbolTextBox
        '''

        if symbol is None:
            # Get the symbol
            symbol = self.newSymbolTextBox.getText().upper().trim()
            self.newSymbolTextBox.setText('')
            # Don't add the stock if it's already in the table
            if symbol in self.stocks:
                return
            # Tell the server that we're adding this stock
            self.remote.addStock(symbol, self)
            self.newSymbolTextBox.setFocus(True)
            # Stocks code must be between 1 and 10 chars that are numbers/letters/dots
            p = re.compile('^[0-9A-Z\\.]{1,10}$')
            if p.match(symbol) == None:
                Window.alert('"%s" is not a valid symbol.' % symbol)
                self.newSymbolTextBox.selectAll()
                return    

        # Add the stock to the table
        row = self.stocksFlexTable.getRowCount()
        self.stocks.append(symbol)
        self.stocksFlexTable.setText(row, 0, symbol)
        self.stocksFlexTable.setWidget(row, 2, Label())
        self.stocksFlexTable.getCellFormatter().addStyleName(row, 1, 'watchListNumericColumn')
        self.stocksFlexTable.getCellFormatter().addStyleName(row, 2, 'watchListNumericColumn')
        self.stocksFlexTable.getCellFormatter().addStyleName(row, 3, 'watchListRemoveColumn')

        # Add a button to remove this stock from the table
        def _removeStockButton_Click(event):
            if symbol not in self.stocks:
                return
            removedIndex = self.stocks.index(symbol)
            self.remote.deleteStock(symbol, self)
            self.stocks.remove(symbol)
            self.stocksFlexTable.removeRow(removedIndex + 1)
        removeStockButton = Button('x', _removeStockButton_Click)
        removeStockButton.addStyleDependentName('remove')
        self.stocksFlexTable.setWidget(row, 3, removeStockButton)

        # Get the stock price
        self.refreshWatchlist()

    def refreshWatchlist(self):
        '''
        Update the price change for each stock
        '''

        MAX_PRICE = 100.0
        MAX_PRICE_CHANGE = 0.02

        prices = []
        for i in range(len(self.stocks)):
            price = random.random() * MAX_PRICE
            change = price * MAX_PRICE_CHANGE * (random.random() * 2.0 - 1.0)
            prices.append(StockPrice(self.stocks[i], price, change))

        self.updateTable(prices)

    def updateTable(self, prices):
        '''
        Update the price and change fields of all the rows in the stock table

        prices -- List of StockPrice objects for all rows
        '''

        # Type checking
        assert isinstance(prices, list)
        for price in prices:
            assert isinstance(price, StockPrice)

        # Nothing to do...
        if len(prices) == 0:
            return

        # Update each individual row
        for i in range(len(prices)):
            self.updateRow(prices[i])

        # Display timestamp showing last refresh
        self.lastUpdatedLabel.setText("Last update: %s" % datetime.datetime.now().strftime("%m/%d/%Y %I:%M:%S %p"))

    def updateRow(self, price):
        '''
        Update a single row in the stock table

        price -- StockPrice object for a single row
        '''

        # Type checking
        assert isinstance(price, StockPrice)

        # Make sure the stock is still in the stock table
        if price.symbol not in self.stocks:
            return

        # Find the index of
        row = self.stocks.index(price.symbol) + 1

        # Populate the price and change fields with new data
        self.stocksFlexTable.setText(row, 1, '%.2f' % price.price)
        changeWidget = self.stocksFlexTable.getWidget(row, 2)
        changeWidget.setText('%.2f (%.2f%%)' % (price.change, price.getChangePercent()))

        # Change the color of the text in the Change field based on its value
        changeStyleName = 'noChange'
        if price.getChangePercent() < -0.1:
            changeStyleName = 'negativeChange'
        else:
            changeStyleName = 'positiveChange'

        changeWidget.setStyleName(changeStyleName)

    def onRemoteResponse(self, response, request_info):
        '''
        Called when a response is received from a RPC.
        '''
        if request_info.method in DataService.methods:
            # Compare self.stocks and the stocks in response
            stocks_set = set(self.stocks)
            response_set = set(response)
            # Add the differences
            for symbol in list(response_set.difference(stocks_set)):
                self.addStock(None, symbol)
        else:
            Window.alert('Unrecognized JSONRPC method.')

class DataService(JSONProxy):
    methods = ['getStocks', 'addStock', 'deleteStock']

    def __init__(self):
        JSONProxy.__init__(self, 'services/', DataService.methods)

if __name__ == '__main__':
    pyjd.setup('./StockWatcher.html')
    app = StockWatcher()
    app.onModuleLoad()
    pyjd.run()

Essentially it's all the same as last time, except now we import "JSONProxy" from "pyjamas.JSONService" then subclass it, change the code in the "addStock" method, and add the "onRemoteResponse" method. Now the stocks (if any) will be loaded when the site is viewed, and stocks will be saved/deleted on the server when the user does. Not bad, right?

...and that how one writes an entire web application in Python!

Again, you can view all of the code here and use the actual application here.

Thanks for reading.

UPDATE (6/5/2011): A commenter has notified me that in the most recent version of Pyjamas, you have to change RootPanel('stocksList').add(self.mainPanel) to RootPanel().add(self.mainPanel) in order to get this demo to work. I'll look into this soon and post another update.

Taken from the GWT website:

Google Web Toolkit (GWT) is a development toolkit for building and optimizing complex browser-based applications. Its goal is to enable productive development of high-performance web applications without the developer having to be an expert in browser quirks, XMLHttpRequest, and JavaScript. GWT is used by many products at Google, including Google Wave and the new version of AdWords. It's open source, completely free, and used by thousands of developers around the world.

Essentially, the main goal of GWT is to simplify the development of web app front ends by abstracting it away. You don't even have to write HTML or JavaScript anymore; you write your application front end in Java! That Java code is then compiled down to the HTML/CSS/JavaScript code that comprises your application. It's really very neat.

I began experimenting with GWT some time ago, and have been very impressed by what it has to offer. It really does make the process of developing web applications simpler and more manageable.

As is stated on the GWT website, even medium-sized web apps, with all their hand-written HTML and JavaScript become awkward at best, unmanageable at worst. I've been running into this at work recently, as the size of web app of ours continues to grow in unpredictable ways. GWT is a way of getting around that.

...but this post isn't even about GWT, it's about Pyjamas!

Pyjamas is a port of GWT to Python, meaning it's even more concise (Java vs. Python) and even easier to setup!

To show just how similar the two are, first check out the GWT tutorial here.

And then take a look at a Pyjamas port of it here.

And here is the source for both: GWT, Pyjamas.

If you write web applications and haven't heard of GWT or Pyjamas, I advise you to check them out. They really are a fantastic alternative.

Next I will be showing how to use Pyjamas with Django, which will allow you to write your entire web application (both front and back ends) in Python!

• Video (HD) and music streaming
• Must stream over a wireless connection
• Netflix and Amazon On-Demand access
• Ideally, some extras, like Pandora, Youtube, etc.

This left me with a few options:

• A full-blown desktop or mini-desktop with enough storage to hold all my media (expensive)
• Some kind of thin-client set-top box that can stream from my computer (would have to have my computer on)
• A networked attached storage device and a set-top box (possibly more expensive, but convenient)

After doing some research I decided on the latter option, using the D-Link DNS-323 NAS server. I chose it because of its impressive feature set: RAID 1, BitTorrent downloading, Gigabit LAN connection, FTP and UPnP servers, and rather fast read/write speeds for devices in its class (30MB/s read, 20MB/s write). Perhaps best of all, after rebate, it only costs $100! I put two Western Digital 1TB Caviar Black drives in there in RAID 1 configuration, and it's been going great!

Then I had to decide on some sort of set-top box to connect to my TV. This was much more difficult. There don't seem to be many devices that support all these features, and if they do, they are almost as expensive as an entire mini-desktop. Then I discovered that the awesome Boxee software is going to come bundled on the so called Boxee Box also made by D-Link. The problem is its not coming out until November (after being pushed back a couple times, without any confirmed guarantee that it won't get pushed back again) and its predicted to cost $250. Not all that bad, but its also not that mush less than the mini-desktop options.

I had looked at the Roku player earlier in my hunt, but dismissed it because it only supported streaming from Netlflix and Amazon, but not from a local source. After some digging around I found the Roksbox Roku player addon:

Roksbox gives you the ability to play your own videos and movies, listen to your own music, and show your own photographs on the Roku Digital Video player (from now on referred to as the Roku player). Your media can reside on your computer's hard drive, on an external drive, on a network attached storage (NAS) device located on your home or local area network, or on a USB attached device (XR model only). Although Roksbox requires some initial setup work, once you get your network and media files properly configured, you will be able to enjoy all of your media files on the convenience of the Roku player.

And the Roku player only costs $70! With a few modifications, it will do everything I want it to! But wait, the Roksbox addon requires a local web server to stream from...  uh ohs!

Turns out that the DNS-323 is even more awesome than I originally thought. With some Linux knowledge and a little hackery, you can enable SSH access to the DNS-323 and setup a LightTPD web server! Amongst many other great things. The work-around is called fun_plug, described here, and available for download here.

My Roku player should be arriving in a few days, and I will give the full run down and any sort of guidance that I think might be necessary.

UPDATE (9/18/2010):

So, I received my Roku player about a week ago. I ended up springing for the HD-XR version that costs a bit more ($99), but has support for a faster wireless connection and better video quality. The Amazon reviews that say you can have the player streaming from Netflix in 10 minutes are absolutely true, and I even performed the web setup on my phone!

In short, setup is trivial, Netflix streaming is great, and the number of "channels" available is enormous. I definitely recommend the Roku player.

Now, on to LAN streaming. As I mentioned above, the Roksbox channel requires a web server to stream from, and the DNS-323 NAS server is capable of doing so with a little hackery. So I set that up; quite painless, really. Then all you have to do is go into the Roksbox settings and point it to the IP and port that the server is running on. By default, it will sort things be your directory structure, and seems to be able to read basic metadata values, but the values displayed will be rather basic.

In order to have a fancier display (box-art, synopsis/overview, year, director, etc.) and sorting by genre and the like, you have to enter all of your movies and TV episodes into some software called Personal Video Database (as described here) from which you export an XML file and place it in the root video folder on your server. This process can be tedious, especially if you have a large collection, as the box-art and info need to be downloaded for each title individually. Although, once you do, everything is displayed very nicely in Roksbox. The same sort of thing can be done with music via MediaMonkey (as described here).

Also, it seems as of this month, Roksbox is no longer free. You can try it out for 30 days with no hassle, but after that you must pay a one time fee to keep using it (on as many Roku players and for as long as you like). I haven't hit the 30 day limit yet, so I'm not sure how much it is, but they claim it is reasonable (and it probably is), but I am definitely going to pay for it. The folks behind Roksbox definitely deserve some beer money for their hard work!