jdbates

2012-01-20T02:32:00.000-08:00

How to gather statistics about traffic flowing through a Linksys WRT54GL? Also how to run ntop and OpenWrt?

Here at the Agahozo-Shalom Youth Village about 500 kids and 150 staff share a satellite internet connection. Some statistics will help us plan how to improve internet access:

Is bandwidth shared fairly among users? or do a few users consume most of our limited bandwidth?
What portion of our bandwidth is web traffic? what other protocols?
Is our web traffic all over the web? or do we mostly visit just a few internet domains?

I have experience with ntop to gather statistics like these. Can anyone recommend other tools?

Our satellite internet connection is hooked up to a Linksys WRT54GL router which is connected to a D-Link DGS-1024-D gigabit switch. Normally ntop gathers statistics about traffic on a local network interface. Because we want statistics about internet traffic, it would be simplest to install ntop on the router

But ntop needs lots of memory: a WRT54GL hasn't got enough by far. So we either need a router with lots of memory, or we install ntop on a separate device. Employing a separate device is attractive because then we have more choice in which devices to use, e.g. a WRT54GL is cheap and simple to configure, ntop needs only one network interface. Employing a separate device is also attractive because then ntop can't possibly jeopardize our internet connection

But how can ntop on a separate device monitor internet traffic? Normally a switch will only send internet traffic out the switch port that is connected to the router. Many switches feature port mirroring whereby traffic on one switch port (or an entire VLAN) is mirrored to another switch port. This way ntop on a separate device could monitor traffic flowing through the router, or all traffic flowing through the switch. Unfortunately the D-Link DGS-1024-D lacks this feature. Not to mention other challenges of monitoring the volume of all traffic flowing through the switch

Alternatively, ntop can gather statistics from NetFlow or sFlow data. NetFlow decouples the traffic monitoring from the statistics generation. Only a lightweight "probe" monitors traffic and sends NetFlow data to ntop, which can then be installed anywhere in the network. To gather statistics from NetFlow, activate the NetFlow plugin via the ntop web interface

Lots of network equipment can generate NetFlow data. One way a WRT54GL can generate NetFlow data is to install OpenWrt, which features at least two software probe packages: nProbe and fprobe

So I installed OpenWrt 10.03.1, but every time I installed nProbe the router locked up and rebooted:


root@OpenWrt:~# opkg update
root@OpenWrt:~# opkg install nprobe

Maybe a WRT54GL hasn't got enough memory for nProbe? Maybe I could install nProbe if I built a custom image?

fprobe comes in two flavors: libpcap and libipulog. The libipulog flavor reportedly features better performance and more features. Plus the OpenWrt fprobe-ulog package is smaller than the fprobe package. I installed fprobe-ulog no problem, but I got the following error until I also installed kmod-ipt-ulog:


root@OpenWrt:~# opkg update
root@OpenWrt:~# opkg install fprobe-ulog
root@OpenWrt:~# fprobe-ulog 192.168.100.7:2055
libipulog initialization error: Unable to bind netlink socketroot@OpenWrt:/#

It ran no problem after I also installed kmod-ipt-ulog, but ntop received no NetFlow data. I guess this is because I must also add some "-j ULOG" rules to /etc/firewall.user? Instead I switched to the libpcap flavor. I installed fprobe no problem, but ntop still received no NetFlow data until I added the "-i" flag:


root@OpenWrt:~# opkg update
root@OpenWrt:~# opkg install fprobe
root@OpenWrt:~# fprobe 192.168.100.7:2055
root@OpenWrt:~#

-i Listen on interface. If unspecified, fprobe will use result of pcap_lookupdev() function. On Linux systems with 2.2 or later kernels, an interface argument of `any' can be used to capture packets from all interfaces. Note that captures on the `any' device will not be done in promiscuous mode.


root@OpenWrt:~# fprobe -i br-lan 192.168.100.7:2055
root@OpenWrt:~#

(I chose the br-lan interface based on this example)

At last! ntop gathered some statistics about traffic flowing through the router. ntop configuration still needed some refinement. Also the statistics lacked some detail, like the internet domain of web traffic. I guess this is because NetFlow communicates only layer 3 data?

So the advantages of gathering statistics from NetFlow are that it decouples the traffic monitoring from the statistics generation. ntop can be installed anywhere in the network and can gather statistics from lots of network equipment. The advantages of gathering statistics about traffic on a local network interface are simplicity: no NetFlow to configure, no probe to install. Also more detailed statistics

Hacking port mirroring on the WRT54GL is the only remaining way that I can think to:

Run ntop on a separate device
Use a local network interface vs. NetFlow
Gather statistics about internet traffic

Apparently iptables can be configured to achieve something like port mirroring with the "-j ROUTE" target. There are just three obstacles:

The "-j ROUTE" target at some point evolved into the "-j TEE" target
The "-j TEE" target is unusable without a bunch of large IPv6 packages. Based on my experience installing software probe packages, I suspect that installing these packages on a WRT54GL will be challenging
The kmod-ipt-tee package is missing from OpenWrt 10.03.1

There was some support for the "-j ROUTE" target in OpenWrt 8.09: I hope it has fewer requirements than the "-j TEE" target

I haven't got the "-j ROUTE" target working yet

This site is not affiliated with or sponsored by the Agahozo-Shalom Youth Village nor has Agahozo-Shalom approved, reviewed or confirmed any of the data and information provided herein

2011-10-26T15:40:00.000-07:00

Recently I'm updating all my accounts with my new email address, and at the same time updating my passwords based on insight like this and this. Thanks XKCD :-)

Reusing passwords for different services is dangerous but I want to minimize the number of different passwords I must manage. OpenID is a common tool for logging into sites without sharing a password with the site. But I quickly discovered at least four common errors in OpenID relying party implementations. I patched a couple implementations:

Direct Verification

If the Relying Party has stored an association with the association handle specified in the assertion, it MUST check the signature on the assertion itself. If it does not have an association stored, it MUST request that the OP verify the signature via Direct Verification.

Some relying party implementations don't implement direct verification. They fail to verify an assertion signed with a private association. I discovered this issue because my OpenID provider stores associations in memory. When I restart it, associations are cleared. When I subsequently request authentication, relying parties correctly use previously established shared associations, but my provider responds with an assertion signed with a private association. Instead of performing direct verification, some relying parties reject this assertion. I can't log into these relying parties until our shared association expires (two weeks in the case of my provider)

Assertions sent as a POST (vs. GET)

Indirect messages may be sent as a GET or as a POST

When a message is sent as a POST, OpenID parameters MUST only be sent in, and extracted from, the POST body.

But some relying party implementations don't support assertions sent as a POST, e.g. the Python OpenID Library example relying party ignores all POST requests. Meanwhile the Python OpenID Library relying party implementation checks that any query parameters that are present in the openid.return_to URL are also present with the same values in the POST body, if the assertion is sent as a POST. It should instead always check that they are present with the same values in the URL of the HTTP request the relying party received. Consequently it rejects assertions sent as a POST

OP Endpoint URL

When verifying assertions, some relying party implementations don't verify that the OP endpoint URL is present in the discovered information, or that the shared association with which the assertion is signed is between the OP endpoint URL and the relying party. This error is a biggie because it allows a malicious user who controls her own OpenID provider to impersonate any claimed identifier

If relying parties don't index associations by OP endpoint URL, as well as by association handle, then a malicious user who controls her own OpenID provider can forge assertions from other providers by signing forged assertions with a shared association between a provider she controls and the relying party. She can also invalidate shared associations between other providers and the relying party by sending these already established association handles in response to association requests from the relying party to a provider she controls

If relying parties don't verify that the OP endpoint URL is present in the discovered information, then a malicious user can make assertions about any claimed identifier, and cause the relying party to verify them with a shared association between a provider she controls and the relying party, or by direct verification with a provider she controls

OP-Local Identifier

Some relying party implementations mistakenly trust OpenID providers to verify the claimed identifier. In fact, assertions are independent of the claimed identifier. In case of 1.1 compatibility, providers can't even determine the claimed identifier. Instead, providers make assertions about the OP-local identifier. This mistake is also a biggie because it allows a malicious user to impersonate any claimed identifier that a provider is authorized to make assertions about, if she controls one OP-local identifier at that provider

Relying parties must identify users by the claimed identifier, not the OP-local identifier. To verify that a user controls a claimed identifier, relying parties must perform discovery on the claimed identifier to obtain an OP endpoint URL and an OP-local identifier. Relying parties must then verify that the user has obtained from the provider, an assertion about the OP-local identifier

When the OP-Local Identifier ("openid.identity") is different from the Claimed Identifier, the Relying Party MUST keep track of what Claimed Identifier was used to discover the OP-Local Identifier, for example by keeping it in session state. Although the Claimed Identifier will not be present in the response, it MUST be used as the identifier for the user.

OpenID adoption is slow, causing some to wonder if it's dead. There are now alternatives to OpenID which claim among their advantages to be more developer friendly

A feature of OpenID 2.0 that I really appreciate is that the provider controls the identifier as well as the account information which is shared with the relying party. This enables me to implement a provider which automatically manages pseudonyms for every site I log into. I'm free from managing these pseudonyms manually. They work without any browser customization, with any browser, on any device, with any site that implements OpenID 2.0

I can't think of any substantial simplification of the OpenID protocol which still enables this functionality. Nor are any substantial improvements to the clarity of the OpenID specification obvious to me. Common errors in relying party implementations could be corrected, however. This would reduce incidents where some users can't log into some sites with OpenID, or where some sites don't interoperate with some providers. Maybe OpenID could then too claim to be more developer friendly

2011-10-03T14:44:00.000-07:00

In response to SIP INVITE where the connection address in SDP content equals the "sent-by" address of the Via header, send RTP media to the transport source address (and the media port in SDP content) - this is how I got my Android to call Asterisk running at EC2, without manually mucking with the firewall. My Android is behind a NAT. Asterisk is behind EC2's NAT

SIP INVITE references three peer addresses and three peer ports:

transport source address - RFC 3261 specifies that this is copied to the "received" parameter of the Via header:
If the host portion of the "sent-by" parameter contains a domain name, or if it contains an IP address that differs from the packet source address, the server MUST add a "received" parameter to that Via header field value. This parameter MUST contain the source address from which the packet was received
"sent-by" address of the Via header
connection address in SDP content - in exotic cases there may be many connection addresses in SDP content

transport source port - this may get copied to the "rport" parameter of the Via header, as per RFC 3581:
When a server compliant to this specification (which can be a proxy or UAS) receives a request, it examines the topmost Via header field value. If this Via header field value contains an "rport" parameter with no value, it MUST set the value of the parameter to the source port of the request
"sent-by" port of the Via header
media port in SDP content - there may be many media ports in SDP content, e.g. for audio and video media

In absence of NAT, transport source address and "sent-by" address of the Via header are equal, as are transport source port and "sent-by" port of the Via header

In absence of SIP proxy and in common case where SIP messages and RTP media are exchanged with a single peer with a single address, "sent-by" address of the Via header and connection address in SDP content are equal

Send SIP responses to the transport source address (via the "received" parameter of the Via header), as per RFC 3261:

Otherwise (for unreliable unicast transports), if the top Via has a "received" parameter, the response MUST be sent to the address in the "received" parameter, using the port indicated in the "sent-by" value, or using port 5060 if none is specified explicitly

Asterisk versions earlier than 1.8 get this *wrong* - they send SIP responses to the transport source address only if "nat = yes" is enabled (it's disabled by default). This is addressed by issue 8855, and fixed in Asterisk versions 1.8 and later

If the client routes SIP messages symmetrically (it listens for responses on the same port from which it sends requests), and a NAT doesn't remap the source port, then this is sufficient to deliver SIP responses, even if a NAT remaps the source address

If the client routes SIP messages symmetrically, then send SIP responses to the transport source port - this can be achieved via the "rport" parameter of the Via header, as per RFC 3581, or Asterisk can be configured with either "nat = force_rport" or "nat =yes", to work around clients which don't support RFC 3581. This is sufficient to deliver SIP responses, even if a NAT remaps the source port

Lots of work has been done to deliver RTP media, if a NAT remaps the source address or source port. The standards endorsed solution is Interactive Connectivity Establishment (ICE) - but it must be supported by the client to work. And Asterisk doesn't implement it yet

Instead, Asterisk implements a workaround which needs no client support - except that the client must route RTP media symmetrically (it listens for media on the same port from which it sends media). Asterisk calls it "connection oriented media" or "comedia". It's enabled by configuring Asterisk with either "nat = comedia" or "nat = yes". Asterisk sends outgoing RTP media to the transport source of incoming RTP media

e.g. in response to SIP INVITE:

transport source address 184.66.112.45 and port 44859
"sent-by" address of the Via header 192.168.1.6 and port 44859
connection address in SDP content 192.168.1.6 and media port 45112

- SIP response is sent to address 184.66.112.45 and port 44859, and Asterisk starts sending outgoing RTP media to address 192.168.1.6 and port 45112

If Asterisk later receives incoming RTP media from transport source address 184.66.112.45 and port 45112, then it starts sending subsequent outgoing RTP media to 184.66.112.45 and port 45112

This works great - unless Asterisk is also behind a NAT or firewall:

RTP media sent to 192.168.1.6 might not cross this NAT or firewall, and a "full cone" NAT or firewall rejects incoming traffic sent to an address and port which weren't previously a source of outgoing traffic
If RTP media sent to 192.168.1.6 does cross this NAT or firewall, a "restricted cone" NAT or firewall rejects incoming traffic sent to an address and port which weren't previously a source of outgoing traffic, or if the outgoing traffic destination address didn't match the incoming traffic source address

Asterisk never receives incoming RTP media, and continues sending outgoing RTP media to 192.168.1.6

One way for this workaround to work if Asterisk is also behind a NAT or firewall is to manually configure this NAT or firewall to forward to Asterisk all ports which Asterisk might potentially use for incoming RTP media (as defined in rtp.conf). This is a blunt solution, and requires keeping Asterisk and NAT or firewall configuration synchronized

I propose an additional workaround: If the connection address in SDP content equals the "sent-by" address of the Via header, send RTP media to the transport source address (and the media port in SDP content)

In above example, connection address in SDP content and "sent-by" address of the Via header are both 192.168.1.6, so start sending outgoing RTP media to address 184.66.112.45 and port 45112, vs. address 192.168.1.6 and port 45112

If the client NAT doesn't remap the client RTP media port, then this is sufficient to deliver RTP media - connection oriented media won't change the outgoing RTP media destination

If the client NAT does remap the client RTP media port, then this, in conjunction with connection oriented media, is sufficient to deliver RTP media. Neither a "full cone" nor a "restricted cone" NAT will reject incoming RTP media, because it is sent to an address and port which are a source of outgoing traffic, and the outgoing traffic destination address matches the incoming traffic source address. Connection oriented media can then correct the outgoing RTP media destination port

The rationale for this workaround is that if the connection address in SDP content matches the "sent-by" address of the Via header, then we should send SIP responses and RTP media to the same address

Like connection oriented media, this workaround isn't endorsed by any standard - like connection oriented media, this workaround enables Asterisk to interoperate with existing clients and existing NAT and firewall. It doesn't impact any configuration which already works

The advantage of this workaround is that it enables Asterisk to work with NAT and firewall, without manually mucking with the firewall. This enables Asterisk to work "out of the box", and in situations without resources to deeply understand SIP and NAT, or where administrators don't have access to firewall configuration

I implemented this workaround as issue 18659 and use it to call Asterisk running at EC2 from my Android, without manually mucking with the firewall

2011-09-26T11:43:00.000-07:00

In Python, how to reliably call code when an object no longer exists, or is "unreachable" - this is a followup to this post

In summary, don't use .__del__() because it's way easy to make an "uncollectable" reference cycle

Instead, a good solution is described here and here: use weak references

Weak references support a callback which gets called when the target of the reference is finalized:



#!/usr/bin/env python

import weakref

class sample:
  pass

def callback(_):
  print 'Goodbye, world!'

weakref.ref(sample(), callback)


$ ./weakref
Goodbye, world!
$

- but the weak reference must outlive the target, or callback won't get called:



#!/usr/bin/env python

import weakref

class sample:
  pass

target = sample()

def callback(_):
  print 'Goodbye, world!'

weakref.ref(target, callback)


$ ./outlive
$

Maintaining a reference to the weak reference prevents it from getting finalized - but to avoid leaking memory, we probably *do* want it to get finalized after the target is finalized and callback is called. Maintaining a reference from the target to the weak reference is one solution:



#!/usr/bin/env python

import weakref

class sample:
  pass

target = sample()

def callback(_):
  print 'Goodbye, world!'

target.outlive = weakref.ref(target, callback)


$ ./attribute
Goodbye, world!
$

Sadly this falls down if the target participates in - or even hangs off of - a circular reference:



#!/usr/bin/env python

import weakref

class sample:
  pass

target = sample()

def callback(_):
  print 'Goodbye, world!'

target.outlive = weakref.ref(target, callback)

target.cycle = target

del target


$ ./cycle
$

Maintaining the reference to the weak reference separately from the target works even if the target participates in a circular reference - but only if the target is finalized before Python exits and the global namespace, garbage collector roots, etc. are finalized

This is the solution I employ here - I avoid leaking memory by maintaining the reference to the weak reference with a weakref.WeakKeyDictionary



#!/usr/bin/env python

import weakref

class sample:
  pass

target = sample()

def callback(_):
  print 'Goodbye, world!'

outlive = weakref.ref(target, callback)

target.cycle = target

del target


$ ./separate
Goodbye, world!
$



#!/usr/bin/env python

import other, weakref

class sample:
  pass

other.target = sample()

def callback(_):
  print 'Goodbye, world!'

outlive = weakref.ref(other.target, callback)


$ ./exit
$

I guess this is because, when Python exits and the global namespace, garbage collector roots, etc. are finalized, the reference to the weak reference is dropped before the reference to the target?

An alternative that works even if the target participates in a circular reference *or* Python exits before the target is finalized is to deliberately make an uncollectable reference cycle:



#!/usr/bin/env python

import weakref

class sample:
  pass

target = sample()

uncollectable = sample()
uncollectable.__del__ = uncollectable

def callback(_, uncollectable=uncollectable):
  del uncollectable.__del__
  print 'Goodbye, world!'

uncollectable.outlive = weakref.ref(target, callback)

target.cycle = target

del target


$ ./uncollectableCycle
Goodbye, world!
$



#!/usr/bin/env python

import other, weakref

class sample:
  pass

other.target = sample()

uncollectable = sample()
uncollectable.__del__ = uncollectable

def callback(_, uncollectable=uncollectable):
  del uncollectable.__del__
  print 'Goodbye, world!'

uncollectable.outlive = weakref.ref(other.target, callback)


$ ./uncollectableExit
Goodbye, world!
$

Sadly this still falls down if the target participates in a circular reference *and* Python exits before the target is finalized:



#!/usr/bin/env python

import weakref

class sample:
  pass

target = sample()

uncollectable = sample()
uncollectable.__del__ = uncollectable

def callback(_, uncollectable=uncollectable):
  del uncollectable.__del__
  print 'Goodbye, world!'

uncollectable.outlive = weakref.ref(target, callback)

target.cycle = target


$ ./cycleExit
$

How can this be? The target is a *collectable* reference cycle so it should get finalized when Python exits. The weak reference hangs off an *uncollectable* reference cycle, so it must outlive the target because it can *never* get finalized before callback is called. WTF?

My guess is that when Python exits, the garbage collector runs *before* the global namespace, garbage collector roots, etc. get finalized. That would explain why callback is called even if the target participates in a circular reference - so long as it's not reachable from a garbage collector root when Python exits : P

2011-08-24T08:32:00.000-07:00

Yadis specification is awesome example of defining something simple, in the most unnecessarily complicated way! No wonder Yadis is dead

First, normative references go at the end of the document. I don't wanna scan a name dropping list of other standards before getting to the meat of the specification - before figuring out what the specification is even about, since the introduction is so abstract, it gives no clue: e.g. "Section 1 describes the scope of this Specification" - thanks: Maybe I'm not the brightest crayon in the box, but that's usually what the "Scope" section describes : P

A whole section, "Abbreviations", is dedicated to defining HTTPS: "HTTP protocol when that protocol is used, in accordance with current conventions, so that the session data is encrypted using a version of the Transport Layer Security or Secure Socket Layer protocols"

Let me break it down for you: Yadis specifies how to resolve a URL into a special document, an XRDS document. This is done with an HTTP request. This request includes an HTTP Accept header specifying MIME media type, application/xrds+xml

The response to this request either,

Is a document of MIME media type, application/xrds+xml
Includes a special HTTP response header, X-XRDS-Location
Is an HTML document with a <head> element that includes a <meta> element with http-equiv attribute, X-XRDS-Location

If the response is a document of MIME media type, application/xrds+xml then this is the special XRDS document. Otherwise if the response includes a special HTTP response header, or if the response is an HTML document with http-equiv attribute, then this is a URL. This URL MUST locate the special XRDS document

Simple. Calling this a protocol is a stretch. Standardizing an internet protocol is an estimable accomplishment - but an unnecessarily complicated specification for a simple URL indirection is not. Specifying this URL indirection in terms of "initiation", "termination", and "failure" is grandiose - it only obfuscates the specification

Using MediaWiki to publish standards, OTOH, is awesome! yadis.org is super awesome - very clear - the information would all be very useful, if Yadis had any substance. The way it uses MediaWiki to organize change proposals is very nice!

More standards published with MediaWiki, please! Less unnecessarily complicated specifications, please : P

2011-08-12T11:21:00.000-07:00

Am thinking about food and power, and themes of corporate supremacy over the state in science fiction. Like would the following make interesting sci fi? Corporations subsidize school lunch programs for self promotion to children: to accustomize susceptible consumers to their product. Profits are used to lobby the state, and long term costs of their product to the population (maybe health care, maybe pollution) are paid by the state (erasing any savings gained from subsidies)

These days energy is the most debated power broker - but food is still the oldest lever to control a population?

2011-08-02T08:01:00.000-07:00

Want Postfix to deliver all @nottheoilrig.com mail to DBMail "nottheoilrig" mailbox

In addition to the IMAP protocol, DBMail implements the LMTP protocol for mail delivery. Addresses may be mapped to mailbox names and some catchalls are supported - though notably not a catch *all* catchall. But if the envelope recipient *is* the mailbox name, then no mapping is needed: DBMail delivers the mail to the mailbox

Just like SMTP, LMTP specifies that the envelope recipient is an address. If the mailbox name isn't an address, then if the envelope recipient is the mailbox name, DBMail still delivers the mail to the mailbox, but this behavior is nonstandard

But I prefer to map addresses to mailbox names with Postfix vs. DBMail. Postfix' map support is very mature - it supports many map backends and map configuration is internally consistent. Postfix must be configured to map all @nottheoilrig.com mail to DBMail - configuring Postfix to map all @nottheoilrig.com mail to "nottheoilrig" mailbox consolidates configuration and minimizes chance that DBMail rejects any mail delivered to it by Postfix

But getting Postfix to violate LMTP standard and deliver mail with envelope recipient that isn't an address is almost impossible!

I think Postfix maps mail to transports separately from address rewriting, so we must either rewrite the envelope recipient after we map all @nottheoilrig.com mail to DBMail, or map the rewritten address to DBMail. I think the only address manipulation *after* Postfix maps mail to transports is lmtp_generic_maps

If we configure "lmtp_generic_maps = static:nottheoilrig" then the rewritten address is nottheoilrig@$myorigin. Disabling append_at_myorigin in master.cf, for just the lmtp service, has no effect. If we disable append_at_myorigin in main.cf, then we succeed in getting Postfix to violate LMTP standard: the rewritten address is "nottheoilrig"! But lmtp_generic_maps manipulates the envelope *sender* - not the *recipient* : (

Next I tried virtual_alias_maps. To avoid rejecting messages, virtual_alias_domains must include nottheoilrig.com. The virtual_alias_domains default value is $virtual_alias_maps, to supposedly consolidate configuration. But if we add nottheoilrig.com to virtual_alias_maps, won't nottheoilrig.com@$mydestination get rewritten?

I configured "virtual_alias_domains = nottheoilrig.com". I think we could configure "virtual_alias_maps = static:nottheoilrig"? I configured "virtual_mailbox_domains = $myhostname" and "virtual_transport = lmtp:localhost". To avoid, "do not list ... in BOTH mydestination and virtual_mailbox_domains", I configured "mydestination ="

Unfortunately this didn't avoid, "do not list ... in BOTH mydestination and virtual_mailbox_domains" - for better or worse, mail is mapped to local transport, regardless of mydestination and virtual_mailbox_domains. Maybe this is a bug? I successfully overrode mydestination and virtual_mailbox_domains with transport_maps, but failed to violate LMTP standard: the envelope recipient is nottheoilrig@$myhostname : (

2011-07-23T10:39:00.000-07:00

In Python, how can you associate one value with another value,

without touching the "associatee" value,
without leaking memory,
and if the associated value might reference the associatee value?

One obvious way to associate one value with another value is to add to the associatee value, a reference to the associated value,


#!/usr/bin/env python

associatee.associated = associated

- however this fails my first condition: It touches the associatee value

Another obvious way to associate one value with another value is with a mapping, like a dictionary,


#!/usr/bin/env python

mapping = {}
mapping[associatee] = associated

- however this fails my second condition: The dictionary will continue to grow, unused values will never get finalized, leaking memory

PEP 205 and the weakref.WeakKeyDictionary standard library class attempt to address this. Normally a value will never get finalized while there exist references to it (except if the only references to it are part of cycles, and there exist no references to these cycles). Unlike normal references, weak references don't prevent values from getting finalized

Normal dictionaries make references to keys and to values, which prevents keys and values from getting finalized. weakref.WeakKeyDictionary makes normal references to values, but weak references to keys, which doesn't prevent keys from getting finalized. When a key gets finalized, it and its value get dropped from the mapping,


#!/usr/bin/env python

import weakref

mapping = weakref.WeakKeyDictionary()
mapping[associatee] = associated

len(mapping) # 1

del associatee

len(mapping) # 0

But what if the associated value might reference the associatee value?

If instead of using weakref.WeakKeyDictionary, we add to the associatee value, a reference to the associated value, then at least we won't leak memory (although we fail my first condition: We touch the associatee value). When the only references to the associated value are from the associatee value (or part of cycles to which no references exist), and the only references to the associatee value are from the associated value (or part of cycles to which no references exist), then the garbage collector will eventually recognize an unreachable cycle and finalize both the associatee and associated values (with at least one, avoidable, exception)

weakref.WeakKeyDictionary makes weak references to keys, but normal references to values. This prevents values from getting finalized while each key (the associatee value) exists. If the associated value references the associatee value, then the associatee value (the key) will never get finalized while the associated value exists. Because a normal reference exists from the mapping to the associated value, the associatee and associated values will never get finalized by the garbage collector - they aren't an unreachable cycle

So if the associated value might reference the associatee value, then weakref.WeakKeyDictionary leaks memory

We might attempt to address this by making weak references both to keys and and to values. If we do this, then the mapping won't prevent values from getting finalized, plugging the memory leak. But the associated value might then get finalized before the associatee value,


#!/usr/bin/env python

import weakref

mapping = weakref.WeakKeyDictionary()
mapping[associatee] = weakref.ref(associated)

mapping[associatee]() # Associated value

del associated

mapping[associatee]() # None

How then can you associate one value with another value,

without touching the "associatee" value,
without leaking memory,
and if the associated value might reference the associatee value?

I want to do this because I've written a descriptor that returns new values, specific to the instance - but the descriptor must always return an identical value, each time it's called with the same instance,


#!/usr/bin/env python

class descriptor:
  class __metaclass__(type):
    def __get__(self, instance, owner):
      return ...

class owner:
  descriptor = descriptor

instance = owner()

instance.descriptor is instance.descriptor # True

instance.descriptor.sample = 'Sample'

# If descriptor didn't return an identical value, then this might raise an AttributeError
instance.descriptor.sample # 'Sample'

My best solution is currently a cache - a mapping from instance to descriptor return value. The descriptor return value references the instance, so I make weak references both to cache keys and to cache values (or the cache would continue to grow, unused values would never get finalized, leaking memory). But the return value might then get finalized before the instance

In practice, the return value doesn't get finalized, and the descriptor always returns an identical value, each time it's called with the same instance. Maybe this is because I make enough other references to the return value, or maybe because the return value is part of an unreachable cycle, and the garbage collector just doesn't run often enough to get unlucky : P

But I worry that it *might* happen - that the descriptor *might* not return an identical value. This behavior would be horribly unexpected, and any resulting bugs would be super painful to diagnose,


#!/usr/bin/env python

import gc, weakref

cache = weakref.WeakValueDictionary()
class descriptor:
  class __metaclass__(type):
    def __get__(self, instance, owner):
      try:
        return cache[weakref.ref(instance)]

      except KeyError:
        result = cache[weakref.ref(instance)] = ...

        return result

class owner:
  descriptor = descriptor

instance = owner()

instance.descriptor.sample = 'Sample'
instance.descriptor.sample # Cross your fingers

gc.collect()

instance.descriptor.sample # AttributeError

2011-05-24T07:40:00.000-07:00

What's there to know about event loop programming?

Promises and coroutines/continuations are significant to event loop programming, and recently I discovered these topics for the first time: they were omitted from my computing science undergrad

What other topics are significant to event loop programming? The actor model? Process algebra? Stackless?

Why's an event loop all the rage?

Because currently JavaScript and web browsers are the most common programming environment, and they manifest an event loop - like HyperCard before them

Because currently real time communication is all the rage, and real time communication needs persistent connections. The number of connections an event loop can handle is far greater than can be handled by threads or processes, and the performance of an event loop is affected less than the performance of threads or processes as the number of connections grows

Besides web browsers, many popular tools implement an event loop,

Topics significant to event loop programing promise to be a big deal because they're applicable to many common programming environments

So far I observe three styles of event loop programming:

Callback hell

In this style, a callback is defined when an action is initiated,


jQuery.ajax({

  url: 'http://nottheoilrig.com/untwisted',

  success: function (data)
    {
      ...
    } });

Sometimes an object is defined which implements an interface prescribed by the action - this is just a variation of callback hell,


from twisted.internet import protocol, tcp

class factory(protocol.Factory):
  def buildProtocol(self, addr):
    ...

tcp.Port(1234, factory).startListening()

Sometimes only one callback can be naturally subscribed to an action. Sometimes behavior which depends on the sequence of callbacks must be explicitly represented, instead of implicitly represented by the order of statements - like if .buildProtocol() should behave differently the first and second times it's called, then it must express this condition and count the number of times it's called, instead of sequentially defining first one behavior and then the second

Sometimes code which initiates an action is separate from code which defines a callback. Even if a callback can be subscribed after the action is *initiated*, if it's subscribed after the action is *complete*, then sometimes it won't get triggered,


// What if document is already complete?
$(document).load(function (event)
  {
    ...
  });

Promises

Promises are a consistent interface which allows callbacks to be subscribed after an action is initiated,


from untwisted import tcp

listen = tcp.listen(1234)

# First connection

@listen().connect
def _(transport):
  ...

# Second connection

@listen().connect
def _(transport):
  ...

A promise is like an event - it can be triggered and callbacks can be subscribed. Unlike an event, a promise can be triggered only once. When a callback is subscribed to a promise which is already triggered, then the callback is executed immediately

Coroutines/continuations

Structured code is like a recipe - a sequence of actions, structured with branches and loops. Promises style and callback hell introduce discontinuity between initiating an action and a callback, making programmers chase control flow around programs

On one level, this discontinuity is real - event loop programming achieves concurrency without threads or processes by jumping from callback to callback. Expressing callbacks explicitly emphasizes this control flow

But at a higher level, we're still representing a sequence of actions, structured with branches and loops. Coroutines/continuations allow this to be emphasized instead,


from untwisted import promise, smtp

class client(smtp.client):

  @promise.continuate
  def __init__(self, transport):
    smtp.client.__init__(self, transport)

    # Greeting
    yield self.reply()

    try:
      yield self.ehlo()

    except reply as e:
      if int(e) not in (500, 502):
        raise

      yield self.helo()

    while True:
      try:
        yield self.mail()

      except StopIteration:
        break

    #return ...
    raise StopIteration(self.quit())

Coroutines and continuations allow subroutines to be explicitly suspended and resumed in mid execution. I'm still gaining confidence understanding coroutines and continuations:

A coroutine is a subroutine with multiple entry points - when it's called, it needn't start executing at the top of the subroutine - it may start executing where last it suspended

A continuation represents a program in mid execution. Like coroutines, continuations can be resumed

I imagine the difference between coroutines and continuations is that a coroutine is mutable: it changes state when called, so two calls may resume executing at different points, or with different state. In contrast, the same continuation always resumes executing at the same point, with the same state. A continuation which resumes executing at a different point or with different state is a different continuation

Instead of expressing callbacks explicitly, coroutines/continuations allow the current subroutine to be suspended and subscribed to an action

Promises and coroutines/continuations are significant to event loop programming, and so they're a big deal because they're applicable to many common programming environments. What other topics are significant to event loop programming?

2011-05-01T07:48:00.000-07:00

Several of the most interesting things I recently read or watched all referenced "distributed production" or "social production", or industrial vs. non-industrial economies. I haven't much to add - I'm still thinking about it. But I found each thing interesting, and they got me thinking, so here are links,

Collaboratively building farm equipment - five minute TED talk
Yochai Benkler on how the internet changes production - twenty minute TED talk
The Omnivores Dilemma - 400 page book with topics of pre-industrial, industrial, and post-industrial agriculture
Fight of the Century: Keynes vs. Hayek Round Two - awesome ten minute economist rap video
Paul Prescod's Politics - my friend's politics
Money Just A Symbolic, Mutually Shared Illusion - news article

Something I struggled thinking about is capitalism. It seemed obvious to me that "money" was the source of plenty of suffering in the world. I sympathized with anti-capitalist organizations and regarded anti-capitalism as a progressive, social justice attitude

Then I learned that several folks I admire and regard as progressive, and who fight for social justice, endorse capitalism. They not only think capitalism and social justice are compatible, but think capitalism is part of the solution, vs. part of the problem

I struggle to reconcile these two views. Maybe these two groups - anti-capitalists and "progressive capitalists" - are actually talking about different things? Like anti-capitalists are fighting consumerism and commodification - trading commodities like drinking water, or ownership of organisms' genetics. Whereas "progressive capitalists" want to fix the rules of the game that is the market, so things reflect their true cost, e.g. so driving reflects the cost of maintaining highways, and so the price of fast food doesn't ignore subsidized production of commodity corn

I think capitalism and communism are often juxtaposed - capitalism is distributed, in contrast to communism which is centralized. "Distributed production" reminds me of the benefits of capitalism - better, less expensive decisions. Collaborative farm equipment video describes their goal as lowering barriers to production, unleashing massive amounts of human potential. But same video also mentions that a significant implication is greater distribution of means of production, which sounds rather like workers' control of the means of production. Which ideology is "distributed production" or "social production" more a realization of?

My takeaway is that the information and collaboration innovation which is the internet enables non-industrial economies to equal and surpass the productivity of industrial economies?

2011-04-26T08:36:00.000-07:00

In Python, how can you reliably call code - but wait until an object no longer exists or is "unreachable"?

I want to ensure that some code is called (excluding some exotic situations like when the program is killed by a signal not handled by Python) but can't call it immediately. I want to wait until there are no references to an object - or the only references to the object are from unreachable reference cycles


#!/usr/bin/env python

class Goodbye:
  def __del__(self):
    print 'Goodbye, world!'

ref = Goodbye()


$ ./goodbye
Goodbye, world!
$

Python's __del__ or destructor method works (above) - but only in the absence of reference cycles (below). An object, with a __del__ method, in a reference cycle, causes all objects in the cycle to be "uncollectable". This can cause memory leaks and because the object is never collected, its __del__ method is never called

Circular references which are garbage are detected when the option cycle detector is enabled (it’s on by default), but can only be cleaned up if there are no Python-level __del__() methods involved.


#!/usr/bin/env python

class Goodbye:
  def __del__(self):
    print 'Goodbye, world!'

class Cycle:
  def __init__(self, cycle):
    self.next = cycle
    cycle.next = self

Cycle(Goodbye())


$ ./cycle
$

In PEP 342 I read that an object, with a __del__ method, referenced by a cycle but not itself participating in the cycle, doesn't cause objects to be uncollectable. If the cycle is "collectable" then when it's eventually collected by the garbage collector, the __del__ method is called

If the generator object participates in a cycle, g.__del__() may not be called. This is the behavior of CPython's current garbage collector. The reason for the restriction is that the GC code needs to "break" a cycle at an arbitrary point in order to collect it, and from then on no Python code should be allowed to see the objects that formed the cycle, as they may be in an invalid state. Objects "hanging off" a cycle are not subject to this restriction.


#!/usr/bin/env python

import sys

class Destruct:
  def __init__(self, callback):
    self.__del__ = callback

class Goodbye:
  def __init__(self):
    self.destruct = Destruct(lambda: sys.stdout.write('Goodbye, world!\n'))

class Cycle:
  def __init__(self, cycle):
    self.next = cycle
    cycle.next = self

Cycle(Goodbye())


$ ./dangle
Goodbye, world!
$

However it's *extremely* tricky to ensure that the object with a __del__ method doesn't participate in a cycle, e.g. in the example below, the __del__ method is never called - I suspect because the object with a __del__ method is reachable from the global scope, and this forms a cycle with a frame's f_globals reference? storing a generator object in a global variable creates a cycle via the generator frame's f_globals pointer


#!/usr/bin/env python

import sys

class Destruct:
  def __init__(self, callback):
    self.__del__ = callback

class Goodbye:
  def __init__(self):
    self.destruct = Destruct(lambda: sys.stdout.write('Goodbye, world!\n'))

class Cycle:
  def __init__(self, cycle):
    self.next = cycle
    cycle.next = self

ref = Cycle(Goodbye())


$ ./global
$

Faced with the real potential for reference cycles, how can you reliably call code - but wait until an object no longer exists or is "unreachable"?

2011-04-18T15:22:00.000-07:00

Ow ow OW!!! Python! Why do you hurt so hard?!


#!/usr/bin/env python

from twisted.internet import reactor
from twisted.web import client

@client.getPage('http://nottheoilrig.com/').addCallback
def ignore(result):
  print result

  reactor.stop()

reactor.run()


$ ./ow
  File "./ow", line 6
    @client.getPage('http://nottheoilrig.com/').addCallback
                                               ^
SyntaxError: invalid syntax
$

Wish Python didn't work unexpectedly

PEP 318

The decorator statement is limited in what it can accept -- arbitrary expressions will not work. Guido preferred this because of a gut feeling.


#!/usr/bin/env python

from twisted.internet import reactor
from twisted.web import client

identity = lambda value: value

@identity(client.getPage('http://nottheoilrig.com/').addCallback)
def ignore(result):
  print result

  reactor.stop()

reactor.run()

^ The former code is more readable than the latter, yet only the latter is legal : P

2011-03-02T17:47:00.001-08:00

Volcanoes National Park was the highlight of our three months in Hawaii. There's *lots* to do in this park, which stretches from coconut oases by the ocean, to the top of Mauna Loa at 13,679 feet - with rainforest and boiling rock in between. We spent seven days hiking and camping around the park, and there was still more to see when we left

Camping

There are two drive in campsites in the park - these are also the two campsites for which permits aren't required. Camping in the park is free (!!!) - park entrance fees apply. On Tuesday we hitched a ride to Namakanipaio campground because it's right on the highway

Campground is at 4,000 feet and we were glad we brought our down sleeping bags from Canada, because even in Hawaii, it gets *cold* at 4,000 feet

Interpretive programs

The park offers outstanding hiking and camping, and outstanding interpretive programs. We're spoiled by British Columbia's provincial parks, and expected outstanding hiking and camping all over the Big Island. Instead, we found it at Volcanoes Park - beyond even our expectations

The day's interpretive programs are announced at 9 am on notice boards at the visitor center and Jaggar Museum, and vary from 20 minute talks to guided hikes lasting several hours. Interpretive programs are free (!!!)

We arrived middle of the day, so checked out a shorter program that afternoon: a rainforest walk with ranger Dean Gallagher. We quickly discovered Dean Gallagher is an elite interpretive guide - both an active biologist researcher and talented at leading and entertaining groups. He has encyclopedic biological knowledge, speaks Hawaiian, and plays the nose flute. Also he memorized group members' names - /me is very impressed! What a privilege to receive a tour from such talent!

Crater after dark

Visitors probably overlook the park's many remarkable features because everyone comes to see one thing: boiling rock! It's probably better to come without this expectation: At the time we visited, the closest we got to boiling rock *in the park* was the view from Jaggar Museum after dark. It's also the easiest thing to do, so gets our thumbs up! A short, 1/2 mile trail leads from Namakanipaio campground to the museum, which overlooks Halemaumau crater. In daylight a sulfur dioxide cloud rose from a lava lake hidden somewhere at the bottom of the crater - but after dark the lava illuminated the crater and clouds of gas in the sky. The red glow grew or dimmed as the surface of the lava broke or cooled, and sounds of cracking, boiling rock came from the crater. Crowds gathered around 8 pm to watch, but camping 1/2 mile away makes it easy to observe the crater any time at night. At 4 am, we were alone. Bring a sleeping bag and full thermos for most enjoyment : )

Pua Po'o lava tube

We planned our trip to Volcanoes Park around a tour of Pua Po'o lava tube, which is only offered on Wednesdays, and to only twelve visitors each week. Our friends Jessie and Jeremy visited Hawaii last year and among lots of great advice that they gave us, they told us DO NOT MISS THIS TOUR - thanks Jessie and Jeremy for this sound advice! Reservations are made by calling the visitor center, (808) 985-6017, and aren't taken any earlier than 7:45 am on Wednesday, one week before the tour. Reservations aren't usually taken any later than 8:00 am because the tour is full by that time. So at 7:40 am one week ago I started dialing like crazy and was lucky to make reservations!

Who should lead this special tour? None other than elite interpretive guides Dean Gallagher and Ruthie Rodelander! Weeks earlier, Ruthie lead an outstanding tour of Kilauea Iki crater on which Melissa and her family were the only visitors

Mauna Loa

We packed every scrap of warm clothing with us to the Park because we wanted to climb Mauna Loa, weather permitting. But when we sought permits, we were warned of a storm, snow, lightning, and possible whiteout visibility. Also there were obstacles of getting to the trail head, 13.5 miles and 2,000 feet elevation from the highway, without a vehicle, and of stashing our extra luggage, like my computer, or lugging it up the mountain. We made up our minds to hike Napau and the coast trails instead - until we met French Canadiens Gael Doyon and Yannick "le proprietaire" Beaudoin at the campground that night. Gael and Yannick planned to climb Mauna Loa as far as Red Hill cabin the next day and invited us along. And we met Ruthie again, who generously offered to store our extra luggage! Thank you so much - Ruthie is pretty much the nicest ranger *evar*!

What choice did we have? We joined Gael and Yannick (team Canada) and had fun times! The sunset at Red Hill cabin lit up Mauna Kea and shown off the telescopes on top. After dark Halemaumau crater glowed below. Yannick is an artist and left this vision of our spirit animal, the unicorn-dolphin-pegasus: "unidolsus", in the cabin log. It was very cold and a little harder to catch our breath

Thank you for awesome times team Canada!

Weather was perfect and clear, so it was a hard decision not to continue to the summit the next morning. But the trip had been so good, the sunset and views so beautiful, that we decided to quit while ahead. Also we were in the park for a limited time, and decided to trade two days struggling to the top of Mauna Loa for a chance to explore the coast trails

Kilauea Military Camp

We returned from Mauna Loa too late to start that afternoon, so spent another night at Namakanipaio, where we learned that the Kilauea Military Camp government kitchen serves affordable breakfast, lunch, and dinner. When I investigated, I also discovered cosmic bowling is available on Friday and Saturday nights! Also met nice folks from a nearby community, doing laundry while their kids were entertained in the small theater. The KMC is like a community center

At the KMC, if you can eat a burger with five 8 oz paddies, plus 2 lbs of fries in 30 minutes, you get a t-shirt, your picture on the wall, and an opportunity to rename the challenge. If not, you pay $20 for the food. I badly wanted to rename it the "unidolsus" challenge, to honor team Canada, but sadly there wasn't an opportunity : ( Probably for the best - dunno if even my formidable eating skills are equal to the challenge of 4 1/2 lbs of food in 30 minutes

Coast trails

Without a vehicle, we needn't end our hike where we started, so we chose to start high above the coast at Hilina Pali overlook and hike down to the end of Chain of Craters road. There's lots of traffic on Chain of Craters road, but we were lucky to get a ride to Hilina Pali overlook - it's 9 miles down a one lane road which few people drive. We stopped at Ka'aha and Keauhou campsites, and camped at Halape and Apua Point. All were beautiful, with opportunities to swim in the ocean, but Halape was amazing - probably the most beautiful spot I saw in Hawaii

The whole coast is lava flows of various ages, but in a couple places there are pools and pockets of sand, variously surrounded by jagged black lava. Halape is such a place, and the contrast between clear water and white sand under coconut palms, embraced by jagged black reefs, in the shadow of tall cliffs and on an erupting volcano, under electric blue skies, was dramatic. A short walk from the campsite, along the ocean, toward the cliffs, leads to another stand of coconuts, "Halape Iki", possibly even more beautiful than Halape itself

Composting toilets

Trails we hiked were all exceptionally well managed: Permits were well organized and rangers were friendly and informative. And we saw almost no one else on them! But the single best thing about Volcanoes Park were absolutely the Phoenix composting toilets. We found them at Red Hill cabin at 10,035 feet, and at every campsite along the coast. Not only is composting environmentally sophisticated, the user experience was excellent! The toilet at Red Hill cabin was more insulated than the cabin - almost warm. On the coast, toilets featured half height walls, affording fresh air and views. All were immaculate, with clear instructions and tasteful bins of bulking agent. Nice thrones!

We collected our luggage from Ruthie - thanks again! Our visit to Volcanoes Park was awesome thanks significantly to your hospitality

Finally, we hitched rides to Kalapana to see boiling rock

Boiling rock

Kilauea has been erupting since 1983 but at the time we visited, lava was flowing only outside Volcanoes Park, at Kalapana. At sunset, crowds gather at what's now the end (because it's covered by lava) of highway 130. It's controlled by the county - open only between 5 and 10 pm and no one is allowed to walk far on the lava, except on a guided tour. From there, at the time we visited, lava could be seen glowing one or two miles in the distance - but it could be mistaken for house lights. Guided tours walk right up to flowing lava and cost $50

But our friend Mighk explained the lava viewing circus shuts down after 10 pm and during the night it's possible to walk out, in pitch darkness, on uneven lava, prone to collapse. People will tell you that it's private property and you're trespassing, or that it's dangerous and life threatening - that you may misstep on fresh, thin crusts of rock and fall through, injuring yourself or dieing in boiling rock. Listen to them: they're correct!

On our way to Kalapana we stopped in Pahoa for Thai food - both Thai restaurants in Pahoa are exceptional

Kaimu beach near Kalapana was once the most famous black sand beach on the island - until it was covered by lava. There's a small, young beach now, where young coconuts have been planted. Camping at this beach is "kapu", but we arrived after dark and had few options. We pitched our tent in the least conspicuous spot that we found. At 4 am we packed the tent, stashed our packs, and walked across the lava, seeking boiling rock

We carried water, snacks, and lots of lights - headlamps, spare lights, and backup lights. We could see lava glowing up the hill, but you don't need to go far up the hill to see it flowing. We stayed low, parallel to the coast, without walking next to the ocean because this lava occasionally collapses into the ocean, which may be boiling hot and acid. As we walked we saw other lights, which turned out not to be county officials or property owners, but other folks seeking boiling rock. At 6 am we came to a breakout which we watched for hours, while the sun rose. Arriving at flowing lava before sunrise and returning in daylight is ideal

On our way back to our packs, we followed the highway from Kalapana to Kaimu - this was a mistake: Lava reaches all the way to Kaimu, and to follow the highway is much farther. The cafe in Kaimu opens at 7 am and we enjoyed another exceptional meal. A kava bar is also in Kaimu

Here are more pictures

2011-01-13T21:43:00.000-08:00

Melissa and I enjoyed hospitality at beautiful Kohala Sanctuary for six weeks - we're about to carry on visiting other farms here in Hawaii. Thanks Joel and Michelle!

A reality we discovered, working with the garden for such a short time, is that it's hard to know the history - what areas have held what crops and in what areas the soil needs building/resting. In case it's helpful to caretakers or future stewards of the garden, we leave this retrospective

Fersher, everyone will bring their own approach to gardening - these are some of our stories. It's valuable to build relationships with the land. Remember the first principle of permaculture: first observe and then interact : )

When we arrived, the garden already contained,

Parsley hedge
Hawaiian chillies, possibly tabasco chillies?
Cilantro
Basil
Kale
Collards
What do ya call 'em tomatoes?
Sage
Dill
Male papayas
Dragon fruit
Curry leaf
Unusual strawberry

Changes

Started a trellis for the tomatoes, but still needs the trellis material - hopefully something suitable can be got from the Hilo garden center
Planted more dragon fruit. Dragon fruit are over planted because the plan is to nurse them in the garden, then possibly establish them elsewhere around the property?
Reduced the parsley hedge
Transplanted lemon grass and galangal into the box
Planted more kale by simply sticking stems into the ground, along the edge as a "brassica border", so pests will attack these hardy, plentiful plants first, and hopefully spare more delicate plants. Kale is producing many leaves
Relocated herbs into one section
Planted cassava under the papayas and squash under the cassava, as a guild. Also planted kava
If you consider removing the the black landscape cloth and planting some understory thing, maybe look into dwarf clover or something that can create a good ground cover. Check Masanobu Fukuoka and others who have experimented with his ideas
Planted bush beans and pole beans by the tomato trellis poles
Planted taro, these are hopefully an edible variety? Maybe wanna get some identification tips from someone. Aunty Leah was concerned about some in the kitchen garden
Taro is growing rapidly - when it starts to die back, it could be harvested and boiled
Placed a second compost bin in the garden, so the first has a chance to cook down when full. Started cooking down the first at the end of December, might wanna give it a stir at the beginning of February to see how it's doing
Thinned and transplanted swiss chard
Planted yacon, sunflowers, and hollyhock
Sunflowers and hollyhock haven't come up yet
Planted lettuce, bok choy, beets, carrots, daikon, basil, calendula, marigold, coriander, etc. but still places in the garden that *aren't* planted and could either use some cover crops, compost, etc. and then planting or just keep fallow for a while
If you're trying to keep yourself with lettuce etc. all the time, plant a little bit every two weeks for continual harvest
Some of the these seeds did fine, others came up but then got munched (beets) or just kinda stayed stunted (brassicaceae). Could probably plant other things where the markers are if seeds haven't come up in three weeks from the planting date
Spinach is not really working - maybe ask around what varieties people grow here. Typically it's a cooler weather crop, but there're probably some that are bred for the tropics. Think Tom Baldwin had some in his garden when we visited
There're some things in pots that need homes. It's best not to retard plants' growth by keeping them in pots too long - just long enough to develop one or two of it's true leaves
Morigia and piegon peas need homes
There're brocolli that need a place in the garden - give them compost and one foot centers and hopefully they will do ok
Planted zuchinni
Squash we planted came from a store bought fruit so it might not be true to its parent. Could be good, could be bad. There're two squashes in the garden labelled "tahitian squash" that came from the seed cupboard and are hopefully reliable. Tahitian squash is supposed to be a really yummy crookneck squash. If you wanna save seeds from squash, keep varieties separate. Look up some seed saving books in the library. Squashes are notorious for interpollination
Added a "hose path" - short bamboo stakes to pull the hose around, so it doesn't slide over and squish plants
Increased the paths to make it clearer where to walk, to avoid unnecessarily compacting soil where plants are growing
Transplanted asparagus to the garden
Right now there's lettuce growing right next to the asparagus. When that's pow you could make a bed (it's obvious) so the asparagus insn't being used as a path

Kitchen garden

Planted the yacon in rows
Planted some turmeric
Need to check if that grass that was left after the weeding party is actually sugar cane or just the big cane grass
Bare areas need to be mulched so they're not taken over by weeds again

Recommendations

Check pH of the garden
Maybe also get a soil test of the garden to see if there are any nutrients missing, because the culture of composting has apparently been hit and miss
Make a garden map and rotational scheme so that pests, diseases, and nutrient deficiencies aren't building up in the garden
Maybe grow crops that don't need as much fertility, like legumes, for a while to build up the soil. In the yurt, Jon Jeavons books' have chapters with ideas about building soil in the first few years of starting a garden
Kaffir lime tree should be moved out of the garden at some point because it can grow too large

Ways to build soil fertility

Make compost! You can never have too much. If animal manure isn't an option, try experimenting with making compost piles with the weeds you have, the grass clippings and other materials from the property. Here's more info on building compost
Also try incorporating green manures (cover crops) into the garden to build it up - Ben from Sage farms has a mix that works for him. Maybe ask him where he gets his seeds
Also keep mulching, mulching, mulching... See Emilia Hazelip Synergestic garden video for more info

2011-01-02T09:41:00.000-08:00

Recently required little research to configure Apache authentication with username/passwords stored in MySQL - here's a summary

Motivation

Am configuring authentication to DBMail, Postfix, and Apache. One fine day I want to authenticate to all three with my GPG key and GnuTLS. Til then I settle for configuring each to authenticate against the same username/passwords

Apache is flexible - lots of modules are available for authenticating against variously stored username/passwords. Postfix is also flexible - it supports both Cyrus and Dovecot SASL implementations, which in turn authenticate against variously stored username/passwords. DBMail isn't yet as flexible. Since version 2.1.0 DBMail supports accounts stored in LDAP - the only other option is SQL

Regarding LDAP, my experience is that there are two general styles of LDAP integration,

presumes an existing schema is already in use (possibly a standard schema like inetOrgPerson) and provides options for mapping to it. Apache mod_authnz_ldap, SASL LDAP, and pam_ldap are examples
configuration backend extends LDAP schema to accommodate application specific semantics. Asterisk is a prime example

DBMail is the latter style. LDAP for all semantics could be more powerful, but is overkill when the only goal is to authenticate against the same username/passwords as Postfix and Apache. DBMail is inoperable without SQL, regardless of whether accounts are stored in LDAP or SQL, so accounts stored in LDAP increases the points of failure. And I think it's more common to store DBMail accounts in SQL

Apache ships with support for authenticating against username/passwords in SQL via mod_authn_dbd, but after configuring Apache like so,


  DBDriver mysql
  DBDParams "dbname=dbmail user=dbmail"

  <Location />

    AuthType Basic
    AuthName nottheoilrig
    AuthBasicProvider dbd
    AuthDBDUserPWQuery "SELECT passwd FROM dbmail_users WHERE userid = %s"
    Require valid-user

  </Location>

- I can't log in : ( error.log contains, "authentication failure for "/": Password Mismatch"

SELECT returns the correct password, the problem is the password format

In current mod_authn_dbd documentation, only mention of the password format is, "The first column value of the first row returned by the query statement should be a string containing the encrypted password." Some Googling revealed this explanation of Apache password formats. Latest mod_authn_dbd documentation is updated with a link to this explanation

DBMail supports many password formats,

plaintext
crypt
md5-hash
md5-digest
crypt-raw
md5-hash-raw
md5-digest-raw
md5-base64
md5-base64-raw

- however SASL SQL supports only passwords stored as plaintext

On *nix, Apache doesn't support passwords stored as plaintext, but the password format explanation includes examples of PostgreSQL expressions for some formats, in terms of the plaintext password. It omits examples of MySQL expressions

I first attempted a MySQL expression for the Apache SHA1 format. Happily MySQL ships with a SHA1() function, but lacks a base64 function. The workaround, base64 implemented as a user defined function, is clever but fugly!

I didn't attempt a MySQL expression for the Apache MD5 format because it's apparently Apache specific

Finally I attempted the Apache CRYPT format, which the MySQL ENCRYPT() function seems to return. I updated Apache configuration,


  DBDriver mysql
  DBDParams "dbname=dbmail user=dbmail"

  <Location />

    AuthType Basic
    AuthName nottheoilrig
    AuthBasicProvider dbd
    AuthDBDUserPWQuery "SELECT ENCRYPT(passwd) FROM dbmail_users WHERE userid = %s"
    Require valid-user

  </Location>

- and am able to log in!

2010-12-17T09:15:00.000-08:00

Working on a Postfix "content filter" to rewrite the sender address of messages I send based on the recipient address. The rewritten address will also carry some information, such as identifying the recipient address. It's purpose is that I wanna use a unique address for e.g. each mailing list I join, so if one address starts getting abused, I can send it to a black hole. But I don't wanna remember which unique address I used when I post messages. This content filter will automatically rewrite the sender address with the correct address. It might also be interesting to know which list an address was harvested from

Here's my work in progress

Got the idea to rewrite the sender address based on the recipient address from TMDA, but addresses rewritten by TMDA carry only a hash of the recipient address, so it's easy to check if a rewritten address matches a given address, but hard to recover the address. So I chose to implement a custom content filter

Implementing a milter would simplify Postfix configuration because a milter doesn't require a second SMTP server, to accept messages after being filtered. Unfortunately a milter can produce at most one message for each message it receives. Because each recipient gets a different sender address, messages to multiple recipients must be split into multiple messages, each with one recipient and different sender addresses

Would like to implement this content filter in my current favorite language: JavaScript, with node.js. However I found no mature node.js SMTP implementation, so used Python and Twisted instead

I wanna recover information carried by an address, but hide that information from others. I also wanna distinguish addresses I generate from addresses forged by others. I thought of two alternative solutions:

Encrypt the information and embed it all directly in the address
Store information in a database and embed only a key in the address

A database is maybe the best way to hide information from others, because an address doesn't directly carry any information - just a database key. However AES encryption is probably also effective, and the information is low risk. At first I thought encryption was simpler, more direct, and distributed: addresses could be rewritten and information recovered without access to a central database, only a copy of the key is required

Some challenges I discovered were,

Padding the information so it's a multiple of the cipher block size
Representing the cipher text with the shortest address. RFC 5322 allows 81 possible letters in an address "local part", without quotation. Actual SMTP implementations may allow less. AES cipher text is a multiple of 16 bytes
Distinguishing addresses I generate from addresses forged by others

Given the key, is it inherently possible to distinguish an AES cipher text from a forged cipher text? Does prepending a second key to the plain text, for the purpose of recognizing forged cipher text, weaken encryption? Is a "message authentication code" what I want?

I now think a database is the simplest solution. It'll be easy to recognize a forged address because the key won't exist in the database. A database also solves another hypothetical challenge with encryption: If I exchange messages with someone who also likes to rewrite their sender address such that it carries my address, then each time we reply to each other, we'll reply to a different address and our replies will be rewritten with different sender addresses. The address will grow each time we reply

With a database I can use the same key for different recipient addresses, in case a recipient uses different addresses. I can even use different keys for the same recipient address, based on the In-Reply-To message header. Probably some MUA are sophisticated to do this, but I could use any MUA and handle it server side

How can I generate new database keys unpredictably? Non-existent keys can't be forged, but if keys are generated predictably, then different, existent keys could be forged

2010-12-08T20:39:00.000-08:00

Had heard that mobile service in Canada is some of the most expensive in the world - but I now think mobile data in US is same or worse

Am in Hawaii for a couple months and I brought my Nexus One. Data roaming with my carrier (Rogers) apparently costs up to $30.72 per MB, so I searched for a local carrier. AT&T apparently uses the same frequencies as Rogers (GSM 850 MHz, 1900 MHz), therefore the same frequencies as my N1

Actually there's absolutely zero mobile coverage where I'm staying ATM - I can only use service while traveling other places with coverage - so I want a cheap plan

AT&T never presents all their options in one place: monthly data only plans, monthly voice plans with data, "GoPhone" pay as you go, and "DataConnect Pass" pay as you go. Also, sales reps invariably ask about your situation ("Got a phone or a netbook?") and prescribe a plan, vs. presenting all the options and letting you choose. Here's my summary,

Monthly

Transfer	Cost	Cost per MB
200 MB	$35	$0.175
5 GB	$60	$0.012

Monthly with cheapest voice plan ($39.99 for 450 minutes)

Transfer	Cost	Cost per MB
200 MB	$39.99 + $15	$0.075
2 GB	$39.99 + $25	$0.012

GoPhone

Duration	Transfer	Cost	Cost per MB
1 month	1 MB	$4.99	$4.99
1 month	100 MB	$19.99	$0.2

DataConnect Pass

Duration	Transfer	Cost	Cost per MB
1 day	100 MB	$15	$0.15
1 week	300 MB	$30	$0.1
1 month	1 GB	$50	$0.0488

What I noticed,

$35 per month buys more from Rogers than from AT&T - I paid Rogers $35 per month for 1 GB - $35 buys only 200 MB from AT&T
AT&T monthly data plans are very limited options: 200 MB is too little data, $60 is too much money, and there are no options in between

My choice: $19.99, 100 MB "GoPhone" pay as you go. The cost per MB is roughly the same as $35, 200 MB monthly data plan ($0.2 per MB) and I can't pay less per MB without paying $50 or more per month. At the same time I bought an additional $25 of "GoPhone" credit, which expires in three months. With this credit I can make calls for either $0.1 per minute, or $2 per day of unlimited minutes - because making calls might be handy while traveling

There was no additional charge for the SIM, and it worked immediately in my N1

Anyone know how to install the AT&T Android widget that displays usage and sells credit?

After the fact I wondered if any Rogers "travel pack" was competitive with local service. Rogers offers three options for travel in Canada and US: one time "travel packs", $10 per month for $1 per MB roaming in Canada and US, and $60 per month for 1 GB in Canada and US without roaming charges

At best, a "travel pack" costs $0.8 per MB with a $60 pack, and packs expire in 1 month. You can technically add and remove the $10 per month for $1 per MB feature whenever you need it - but Rogers sales reps are only supposed to sell it with a commitment of at least 3 months. Apparently you can get a plan with 1 GB in Canada and US without roaming charges, for $60 per month

2010-11-08T16:56:00.000-08:00

I'm a Cooperative Auto Network member and decided it'd be fun to hack on a Car Coop Android app. I learned from the Car Coop that no app already exists, but neither does an API that an app could use

So my first step was to get car coordinates. Karen helpfully pointed me at car coordinates in JavaScript source and I hacked a script to convert them to KML

Here it is in action

Also wrote a quick test with jQuery and QUnit - it's awesome how easy and fun web programming keeps getting!

I think this map already works on my mobile better than this one because info windows are full screen vs. balloons, and link to transit directions and Street View, which helps locate cars

There's little on the server side. I considered trying node.js and Joyent, to have JavaScript front and back, and because Node is the rage. But I settled on Python and App Engine because I'm familiar

Google Maps' Developer's Guide explains how to detect the user's location

KML layers look different in the Maps API than in Google Maps or the Mobile Maps app, e.g. info windows appear as balloons vs. full screen and lack links to transit directions or Street View

Links like http://maps.google.com/maps?q=http://carcoop2.appspot.com/index.kml%23nw-ash-fourth-avenue sent to Android are handled with the Mobile Maps app

Google Earth respects same-document references, but Google Maps and the Mobile Maps app don't

Next steps

Color <Placemark/>s according to whether cars are booked or not
Client side filters for various car features, like minivans, iPod auxiliary jack, etc.
Center "Loading" <div/> with CSS3 and flexible box layout module
Handle URL fragments and call Maps API accordingly
Is it possible to write plugins for the Mobile Maps app? Like clustering?

I still wanna hack on a Car Coop Android app with this KML - any other Car Coop members also interested in hacking on apps? Please use any code that's helpful!

2010-06-21T09:22:00.000-07:00

We celebrated my Gran's life on Saturday

Margaret Clarries Bates nee Edser, October 3th, 1920 - April 12th, 2010

Before she died, Gran said she lived a full, happy life, and has four accomplished children and six accomplished grandchildren - what more could she ask for? I agree Gran lived an admirable life - she's an inspiration to me

My memories of Gran are of her character, her sense of humor, her intelligence, how easily she made conversations, and how smartly she dressed - and that she maintained this character and dignity to her very end

I struggle to make conversations - to discuss meaningful subjects - to get off topics when they've been exhausted. I'm especially afraid of conversations with people of other generations, because we have even less shared experience. Conversation with Gran was a pleasure - she could have an interesting conversation on any topic. Gran was hip - up on current events and changing times. She was direct

I imagine Gran was an excellent facilitator - I bet she kept meetings focused and productive - I'd like to have served on a board with Gran. I dunno if Gran was ever the chair of a board, but she volunteered for many years with the Peach Arch Hospital Auxiliary

Gran was very intelligent and enjoyed playing contract bridge - I bet she was very good. After suffering a small stroke about a year ago, Gran complained that she couldn't do her figures like she used to - until then she balanced her cheque book by hand

Gran was impressively stoic in the face of pain - she suffered from crippling arthritis for many years and yet was always charming and dignified. Even in the hospital, suffering from a perforated bowel, she complained that the morphine dulled her senses and requested it not be administered. "They don't want you to feel any pain," she complained

Please someone correct me if I'm wrong - I gather Gran and Grandpa met while stationed on the same aircraft carrier in the second world war. Grandpa was a radar technician and Gran was a radio technician - so hardware hacking is part of my heritage : )

Gran and Grandpa relocated their family by boat three times: From Britain to Canada, back to Britain, and finally back to Canada

Uncle Chris generously scanned and shared pictures from many old albums with the family. What stands out looking at these pictures is that Gran is always smiling - and that she and Grandpa did lots of traveling together. Amongst raising four children, relocating three times, and Grandpa teaching math at UBC, they made time to explore British Columbia - before many of the roads were paved. They traveled to Switzerland and Yugoslavia, and later took many bus trips and cruises together

The best advice I received from Gran:

I'm inspired by the length and strength of Gran and Grandpa's partnership - we celebrated their 60th wedding anniversary a couple years ago. 60 years is a *long* time! I'm curious what it's like to share 60 years of experiences with another person - Gran and Grandpa looked like they enjoyed each other's company very much

Once I asked Gran what was the secret of her's and Grandpa's relationship - what I should look for in a partner. She said the most important thing was a sense of humor, which I think is wise

At our celebration of Grandpa's life a couple years ago, Gran said she "picked a winner"

Gran is an inspiring woman - a role model - I hope I realize some of her strength and character in my own life

2010-06-06T09:00:00.000-07:00

Had great time at second "Luke's Retreat" yesterday. Then cycled home in the sunshine. Perfect

Generally I feel inspired - not sure what I feel inspired to do specifically - but a day of dialogue with high functioning people is inspiring - inspiration to be high functioning. Thanks everyone!

Format was like the first retreat - full day of group dialogue. Luke skillfully and subtly facilitates - making and holding space

Retreats are an opportunity for,

Personal reflection, which I don't otherwise make time for
Relaxation, outside normal routine - stepping outside normal routine also creates perspective
Dialogue

Sound bites

Analysis paralysis - when I'm stuck analyzing a problem instead of taking action
Spaghetti on the wall - eventually we gotta make a meal - reminded me of times at my housing coop when everyone's brainstorming, but eventually we gotta formulate a plan
Heroics a sign of dysfunction in an organization
Trim tab - miniature rudder on the trailing edge of a huge rudder - metaphor for leadership
Capitalism isn't the problem - consumerism is the problem

Feedback

Noticed tone was bit different compared to last retreat - jumped quickly into agenda and then dialogue this time. Last retreat began in total silence, with round of eye contact with each participant - feelings of anticipation, focus, meditation - maybe this difference was actually in myself vs. the retreat

Delicious food - sandwiches, blackberry banana bread, honey buns, loads of fruit - thanks Jen and Luke!

Wished for longer lunch break - went for a walk with David Ascher at lunch last retreat. Because whole day is one format: whole group discussion, lunch is opportunity for contrast: small group discussion. Initially felt tired after lunch

Luke mentioned in his last retreat wrap up the importance of diversity to the dialogue, and there was good diversity again this time - though no one confessed to hating critical mass : ) There was good gender mix but mostly young people. Wonder if Rob Cameron would enjoy attending a future retreat? or maybe Tim Bray? or Margo Leight?

Like last time, I really enjoyed the physical game during afternoon break : )

Snacks are an important part of retreats and another opportunity to participate - would be fun to contribute a snack next time

Holding retreats between November and April, folks might be less tempted by sunshine outside? or summer retreats could be held outside? Holding retreats after January coincides with New Year's catharsis

Dialogue

Anthony mentioned his interest in interdisciplinary dialogue at school, and my housemate Maria just completed the Undergraduate Semester in Dialogue, but I didn't get a chance to talk about it with Anthony

Wonder if Maria would enjoy attending a future retreat?

Language translation

Got to meet Ingy döt Net IRL! Ingy showed me C'Dent - a project he's working on to translate modules between scripting languages

It reminded me of this thesis

Also, Ingy was wearing the most awesome belt!

Productivity

Lots of productivity topics discussed in the morning - consensus: no good way to organize ideas?

Kanban - limit the amount of work in progress - the more things on the go, the slower they all go

I plan to give the hipster PDA a try - cue cards held together with an alligator clip

Pomodoro - technique sworn to by Audrey Tang - write one task on a sheet of paper, set physical egg timer to 24 minutes (Audrey's patch - original Pomodoro prescribes 25 minutes, but 24 has common factors with more numbers), work *only* on this task. Then relax for six minutes - possibly meditate. Possible benefits of contrasting intense and relaxed mental activity

Anthony recommends identifying the motivation behind tasks, then recursively identifying the motivation behind motivation, til there's one clear reason or goal. Focusing on this reason or goal helps avoid distraction

Lots of us struggle with saying "no" - instead of seeing "no" as a negative - as a refusal, or failure to fulfill a request - Laura suggests seeing it as a point of information which saves us from drawing others' time and energy, allowing them to focus it elsewhere. I know the quotes, "When everything's important, nothing's important" and, "Saying yes to something is saying no to something else." Likewise, saying "no" may be saying "yes" to something else?

Scaling up

I hadn't heard of virtual assistants, but they and Amazon's Mechanical Turk are serious options for accomplishing more than one person can alone. Danielle LaPorte of White Hot Truth uses Monday Morning VA

Alternatively, instead of hiring a cog, scale up by hiring or otherwise working with one good person with same values at a time. I hear this is how Google did it?

Scale up by radical openness

How to scale up without institutionalizing? or how to stay an effective and nimble institution? Google has so far managed this

Takeaways

You are today everything that you thought and did in the past, so to change your future self, start with what you think and do today

A good day's work can't be it's own reward - think I took this differently than it was intended

Intrapreneur - how to change existing culture - go with the flow for a year, build social capital, then be the trim tab

Try not to judge people day to day, but on a long term basis

Sustainability is necessarily about reducing (deficit model) - need also to focus on the goal. Need to focus e.g. on what future generations will accomplish (asset model), vs. focusing on avoiding extinction

I really appreciate being invited to these retreats and often feel I haven't much to offer in return. Luke was interested in a bike trip I recently took - maybe he'd like to go bike touring together?

2010-05-27T11:24:00.001-07:00

Just got my first ever mobile - welcome 21st century : )

Got a Nexus One

For years I actively ignored the ubiquity of mobiles, mostly because of the high monthly service charges here in Canada

But I started spending about nine hours each way commuting to Cortes Island - most of which is idling on a ferry or bus. Using my netbook is nice, but what I really need is the internets!

Hardware

USB sticks that work with cellular carriers to provide mobile internet are now pretty common - but as long as I'm spending up to $200 on a USB stick, plus monthly service, might as well get an Android and tether my netbook?

Among advantages of a droid are I can check ferry schedules and play mp3s w/o hauling out the netbook (this is also my first ever portable mp3 player)

One advantage of USB sticks is that they support "HSPA+" (reportedly 21 Mbps) vs. "HSPA" (reportedly 7.2 Mbps) - don't think any HSPA+ droids are currently available

I considered the Motorola Milestone, Nexus One, or an older droid - maybe off Craigslist

Big thanks to the Vancouver Hackspace list for all the advice - it's *so awesome* that the Android Developer Advocate is a VHS member!

Milestone and Nexus One are about the same - same size, screen, OS, price. Same battery performance (not great?) Was about to get a Milestone because it features a hard keyboard, so more screen is visible while typing

Thanks Joe Bowser for convincing me to get a Nexus One instead, because it comes with root access. Getting root access on the Milestone is apparently a total hassle. Also, although the Nexus One lacks a hard keyboard, the Milestone hard keyboard gets terrible reviews, and I reckon a mechanical keyboard is a potential point of failure?

Painful root access isn't worth a good hard keyboard - so forget about a bad hard keyboard

Also the form of the Nexus One is quite stylish : )

Carriers

Rogers, Telus, Fido, and Bell all offer similar data-only plans - and similarly all hate customers

Plans are all about $30 - $35 /month for 500 MB or $35 - $40 /month for 1 GB

Can't tell the difference between Rogers' and Telus' coverage from the maps on their sites

I got equally negative reviews of each carrier

Wind looks awesome! Theirs was the only site where I could find the info I wanted without feeling trapped in a sales tunnel - a water slide except no fun and with sharks at the bottom (thanks for that one Jeremy : )

Too bad I think it'll be a *long* time before Wind offers service on the Gulf Islands

Signed up for $35 /month for 1 GB from Rogers because they're slightly cheaper and because most folks I asked use Rogers

Rogers, Telus, Fido, and Bell all hate customers because they refuse to sell data-only plans except with USB sticks - or without the customer dancing the following game. At least it's not a hard game:

Go to a Rogers store, buy a SIM card for $10. Claim you have a "USB stick" at home. If necessary, claim you bought it off e.g. Amazon. Sign up for a data-only plan. At home, insert the SIM card into the droid

Rogers and Telus also charge a $35 activation fee - possibly this is waved if you activate online, or sign a contract during a promotion. But the freedom to switch carriers without a contract is probably worth much more than $35?

First impressions

After playing with it for just a day, I think the Nexus One is polished and featureful - but the bundled apps are individually quite basic. Lucky it's open for extension?

The user experience is subtle and intuitive - finally I understand why folks love their mobiles like a junky needs junk!

Am already glad I got the Nexus One, so I can run the latest Android OS. It came with 2.1, and 2.2 is apparently in the process of being released this week? Not sure when 2.2 will be available for the Milestone, or for older droids

Two significant features of 2.2 are,

Built in tethering - so I'll probably wait a little for 2.2 before investing any effort configuring that
Install Market apps on the SD card - I hear this is useful for saving internal storage for things that can't use the SD card

Quickly discovered that the bundled apps are quite basic - probably this is a sound strategy:

stay focused on the 20% of features used 80% of the time
be open for extension

First thing I tried was playing mp3s shared from my netbook - I expected to find a list of DAAP servers on the network, as in iTunes, GNOME Rhythmbox, etc. but no joy. If the "music" app is open source, maybe an extension could be developed? Maybe one already has : )

Next I tried to join the "openarchives-dev" chat room on conference.jabber.org, but the bundled "talk" app doesn't support multi-user chat?

Really surprising was that the bundled "talk" app doesn't support voice chat - it's bundled on a phone after all! As well as surprising to find missing, voice chat is an important feature - I have data-only service and plan to make occasional calls by voice chatting with my Asterisk gateway

Haven't yet figured out

Haven't decided which "notes" apps to try - none are bundled with the Nexus One. Ideally it'll sync with Tomboy, so probably will try Tomdroid

Posting bookmarks to Delicious is important because I mostly bookmark while reading blogs and Twitter, which I plan to do more on the droid. I think Peter Baldwin's open source app is most populare (10,000 - 50,000 downloads). Wish Delicious mentioned Android on their tools page

Haven't yet figured out how to drag and drop - like several sites feature drag and drop interfaces, but dragging with the droid scrolls the page instead of dragging the interface element

At first there was a bundled "voice" app which promised to send free SMS messages - this is great since I have data-only service! but the app disappeared as soon as I inserted Rogers' SIM card - probably because Google voice is only available in the US?

2010-04-12T12:30:00.000-07:00

I had an incredible time yesterday at the Day of Workshops presented by the Collective House Network! These are some of my reflections on the event - my "retrospective",

Awesome things that really stood out for me

That it was held in collective houses, instead of e.g. a community center. It was a challenge to fit everyone - sixty people registered, an additional fifteen were wait listed, and some unregistered folks were welcomed - but exhibiting functioning collective houses, as well as discussing them, was awesome
Signs - washrooms etc. were all marked with signs which made the space very usable
Graphic design - the design of the promotional poster is *wicked*
Workshops - the workshops I attended all offered totally useful content and were expertly facilitated
Food! So much delicious food! Such modest budget!
Media - the Georgia Straight was invited - and attended!

These are thoughts I had for the next Day of Workshops

Name tag stickers - some conferences I attended, e.g. BarCamp Vancouver, had small, fun, but well discriminated stickers - e.g. animal faces but all with discriminated background colors. Attendees self-add zero or more distinct stickers to their name tags to indicate, e.g.
- Am a presenter
- Live/lived in a collective house
- Seeking available space to move into a collective house
- My house has available space
- etc.

These are workshop topics I'd love to attend at another Day of Workshops

Pitfalls of collective living and workarounds
Artifacts of houses that support collective living, like signage, linen service, dish sterilizer? This could be a wiki page?
Who doesn't wanna live collectively?

"Who doesn't wanna live collectively?" - this is something I'm thinking about recently - ever since a long conversation with my housemate Ty

I think my house has endured for (45 yeas?) because of membership diversity - when it's all Leninists or all Trotskyites, changing times can destabilize the whole group - all "isms" are "wasms"

But new applicants to my house are all young and unmarried - often students. And people are suspicious of mature singles. Not sure how to put this... People suspect that mature singles are anti social. There. I said it.

So what is it about collective living that doesn't appeal to other demographics? How can/should we fix this?

Things I wanna follow up on

Get electronic copies of the Cherry Tree Fort visioning mind maps and presentation notes

Takeaways

These are my "takeaways" from Colin's presentation,

Real priorities

Look at how you spend your,

Time
Money
Energy

e.g. many people love organic gardening and say they want a garden. But look at how you spend your time - e.g. studying - to identify *real* priorities

Cooperative structures

These are systems like,

Taking turns cooking
Rotating chores
Quarterly work parties
Annual *party* parties
etc.

There's no "one true" cooperative structure - difference structures work for different groups. But almost all groups require at least some cooperative structures

Cooperative structures facilitate participation in the group, and are a framework to promote equality, e.g. without any structures, often folks who like cooking quickly end up cooking lots, and folks who don't end up cooking little. This can lead to unfairness - real or perceived

This makes me wish for resources of various cooperative structures - like "A Practical Guide to Collective Living". It makes me think there might already be such a book. I should ask on the CHN discussion group if anyone knows of such resources. Colin offered to send me a list of some books by email

I think a resource like this could be really valuable because, e.g. Melissa and nine classmates recently moved into a house together. There were no cooperative structures already in place, so they made their own. Melissa said some examples could have helped. Maybe one benefit of the CHN could be to provide this resource? Similar to these Hackerspaces Design Patterns?

This too could be a wiki page?

Ministries

At Colin's collective house, instead of "committees", they have "ministries",

Ministry of small projects
Ministry of the interior - decorating committee
Ministry of agriculture - garden committee

I think this is a fun name change because "committees" has negative, bureaucratic baggage. Well, so does "ministries" - but its use in this case is ironic : ) I wonder if my house would embrace this change?

Pitfalls and workarounds

Totally interesting that in our brief discussion of pitfalls of collective living and workarounds, the same issues which come up again and again at my house also came up,

Dishes - many collective houses have zero tolerance for leaving dirty dishes. At others, the "daily reset" of full kitchen cleanup after dinner solves the problem
Guests - some perceive unfairness in others frequently having guests, who use common food and possibly common space. Others don't want to track how often their boyfriend/girlfriend/partner visits. At Linnaea Farm they have a five day maximum stay. This is my houses' guest policy
Common space - some think it's wasteful not to take over unused common spaces for projects like bike building. Others are resentful because they *might* use common space if it weren't occupied. At my house we have a "notice of common space use" form, to be used to notify other members of common space use and anticipated project timeline

Interesting to hear that at one house they use a "communication book" as a safe space for writing things to each other. They try to maintain a balance of positive and negative comments

Conflict and communication

Somehow these amazing presenters managed to pack a three hour workshop on effective communication into like one hour?

Five basic types of communication,

Request
Offer
Statement
Judgement
Promise

Many communication breakdowns result from inconsistent grammar and "intent", e.g. "I'm always washing the dishes" is intended as a request, but grammatically it's a statement

"VOMP"ing is a method of conflict resolution,

Vent - both parties get a chance
Ownership - both parties take appropriate ownership - often the hardest part
Moccasins - both parties empathize - walk a mile in each other's shoes (moccasins)
Plan - create a plan that resolves the conflict

VOMPing takes time

Buying a house collectively

To incorporate a cooperative in BC, there's a helpful document, "starting a cooperative in BC". Link?

Canadian Cohousing Network is a useful resource

The Walnut Street Coop is an excellent example of revolving loans

I had an "aha" moment at Colin's comment that you don't need to have a mortgage or own property to incorporate a cooperative. You could incorporate a cooperative for the purpose of eventually owning property. The cooperative could continue to rent property from a landlord, but any extra funds could be invested. The memorandum could stipulate that, should the cooperative ever dissolve, assets be transferred to another, similarly purposed cooperative

To me, this seems like a good idea for any collective house!

Collective house directory

I think some folks are working on a directory of collective houses - which is awesome! because I think this list of collective houses is largely unused

I wonder if I should promote that list a bit more? or wait for the new directory

Would to cool to have facets of collective houses in that list/table, e.g.

Diet: meat, vegetarian, vegan
Size: maximum people/rooms
Ownership: Rental, cooperative
Age: years the group has existed
etc.

What would be some interesting facets?

Mashups

Someone mentioned that one of the Collective House Network's benefits is connecting people seeking available space to move into a collective house, with houses that have available space. Someone else mentioned that they used both the CHN and Craigslist to make this connection, and actually it was Craigslist that made the connection

Could we automatically embed some relevant Craigslist postings in the CHN website? So the CHN could better connect people and houses by not splintering this existing housing forum?

Speaking of mashups, Grounded TV embeds Facebook friends in their web site - the CHN has a Facebook group

2010-03-19T14:12:00.000-07:00

Checked out Open Data Drinks and Apps last night

Chatted with Boris Mann about how the City of Vancouver Open Data Catalogue should allow user contributions - citizen collected data. I think this is an awesome idea - Pete Lypkie impressed me earlier this week by demonstrating what's possible with individuals contributing GPS tracks to OpenStreetMap

Another theme discussed by Boris and Roland was applying lessons from the open source movement to the open data movement - for example that communities around assets like code - or now data - are as important as the assets themselves

Also the freedom to fork is important for free development, but a home where the code/data lives, and contributors gather, is also important

By allowing user contributions, the Open Data Catalogue could be this valuable home

An interesting project might be to import data from the Open Data Catalogue into OpenStreetMap - perhaps eventually automate a process whereby data posted to the catalogue were shared with OpenStreetMap

Zak Greant made some great points about open data currently being simple data sets, while lots of interesting data is in the form of documents - coincidentally an area where Microsoft, who hosted the event, make lots of money (Office, SharePoint, etc.) Adding semantics to documents is hard

- and I really enjoyed chatting with Sue Bigelow about the challenge not only to make archival records accessible, but also relevant - could social technology and user contributions help? or are they inapplicable to carefully cataloged historical records?

Conflating open data with open source is a trick which leads folks to think they need to adopt open source to join the open data movement - not necessary a bad thing, and a perception which Microsoft is apparently fighting hard. Is there a difference between doing open data with open vs. proprietary tools? I guess it wouldn't be open data if there were - but open tools have a greater legacy of promoting open data formats - and arguably open data

Urbanspoon is a great iPhone app and Sushi Sushi apparently has a great vegetarian menu : )

2010-03-15T18:43:00.000-07:00

What's the best way to achieve the behavior described in this W3C working group note on XHTML media types with Apache?

I want to serve up this HTML document with content type "application/xhtml+xml", so Firefox respects the @xml:base attributes

However Internet Explorer totally ignores application/xhtml+xml documents - visiting an application/xhtml+xml document with Internet Explorer throws up a "save file" dialog

So I'd like to follow the advice from this W3C working group note and use content type "applicatin/xhtml+xml" if the user agent advertises "application/xhtml+xml" in the "Accept:" request header, otherwise use content type "text/html"

I tried assigning both mime types to the document with the Apache AddType directive, hoping mod_mime would sort it out. No luck

What's the best way to achieve this behavior?

2010-02-22T16:49:00.000-08:00

Last weekend I wished I could simply paste CSV into MediaWiki and render it as an HTML table

With a little JavaScript, jQuery, and regular expressions, now I can. This script finds all elements with class "csv", transforms their content from CSV to an HTML table, and replaces the original element with this table

Now I can paste CSV into MediaWiki and wrap it in an element with class "csv",


<div class="csv">
7 grain cerial,,$7.39
Oyster sauce,,$2.09
Sweet chili sauce,,$2.99
[...]
</div>

Here it is in action

Anyone know if there's a CSS style to align the contents of a table column at the decimal point? or control how many significant digits are displayed?

2010-01-16T11:45:00.000-08:00

I *hate* "reward points" - those points you collect for using a credit card or loyalty program. I hate them even more than gift certificates

Reward points waste my time figuring out what I can spend them on and what I can't
I never *really* know what a "point" is worth (probably this is by design?)
Unused points sometimes just disappear
(The redemption websites sometimes suck)

Reward points == terrible user experience. It's like being forced to spend monopoly money in a parallel, soviet economy

My first credit card was a Royal Bank Visa Classic II Student, which I'm pretty sure had no annual fee at the time. Using this card earned reward points

When I got fed up with consistently terrible customer service, I switched to a TD Rebate Rewards Visa, which also has no annual fee. Using this card earns 1% "cash reward" - so for every $100 I spend with the card each year, I get $1 credit in January. Unlike reward points, I think this credit is *actual* money!

I can spend it wherever I use the card
I can intuitively gauge it's value ($1!)
It's not likely to just vanish anymore than is my card balance

- of course I think my consistently terrible customer service was actually from Visa, not the Royal Bank - so switching to a TD Visa hasn't changed that. Because I think a Visa card is basically essential in these times, in this part of the world, e.g. for shopping online, I think I'm stuck dancing to Visa's music : (

When I switched, I didn't actually cancel my Visa Classic II, because I would loose something like 38,000 reward points - whatever that's worth. Instead I just stopped using it

So I was surprised last week to discover a $35 charge on the Visa Classic II. The charge was from Visa and turned out to be an annual fee. Turns out Visa decided I'm not a student anymore and automatically switched the card to a Visa Classic II *with* annual fee. It's annoying because I actually haven't used the card in over a year

Visa explained that the benefit I get from the Visa Classic II, in exchange for $35 a year, is that (if I use the card) I earn reward points "hand over fist". Specifically I earn one reward point for every $1 I spend with the card - unlike the Royal Bank Rewards Visa Gold, which has no annual fee but only earns one reward point for every $2 spent with the card

So how much are these shifty points actually worth? Turns out, at least towards travel - like flights, I can redeem 100 points for $1. So the benefit of the it's-so-awesome-you-gotta-pay-for-the-benefit Visa Classic II is that for every $100 I spend with it, I get a quantity of monopoly money redeemable for $1 towards only *some* things I actually wanna buy in the real world

I think this is identical to the Rebate Rewards Visa, which also earns $1 for every $100 I spend with it, except that with the Rebate Rewards Visa,

I earn *actual* money
I don't have to play games like paying an annual fee which may or may not be less than the value of the rewards

Happy ending: I got great customer service last night from Amanda at the Royal Bank and Tony at Visa (employee number 3744). I switched my Royal Bank Visa from a Visa Classic II to a Rewards Visa Gold

I didn't loose any reward points
I continue paying no annual fee for either the Royal Bank Rewards Visa Gold or TD Rebate Rewards Visa - whether I use either or not
The Rewards Visa Gold includes some insurance on rental cars, which is handy since I choose not to own a car and so rent cars occasionally

2009-12-28T19:57:00.000-08:00

First reaction to Avatar - warning - probably some spoilers in here

I didn't wanna have too high expectations - I expected amazing visuals but extremely cheesy plot. Like FernGully. So cheesy Melissa refused to see it with me. I went with folks from Artefactual for,

Sigourney Weaver - what a great name
floating islands
vertical takeoff military aircraft
telepresence and in vitro alien/human genetic hybrids
acrobatic aliens resembling psychedelic tribal rave
- all in 3D

^ awesome.

I was positively surprised all around - first, it's *gorgeous*. It's so beautiful that I actually unconsciously felt sad and angry to witness parts of Pandora destroyed

And I fully bought into the plot. It's interesting, it has good arc - good symmetry - no, "BTW, what ever happened to person X?" or, "Well that was non sequitur." And it has satisfying "Hollywood" - I found myself again unconsciously needing the Pandorans to win and the lovers to be together in the end

I struggled with the paradox of watching an expensive anti-corporate film, or a violent pro-diplomacy film

Interesting how the theme of wrong-thinking corporations runs throughout James Cameron's films - my friend Jeremy pointed this out. Weyland-Yutani, Skynet (Jeremy pointed out that one could argue even Titanic questions industrial progress - not that Jeremy himself believes this argument : )

I wondered after the film about Jake's guilt over gathering intelligence for the military on the Pandorans. I mean, the closest he comes to admitting his complicity - even to his soul mate - is when he says he knew all along that the military would come to destroy the Pandorans' home - he doesn't say that he personally provided intelligence how best to do it

Jake is a depth-less hero because all the challenges he overcomes are external - happen *to* him - paralysis, death of his brother. He doesn't have any character flaws. He doesn't confront his guilt and overcome it. Jake's personal responsibility in the occupation of Pandora generally and the destruction of "home tree" specifically is totally missing. Like the theme of individual responsibility for collective exploitation of the environment and other people. Theme of change starting with the "man in the mirror" (also recently saw This Is It)

There was a *lot* going on in this film,

floating islands and the properties of unobtainium
tree brain
disability
occupation

- but given the length already, I understand some stuff had to get glossed over

I met Reno, one of Melissa's favorite yoga teachers, on his way out of seeing Avatar as we were on the way in. I know Reno is powerfully into "mother" - the source of life - which is a powerful theme in the film. So I'm totally interested to hear what Reno found in the film and see if he brings it up at a yoga class - maybe uses it as a metaphor. Also the theme of connection between all living things - temporarily borrowing eternal energy - is central to the little yoga that I practiced, and was also a powerful theme in the film. Interesting to see so much spirituality in Hollywood. Besides Scientology and The Passion.

Also met David Suzuki on our way out of seeing the film - and I also wonder what he thought of it - but looked like he already attracted a crowd

I wonder if this film resounded with me especially as a cyclist? I felt real frustration at the un-powerful, horse/pterodactyl? riding Pandorans suffering injustice at the hands of ignorant machine driving military industrialists. It actually reminded me of the frustration I often feel biking in traffic. Hard to explain why I identify biking with Pandorans instead of machine driving. There's something "hybrid" about biking (human/bike, not mountain/road : ), unlike how cars isolate the operator?

Also there was the usual parade of car advertising pre-show (and pre-pre-show) I still haven't been explained why public money bailed out private car makers? I mean, if you have advertising overwhelming every part of culture and you're still not profitable, then - ah - Why are you still here? Whereas free software is slowly succeeding without advertising, despite anti-competitive opposition, with volunteers and experimental revenue models. Can we please have some public money for free software? </rant>

Also there's the theme of military occupation for the purpose of natural resources, which reminded me of the middle east, oil, and private cars

Final thoughts - I wonder how much the making of Avatar was like the story in Avatar? I mean I think the way they animate "human" computer graphic characters is with motion capture? Like an actor controlling an... avatar?

How much is behind the location name, "Pandora"? Pandora's box - the secret of fire (symbolic?) in exchange for a box of evil?

2009-12-21T10:49:00.000-08:00

Although we use Google project hosting, we also run our own master Subversion repository, for greater control over the repository than project hosting supports, e.g. installing this pre-commit hook

Eventually we'd like to authenticate to our master repository with GnuPG keys and GnuTLS - until then we authenticate with LDAP,


  AuthType Basic
  AuthName Example
  AuthBasicProvider ldap
  AuthLDAPURL ldap://ldap.example.com/ou=People,dc=example,dc=com
  AuthzLDAPAuthoritative off

  <LimitExcept OPTIONS>
    Allow from localhost
    Deny from all
    Require valid-user
    Satisfy any
  </LimitExcept>

As our projects grow, maintaining separate LDAP accounts for contributors becomes tedious - we'd like to allow contributors to authenticate to our master repository with their googlecode.com account, if they trust our server

Checking whether a googlecode.com account is a contributor to a project hosting project is easy - one way is to request https://example.googlecode.com/svn/ with the provided username and password. If the response status code is 200 OK then the username and password match and the account is a contributor to the project

There's a range of possible ways to get Apache to perform this check to control access to our master repository, from implementing a custom Apache module to configuring an existing module which supports this check, or supports calling a helper which does

Because making a request and checking the response status code is an unusual case, I guess any existing module which supports it will be very general, and therefore need substantial configuration. Another possibility, in between implementing a custom module and substantial configuration of an existing module, is using a module with a scripting language. A familiar scripting language is a good tool for doing substantial configuration, and scripts are easier to maintain than a custom module

My current favorite and most familiar scripting language is JavaScript, but I haven't found a mature JavaScript Apache module yet, so I chose mod_python

The PythonAuthenHandler directive lets you implement an Apache authentication handler in Python,


  PythonAuthenHandler /home/example/google/authenhandler

I struggled a little to specify the handler before I leaned it "... can be a full module name (package dot notation is accepted) or an actual path to a module code file." Also I wonder why the directive isn't called "PythonAuthnHandler" - all the Apache documentation and AFAICT all the Apache APIs use the terms "authn" and "authz", and don't use the term "authen" - maybe it's a Pythonism?

Here's the short handler - it uses Python's httplib

This is almost good enough to check googlecode.com accounts to control access to our master repository - implementing a mod_auth_basic provider in Python might be even nicer? but I don't think mod_python supports this...

The first trick to get Apache to use this handler was setting AuthBasicAuthoritative off, as mentioned briefly in the mod_python documentation. AFAICT, without AuthBasicAuthoritative off, authentication handlers are ignored except mod_auth_basic providers

The second trick was that setting AuthBasicAuthoritative off in one directory context prevents inheriting other AuthBasic* directives from other directory contexts, e.g. AuthBasicProvider - I once had a similar experience with Rewrite* directives

This means that, given the following configuration, googlecode.com accounts can access /svn/example, but LDAP accounts can't,


<Location />

  AuthType Basic
  AuthName Example
  AuthBasicProvider ldap
  AuthLDAPURL ldap://ldap.example.com/ou=People,dc=example,dc=com
  AuthzLDAPAuthoritative off

  <LimitExcept OPTIONS>
    Allow from localhost
    Deny from all
    Require valid-user
    Satisfy any
  </LimitExcept>

</Location>

<Location /svn/example>

  AuthBasicAuthoritative off
  PythonAuthenHandler /home/example/google/authenhandler

</Location>

The solution is either to copy AuthBasicProvider ldap to the /svn/example directory context, or set AuthBasicAuthoritative off in the / directory context,


<Location />

  AuthType Basic
  AuthName Example
  AuthBasicAuthoritative off
  AuthBasicProvider ldap
  AuthLDAPURL ldap://ldap.example.com/ou=People,dc=example,dc=com
  AuthzLDAPAuthoritative off

  <LimitExcept OPTIONS>
    Allow from localhost
    Deny from all
    Require valid-user
    Satisfy any
  </LimitExcept>

</Location>

<Location /svn/example>

  PythonAuthenHandler /home/example/google/authenhandler

</Location>

Now contributors can authenticate to our master repository with either LDAP or googlecode.com accounts!

2009-12-15T10:07:00.000-08:00

I wish LDAP (actually many protocols) included built-in support for version control. I think version control and real-time collaborative editing are two general areas where we can continue making useful contributions to software systems. These areas are related by having consistency problems

Meanwhile daily dumps of LDAP directories to, e.g. Subversion, are useful. It's an incremental backup. LDIF is reasonably amenable to unified diff, which can give some idea of what, if anything, changed in the directory

We use OpenLDAP, so we use cron and slapcat to make daily dumps to a Subversion repository. We don't want to run the daily cron job as superuser unless absolutely necessary - we prefer to run it as an ordinary user, e.g. "administrator"

The Debian and Ubuntu slapd packages create both an "openldap" user and an "openldap" group. In addition to superuser, the "openldap" user can run slapcat. Other ordinary users can't run slapcat - they can't read/write some of the necessary files/directories. Unfortunately "openldap" group members can't run slapcat either : (

To run slapcat,

/var/lib/ldap must be writable, by default it's 700
/var/lib/ldap/alock must be writable, by default it's 600
/var/lib/ldap/__db.* must be writable, by default it's 600
/var/lib/ldap/dn2id.bdb must be readable, by default it's 600
/var/lib/ldap/id2entry.bdb must be readable, by default it's 600

To run slapcat as the "administrator" user I added "administrator" to the "openldap" group and then used,


$ sudo dpkg-statoverride --update --add openldap openldap 770 /var/lib/ldap

- to make /var/lib/ldap writable by "openldap" group members

I don't think dpkg-statoverride supports globs like /var/lib/ldap/__db.*, so I used,


$ sudo sh -c 'chmod 660 /var/lib/ldap/{alock,__db.*}'
$ sudo chmod 640 /var/lib/ldap/{dn2id.bdb,id2entry.bdb}

I opened a bug against the Debian slapd package a year ago, wishing for "openldap" group members to be able to run slapcat "out of the box", but it hasn't seen any activity : (

Perhaps instead of trying to run slapcat as the "administrator" user, I should use a different tool instead? e.g. is slapcat safe for hot backups? For big directories, instead of one dump file would a directory structure like /etc/ldap/slapd.d be helpful?

2009-12-11T09:58:00.000-08:00

What's the best way to set the umask for cron jobs?

As described here, we set the umask in ~/.bashrc,


# ~/.bashrc: executed by bash(1) for non-login shells.
umask 002

- and include .bashrc from .bash_profile,


# ~/.bash_profile: executed by bash(1) for login shells.

# include .bashrc if it exists
if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi

This is effective in most cases, including login and non-login shells, but not for cron jobs : (

So far I tried the following experiments in our crontab,

```
@daily bash -c '...'
```
```
@daily umask 002 && ...
```

The first experiment doesn't work - the cron job is still executed with umask 022. I guess this is because Bash reads neither .bashrc nor .bash_profile for non-interactive shells. I haven't found any configuration Bash does read for non-interactive shells

The second experiment does work - the cron job is executed with umask 002. The drawback is that the umask is configured in two places, .bashrc and our crontab. I'd prefer there were no opportunity for inconsistency

Is there a better way to set the umask for cron jobs?

This came up because we want new files and directories to be created with permissions 664 and 775, respectively, so all users in our group can edit them, and we noticed a bunch of files and directories that only the owner could edit : P

Some investigation revealed three causes,

tar
scp
cron

We frequently use tar to copy directories,


$ tar cz foo | ssh bar@example.com cd foo/bar \&\& tar xz

The tar --no-same-permissions option applies the umask when extracting permissions from the archive and it's the default when tar's run by an ordinary user - not superuser

However it applies the umask to the permissions from the archive, so it never adds to the file's original permissions, e.g. if the umask is 002, then an extracted file which originally had permissions 666 should have permissions 664, but an extracted file which originally had permissions 644 should still have permissions 644 : P

Maybe tar should support an option to ignore permissions from the archive, but this would also affect whether files are executable or not - not just whether all users in our group can edit them. Maybe it's a failing of the file mode to lump the executable property together with permissions?

When necessary, we work around this with,


$ find foo -perm 644 -exec chmod 664 {} \; -o -perm 755 -exec chmod 775 {} \;

The same is true of scp - it respects the umask set in ~/.bashrc, but applies the umask to the file's original permissions

2009-12-07T12:16:00.000-08:00

PEAR is the PHP package manager, similar to Perl's CPAN. We run a PEAR channel for the Qubit project to post releases and manage plugins

Formerly we used Ciara_PEAR_Server to manage this channel, but it was quite hard to set up

Then last week I heard Fabien Potencier has released Pirum - a simple channel server

The responses from a channel server are static - at least for a simple channel they only change when a release is added. These responses consist basically of some XML and tarballs

Both Chiara_PEAR_Server and Pirum manage this XML - a difference is that Chiara_PEAR_Server uses a web interface while Pirum is command line

Chiara_PEAR_Server also uses a database, e.g. MySQL, for accounts, etc. whereas Pirum uses only the XML files

Unlike Chiara_PEAR_Server, Pirum is just one PHP file

Interestingly, although Chiara_PEAR_Server uses a web interface to manage the channel, it doesn't provide any formatted output for end users (just XML). Pirum does provide some fairly nice end user instructions

So I decided to give Pirum a try. It was quite easy,


$ git clone git://github.com/fabpot/Pirum.git
$ php Pirum/pirum build pear.qubit-toolkit.org
$

To add a new release,


$ php Pirum/pirum add pear.qubit-toolkit.org icaatom-1.0.3.tgz
$

Now Qubit is listed among the projects using Pirum : )

It's interesting that pyrus, the PEAR 2 installer, has built in support for managing a channel

2009-11-27T10:05:00.000-08:00

Initially I had some email addresses, e.g. username@example.com, that need to get forwarded to other addresses, depending on the username, e.g. jbates@campcoop.com -> jack.bates@gmail.com

The usernames and forwarding addresses are available from LDAP directories, so I used Postfix's LDAP map support

I put the following in a new file, /etc/postfix/ldap-mail.cf,


search_base = ou=People,dc=example,dc=com
query_filter = uid=%u
result_attribute = mail
version = 3

- and added "ldap:/etc/postfix/ldap-mail.cf" to the alias_maps parameter in /etc/postfix/main.cf,


[...]
alias_maps = hash:/etc/aliases, ldap:/etc/postfix/ldap-mail.cf
[...]

I also had some messages that were getting sent from username@example.com and want them to get sent from the other address instead, e.g. I had Subversion commit messages getting sent from jablko@artefactual.com and want them to get sent from jack.bates@gmail.com

So I added a canonical_maps parameter to /etc/postfix/main.cf,


[...]
canonical_maps = ldap:/etc/postfix/ldap-mail.cf
[...]

Next I had some email addresses that need to get forwarded to one address, but I want messages to get sent from a second, different address, e.g. I need messages addressed to jbates@campcoop.com to get forwarded to jbates@imap.campcoop.com, but want messages to get sent from jack.bates@gmail.com

To accomplish this I put both addresses in the LDAP directory. I found a "mailRoutingAddress" attribute in the misc.schema distributed with OpenLDAP. "inetLocalMailRecipient" is an auxiliary object class in misc.schema and LDAP entries with this object class may have a mailRoutingAddress attribute. So I added misc.schema to our OpenLDAP configuration and used a mail attribute for the address to send messages from (e.g. jack.bates@gmail.com) and a mailRoutingAddress attribute for the address to forward messages to (e.g. jbates@imap.campcoop.com)

Then I put the following in another new file, /etc/postfix/ldap-mailRoutingAddress.cf,


search_base = ou=People,dc=example,dc=com
query_filter = uid=%u
result_attribute = mailRoutingAddress
version = 3

- and updated alias_maps,


[...]
alias_maps = hash:/etc/aliases, ldap:/etc/postfix/ldap-mailRoutingAddress.cf
[...]

Although this works great, it meant adding the inetLocalMailRecipient object class and a mailRoutingAddress attribute to LDAP entries where, in the majority of cases, the mail and mailRoutingAddress attributes have identical values : P

Then Wietse explained how to configure maps to use one LDAP attribute if it exists (mailRoutingAddress), and fall back on another if not (mail)

So I updated alias_maps again,


[...]
alias_maps = hash:/etc/aliases, ldap:/etc/postfix/ldap-mailRoutingAddress.cf, ldap:/etc/postfix/ldap-mail.cf
[...]

This way, usernames with both mail and mailRoutingAddress attributes get messages sent from the mail attribute address and messages forwarded to the mailRoutingAddress - as before - but usernames with just a mail attribute get messages both sent from and forwarded to this address

Usernames with just a mailRoutingAddress get messages forwarded to this address, but messages won't get sent from this address because canonical_maps isn't configured to use this attribute. So messages will continue to get sent from username@example.com

I have exactly one case, "administrator", where I want messages to get sent from another address, but not forwarded to another address - I need them to be delivered locally instead. In this case I want messages to get sent from administrator@artefactual.com, and administrator@artefactual.com isn't a local destination. Before updating alias_maps with both mail and mailRoutingAddress maps, this username had just a mail attribute. Because it had no mailRoutingAddress, messages weren't forwarded - they were delivered locally. After updating alias_maps however, messages were forwarded to the mail attribute address : (

I inquired on posfix-users what value I could give the mailRoutingAddress attribute such that messages *wouldn't* get forwarded, but would still stop alias_maps lookups before the mail attribute map. Currently I use the value "administrator", which has this effect

I was pretty pleased with this configuration - until I realized what happened to messages addressed to "username". If the username has both mail and mailRoutingAddress attributes, I need these messages to get forwarded to the mailRoutingAddress attribute address - but instead they were forwarded to the mail attribute address : (

This is because alias_maps only applies to messages addressed to a local destination. canonical_maps is applied first, and if the mail attribute address isn't a local destination, then alias_maps is never applied

I discovered this because I need output from the administrator user's crontab to be delivered locally, but it was forwarded to administrator@artefactual.com instead

To correct this I first went down the wrong track - I tried using a recipient_canonical_maps parameter instead of alias_maps. I thought this was a cute solution because Postfix looks up recipient addresses in both recipient_canonical_maps and canonical_maps, in that order. So I didn't need to add the mail attribute map to recipient_canonical_maps for usernames without a mailRoutingAddress, it's used anyway


[...]
canonical_maps = ldap:/etc/postfix/ldap-mail.cf
recipient_canonical_maps = ldap:/etc/postfix/ldap-mailRoutingAddress.cf
[...]

Later I changed my mind - I now use alias_maps as before,


[...]
alias_maps = hash:/etc/aliases, ldap:/etc/postfix/ldap-mailRoutingAddress.cf, ldap:/etc/postfix/ldap-mail.cf
[...]

- and a sender_canonical_maps parameter instead of canonical_maps

Postfix doesn't look up recipient addresses in sender_canonical_maps, so this avoids the problem of applying canonical_maps before alias_maps and then never applying alias_maps if the mail attribute address isn't a local destination

The myorigin parameter must be a local destination for this to work - at artefactual.com it wasn't. Without canonical_maps or recipient_canonical_maps, messages addressed to "username" get forwarded to username@$myorigin and if myorigin isn't a local destination, then alias_maps is never applied

I had configured myorigin with the domain name, e.g. example.com, because in most cases messages addressed to "username" should get sent from and forwarded to username@example.com - but at artefactual.com it isn't a local destination

To correct this I changed myorigin from the domain name to the host name, and to keep messages getting sent from and forwarded to username@artefactual.com, I added mail attributes to usernames

2009-11-14T10:55:00.000-08:00

Because Trac is an integrated wiki, issue tracker, and version control browser, it offers a nice feature where patterns like "issue 123" in wiki pages, etc. are autolinked to the referenced issue page. The link title gets copied from the issue title, which browsers commonly represent as a tooltip, and the link gets line-through text-decoration when the issue is closed. There's an example on this page.

I'd like this feature for one of my projects, but without having to give up choice among wiki software, issue trackers, and version control browsers, for one monolithic package. For example we choose to use MediaWiki and Google project hosting.

Server side integration of open source applications is generally feasible, especially if they're written in the same programming environment and can call one another's code directly. But in our case, Google project hosting isn't even running on a server we can access.

Getting the needed issue data from project hosting isn't hard, however. The issue page for "issue 123" is http://code.google.com/p/qubit-toolkit/issues/detail?id=123, and I found this CSS selector initially sufficient for getting the issue title, "#issueheader span", and this one for getting whether the issue is closed, "#issuemeta tr:has(th:contains(Status)) td:contains(Fixed)".

So AFAICT any implementation of this issue autolinking feature with Google project hosting involves making HTTP requests and parsing HTML - happily something well supported by modern programming environments. So server side implementation of this feature is still feasible - before responding with a wiki page, our server checks for patterns like "issue 123" and possibly makes some requests to Google project hosting.

But why should our server be making HTTP requests and parsing HTML before responding to a *browser*? Modern browsers can do it client side - making HTTP requests and parsing HTML is kinda what they do, after all : ) The other problem with a server side approach is in the words "*before* responding" - if the requests to project hosting are slow, or a page needs lots of them, then the whole page is held up waiting for some tooltips and line-through text-decoration.

So I decided to try replicating this issue autolinking feature by integrating applications on the client side instead of the server side - I think this is called a "mash up"?

I made two JavaScripts, using jQuery - one to turn patterns like "issue 123" into links, and one to add data from project hosting to these links

I could have turned patterns into links server side, and added data from project hosting client side, but this is a client side experiment. A possible disadvantage (or advantage?) is that links could be created in not just the wiki text, but the surrounding MediaWiki layout too. A possible advantage is that this JavaScript could be easily ported to other applications - I'm now trying to add it to our ViewVC.

I initially based autolink.js on this Greasemonkey script. It does regular expression replacements on text nodes, except descendants of <textarea>s. The special characters '&', '<', and '>' aren't escaped in text nodes' .nodeValue, so the script handles this. It supports various patterns,

issue 123
issues 123 and 321
issue 123 and 321
NOT issues 123
issues 123, 321, and 456

The trick to google.js is getting around the same origin policy. Some options include,

Serve the JavaScript from Google project hosting?
Run a proxy and serve the JavaScript from the same domain
Run a JSONP proxy
Access-Control-Allow-Origin: * header

Google project hosting does support downloads, but they're served from a different domain - perhaps for reasons of the same origin policy? e.g. qubit-toolkit.googlecode.com vs. code.google.com

Running a proxy on our server feels like failure of this client side experiment

Andy McKay is defying the same origin policy with JSONP

I requested project hosting add the Access-Control-Allow-Origin: * header to issue pages, which was merged into an issue to add a project hosting issue API. That API was recently released, but it doesn't add the Access-Control-Allow-Origin: * header : P

So I wrote an "allow origin proxy" for Google App Engine. This proxy should add the Access-Control-Allow-Origin: * header to responses from any URL, e.g. http://alloworiginproxy.appspot.com/http://code.google.com/p/qubit-toolkit/issues/detail?id=123

This has the nice property of using Google Code's own infrastructure - requests from App Engine to project hosting are potentially more efficient than from our server to project hosting. I think it's the next best thing to project hosting adding the Access-Control-Allow-Origin: * header directly.

AFAIK the Access-Control-Allow-Origin: * header is supported by Firefox 3.5 or later. Using it for this issue autolinking feature isn't unreasonable because most of the folks for whom it's useful run Firefox 3.5 or later - and it degrades smoothly. Without Firefox 3.5 or later, links are created without tooltip or line-through text-decoration. Without JavaScript, patterns like "issue 123" are unchanged.

Here are the client side issue autolinking and App Engine "allow origin proxy" in action

2009-11-09T10:02:00.000-08:00

Celebrated the Ubuntu 9.10 release with Cakebuntu here at the office

When the cake was still intact at lunchtime, we decided to take it with us to the Tamarind Hill restaurant, where they brought it to our table singing. Terima kasih!

Ducky Sherwood traveled all the way from Vancouver to share in the festivities - and gave an awesome demo of her Canadian Stimulus spending project. Jeremy Holman came to celebrate open source operating systems - and eat cake : )

In preparation, I considered,

Baking and decorating a Cakebuntu
Ordering a Cakebuntu from a bakery
Getting one from a grocery store

I settled on getting a Cakebuntu from Save-On-Foods because it was easiest. But I struggled - and eventually failed - to get the Ubuntu logo drawn on the cake with icing.

Both Safeway and Save-On strongly resisted drawing anything on the cake by hand - despite my insistence that they couldn't go wrong and that any red, orange, and yellow circle-ish decoration would do. Apparently they have a cake printer device for transferring images to cakes, so customers can't complain that the results aren't exactly what they wanted.

But I'm lukewarm on the impersonal output of the cake printer, and thought I finally had a baker at Save-On convinced to risk free hand Ubuntu decoration. I also thought when they wrote down the cake text - "Happy Ubuntu 9.10 Release" - that "H." was bakers' shorthand : P

So I was wrong twice - an Ubuntu sticker was reproduced on the cake with the cake printer - but it was unmistakably Cakebuntu. And very tasty!

For promotion, I invited some friends of open source here in New Westminster, and posted an open invitation on Twitter

I heard feedback that there should be more promotion next time, like some New Westminster blogs,

Also that we should have a machine running the celebrated release, for playing with

I wanna sample another bakery's Cakebuntu

Thanks for celebrating - and for the feedback! Ubuntu 10.04 is scheduled to be released on Thursday, April 29th.

2009-10-05T13:11:00.000-07:00

I started writing a script to make a web page from some information in our Subversion repository. I generally prefer working with Subversion or WebDAV requests and responses to working with Subversion language bindings, or executing the svn command and parsing it's output - yuk! I guess I prefer it because WebDAV is a widely used (if flawed) standard, and because pretty much every programming environment supports making HTTP requests and parsing XML responses

Previously I've handled these requests and responses server side, but of course the modern client side is well equipped to make HTTP requests and parse XML responses - that's what it's made for after all : )

In case the web page is hosted on a different domain than our Subversion repository, I investigated cross domain WebDAV requests, https://developer.mozilla.org/En/HTTP_access_control

I since abandoned cross domain WebDAV requests because I hope to instead generate the web page with XSLT and its document() function. Unfortunately WebDAV metadata isn't available at a URL, instead it must be requested with a special HTTP request method, PROPFIND. This is one flaw in WebDAV, it's not fully RESTful

Before moving on, I thought I'd post the progress I made on cross domain WebDAV requests. Here's some JavaScript to get a PROPFIND response from our Subversion server - this example uses the jQuery library, it does nothing with the response,


$(function ()
  {
    jQuery.ajax({

      type: 'PROPFIND',
      url: 'http://example.com/svn/',

      beforeSend: function (xhr)
        {
          xhr.setRequestHeader('Depth', 1);
          xhr.withCredentials = true;
        },

      success: function (data)
        {
          $('tbody').append('<tr><td>foo</td></tr>');
        } });
  });

On the server side I had to add access control headers to preflight responses,


$ cat /etc/apache2/conf.d/accesscontrol
Header set Access-Control-Allow-Credentials true
Header set Access-Control-Allow-Headers Depth
Header set Access-Control-Allow-Methods PROPFIND
Header set Access-Control-Allow-Origin http://localhost
$

Something to note is that I couldn't use wild carding in the Access-Control-Allow-Origin header with credentialed requests, https://developer.mozilla.org/En/HTTP_access_control#Requests_with_credentials

Finally, our Subversion repository requires a username and password for all requests - I had to turn this off for preflight requests,


$ cat /etc/apache2/conf.d/auth
<Location />

  AuthType Basic
  AuthName "Example"
  AuthBasicProvider ldap
  AuthLDAPURL ldap://example.com/ou=People,dc=example,dc=com
  AuthzLDAPAuthoritative off

  <LimitExcept OPTIONS>
    Require valid-user
  </LimitExcept>

</Location>
$

2009-09-10T10:57:00.000-07:00

Really enjoyed a presentation last night at van.pm by Jeremy Stashewsky on coroutines in Perl, and another presentation last night by Andrea Reimer on the Vancouver open data, open standards, open source motion

After Jeremy's presentation, I think coroutines would be especially useful in JavaScript (my current <3 programming language) because so much event style programming is done in JavaScript. It's currently done in callback style, but coroutine style could be much nicer. A quick Google search turned up, http://openajax.org/runtime/wiki/JavaScript_Pause_Release, among others. I'm excited to look deeper into JavaScript coroutines : )

After Andrea's presentation, I wish all City of Vancouver web pages had a link to their source code, i.e. the code which output the page. Maybe this is a good idea, maybe not. Would it be useful for web pages all over the internet to have links to their source code? The open source project I contribute to might include a link to its source code repository by default, but someone who installed it might customize the link to point at their own source code repository, where they manage the current version and any customizations.

I checked out the <link rel="..."> tag to see if something already exists. Also there's an IANA registry of link relations, and I recall Mark Nottingham collecting a registry of link relations, but so far nothing for source code.

Maybe a standard link relation should be created? Or a microformat? The first thing that comes to mind is one might want to make a distinction between the source code which output the page and the source data which was processed to produce the page? Maybe that resolution isn't initially necessary, and a general link to the source, however you define it, would give the biggest gains for the least effort?

2009-08-15T12:28:00.000-07:00

This is a followup to my last post about marking up web pages with SVG for automating screenshots for documentation. The application we're documenting uses content type 'text/html'. I figured I could convert it from an HTML DOM to an XML DOM with JavaScript, but got stuck trying to associate the XML DOM with a window instance, for use with drawWindow()

Then I found an example of inline SVG in an HTML DOM!

From this I learned that,

Sticking SVG markup in 'text/html' source doesn't work


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <meta content="HTML Tidy for Linux/x86 (vers 7 December 2008), see www.w3.org" name="generator" />

  <title></title>
</head>

<body>
  <svg xmlns="http://www.w3.org/2000/svg">
    <circle cx="64" cy="64" fill="red" r="16"/>
  </svg>
</body>
</html>

Using innerHTML to insert SVG markup doesn't work either


window.onload = function ()
  {
    document.getElementsByTagName('body')[0].innerHTML = [
      '<svg xmlns="http://www.w3.org/2000/svg">',
        '<circle cx="64" cy="64" fill="red" r="16"/>',
      '</svg>'].join('\n');
  }

- but using createElementNS() to build elements programmatically does work!


window.onload = function ()
  {
    var svg = document.createElementNS('http://www.w3.org/2000/svg', 'svg');

    var circle = document.createElementNS('http://www.w3.org/2000/svg', 'circle');
    circle.setAttribute('cx', 64);
    circle.setAttribute('cy', 64);
    circle.setAttribute('fill', 'red');
    circle.setAttribute('r', 16);

    svg.appendChild(circle);

    document.getElementsByTagName('body')[0].appendChild(svg);
  }

What's more, because I like working directly with familiar markup, I tried inserting elements parsed with parseFromString(..., 'application/xml'), and it worked!


window.onload = function ()
  {
    document.getElementsByTagName('body')[0].appendChild(new DOMParser().parseFromString([
      '<svg xmlns="http://www.w3.org/2000/svg">',
        '<circle cx="64" cy="64" fill="red" r="16"/>',
      '</svg>'].join('\n'), 'application/xml').documentElement);
  }

Here's an updated hack to make jQuery support E4X, and the above example implemented with jQuery and E4X,


var clean = jQuery.clean;

jQuery.clean = function (elems, context, fragment)
  {
    jQuery.each(elems, function (i)
      {
        if (this instanceof XML)
        {
          elems[i] = new DOMParser().parseFromString(this.toXMLString(), 'application/xml').documentElement;
        }
      });

    return clean(elems, context, fragment);
  };

$(function ()
  {
    $(<svg xmlns="http://www.w3.org/2000/svg">
        <circle cx="64" cy="64" fill="red" r="16"/>
      </svg>).appendTo('body');
  });

Because they use parseFromString(), I think both this and the hack from my last post work whether the content type is 'text/html' or 'application/xml', which makes marking up pages with SVG convenient : )

A couple bumps marking up pages from a Selenium action,

Stock jQuery appendTo() doesn't support a 'context' argument, so $(<svg/>).appendTo('body', window.document); marked up the test runner instead of the AUT. Here's a patch to fix that,


--- a/jquery-nightly.js 2009-08-15 13:13:01.000000000 -0700
+++ b/jquery-nightly.js 2009-08-15 13:12:27.000000000 -0700
@@ -2358,8 +2358,8 @@
        insertAfter: "after",
        replaceAll: "replaceWith"
 }, function(name, original){
-       jQuery.fn[ name ] = function( selector ) {
-               var ret = [], insert = jQuery( selector );
+       jQuery.fn[ name ] = function( selector, context ) {
+               var ret = [], insert = jQuery( selector, context );
 
                for ( var i = 0, l = insert.length; i < l; i++ ) {
                        var elems = (i > 0 ? this.clone(true) : this).get();

Manipulating CSS on the <svg/> element with jQuery works fine until tried from a Selenium action. Then the CSS just doesn't change,


        $(<svg xmlns="http://www.w3.org/2000/svg">
                <circle cx="64" cy="64" fill="red" r="16"/>
            </svg>)
            .css('left', 0)
            .css('position', 'absolute')
            .css('top', 0)
            .appendTo('body', window.document);

- but using a style attribute,


        $(<svg style="left: 0; position: absolute; top: 0" xmlns="http://www.w3.org/2000/svg">
                <circle cx="64" cy="64" fill="red" r="8"/>
            </svg>).appendTo('body', window.document);

- or even calling parseFromString() upfront and passing jQuery the DOM element instead of E4X,


        $(new DOMParser().parseFromString(<svg xmlns="http://www.w3.org/2000/svg">
                <circle cx="64" cy="64" fill="red" r="16"/>
            </svg>.toXMLString(), 'application/xml').documentElement).css('left', 0)
            .css('position', 'absolute')
            .css('top', 0)
            .appendTo('body', window.document);

- work. This is way strange because it's exactly what the hack does - calls parseFromString(). Any ideas why calling parseFromString() outside jQuery works, but not inside? Any ideas why the hack works until tried from a Selenium action?

2009-08-06T16:44:00.000-07:00

I want to markup a web page with maybe circles and arrows for documentation, and JavaScript and SVG seemed like the right tools. I looked to jQuery and found a jQuery SVG plugin, but at first glance it wraps SVG markup in an API, which turned me off. My current thinking is anti wrapper - provide access to underlying markup or syntax wherever possible (my experience with Doctrine DQL vs. Propel criteria inform this thinking).


$(<circle cx="4" cy="4" fill="red" r="3"/>).appendTo('svg');

^ turns me on : )

Before beginning, it occurs to me that I may be doing something wrong, trying to get DOM manipulation with a combination of jQuery and E4X working. I suspect there's nice E4X syntax for accomplishing the same thing without jQuery. But I'm new to E4X, and comfortable with jQuery.

The first thing I learned is that inline SVG - mixing SVG tags with HTML tags - only works when the content type is 'application/xml'. Here are two identical pages, one with content type 'text/html' and one with content type 'application/xml'. The blue circle only appears on the 'application/xml' page.

Next, jQuery doesn't normally support E4X, but it does support strings of markup, e.g. $('<div>foo</div>').appendTo('body'); Unfortunately strings of markup stop working when the content type is 'application/xml' : P I guess this is because 'application/xml' pages get parsed into an XML DOM instead of an HTML DOM. jQuery supports strings of markup with innerHTML(), which I guess XML DOM doesn't support.

One way to get strings of markup into an XML DOM is with DOMParser(), so here's a naive patch for jQuery to make strings of markup work even if the content type is 'application/xml'. jQuery uses innerHTML() pretty extensively - maybe for performance - so I'm not sure how this patch will be received.

Coincidentally, because E4X and DOM are currently separate beasts, toXMLString() and DOMParser() are the only way I know to get E4X into an XML DOM. So here's a hack to make unmodified jQuery support E4X, whether the content type is 'application/xml' or not,


$ = function ()
  {
    if (arguments[0] instanceof XML)
    {
      arguments[0] = new DOMParser().parseFromString(arguments[0].toXMLString(), document.contentType).documentElement;
    }

    return jQuery.apply(this, arguments);
  };

The last thing I learned is to beware of namespaces. With the above hack, $(<circle cx="4" cy="4" fill="red" r="3"/>).appendTo('svg'); successfully adds a circle element to the DOM! but it doesn't display : ( This is because the circle element belongs to the empty namespace - it needs to belong to the SVG namespace. Once again I found the solution in the Mozilla Developer Center,


default xml namespace = 'http://www.w3.org/2000/svg';

So here's a working example. The blue circle is inline SVG in the XML file, the red circle is added with $(<circle cx="4" cy="4" fill="red" r="3"/>).appendTo('svg');

2009-08-02T11:20:00.000-07:00

Added an extension to the Asterisk dialplan here at the Coop this morning, to access voicemail for the current channel

We've been hosting voicemail mailboxes with Asterisk here at the Coop for several months, but until recently it was super awkward to check voicemail with a phone. We had no FXS interfaces, I had the only VoIP handset, and no one bothered with soft phones. We could reach Asterisk by calling the Coop from another line, but Asterisk only answered after four rings and someone might answer the phone in the meantime. Consequently folks really only received voicemail via email.

Now we've been given two AudioCodes MP-124 FXS gateways we'll be able to reach Asterisk from any phone at the Coop! So some folks are excited to check voicemail with a phone.

The default Asterisk dialplan, which we've mostly stuck to, includes the VoicemailMain() app at extension "8500". VoicemailMain() prompts us for a mailbox and password unless we specify a mailbox as an argument. We want to skip the mailbox prompt by specifying a mailbox based on the current channel.

First we need to associate mailboxes with channels. Depending how we name our channels, this info might be available from our LDAP server, which might be accessible with Asterisk 1.6's LDAP support. Alternatively I found the SIPPEER() function which can return the configured mailbox for a SIP peername. Configure mailboxes for SIP peernames in sip.conf - this is also needed to turn on the message waiting indicator when new voicemails are waiting in associated mailboxes.

SIPPEER() works for us because the gateways connect to Asterisk with SIP. But we need the SIP peername for the current channel. For this I found SIPCHANINFO(). So "VoicemailMain(${SIPPEER(${SIPCHANINFO(peername)}|mailbox)})" accesses voicemail for the current channel.

I added this to the dialplan with extension "*98" because the local telephone company accesses voicemail with "*98" and it might be familiar to some folks. I added it to the "local" context so it's only available to phones here at the Coop and not from another line,


@@ -386,6 +386,8 @@
 ;
 ; eswitch => IAX2/context@${CURSERVER}
 
+exten => *98,1,VoicemailMain(${SIPPEER(${SIPCHANINFO(peername)}|mailbox)})
+
 [macro-trunkdial]
 ;
 ; Standard trunk dial macro (hangs up on a dialstatus that should

A final step, the gateways wouldn't dial "*98" after the Interdigit Timeout, despite the Digit Mapping Rules including "x.T". Adding "*98" to the Digit Mapping Rules solved this problem.

Now dialing "*98" from a phone here at the Coop accesses the mailbox associated with that phone. VoicemailMain() still prompts us for the mailbox password but supports an argument to skip the password prompt. Not sure if folks want to skip the password prompt? or if the consistency of the password prompt when accessing mailboxes from phones here at the Coop and from another line is desirable? or if it's possible to configure whether to skip the password prompt on a per mailbox basis?

Speaking of consistency, maybe it's better to encourage folks to use extension "8500" instead of "*98" because it's consistent with accessing mailboxes from another line and encourages folks to remember their extension?

2009-07-29T07:24:00.000-07:00

Melissa and I are just back from a week's hiking trip in the Southern Chilcotin and it was awesome : ) The weather was great and I think our timing was great because the meadows were full of blooming wildflowers.

Melissa is busy with field work in the summer and time off is precious so I wanted to make it count. Other trips we considered were Cathedral Lakes and Elbow Basin in Manning Park - but our friends Luke and Jen *highly* recommended Taylor Basin in the Southern Chilcotin. Much thanks Luke and Jen - awesome recommendation!

Jen and Luke have a book, Don't Waste Your Time in the B.C. Coast Mountains, with details of Taylor Basin and I assumed I could buy it from Mountain Equipment Coop at the last minute - wrong! Not only was MEC out of stock, also every Chapters and Chapters.ca were out of stock, and every copy in the Vancouver Public Library was on loan or on hold.

By chance, the SFU library has one copy of exactly this book, but my colleague Richard wasn't on campus - he was en route to go fishing when I called. Thankfully Mark Jordan was working at the library and kindly delivered the book to Krisztina Kun at the SFPIRG. Krisztina lives a block away from Melissa, so we picked up the book that night. Thanks a lot Mark! You saved the trip : )

My housemate Norman Ellis was away sailing and very generously loaned us his sports car for driving to the trail head. His red, two door Mazda Precidia has 350,000 km, so we were very careful. The Duffy Lake road with view of Mount Rohr was beautiful. Thank you very much Norman!

We parked at the fork in Tyaughton Lake Road signed "Taylor Creek" and hiked up the logging road. The start of the logging road was in good condition, however, so we'd probably drive to the first parking spot next time.

We followed the trail up Taylor Creek through sub alpine forest with occasional views of red mountains and flowered meadows. The second day we went left at a trail junction, over Camel Pass and into Cinnabar Basin, stopping often for pictures, trail mix, and a swim. Continuing along the trail, we went over an unnamed pass to Eldorado Basin. The third day we followed the trail the length of Eldorado Basin to Windy Pass. We must've passed a junction to return to Taylor Basin, but didn't see it. Anyway, we intended to go to Windy Pass and it was worth it. Views were great : )

To return to Taylor Basin, Don't Waste Your Time recommends some alternatives which seemingly all involve climbing back down into Eldorado Basin. Instead, on the fourth day, Melissa and I left the trail and climbed to the top of the ridge on the east side of Windy Pass. From here we followed game trails along the ridge top in the direction of Eldorado Mountain. The ridge reached 8,160 ft, with great views.

On the ridge we met Mitch, John, Kevin, and Tony who'd been flown into Warner Lake and had on the previous day hiked 20 km from Warner lake to Mount Sheba, and finally Spruce Lake. They recommend it : ) All were super nice to talk with: Tony is a photographer and lives in a communal house here in Vancouver, Mitch was super knowledgeable and had impressive biology and geology vocabulary, and Kevin was the third fastest marathoner in Canada!

Kevin continued west along the ridge to Eldorado Mountain, while the rest of us took the ridge south to rejoin the trail at a pass and return to Taylor Basin. On the fifth day we hiked out the way we came.

The climb from Windy Pass to the ridge top was steep, but the rest of the ridge was much easier to follow, and the scenery was amazing - I'd highly recommend this route : )

An awesome feature of this trip was that each day felt different. The hike to Taylor Basin was mostly sub alpine forest. Taylor and Cinnabar Basins were alpine meadow landscapes. Eldorado Basin felt different again: Grassy meadows amongst forest, and a long, gradual climb up grassy slopes to Windy Pass. At Windy Pass we camped on a big, heathery shoulder with few trees. The return to Taylor Basin along the ridge was rocky with huge views, including views of the way we'd come.

We saw lots of marmots, one deer, and I was very excited to see a herd of eight goats, including some baby goats!

Here are more pictures

For the most part, we didn't forget to pack anything. The three things I wished I'd brought are,

Toque - I brought just a sun hat 'cause the weather forecast was hot - but it gets cold at night, duh! Toque would've made sleeping under the stars more comfortable. Luckily Melissa brought a head tube which was adequate substitute.
Binoculars
Pillow case - to stuff with spare clothes and make a pillow

Here's the list of things we packed, recorded for the benefit of packing next time,

Tent
Sleeping bags
Foamies
Stove and fuel - one liter
Pots and scrubby
Dr. Bronner's Magic Soap in like a 5 ml Nalgene from MEC
Scrappy thing
Tupperware and cutlery
Rope
Headlamps
Bug dope
Sunscreen
First aid
Book
Maps
Bear bangers
Camp pillow
Back packs
Hiking poles
Water bottles - we brought two bottles and two camel backs, about 6 l total capacity
Pristine water purifier
Thermos
Sun hat
Rain clothes
Gators
Mits
Fleecy
Hiking boots
Swim trunks

Food,

Trail mix
Dried fruit
Tea
Granola
Dried humus
Cheese
Meat sticks
Bagels
550 g gummy bears
Bars

Dinners,

Coconut curry
Chili
Gado gado
Dijon veggies
Biriani

We think our packs were unnecessarily heavy, so here's our retrospective on what to change next time,

One cup granola for Jack, 3/4 cup granola for Melissa per day
3/4 cup trail mix per person per day
Two pieces of bread for lunch per person per day
- Black Russian bread crumbles but doesn't mold
- Ancient grains spoils
- Next time try dense rye bread
1/2 meat stick per day
Try different cheese - extra old cheddar got sweaty
Toque - think night time : )
More gummy bears : )
Only one extra socks
Only two extra undies
No extra shorts or shirt
Binoculars
Lighter dishes and possibly cup for tea, instead of mini thermos
Second white rope and reinforced rock pouch for bear hang
No forks
More Jam
Jack wants peanut butter : )
No books but two crosswords
No brown sugar
Bigger camera memory card
Bandanna
Sunglasses for Jack
No deodorant
Pillowcase

2009-07-13T17:09:00.000-07:00

I moved a lot while I was a student - in and out of residence and shared houses. Based on these experiences, I love bins for organizing my stuff.

Also, residence was great for coming efficiently furnished. I saw lots of experiments to improve space utilization in residence and at the housing coop where I live now, but I didn't see much improvement on the stock desk, drawers, shelves, closet, and bed which furnished each room in residence. But in most of the shared houses, I wasn't so lucky with furnishings.

So after my last move, into the housing coop which wasn't furnished, I built these "Rubbermaid drawers". It's a chest of eight drawers made of Rubbermaid bins and almost exactly one standard sheet of plywood. When it's time to move, I can just pull out the drawers and put lids on them, making packing easy. The cabinet is held together entirely with screws, so with a cordless drill, it breaks down into four pieces plus some braces and packs flat.

A couple friends liked the design after seeing it and asked for plans, so after much delay, here they are. I briefly considered drawing the plans with Google SketchUp or Inkscape, but used a tool I'm already comfortable with instead: pencil and ruler : P

I used a standard sheet of 3/4" sanded pine plywood from Home Depot, and got them to make most of the cuts on their panel saw.

I can't see a way to avoid setting the saw to 36" twice, so start with that and cut the 36" piece off one end
Next, set the saw to 23" and cut the two 23" pieces off the board, and two 23" pieces off the 36" piece, leaving about 2" x 36" of scrap
Set the saw to 31 1/2" and cut one piece off the remaining approximately 8" of board, and one off of one of the 23" x 48" pieces of board
Finally, set the saw back to 36" and cut that off the remaining 23" x 48" piece of board

At this point the pieces are small enough to get home on public transit if necessary, so I finished the remaining small cuts with a table saw at home

2009-07-06T10:57:00.000-07:00

Went paragliding for the first time this weekend. Thanks to Steven here are some pictures of me and here's some pictures of where I was paragliding and related commentary.

So I thought of paragliding for a long time, but now my colleague Evelyn is an avid paraglider so I figured it was my chance! After months of dallying, the weather was great this weekend, I borrowed a car, and decided I could miss other activities - so I went paragliding : )

I took the two day beginner paragliding course from Jim and Coleen for about $400. Evelyn said chances were good I would get a solo flight this weekend - but no guarantees. So I think I'm super lucky to have gotten seven!

My first solo flight was around noon on Saturday after practicing forward launching in Jim and Coleen's field for a couple hours. Coleen is a nursing professor at UBC and her teaching skills are evident - tough but encouraging. Between launching and landing I had a chance to look around and think, "What am I doing?!" - nothing above, nothing below - just stuck in the sky about 2,000 feet above the valley.

I was amazed how quickly I got air born - very different from the scuba diving course I took last summer

Before meeting Evelyn, some misconceptions I had about paragliding were that you use the same parachute as sky diving, that you pull a cord to deploy, and that you jump off a cliff or plane. Turns out in paragliding it's always called the "wing" and it's totally different from the kind for sky diving.

When the wing is deployed, it's shaped like an airplane airfoil. There're a whole bunch of strings connecting you to the wing, and these change the shape of the wing to maneuver in the air. It's super cool that leaning left or right is a big part of maneuvering. It could actually be quite subtle - I want to turn left so I lean left - very cool feeling of flying.

To launch, you get the wing above your head like a kite, then run while using the strings to control the wing and keep it above your head. You're running down hill, so eventually the wing isn't descending as quickly as the ground and you're air born - or else you stop, go back, and try again. No "jumping". Also, this launching, running, and flying all happens in a couple seconds, so I found it was lots of inputs to get right.

When will I go next? : )

2009-06-28T12:13:00.000-07:00


RewriteEngine on
RewriteCond %{HTTP_HOST} ([^.]+)
RewriteRule . http://%1.local%{REQUEST_URI} [P]

I want to run the W3C validation service or Sauce Lab's Selenium service against an application I'm developing on my laptop, but my laptop has a dynamic address. Or I have machines with RFC 1918 addresses and .local domain names, and I want to access them from the internets.

I asked a couple local ISPs about IPv6 addresses, which I assume would let me give each machine an accessible address. None offer this service yet - keep toasting to the universal deployment of IPv6 : )

In the meantime, I used the above Apache configuration to proxy requests to inaccessible machines or my laptop. The configuration uses mod_rewrite and mod_proxy. It takes advantage of HTTP multihoming. HTTP 1.1 requires requests include a Host: header, which enables the server to know which domain name is requested. This is important because requests to all inaccessible machines will go to one address, but use different domain names.

I use Debian and put the above Apache configuration in /etc/apache2/sites-available/default. It would be nicer to put this configuration in /etc/apache2/conf.d/ because the former file is distributed in the apache2.2-common package and changes can cause conflicts when this package is upgraded. However, /etc/apache2/conf.d/ configuration is applied to the server config context, and mod_rewrite doesn't apply configuration from the server config context to virtual hosts.

It's possible to work around this by adding to each virtual host,


RewriteEngine on
RewriteOptions inherit

I gave my laptop the hostname "tad", and it runs the Avahi mDNS/DNS-SD daemon. Recent installs of Debian or Ubuntu run this daemon by default. So it's accessible to any machine with an mDNS resolver on the local network at the domain name "tad.local".

I added a wildcard DNS record which points *.campcoop.com at the server running mod_rewrite and mod_proxy. Now when my laptop is at home, on the same network as the server running mod_rewrite and mod_proxy, it's accessible on the internets at "tad.campcoop.com" - even though it has a dynamic RFC 1918 address.

jdbates

Direct Verification

Assertions sent as a POST (vs. GET)

OP Endpoint URL

OP-Local Identifier

Why's an event loop all the rage?

Callback hell

Promises

Coroutines/continuations

Camping

Interpretive programs

Crater after dark

Pua Po'o lava tube

Mauna Loa

Kilauea Military Camp

Coast trails

Composting toilets

Boiling rock

Changes

Kitchen garden

Recommendations

Ways to build soil fertility

Motivation

Monthly

Monthly with cheapest voice plan ($39.99 for 450 minutes)

GoPhone

DataConnect Pass

Next steps

Sound bites

Feedback

Dialogue

Language translation

Productivity

Scaling up

Takeaways

Hardware

Carriers

First impressions

Haven't yet figured out

Awesome things that really stood out for me

These are thoughts I had for the next Day of Workshops

These are workshop topics I'd love to attend at another Day of Workshops

Things I wanna follow up on

Takeaways

*Real* priorities

Cooperative structures

Ministries

Pitfalls and workarounds

Conflict and communication

Buying a house collectively

Collective house directory

Mashups

Real priorities