Viking's Notes
Screen
To change the escape sequence from C-a to `, add this to your .screenrc:
escape ``
Kubuntu on MacBook
Helpful Tutorial
https://help.ubuntu.com/community/MacBook
Dual Display
To get dual display working, it took a lot of time. After much cursing, I finally ended up with this
xorg.conf file, and it actually works! In KDE 4 no less! I didn't think it was possible.
Turn off your desktop effects or you'll get nothing but black/white screens on both monitors.
Git
Ignoring changes in tracked files
To ignore changes in already tracked files, run this:
git update-index --assume-unchanged
Submodules
After cloning a repository with submodules, you still need to clone the submodule respositories:
git submodule init
git submodule update
Importing from Subversion
Here's the command to import from SVN. The --no-metadata
option makes this more viable for a one-shot import.
git-svn clone --stdlayout --no-metadata -Aauthors.txt svn+ssh://biostat.mc.vanderbilt.edu/usr/local/subversion/Ryaml
File Permissions
Git doesn't track file permissions for most files (except for the executable bit). The reason for that is permissions aren't content. If you want the files in your git repository to be world-unreadable, set the core.sharedRepository option like so: =git-config --add core.sharedRepository group=
Clipboard manipulation
To output to the X clipboard on the command-line, use
xclip
(
apt-get install xclip
).
Sweet.
Awk output
To change how
awk
prints things (with
print
), change the
OFS
variable like so:
awk 'BEGIN { OFS="," } {print $1, $2}'
Subversion Diff Wrapper
Modified from the
SVN book:
#!/bin/sh
# Configure your favorite diff program here.
DIFF="/usr/bin/vim-diff"
# Subversion provides the paths we need as the sixth and seventh
# parameters.
LEFT=${6}
RIGHT=${7}
# Call the diff command (change the following line to make sense for
# your merge program).
$DIFF $LEFT $RIGHT
# Return an errorcode of 0 if no differences were detected, 1 if some were.
# Any other errorcode will be treated as fatal.
» Download
Password-less SSH
Shortest passwordless ssh tutorial, ever
Here's a script I wrote based on this tutorial to copy keys to remote hosts:
add-ssh-key.sh
MySQL
Adding primary key field to existing table
ALTER TABLE foo ADD id INT NOT NULL AUTO_INCREMENT FIRST, ADD PRIMARY KEY(id);
Credits to Daniel Schneller.
Paging
To turn on paging, do: pager less -S
To turn it off, do: \n
Fixing other people's code
http://drnicwilliams.com/2007/06/01/8-steps-for-fixing-other-peoples-code/
Exim4
Exim4 configuration sucks. It took me 3 or 4 hours to figure this out.
Exim4 and TLS don't work together well, apparently. To make exim work correctly, edit
/etc/exim4/exim4.conf.template and add this line just under the
driver line in
remote_smtp_smarthost section, so that it looks something like this:
remote_smtp_smarthost:
debug_print = "T: remote_smtp_smarthost for $local_part@$domain"
driver = smtp
hosts_avoid_tls = *
hosts_try_auth = ${if exists {CONFDIR/passwd.client}{DCsmarthost}{}}
tls_tempfail_tryclear = false
DEBCONFheaders_rewriteDEBCONF
DEBCONFreturn_pathDEBCONF
Re-run
update-exim4.conf
and then restart exim4.
Internet Explorer Quirks
IE doesn't support transparent PNG. Stupid. Check out the following page for a helpful script to fix it:
http://homepage.ntlworld.com/bobosola/
If you use this script with Rails, make sure you do something about the
asset ID that Rails puts onto the end of image files. The script file expects the last 3 characters of a PNG filename to match /png/i, so either take off the asset ID or modify the script.
To use in Rails, drop the script into
public/javascripts
and put this into your application layout:
<!--[if lt IE 7]>
<script defer type="text/javascript" src="<%= javascript_path "pngfix" %>"></script>
<![endif]-->
.NET
Links
Page Life Cycle Overview
Deleting Users
To delete users using stored procedures, run this:
DECLARE @RC int
DECLARE @ApplicationName nvarchar(256)
DECLARE @UserName nvarchar(256)
DECLARE @TablesToDeleteFrom int
DECLARE @NumTablesDeletedFrom int
SET @ApplicationName = 'foo';
SET @UserName = 'some guy';
SET @TablesToDeleteFrom = 15;
EXECUTE @RC = [foo].[dbo].[aspnet_Users_DeleteUser]
@ApplicationName
,@UserName
,@TablesToDeleteFrom
,@NumTablesDeletedFrom OUTPUT
SELECT @NumTablesDeletedFrom;
The 15 in TablesToDeleteFrom
is a flag that means to delete from all user-related tables (4 total).
Web Application Design Principles
REST: Representational State Transfer
CRUD: Create/Retrieve/Update/Delete
Firefox and KDE
If Firefox is open when KDE shuts down, Firefox does
not exit gracefully. Therefore, every single time you log back in, you'll get a session restore dialog box. AFAIK, there's no way to turn this off. KDE says it's a Mozilla bug (
https://bugs.kde.org/show_bug.cgi?id=136350 link), and vice versa (
https://bugzilla.mozilla.org/show_bug.cgi?id=354686 link). So, whatever.
HTML redirection
I always forget how to do this, so here it is:
<html>
<head>
<title>Redirecting...</title>
<meta http-equiv="refresh" content="1;/rhubarb">
</head>
<body>
Please click <a href="/rhubarb">here</a> if you are not redirected.
</body>
</html>
PostgreSQL
Authentication
To set up postgresql so that non-ident users can connect, change the
pg_hba.conf
file:
# TYPE DATABASE USER CIDR-ADDRESS METHOD
local all all md5
Logging
To turn off annoying and hateful notices to stdout, play around with
/etc/postgresql/8.1/main/postgresql.conf
.
Passwordless Connection (psql)
Create the
~/.pgpass
file, chmod it to
600
, and add entries that follow this format:
hostname:port:database:username:password
localhost
doesn't seem to work for
hostname
, so use
*
unless you want to manually specify
localhost
on the psql command line via
--host
.
Debian
X Display Manager
To set the default X display manager, change
/etc/X11/default-display-manager
.
Building a Custom Kernel (brief)
Packages needed:
apt-get install gcc kernel-package libc6-dev libncurses5-dev modutils module-init-tools
Steps:
- Get the source
- Unpack it
- Make a symlink
- Configure
- Build it by running:
make-kpkg --append-to-version=-2024-03-29 --revision=1 kernel_image
- Install the package
- Run
update-grub
- Reboot and do a jig
This is handy for oldconfig's:
yes $'\n' | make oldconfig
Double check the permissions on the source tree
Ruby
Snippets
See VikingRuby.
fastri
Here's an init.d script to get the fastri server running on boot. You need to create the fastri user, make the /etc/fastri directory, and change ownership of /etc/fastri to be owned by fastri. Whammo.
» Download
set_trace_func
Here's the set_trace_func example from Pickaxe:
set_trace_func proc {|event, file, line, id, binding, classname| printf("%8s %s:%-2d %10s %8s\n", event, file, line, id, classname) }
Postgresql
To get the postgres gem to compile on debian, do this:
sudo gem install postgres -- --with-pgsql-include-dir=/usr/include/postgresql
Mysql
To compile the mysql gem on Mac (with the macports version of mysql installed), run:
sudo gem install mysql -- --with-mysql-include=/opt/local/include/mysql5 --with-mysql-lib=/opt/local/lib/mysql5 --with-mysql-config=/opt/local/lib/mysql5/bin/mysql_config
(credit)
1.9 on Mac
First, you need an updated readline (probably). Get it from macports. Then, download ruby 1.9 and run:
./configure --prefix=/usr/local --program-suffix=1.9 --with-readline-dir=/opt/local
make
sudo make install
Rails
Cloning Regexp's
Ruby doesn't allow the cloning of literal regular expressions.
$ irb
>> /blah/.clone
SecurityError: can't modify literal regexp
from (irb):1:in `initialize_copy'
from (irb):1:in `clone'
from (irb):1
>>
This is a problem if you serialize a literal regular expression in a model. ActiveRecord clones its attributes before saving, and since you can't clone a literal regexp, you best be changin' that. Use Regexp.new("blah")
or Regexp.new(/blah/)
instead.
Selenium File Uploads
Firefox doesn't allow Javascript to type things into file upload fields. Here's a fix: http://cakebaker.wordpress.com/2006/03/29/file-upload-with-selenium/
Freezing Third Party Gems
I wanted to do this without having to generate a template, like gemsonrails wants, so I wrote a script based on the package's rake task. Credits to Dr Nic Williams for the gemsonrails package.
-
mkdir vendor/gems
- Run this script. The syntax is:
ruby freeze_gem.rb [ -o ENV ] [ -v VERSION ]
- Add this to your
config/environment.rb
:
# yanked from gemsonrails
gems = Dir[File.join(RAILS_ROOT, "vendor/gems/*")]
if gems.any?
gems.sort.each do |dir|
lib = File.join(dir, 'lib')
$LOAD_PATH.unshift(lib) if File.directory?(lib)
init_rb = File.join(dir, 'init.rb')
require init_rb if File.file?(init_rb)
end
end
The -o
(--only
) option lets you specify which environments you want the gem to be loaded in. If you only want to load the gem in development and test environments, do -o development,test
.
The -v
(--version
) options lets you specify which version of the gem you want to freeze. Hip, hip, hoorah.
Redefining Rake Tasks
This should probably be under the Ruby sub-heading, but I'm using this for Rails, so there. If you want to re-define an existing rake task, you have to first remove the old one, or both will run. Here's my Rakefile that adds this capability:
# Add your own tasks in files placed in lib/tasks ending in .rake,
# for example lib/tasks/capistrano.rake, and they will automatically be available to Rake.
require(File.join(File.dirname(FILE), 'config', 'boot'))
require 'pp'
require 'rake'
require 'rake/testtask'
require 'rake/rdoctask'
# credits to Blair Zajac (blair@orcaware.com) for this idea:
module Rake
class Application
def remove_task(name)
@tasks.delete(name)
end
end
end
require 'tasks/rails'
Then in your task file, add Rake.application.remove_task 'test:functionals'
, or whatever the relevant rake
task is.
Customizing RSpec Behavior Type
If you want to run <insert some other behavior type> tests in a spec file that isn't in the <that other behavior type you inserted before> spec folder, you can specify the behavior manually by using :behaviour_type => :<yes, that same behavior type again>.
For example, if I want to run a controller test in my spec/libs
directory, I do:
describe PockyTestController, :behaviour_type => :controller do
# do stuff here
end
Look at Spec::DSL::BehaviourFactory::BEHAVIOUR_CLASSES
to see what the behavior types are. Actually, the BEHAVIOUR_CLASSES
constant is in the metaclass of Spec::DSL::BehaviourFactory
, which makes it a little tricky to get at if you're not in the right context. First install the metaid gem, then run irb
(not script/console) from your Rails root directory, and paste this in:
$LOAD_PATH.unshift(File.expand_path(File.dirname(FILE) + "/vendor/plugins/rspec/lib"))
require 'spec'
require 'rubygems'
require 'metaid'
require 'spec/spec_helper'
p Spec::DSL::BehaviourFactory.metaclass.const_get("BEHAVIOUR_CLASSES")
Or you could just take my word that BEHAVIOUR_CLASSES
looks like this:
{:default=>Spec::DSL::Behaviour,
:helper=>Spec::Rails::DSL::HelperBehaviour,
:controller=>Spec::Rails::DSL::ControllerBehaviour,
:view=>Spec::Rails::DSL::ViewBehaviour,
:model=>Spec::Rails::DSL::ModelBehaviour}
Asset ID
Rails likes to put a timestamp on the end of filenames (i.e. /images/edit.png?1159723626). To change this, set the RAILS_ASSET_ID environment variable. You can do this in config/environment.rb
like so:
ENV['RAILS_ASSET_ID'] = ""
Converting Specs to Test::Unit
Here's a script I wrote to convert RSpecs back to Test::Unit::TestCase.
Connecting to MS SQL
See instructions on the Rails Wiki. Ubuntu packages needed:
- odbcinst1debian1
- unixodbc
- unixodbc-dev
- tdsodbc
- freetds-dev
- libodbc-ruby1.8
- libdbd-odbc-ruby
- sqsh
apt-get install odbcinst1debian1 unixodbc unixodbc-dev tdsodbc freetds-dev libodbc-ruby1.8 libdbd-odbc-ruby sqsh
The instructions actually work!! Wow.
Database connections
The magic happens when something calls one of the <adapter>_connection methods from ActiveRecord::Base. Most (if not all) of the time these methods are called as a result of a call to the connection() method, which tries to retrieve a connection. The establish_connection() method doesn't actually connect to a database, it just records the information needed to connect to a database.
ActiveRecord::Associations magic
Let's say we have a model Foo, which has_many :bars
. The has_many
class method defines the instance method bars
, which returns a HasManyAssociation
instance object. So what the hell is going on with the code below?
f = Foo.find(1)
f.bars.class #=> Array; wth?
Why isn't f.bars.class
HasManyAssociation
?? The reason why this happens is because of some tomfoolery by AssociationProxy
, the superclass of the association classes. AssociationProxy
does this number:
instance_methods.each { |m| undef_method m unless m =~ /(^__|^nil\?$|^send$|proxy_)/ }
This deletes most methods (including class
, inspect
, to_s
, etc.). So when I do: f.bars.class
, method_missing
is called. Here's what method_missing
essentially does:
if load_target
@target.send(method, *args, &block)
end
It sends the method right along to @target
, which is the Array
. Jeez. So, the class of f.bars
really is
HasManyAssociation
.
RSpec
How to make deprecations flunk a spec
Put this in spec/spec_helper.rb:
ActiveSupport::Deprecation.behavior = Proc.new { |message, callstack|
raise message
}
script/spec customizations
Here's a custom script/spec that automatically uses the options in spec/spec.opts. Also, if called without arguments, it runs the spec file that was most recently changed.
#!/usr/bin/env ruby
$LOAD_PATH.unshift(File.expand_path(File.dirname(FILE) + "/../vendor/plugins/rspec/lib"))
require 'spec'
args = ["-O", File.expand_path(File.dirname(FILE) + "/../spec/spec.opts")] + ARGV
# run the last touched file if no arguments
if ARGV.empty?
spec_files = Dir[File.expand_path(File.dirname(FILE) + "/../spec") + "/**/*_spec.rb"]
spec_files.sort! {|a, b| File.mtime(b) <=> File.mtime(a) }
args << spec_files[0]
puts "====== running #{spec_files[0]} ======"
end
#exit ::Spec::Runner::CommandLine.run(::Spec::Runner::OptionParser.parse(ARGV, STDERR, STDOUT))
exit ::Spec::Runner::CommandLine.run(::Spec::Runner::OptionParser.parse(args, STDERR, STDOUT))
Rails Logger Custom Formatter
Here's a custom rspec formatter that puts example names in the Rails logger:
class CustomFormatter < Spec::Runner::Formatter::SpecdocFormatter
def add_behaviour(name)
super(name)
@behaviour_name = name
end
def example_started(name)
RAILS_DEFAULT_LOGGER.info
RAILS_DEFAULT_LOGGER.info "=============== #{@behaviour_name} #{name} ==============="
RAILS_DEFAULT_LOGGER.info
end
end
Windows text file conversions
Sometimes Windows likes to save text files in the
UCS-2 encoding. You can convert this back to latin1 in vim by setting
fenc
to be 'latin1' (instead of 'ucs-2le'). Or, do:
iconv -f UCS-2LE -t LATIN1 somefile.txt
File creation permissions (umask)
To get X and KDE (and everything else that uses PAM for that matter) to use a default umask, install the libpam-umask package (in Debian). Then put this in
/etc/pam.d/common-session
:
session optional pam_umask.so umask=027
This doesn't really seem to work.
EncFS
To encrypt folders, install the encfs and fuse-utils packages. Then, run
encfs ~/.crypt ~/crypt
or something similar to create a virtual filesystem that's encrypted!
Regular Expressions
RFC2822-compliant E-mail Validator
Here's a regular expression to validate an e-mail address, mostly compliant with
RFC 2822 (see also
Wikipedia:E-mail_address):
/^[\w!#\$%&'*+\-\/=?^`{|}~][\w!#\$%&'*+\-\/=?^`{|}~.]{0,62}[\w!#\$%&'*+\-\/=?^`{|}~]@([\w\d\-]+\.)+[\w\d\-]+$/
Bash Scripting
To change the internal field seperator ($IFS) in Bash to just a newline, do this:
export IFS=$'\n'
R
Last.fm analysis
Use this bit of code to download the last.fm weekly artist chart and convert it into a data frame. Note: you need the XML package.
library('XML')
user <- 'kindlyviking'
url <- paste("http://ws.audioscrobbler.com/1.0/user/", user, "/weeklyartistchart.xml", sep="")
xml <- xmlTreeParse(url)
num <- length(xml$doc$children$weeklyartistchart$children)
artists <- data.frame(name=I(character(num)), count=integer(num))
compileInfo <- function(i, parent) {
artists[i,'name'] <<- xmlValue(parent$childreni$children$name$children$text)
artists[i,'count'] <<- as.integer(xmlValue(parent$childreni$children$playcount$children$text))
}
lapply(1:num, compileInfo, parent = xml$doc$children$weeklyartistchart)
Local install
To install a local copy of R without optimization, lots of debug information, and as a shared library, run this:
CFLAGS="-g3" FFLAGS="-g3" CXXFLAGS="-g3" FCFLAGS="-g3" ./configure --enable-R-shlib
make
My script to run my local copy of R (~/bin/myR):
#!/bin/sh
R_VERSION=2.3.0
R_EXEC=~/src/build/R-${R_VERSION}/bin/R
R_ARGS=
if [ -z $1 ]; then
R_ARGS="--no-save $*"
else
if [ $1 = "CMD" ]; then
R_ARGS="$*"
else
R_ARGS="--no-save $*"
fi
fi
R_LIBS=~/src/build/R-${R_VERSION}/library
LD_LIBRARY_PATH=~/src/build/R-${R_VERSION}/lib
R_HOME=~/src/build/R-${R_VERSION}
eval "$R_EXEC $R_ARGS"
VIM
Paste mode
To paste in VIM (console) without getting a mess, put VIM in paste mode like this:
:set paste
For a toggle, put this in your ~/.vimrc:
set pastetoggle=<F9>
I got this tip from
here.
Bootstrapping
(information taken from Frank's book, chapter 5)
The idea behind bootstrapping is taking several samples (with replacement) of your original data set and performing calculations on those samples (such as mean, median, etc). You can then figure out the behavior of those calculations. Use this method when you don't want to make any assumptions about what distribution your data conforms to.
Validating the accuracy of models when predicting stuff about future data is possible with bootstrapping. Efron's "enhanced bootstrap" technique involves estimating the "bias due to overfitting".
Overfitting occurs when a model takes too many variables into account. The bias estimated by bootstrapping validation can be subtracted from the original accuracy index (like R-squared) in order to produce the "expected bias-corrected index".
Receiver Operating Characteristic (ROC) Curves
An extremely helpful webpage about this can be found
here.
ROC curves can be used to determine how good a scoring algorithm is. If you have some data you want to test, you first need some "gold standard" to separate your data into true positives and true negatives. First you run your test on the data and get scores. Then, you need to put each element in your data into a positive or negative subset depending on what the gold standard says. Next, you plot those two subsets on the same graph. The ROC curve is the plot of true positives against false positives based on the threshold you set for your test. If the area under the ROC curve is close to 1.0, your test is close to the "gold standard".
MAST's validation technique
(information taken from the MAST paper)
To test their algorithm, the MAST folks did the following (almost verbatim from page 6 of their paper):
- select a large number of protein sequence families
- construct a set of characteristic motifs for each family
- create a database of pseudo-random sequences
- calculate the combined p-value of each pseudo-random sequence and a family of motifs
- measure the classification accuracy of combined p-values when searching a database of real proteins
They took 75 families of proteins from
Prosite and scored each pseudo-random sequence (about 1000 total). They plotted the computed scores against the expected scores using negative logarithms. They also tested real sequences (from SWISS-PROT) and computed the ROC
50 score based on the Prosite standard.
From the footnote about the ROC
50 metric:
The ROC50 metric considers only the top of the sort down to the fiftieth non-family member. The metric has a value of one if all the true family members come before any non-family members in a sort of the sequences in the database. It has the value zero if fifty non-family members appear before the first family member.
Other
When installing MEME, make sure the MEME path line goes at the end, since it has it's own
install
program. They really should have called it
meme.install
instead, since if you run ./configure with the MEME path line before the path to the actual
install
program, ./configure will make its Makefiles to call that install instead of the real one. Bad idea.
If gdb stops working, make sure the SHELL environment variable is set! Yarrrr.
Data Capturing
- Unique field names across tables (avoids problems when merging)
- Camel case field names (avoids periods and underscores, which can conflict with some languages)
- Include backup keys
- Use strings for categorical values (easier to read, harder to change)
Running Latex through RApache
LaTeX might crash when running through RApache because it can't create the temporary files it needs. Check out a solution here:
http://stackoverflow.com/questions/11240725/how-can-i-get-php-to-compile-a-latex-document-if-it-www-data-cant-get-access
Scrypt Encryption
This section describes how the
scrypt utility encrypts files.
Setup Phase
- Calculate parameters from the following options: maxmem (maximum memory to use), maxmemfrac (maximum fraction of memory to use), maxtime (maximum time to spend)
- Get random 32-byte salt from /dev/urandom
- Generate a 64-byte derived key from the password, salt, and parameters
- Prepare a header to write to the front of the output file, which contains the
scrypt
magic word, values of parameters, and the salt
- Create a checksum for the header using SHA256
- Create a header signature using HMAC with the header data, the header checksum, and the last 32 bytes of the derived key (the MAC is used to verify the password)
After the setup phase, we have the derived key and a header that is 96 bytes long.
Encrypt Phase
- Start a new HMAC and add the header data to it
- Write the header to the output file
- Initialize AES encryption algorithm with the derived key
- Read in unencrypted data, encrypt, and write encrypted data a block at a time. The encrypted data is also added to the HMAC.
- Finalize the HMAC and write it to the output file
- Zero out the memory containing the derived key and AES key
LDAP search example
ldapsearch -W -H ldaps://ldap.vunetid.vanderbilt.edu/ -D uid=vunetid,ou=people,dc=vanderbilt,dc=edu -b ou=people,dc=vanderbilt,dc=edu "(uid=vunetid)"