How to install Elastix onto an Amazon EC2 Instance

We recently received an enquiry asking us if we could help install Elastix onto an EC2 instance. It's been a couple of years since I last installed Asterisk onto EC2, so I couldn't help but give Elastix a go.

Read more of this entry

Modifying Vagrant box memory settings

To increase the default Vagrant box memory with the VirtualBox provider there are VirtualBox specific configuration options. We are currently using the following:

config.vm.provider "virtualbox" do |vb|      
  vb.customize ["modifyvm", :id, "--memory", "1024"]
end

The latest documentation also lists the following shortcut for VirtualBox memory configuration:

config.vm.provider "virtualbox" do |v|
  v.memory = 1024
end

If you are using the VMWare Provider the configuration is different.

config.vm.provider "vmware_fusion" do |v|
  v.vmx["memsize"] = "1024"
end

If using VMWare Workstation you would replace "vmware_fusion" with "vmware_workstation".

Discuss this entry

Amazon wishlist - Please give us an EC2 Recovery Console!

In the olden days with our racks of Dell servers we used to have the luxury of the iDRAC card - Integrated Dell Remote Access Controller (I often wondered what the acronym meant!) - other manufacturers like HP, IBM, etc would all have something similar. Essentially, it lived on a different network connection, and gave you remote access to the machine - even when things had gone badly wrong. You could even mount ISO files over the network - meaning you could install Operating Systems over it - great for when things had all gone a bit wrong, or even for upgrading with some peace of mind that you could access the machine regardless of what happened. When working with XenServer it was a pretty useful way of upgrading between major releases (for some reason it didn't allow remote upgrades).

In the virtual server space, one of the great things about Linode was LISH - an out of band access tool to your virtual server instance. Again, this provided console access - great for resolving boot time issues, allowing you to drop into single user mode.

But, with EC2 there isn't yet an equivalent - if you've used an EBS backed instance then you can stop your instance, mount the volume elsewhere and attempt to diagnose and fix any issues - there is a guide over here. This doesn't allow you to drop into single user mode, and is a bit more effort - it would be great if Amazon could provide something more like LISH...

Why would it be useful? Sometimes you can be a dork and make changes to the system that don't become a problem until you restart - changes to fstab are a classic example - you can see the log telling you to press a button, but you can't get to it. As above - there are workarounds, but it would be great if you could jump straight in and connect to the console. Likewise when creating new AMIs - it would aid debugging of issues to have console access and single user mode without having to start over or mount/unmount volumes from other instances.

So, Dear Amazon Web Services.. please give me a console!

Discuss this entry

The Defensio Gem versus HTTParty YAML Deprecation!

As part of the recent round of security issues relating to Rails and YAML, support for using YAML has been removed from the HTTParty Gem (in this commit).

This has a knock on effect on any Gems that use YAML with HTTParty and don't set a specific version of the YAML-friendly HTTParty to use - one such example is the gem for Defensio.

Luckily, the fix is simple - the gem uses the Defensio 2.0 API - which already supports XML or JSON (as well as YAML) - a quick fork and 5 minutes later we have a JSON consuming version of the Gem.

UPDATE! My pull request has been accepted so the official repository is now using JSON rather than YAML.

Discuss this entry

Ebook DRM - a closed book that needs to be opened?

In a departure from the normal format of our blogs, i'm going to attempt a more commentary based post!

I've spent a significant amount of the last 18 months working with Ebooks - from integrating ONIX (and proprietary) data feeds through to working on displaying epubs in a browser. One area that frustrates me is the mandatory inclusion of DRM in many contracts - I don't think I am alone in this frustration.

Read more of this entry

Using Resque with short-lived (i.e. quick) workers

We use Resque all the time - but on a recent project we came across a slight issue - our jobs were executing so quick, that the overhead of forking on every job being processed was massively effecting the time taken to process the queue.

The first line of thought was to see if we could make the jobs 'do more' - but they really were both as concise and as complex as they needed to be!

Luckily, it looks like our problems have already been solved - there are a number of options already in the wild for improving worker performance with fast running jobs.

Read more of this entry

Amazon DynamoDB - SSD backed non-relational DB is here!

So - pretty exciting stuff, for us at least - Amazon Dynamo DB (previously mentioned here) - has now seen the light of day, and is appearing within the console. Instructions are linked from within the console.

We're already getting cracking on giving it a good play - we'll report back findings of interest!

Discuss this entry

We're hiring!

Bucking the trend of interplanetary meltdown, we're looking for some 'awesome' people to join our team. We're fleshing out the full details right now and will be posting ads in all the predictable places in the next few weeks, but in the meantime if you fall into any of the following job descriptions and are interested in applying send your CV and examples of your work through to .

These are all permanent positions, salaries are to be decided based on experience and skills but will be based on market rates, as will the overall package on offer. We're currently based in Fareham, Hampshire but may also be relocating soon - wherever we go will be within 15 miles and will be near a mainline station. So if you're currently London based then you'll need to suffer a nice commute (we've all done it the opposite way round!) or perhaps a nice reloaction to the Hampshire countryside.

Openings:

User Interface/Web Designer
We need someone with strong skills in designing web based interfaces and graphics. Our clients require designs that are clean, professional and modern with a sprinkling of innovation and spark. We're also interested in people who dabble in pixel art as it's a current favourite of ours and will give us a head start on some of our current internal projects!
Ruby Developer (x2)
We predominantly work with Ruby based applications. Whether it be Rails, Sinatra or anything else, we need someone proficient in Ruby as a language with a strong foundation in clean, sensible programming. The more skills you have, the more chance you have of getting a job with us - make sure your CV's are full of the things you can do rather than the names of as many technologies you can think of!

Experience is beneficial, but if you can demonstrate a keen background in the scene or can bring along a strong portfolio of homegrown projects for review then we're happy to consider you. Qualifications are secondary to skill, attitude and enthusiasm.

We are also happy to accept interns/gap year students falling into any of the above skillsets - send us your details and we'll take a look!

Discuss this entry

Revisited: Tamper-proof cookies in Rails 3

Here's a revisited post that's fairly short and sweet: way back in 2008 I blogged about my implementation of tamper-proof cookies which used a similar technique to that used by Rails for its cookie-based session store. Back then the solution involved a custom cookie jar, the OpenSSL library to generate a HMAC, overriding the ApplicationController#cookies method and a slightly unorthodox method signature for reading cookie values.

Read more of this entry

Revisited: roll your own pagination links with will_paginate and Rails 3

With a final release of Rails 3 edging closer every day it seems like a good time to revisit some of my old articles from the last few years and bring them up to date.

Back in the summer of 2008 I wrote about custom link renderers using will_paginate and, as it is still one of the most popular posts on the blog, it’s the one I’ve decided to refresh first. Don’t worry if you haven’t read the original article as I’ll be covering the same things here. So without further ado, let’s get stuck in!

Read more of this entry

Rails 2.3.8, Rack 1.1 and the curious case of the missing quotes

If you're using Rails 2.3.8 for your application and thought that you were safe after May's comedy of errors produced three point updates in as many days, think again. Unfortunately there's a little bug that can lead to parameters being altered or potentially even truncated without warning.

Read more of this entry

Create a bootable EBS AMI from a running instance

A quick set of notes on how to create a bootable EBS snapshot from a running EC2 instance - for example, an instance that has been started from an S3 backed AMI.

We've had to do this a few times over the last few months - for the benefit of others, we've outlined how we currently do it - this is based on a number of articles that were surfacing at the time of our research, but I don't have the links to hand. If there's a better way out there feel free to jump in!

Read more of this entry

Ooh la la: Paperclip et les European S3 buckets

At the end of my last blog about Paperclip I mentioned that you need to do some patching if you want to use European S3 buckets to store your files. The problem was introduced when Paperclip made the move from RightAWS to Marcel Molina’s AWS::S3 gem. Unfortunately despite several forks containing patches to AWS::S3 and a 4 month old bug report nothing has been done to officially fix the problem.

So my fellow Europeans, what are we to do?

Read more of this entry

Conditional duplicate key updates with MySQL

In one of our larger Rails apps the sheer volume of data we process means we’ve had to rely more and more on direct SQL queries, denormalised tables and summary tables to speed things up. When updating summary tables we typically use ON DUPLICATE KEY UPDATE, a MySQL extension to INSERT statements since version 4.1, that allows a record to either be inserted or updated in one query.

Read more of this entry

InfiniDB, Infobright and MonetDB - Day 3: MonetDB

Day 3 of my database exploration mission brings me to MonetDB. Binary downloads are available for Debian, Fedora, Ubuntu and (strangely!) Windows! If we still had any Windows users left here at HQ then it'd be a rare treat, but instead (as usual) our platform of choice (Centos 5) isn't directly available in binary form. We downloaded the Fedora source RPMs and built our own - in case they're of any use then i've put them up on a Google Code site for others to download.

After installing the RPMs then you're ready to get started - before you can do anything you have to start the merovingian process (you could either setup an init script, or run the binary manually for now). For information, the instructions say:

merovingian is a daemon process that controls a collection of database servers, i.e. mserver5 processes, each looking after a single physical database. Start this program to gain access to your MonetDB database farm. merovingian is designed to be used in a system initialisation script in production environments.

With merovingian running then you're ready to create a database - for this you use monetdb - and then start the database using the same command for example:

> monetdb create twf
successfully created database 'twf'

> monetdb status
     name        state     
twf            stopped              
  
> monetdb start twf
starting database 'twf'... done

> monetdb status
     name        state     
twf            running

You now have a running database and can connect to it using mclient. This is similar to most command line clients where you can perform changes to your database as well as query for data.

The first step to transferring the database was as usual - inspect the schema on our MySQL database and update it to make the correct use of the supported data types. As with the other systems, there's no support for unsigned values, it also wasn't immediately obvious to me what the maximum length of a varchar is.

With the tables created it was time to try and migrate some data. Given MonetDB has been around for quite a while then there seemed to be pretty scarce resources with any detailed instructions - I couldn't, for example, find any simple migration tools or documentation detailing the best path for migration. I guess this could be because MonetDB is more often tackled by people with bigger brains or with more time to figure things out.

I attempted to use the following to dump data from MySQL:

select * from h into outfile '/dbtmp/tmp/h' fields terminated by "|" enclosed by '"';

And then the following to import into my MonetDB table:

copy 1000000 records into h from '/dbtmp/tmp/h' using delimiters '|','\n', '"'  null as '';

This yielded reasonable results - though I did have to do some tidying up in the middle with sed - in the end I gave up as there were some string values causing me problems, so I decided to rest on it and went to bed!

In the morning I came back to find the merovingian process was dead, and the status of the database was showing as crashed. I started up the processes and took a look at the status - it said the health was 67% so i'm not really sure what's going on with it!

Performance

In the time I had available I was only able to get a 1 million row table imported successfully to play with - a shocking performance I know, but MonetDB was being quite fussy and I wasn't pressing the right buttons! I did run a few tests and also ran them against the same dataset in MySQL for comparison, all are run from cold - i.e. MySQL and MonetDB are both restarted before each query. I don't expect these queries to be representative of real world cases, I was just thinking of some nasty queries that I could throw at a single table in order to cause some pain.

Query 1

MySQL takes 250msec:

sql>select count(*) from h;
+---------+
| L1      |
+=========+
| 1000000 |
+---------+
1 tuple
Timer       1.532 msec 1 rows
Query 2

MySQL takes 420msec:

sql>select count(*) from h group by intcolumn;
+-------+
| L2    |
+=======+
+-------+
65 tuples
Timer     142.260 msec 65 rows
Query 3

MySQL takes 44,000msec:

sql>select count(*) from h group by varcharcolumn;
+-------+
| L1    |
+=======+
+-------+
12743 tuples
Timer    1464.389 msec 12743 rows
Query 4

MySQL takes 37,500msec:

sql>select count(*) as total from h group by varcharcolumn order by total;
+-------+
| L1    |
+=======+
+-------+
12743 tuples
Timer    1496.537 msec 12743 rows
Query 5

MySQL takes 373,000msec (not a typo, it's more than 6 minutes):

sql>select count(*) as total from h group by varcharcolumn, anothervarcharcolumn order by total;
+-------+
| L1    |
+=======+
+-------+
69696 tuples
Timer    4170.520 msec 69696 rows

Summary

Obviously this quick trial of each of these is not comprehensive enough to make any solid comparisons of performance - the next step will be for me to go through and come up with a proper test plan in order to be a little more methodical about things. However, it has given me a good grounding in how the 3 systems compare with respect to installing and getting started. I'll be keeping a close eye on InfiniDB - while not stable enough right now, i'm sure they'll keep things rolling and I look forward to taking another look. If I can overcome the import obstacles and also the different 'feel' of MonetDB then the basic query results make a compelling case for taking a further look - there's also more to learn here with respect to architecture, deployment techniques, monitoring, etc. Finally, Infobright - it would make my life easier if we could use it on an insert/update/delete basis - as it is I think we'd have a tough time getting clients to pay the license fee - perhaps if bundled with something like EC2 instances with a smaller incremental cost then it may be more palletable and help to increase adoption (it may be that Infobright have lots of customers with open wallets - in which case please share them!). In terms of immediate ease of use, with some visible performance improvements, Infobright fits the bill - but until i've had a chance to compare MonetDB and Infobright in a bit more detail then i'll reserve my final judgement!

Discuss this entry

Archives

  1. December 2013
  2. January 2013
  3. December 2012
  4. January 2012
  5. May 2011
  6. January 2011
  7. August 2010
  8. July 2010
  9. April 2010
  10. January 2010
  11. November 2009
  12. September 2009
  13. August 2009
  14. July 2009
  15. June 2009
  16. May 2009
  17. April 2009
  18. March 2009
  19. February 2009
  20. December 2008
  21. November 2008
  22. September 2008
  23. August 2008
  24. July 2008
  25. June 2008
  26. May 2008
  27. April 2008
  28. March 2008
  29. February 2008
  30. January 2008
  31. December 2007
  32. November 2007

Tags

  1. 37signals
  2. actioncontrollerdispatcher (nameerror)
  3. actionview
  4. active messaging
  5. activerecord
  6. activesupport
  7. actverecord
  8. adobe content server
  9. aes
  10. aggregation
  11. ajax
  12. akismet
  13. amazon
  14. amazon sqs
  15. ami
  16. apache
  17. api
  18. apple
  19. apr
  20. apr-util
  21. asterisk
  22. async
  23. attachments
  24. attachment_fu
  25. attr_accessible
  26. auto scaling
  27. autotest
  28. availability
  29. aws
  30. backgroundrb
  31. beanstalkd
  32. bindings
  33. bj
  34. block
  35. boot failure
  36. branding
  37. buckets
  38. bug
  39. bugmash
  40. cache
  41. caching
  42. callbacks
  43. cancer research uk
  44. cdn
  45. centos
  46. charity
  47. cloud
  48. cloudfront
  49. clusters
  50. column information
  51. columns
  52. community
  53. company name
  54. compatibility
  55. compiler
  56. composed_of
  57. config.ru
  58. consultancy
  59. content
  60. content delivery
  61. controller
  62. convert
  63. cookies
  64. csrf
  65. css
  66. data warehouse
  67. database
  68. dates
  69. defensio
  70. deployment
  71. design
  72. development
  73. dhtml
  74. digital editions
  75. docrails
  76. documentation
  77. donations
  78. drdb
  79. drm
  80. duplicate key
  81. dynamodb
  82. ebooks
  83. ebs
  84. ec2
  85. elastic
  86. elastic block store
  87. elastic load balancing
  88. elastix
  89. em-resque
  90. encoding
  91. encryption
  92. erb
  93. error
  94. european
  95. events
  96. exalead
  97. ezcrypto
  98. facebook
  99. fckeditor
  100. feedburner
  101. feeds
  102. ffmpeg
  103. filter
  104. fixes
  105. flash
  106. flickr
  107. flickr api
  108. flickr_fu
  109. fuse
  110. geekup
  111. gem
  112. gems
  113. geocode
  114. git
  115. github
  116. god
  117. great south run
  118. greenplum
  119. growl
  120. hacker
  121. haml
  122. haproxy
  123. helper
  124. hmac
  125. holiday
  126. hooks
  127. hosting
  128. howto
  129. hpricot
  130. html
  131. identity
  132. imagemagick
  133. imagescience
  134. infinidb
  135. infiniteftp
  136. infobright
  137. init.d
  138. insert
  139. invalid authenticity token
  140. italy
  141. javascript
  142. jobs
  143. jobs per fork
  144. jquery
  145. json
  146. leeds media
  147. limit
  148. linkrenderer
  149. linode
  150. linux
  151. lish
  152. load balancing
  153. logo
  154. mac
  155. markaby
  156. mass-assignment
  157. memcached
  158. mephisto
  159. messageverifier
  160. messaging
  161. middleware
  162. migrate
  163. migration
  164. model
  165. mod_rails
  166. mod_ruby
  167. monetdb
  168. mongrel
  169. mongrel_cluster
  170. monit
  171. monitoring
  172. mootools
  173. mp3
  174. mq
  175. multiple gems
  176. multiselect
  177. mysql
  178. neon
  179. new site
  180. nginx
  181. observer
  182. offset
  183. open source
  184. opensolaris
  185. openssl
  186. optimisation
  187. pagination
  188. paperclip
  189. parameters
  190. params
  191. passenger
  192. patch
  193. pbx
  194. performance
  195. permanentredirect
  196. persistence
  197. persistent storage
  198. persistentfs
  199. php
  200. phusion
  201. plugin
  202. plugins
  203. post commit
  204. post-commit
  205. pow
  206. protomultiselect
  207. prototype
  208. query
  209. queues
  210. quotes
  211. race for life
  212. rack
  213. rails
  214. rails development
  215. rails patch
  216. rails plugin
  217. rails-doc
  218. rails3
  219. rake
  220. recovery
  221. refresh
  222. renderer
  223. respond_to
  224. resque
  225. rich text editor
  226. rmagick
  227. ruby
  228. ruby on rails
  229. rubyinline
  230. running
  231. rvideo
  232. s3
  233. s3fs. elasticdrive
  234. scaling
  235. schema
  236. schwarzenegger
  237. scm
  238. search based applications
  239. security
  240. services
  241. session
  242. shorthand
  243. signed
  244. snarl
  245. social
  246. solaris
  247. spam filter
  248. sparrow
  249. specify
  250. sponsorship
  251. sql
  252. sqlite3
  253. sql_logging
  254. starling
  255. starter kit
  256. storage
  257. streaming
  258. subversion
  259. sue ryder care
  260. survey
  261. svn
  262. swfupload
  263. swig
  264. sysadmin
  265. tables
  266. tamper
  267. templates
  268. the webfellas
  269. thewebfellas
  270. thin
  271. thumbnail
  272. time zone
  273. tinymce
  274. tip
  275. tips
  276. to-done
  277. training
  278. transcoding
  279. twitter
  280. tzinfo
  281. ui
  282. uk
  283. uk rails
  284. unsigned
  285. update
  286. uploads
  287. url
  288. ux
  289. vagrant
  290. vagrant configuration
  291. vagrant memory
  292. validation
  293. version
  294. video
  295. view
  296. vmdk
  297. vmware
  298. voip
  299. webfellas
  300. webfellows
  301. wedding
  302. welcome
  303. widgeditor
  304. will_paginate
  305. win32
  306. windows
  307. wysiwyg
  308. xen
  309. xhtml
  310. xvm
  311. yaml
  312. youtube
  313. zenoss
  314. zentest
  315. zfs

Flickr snaps