Ruby on Rails, JRuby, AWS, EC2, Exalead - Creating a new EC2 AMI from within VMware or from VMDK files Comments

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by rajkumar

2010-05-19T10:24:48Z

Thanks for this recepie. I successfully created AMI image from vmdk. I started it on eucalyptus. VM is getting started but IP is not getting assigned to it.......

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by David

2010-03-09T08:03:13Z

I'm also interested in knowing whether anyone got Windows VM converted to AMI and running successfully in EC2.

One question though, don't you need to have the right drivers (network, video, etc.) to run on EC2 hardware, especially for Windows?

If that is true, that might be one blocker for getting customized Windows to run on EC2, or connecting to it over the internet if no have network driver. Though perhaps we could install or slipstream drivers into the VM before conversion to AMI. Could find out the drivers by booting up the public Windows AMI and extracting them out or tracking down those drivers for download.

Also, I believe you can send Amazon your custom image on physical disk (USB, HD, etc.) for conversion to AMI. Especially useful for Windows, but don't know what Amazon would officially accept for OSes, Windows 2003/2008 only?

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by jm

2010-01-15T20:38:21Z

I dont see an answer for JT's question. Has anyone been able to successfully convert a Windows VM to run on Amazon EC2? The ec2-bundle-vol doesnt seem to run on windows. Tx in advance.

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by JT

2010-01-05T18:59:44Z

Has anyone been able to successfully convert a Windows VM to an Amazon AMI and then successfully run it on the EC2 cloud? I have been looking all over the web, and I cannot find anyone who was successful?

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Jeremy Cothran

2009-11-20T15:25:32Z

Got around the UUID issue by using the ec2-bundle-vol command with --generate-fstab as mentioned in earlier comments, but still had multiple filesystem problems reported in the system.log

./ec2-bundle-vol -k pk-<xxx>.pem -c cert-<xxx>.pem -u <xxxx-xxxx-xxxx-xxxx> -d /mydir -r x86_64 --no-inherit --generate-fstab

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Chris Anderton

2009-11-16T16:03:49Z

I intend to revisit this topic to explore some of the issues and also make the instructions a bit more comprehensive - it's been on my list for a while but i expect to get round to it in the next couple of weeks!

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by jhonn

2009-11-12T10:32:36Z

I succesfully created and uploaded windows image to EC2 cloud following this blog's instructions. I'm able to start and stop my new AMI but i cannot connect to it any way. The EC2 platform doesn't recognize my AMI as a windows istance but mark it as linux one. Does anybody know how to set image operating system type to window at creation time?? Tanks to all.

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Jeremy Cothran

2009-11-11T16:33:29Z

Hi Chris,

Thanks also for the article. I'm also having the same issue as Hugo regarding Ubuntu OS and UUID references not being a valid device name leading to a kernel panic for Amazon/AMI. Have searched but not found any documentation yet on the web where there might be a solution to this(converting Ubuntu/UUID vmdk to a running AMI).

Thanks Jeremy

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by abhi

2009-10-27T16:44:26Z

hello Chris, my vmware image have 2 vmdk disk files (1 for os the other for user data/programs), anything special I have to do [ in method1: Converting VMDK files for use on EC2 or in method2: Creating the EC2 AMI from within VMware] so that, bundling will bundle both disks, and once completed the image will work correctly on EC2 cloud?

Thanks

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Diego

2009-09-14T17:10:04Z

Hi Chris,

Thanks for the article. I almost succeeded creating my own instance, but I got the same error from Hugo. Have you found the solution for this problem?

“nd_request: I/O error, dev sda1, sector 2 EXT3-fs: unable to read superblock end_request: I/O error, dev sda1, sector 2 EXT2-fs: unable to read superblock end_request: I/O error, dev sda1, sector 64 isofsfillsuper: bread failed, dev=sda1, iso_blknum=16, block=32 Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,1)”

Thanks, Diego

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by randy

2009-09-07T05:47:06Z

has any one used rpath (www.rpath.com) to create images for both vmware and AMI and used for dev and production respectively.

It looks like, rpath can create images for both virual envrioments simultaneously with your own packages... all from the web!

would love to hear your experiences.

randy

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Rob Ristroph

2009-08-31T15:05:27Z

These are wonderful directions, a great article.

I would add that you should add the --retry option when running the ec2-upload-bundle command, so that you don't have to wait for hours only to have it time out on one of the files near the end.

Also, in the current ( Aug 2009 ) versions of CentOS and EC2, there is some additional step you have to do with /etc/fstab.

The output of my ECS instance ended in this:

Checking filesystems Checking all file systems. [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a LABEL=/ fsck.ext3: Unable to resolve 'LABEL=/'

[FAILED]

* An error occurred during the file system check. * Dropping you to a shell; the system will reboot * when you leave the shell. Give root password for maintenance (or type Control-D to continue):

I editted /etc/fstab to refer to the actual devices for / and swap, i.e., /dev/sda1 and /dev/sda2, instead of the "LABLE=" notation. That seemed to fix it, and I now have a running EC2 instance !

Thanks again for documenting and posting this !

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by dipopo

2009-08-08T15:31:50Z

Dude....you're a star for blogging this, searched high and low for this info...thanks!!!

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by akabdog

2009-08-07T23:49:31Z

Great howto, I noticed something odd, when I do ec2-bundle-vol to a cifs mount i get an error: Bundling image file... tar: image: Cannot close: Bad file descriptor tar: Exiting with failure status due to previous errors Execution failed, pipeline: image-bundle-pipeline, stage: encrypt.

But if I install a second drive, mount it and bundle to there- no error.

Also, my centOS image didn't work unless I used --generate-fstab, i think this is because I was using LVM

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Chris Anderton

2009-08-04T00:03:50Z

Hi Hugo

It's on my list to revisit this whole subject when I get chance - whenever i've used it then it's been plain sailing, but I think it'd be good for me to try a few different options and come up with some more useful recipes for people to follow.

I'll send you an email with my contact details and I can then try and help you offline..

Thanks

Chris

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Hugo Troche

2009-08-03T19:43:52Z

Chris, Thank you for posting this great article. I am having the common kernel panic problem after doing the conversion.

The error I get when I try to boot my AMI is:

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,1)

This the command that I am running to create the image:

ec2-bundle-image -i MyAppliance.raw -r i386 -c MyPublicCert.pem -k NyPrivateCert.pem --user MyUserAccountNumber

I am not using any --block-device-mappings since I actually don't understand exactly what they do.

Here is my /etc/fstab file:

/etc/fstab: static file system information.

<file> <mount> <type> <options> <dump> <pass>

proc /proc proc defaults 0 0

/dev/sda1

UUID=0e052cb7-9667-4853-9c05-275dca11cc43 / ext3 relatime,errors=remount-ro 0 1

/dev/sda5

UUID=cd7cb797-32dd-4252-ad7d-a6a78bf50b6c none swap sw 0 0 /dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec,utf8 0 0

The results of a df -h command:

/dev/sda1 9.4G 3.0G 6.0G 34% / tmpfs 252M 0 252M 0% /lib/init/rw varrun 252M 340K 251M 1% /var/run varlock 252M 0 252M 0% /var/lock udev 252M 2.6M 249M 2% /dev tmpfs 252M 0 252M 0% /dev/shm lrm 252M 2.0M 250M 1% /lib/modules/2.6.27-7-generic/volatile

What do you think settings should be?

Thank you, Hugo

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Chris Anderton

2009-06-22T20:20:13Z

Given the infinate number of ways the various linux distributions can be setup, then i doubt a one click converter will ever be created.

Typically, the conversion process is thwarted by one of two things - kernel and module settings, or the fstab.

For the first, you need to make sure you use features provided by the base Amazon kernel and supplied loadable modules.

For the second you need to consider what type of EC2 node you are targetting to know what drives it has available - combined with correct usage of block-device-mapping options you will be fine.

As I said, I doubt it will get any easier than this - with some reading of the documentation then it gets easier to understand.

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Zee

2009-06-22T13:06:00Z

Well, I took a VMWare image, converted it to RAW, packaged it with AMI tools and uploaded it just to get this typical KERNEL PANIC error.

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,1)

Oh, you forgot to do this, oh btw, you need to do that, and when you get done you need to do these other steps and when you are finish with those, jump thru this set of hoops..CMON now...

Has anyone found a straight forward set of directions to getting this process up and running without a bunch of here for, what not, and where to after you get to the final step???

Install, convert, upload, launch <-- this too hard to ask for?

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Chris Anderton

2009-05-20T14:57:54Z

Hi Doc

The honest answer is I don't know - I haven't tried ESXi. A quick scoot around Google leaves things as clear as mud - although comments about the Converter (which appears to be downloadable for free) suggest it can convert both ways between ESXi and VMWare Workstation (and others). Give it a try and let me know if it works!

Chris

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Doc Rice

2009-05-20T09:04:40Z

In regards to VMDK "conversions" to raw / AMI, does it make a difference which VMware product was originally used? I use ESXi, and I believe there are subtle disk format differences between Workstation, Server, and ESX (Infrastructure) virtual disk formats (hence why one needs to use their Converter product to use a Workstation image in ESX).

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Chris Anderton

2009-05-06T20:02:06Z

Hi Jimmy

If the instance appears to be alive, then have you made sure you've setup the firewall rules (on the amazon level)?

You need to setup a security group (if you haven't before) and assign it to the AMI:

http://docs.amazonwebservices.com/AWSEC2/latest/DeveloperGuide/index.html?distributed-firewall-topic.html

If you want to manage security groups via a UI then ElasticFox can do it (it's a Firefox plugin):

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=609

Hope this helps

Chris

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by weput

2009-05-06T17:31:58Z

Hello chris...

I've been trying this and everything worked fine (using the vmdk file option)... until i launched my first instance with the ami i created....

when i try to ssh the instance i get no response; like if the sshd is not on. ofcourse i've checked the local vmware application and made sure the service is up.

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Chris Anderton

2009-04-28T13:09:13Z

Hi nanao

The ec2-bundle-image tool should come with the AMI tools - make sure you have these installed. Documentation for the command is here:

http://docs.amazonwebservices.com/AWSEC2/latest/DeveloperGuide/index.html?CLTRG-ami-bundle-image.html

It sounds like you only have the API tools installed, so you need to go and fetch the AMI tools and install them.

You can then proceed as instructed - the main source of problems with a bundled image will either be around configuring the fstab, or using options that are not compatible with the kernels available on ec2.

Thanks

Chris

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by nanao

2009-04-27T23:00:00Z

hello, i'm trying this but now stuck because I don't see a ec2-bundle-image command; my etctools/bin only have ec2-bundle-instance command and ec2bundle command. how do i proceed to try and bundle the image and upload it to see if it works or not or just get kernel panic like most people here? thanks.

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Chris Anderton

2009-03-19T10:18:25Z

Hi Barry - Send me an email (via the contact email) with the trace in and i'll take a look.. Chris

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Barry

2009-03-19T04:10:51Z

I am having the same kernel panic problem.
But, my guest OS is RHEL 5.2 x64. I have edited my /etc/fstab as suggested by Amazon and other sources.

This is truly driving me crazy. Do you have any suggestions ?

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Chris Anderton

2009-01-07T22:31:41Z

Hi Tony

That's right - it would be a pre-requisite that you used an OS that could use one of the supported kernels - you can download these so could use them locally for testing. Also, as you say, if you don't specify an AKI then it will still use one as default - and in some cases not having the modules installed may cause you problems, or using functions that require non-built in or loadable kernel modules will again cause pain.

One of my questions when we attended an Amazon Web Services event in London was around allowing us to create our own kernels - i.e. give us some automated access to menuconfig for the kernel to allow us to specify our own built in options, but within a restricted environment to avoid risk to the service. I didn't get to speak to Werner about it, but the response i did get was that this was likely to remain as something for them and vendors and not us normal folk.

I've corresponded with Shiju offline and i believe the issue lies in two places - firstly the panic is likely caused by a DevFS vs udev issue - the Xen virtual device drivers don't support DevFS and the error experienced is symptomatic of this - and secondly the abscence of a correct fstab in the ec2 environment (which could be overcome with --block-device-mappings and then modified in situ and a new bundle built on ec2)

Thanks for your feedback..

Chris

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Tony S

2009-01-05T19:23:57Z

Hello,

A correction to my last post...

Just received word on the EC2 forums that regardless what kernel is installed in your AMI, Amazon will apply and run its own kernel. You can optionally specify an AKI and then optionally again an ARI, but those seem to not be necessary in many situations.

Still, it seems that Shiju V. Joseph's kernel panic could be caused if an incorrect kernel is applied by Amazon or there just might not be a supported kernel.

If I understand the situation correctly, you can't convert just any VMDK running just any OS kernel... Amazon must already support a similar kernel.

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Tony S

2009-01-05T05:16:30Z

Howdy, Am just beginning to look at AMI creation, your recipe looks interesting but I wonder how the procedure handles the fact that Amazon is very restrictive what kernels and possibly what memory images are permitted, and assign ID numbers (AKI and ARI) accordingly.

In fact, this may relate directly to the kernel panic problem Shiju V. Joseph is reporting.

I was hoping that somewhere in your recipe there is an option/configuration that might specify and merge a supported AKI and ARI into the AMI being created.

In fact, I'm wondering where/if somewhere in your recipe a Xen appropriate kernel is substituted for the VMware-modified kernel, and just musing a bit further whether it might be important that Xen's kernel supports only hardware-assisted virtualization while VMware supports not only hardware-assisted but software only kernel virtualization (x86 32-bit only of course).

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Chris Anderton

2008-12-23T22:00:03Z

Hi Shiju

What size image are you trying to start up?

Definitely looks like a file system problem with your fstab - send me an email with the full command you're running to bundle the image and the fstab on your vmware image and i'll try to help you out.

Thanks

Chris

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Shiju V.Joseph

2008-12-22T06:22:38Z

Hi chris,

I tried converting VMDK file and bundled an AMI , but it shows kernel panic errors

NET: Registered protocol family 17 NET: Registered protocol family 15 Using IPI No-Shortcut mode md: Autodetecting RAID arrays. md: autorun ... md: ... autorun DONE. Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,7)

df-h output on the VMware VM looks like this /dev/sda7 819M 123M 655M 16% / none 819M 123M 655M 16% /dev/pts /dev/sda1 145M 7.0M 130M 6% /boot /dev/sda2 3.4G 307M 3.0G 10% /opt none 502M 0 502M 0% /dev/shm /dev/sda5 1.4G 580M 710M 45% /sysimg /dev/sda3 2.2G 331M 1.7G 17% /var

so I added --block-device-mapping ami=sda7,root=/dev/sda7 option in the ec2-bundle-image command , but now also I am getting kernel panic message and the instance is not booting up ,am I doing anything wrong?

Thanks Shiju V.Joseph

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Chris Anderton

2008-10-01T15:50:08Z

Hi Vinchu

This doesn't directly help with getting Windows running on EC2 - there are two approaches to this:

Use QEMU on your EC2 instance - you may be able to install and test windows on QEMU on VMware on Windows/Linux - I haven't tried it but it might be worth a shot!
Wait a while - Amazon announced EC2 with Windows Server support today..

Chris

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Nikunj

2008-10-01T12:20:16Z

Oops I can't seem to find out where did I see that vmdk as supported type. I thought it was help text of the ec2-bundle-image command, but now I can't find it too.

Will let you know if I find it. Till that time, my comment should be considered invalid. :)

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by vinchu

2008-10-01T10:15:40Z

will that be useful for hosting windows VM in EC2? any idea?

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Chris Anderton

2008-09-27T20:37:26Z

Sounds interesting - I can't see it documented anywhere (but what's new!) - did you just find it out by trying, or have you seen it documented? I'll give it a try myself as I'm intrigued!!

Comment on 'Creating a new EC2 AMI from within VMware or from VMDK files' by Nikunj

2008-09-24T11:53:40Z

Informative post. While creating an AMI from vmdk file, I found out that the ec2-bundle-image command accepts image format as vmdk(in the -i option). so I guess the "qemu-convert" step to convert vmdk to raw image may not be necessary. As raw images seem a lot bigger than vmdks, it saves time as well as disk and bandwidth requirements to upload the bundle to s3.

May be the support for vmdk in the ec2-image-bundle command is added recently.