Setting up a Ubuntu mirror manually

Ok. So I've confirmed that my mirror works. So I'll describe the manual process of building your own mirror.

The simplest way to build your mirror is to follow the instructions provided by others (google: ubuntu mirror). The drawback is that you will need to download roughly 5.5 GB for the main repository alone (the multiverse universe is even bigger).

As I wrote earlier, I purchased the entire repository from a place called LinuxStore.ca.

1. Follow the instructions from http://www.howtoforge.com/local_debian_ubuntu_mirror to install apt-mirror and apache2.

2. Comment out all the entries in the /etc/apt/mirror.list file. Add a single entry like so (for easy testing):

deb http://closestmirror.ubuntu.com/ubuntu main jaunty

3. Run apt-mirror and break out when apt-mirror starts downloading packages.
Do a line count of the following files:

/var/spool/apt-mirror/var/NEW
/var/spool/apt-mirror/var/ALL
/var/spool/apt-mirror/var/MD5

The file NEW and the file ALL should have the same number of lines. (Which means apt-mirror would download all the files specified in NEW).

4. Copy the directory and all it's subdirectories and files (e.g. from the install CD):

cp -R /media/cdrom/pool/main /var/spool/apt-mirror/mirror \
/closestmirror.ubuntu.com/ubuntu/pool/

5. Run apt-mirror again and break again when it begins downloading. Check out the NEW and ALL files again. Notice that NEW should have fewer lines than ALL. This means that the mirror has accepted the files that you copied to your mirror. Which means that you will be able to update your mirror regularly (as designed by the creator of apt-mirror) as if it downloaded all the files from the internet.

6. Lastly, don't use the clean.sh script. I believe it builds a script which cleans up *everything* from your mirror. I thought it would clean up all the packages that is outdated. This is mistaken. The apt-mirror script makes a file size comparison between the file that exists in your mirror directory with an index file which it downloads from the external mirror. If they don't match then NEW gets updated with the name of the package to download.



And that's it. Pretty simple, huh?

Virtualization

I've been exploring virtualization technology for a year or so. At first I was interested in checking out Xen (since it was open source). But it appeared complicated and I didn't want to get involved in a compilation/setup cycle just yet. Recently, I've had the chance to actually try out a few products and here are some of my experiences with it.


I've tried VMware workstation in the past. It works well and it allows you to explore working with VMs on your own desktop. My problem with it is that when given a choice of working through a VM versus through the host you would choose the host naturally. I wanted a hypervisor (or a light-weight OS) which would not distract me from working with the VMs.


So I downloaded and tested different hypervisors available. XenServer and KVM. There is also VMvisor from VMware but I figured it was similar to XenServer. They are all basically the same.


XenServer and VMvisor are hypervisors which means they are light-weight OS managing and running on top of physical hardware. They provide the resources for running VMs on top of the hypervisor. I installed XenServer first. It was an easy installation on my workstation. After installation you have a curses interface to the OS.


But I didn't know how to create VMs. I searched for the documentation online (looks like good documentation) and I discovered that I have to use a client to access the VM manager to create VMs. So what I was losing in deciding to go with a hypervisor was the use of a desktop. The hypervisors are for servers or any headless system that will run the software. Losing the use of my new monitor wasn't part of the plan. So I decided to ditch XenServer.



I moved on to KVM. KVM stands for Kernel-based virtual machine. KVM was the chosen virtualization technology for Ubuntu. Xen technology was purchased by Citrix systems a while ago. Xen is still provided as open source but the enterprise product is requires a license. So I wondered why KVM for Ubuntu? The reason as provided by the KVM site is that the development effort was minimized because it utilized the Linux kernel. Why build a light-weight OS with scheduling of processes and workloads when there is an excellent open source OS available already? So what KVM does is it adds a module to the kernel to trap whenever the kernel encounters certain instructions. Oh yeah, KVM assumes you have virtualization capabilities provided by the CPU. So I tried it.



It took some time but I enjoyed playing with KVM. It's an interesting set-up. And the documentation is Ok. KVM by itself only allows you the capability of running VMs. But to run a VM you require some sort of emulator. So how this solution works is that you require either VMware player, Xen, or QEMU. In other words, KVM turns your Linux host into a hypervisor. The good thing is that you still have the desktop (GUI) where you can run VMs. The difficulty is that there are extra software layers involved that makes the solution less than simple. I.e. libvirt. Libvirt is a library which permits access to VMs. Libvirt isn't necessary of course. All it does is gives you a standard front-end to your VM solution. So KVM => Libvirt => emulator.

What's cool about Libvirt is that you manage the VMs via a shell console (virsh) or a gui console (virt-manager). Anyway, I enjoyed working through the solution.



I didn't settle on KVM (although I'd like to). It has alot going for it. Principally, it's open source and second it doesn't shield you from it's capabilities. The only drawback is it's complexity. It took a little time to get started. But the reason I went back to VMware workstation is the performance of QEMU. With non-Linux OSs QEMU just didn't perform very well. I installed Windows XP and Solaris 10 using QEMU and it was slow. Very slow. Plus the fact that converting a QEMU created VM (i.e. qcow2 format to vmdx format) didn't necesarily mean that I could take it to a VMware player and play it back. I could have tried KQEMU but it was another level of complexity that was eating up more of my time. So I'm back to using VMware.



Long story short, I would prefer to use KVM/libvirt/QEMU if QEMU performed better out-of-the-box. I would also prefer to have a document which described the architecture of the solution to ease new users into using these technologies. Finally, based on some of the documentation I researched, it opens up clustering and cloud computing which I'm also interested in exploring. So it has alot of interesting technologies going for it. But for my current purposes it's complicated to setup and maintain. I hoped to use both KVM and VMware at the same time but it's not possible as both KVM/VMware add a module to the kernel to run virtualization. So they are mutually exclusive.

Ubuntu Mirror

Working with Ubuntu exclusively on my workstation now since April 24th (Jaunty release). I've also been exploring virtualization technology as well. I gave XenServer, KVM a try. Settled on VMware only because the files were portable (so I can take it to work). I'll write about the different products another time.

My intent was to create a VM so that I can run some genealogy software. The VM for the obvious reason of backing it up easily. Funny thing is that after installing Ubuntu (a few times) it appeared to me that Ubuntu *required* a connection to the Internet to install. I found out after some experimentation that this isn't the case. Anyway, I wanted to set up a mirror so that I didn't need to do this. However, I didn't want to set up a mirror by downloading the entire Ubuntu repository (22+ GB).

So I experimented with apt-mirror and here are some of my results.

- I purchased the entire Ubuntu repository from a store called LinuxStore.ca (LinuxStore.org.uk).
- So instead of downloading the mirror I wanted to copy it from my set of DVDs
- I followed the instructions in setting up apt-mirror
- Instead of pointing mirror.list to a mirror like http://ca.archive.ubuntu.com/ubuntu I followed an instruction to add the line: deb file:/media/cdrom/ubuntu jaunty main
- This didn't work so I read the /usr/bin/apt-mirror perl script
- Problem with the script is that it strips "http://" from the mirror name and creates a folder under /var/spool/apt-mirror/skel/. Since my URL = file:/media/cdrom/ubuntu this just didn't work.
- So what I ended up doing was keeping a single entry in the mirror.list file and mirroring that (i.e. jaunty-security main - 66 MB).
- I then confirmed the file structure and copied the pattern for the rest of the files
- I then confirmed file structure of the locally (DVD) retrieved files by confirming that apt-mirror when executed would update the packages
- Therefore I know that copying universe and multiverse components (from the DVD) of the repository will work correctly

Quick summary

- If I could do this again I would not buy the entire repository. Downloading "main" and "restricted" is good enough.
- Creating a mirror isn't necessary. You can point any VM sources.list file to a local directory or a NFS mounted file system or a DVD mounted iso and apt-get will retrieve the files.

Ubuntu Install

/dists/jaunty/binary-i386/main/Release.gz contains the list of packages for the component (i.e. main)
/pool/main/a
/pool/main/b ... etc.
contains all the packages.
Running apt-mirror for a different component, eg. main, will check the most recently available packages from the repository and update your mirror. So this is the benefit of using apt-mirror.

Question: Is the archive of apt-mirror the same as an archive stored at /var/cache/apt/archive ?
When running Update Manager it saves the updated packages to /var/cache/apt/archive .

Chinese computing

I've been learning Mandarin Chinese for 2 years now. I've been attending a Mandarin language school with Tzu Chi http://en.tzuchi.ca/canada/home.nsf/home/index . My teacher is 傅老師 (Teacher Fu) and we've been learning from a book named 五百字說華語 (Speak Mandarin in Five Hundred Words).

Problem: I recently installed Ubuntu Linux on my computer at home. I figured out how to install and use the Chinese input method offered in the Windows OS. I used the Windows New Phonetic 2002a code table for input because it allowed me to use pinyin for input and obtain traditional chinese characters for output. Ubuntu has a input method (IM) framework so that you can add multiple language input methods and create your own easily. It's called SCIM (Smart Common Input Method). Unfortunately it has multiple input methods but not one suited for me. It does have Pinyin but it produces simplified characters. Not being able to find the one I wanted I went online to read about all of them. I found that 五筆字型 (wu bi zi xing) was the coolest. It's difficult that's for sure but it's also efficient when you've acquired the skill and the memory for the character strokes. Since the system is built around how the characters are written with a brush it means that I kill 2 birds with one stroke (if you will). I can learn characters, character strokes and an input method all at once. And when I finally build up the repertoire of characters I won't be (necessarily) defficient in any particular area (that is the hope anyway).

It was a problem at first. Because 五筆字型 produces simplified characters. Further research indicated that there are more than one wubi method. wubi86 supported the GB86 character set. There was an update called GBK which included traditional characters as well. However, GBK wasn't very popular and it was because GBK wasn't backward compatible with wubi86. And so wubi 2000 provided support for GB 18030-2000 which was backward compatible with GB86 but also contained traditional characters. I was looking for a wubi code table which supported GB 18030-2000. I couldn't find it.

Lucky for me, after starting this entry for my blog I discovered a method for looking up input codes for various chinese characters (until I actually know some of these codes) and I discovered that the wubi input in Linux does have traditional characters contained in the table. So that the character 謝 (traditional) and 谢 (simplified) both have the same wubi code "ytmf". So I'm not limited to a particular character set after choosing a specific input method.

My next step is to begin cataloguing the characters I've learned from my chinese textbook and inputting them into a table for quicker lookup. I'm also contemplating how to produce a database to store the various methods of locating characters.
There are various systems available for looking up characters. Radicals, pinyin, ... I will probably begin cataloguing these methods to get a better picture of how to go about searching.

More later.

Computer graphics, Intro

I started a business after coming back from Hong Kong last year. It's called FriendlyComputers and we have a website at FriendlyComputers.ca. I started it with a friend of mine (Ted) in July 2008 and we hoped to have an online presence at least by October/November. The business is to sell computer products and services online. The goal is to create enough clientele that by the time we had sufficient funds (and a feasible business supply/demand) then we would move the business to a physical location (preferably in the Mississauga/Etobicoke area).





Unfortunately our plans didn't come to fruition. We've experienced delays and my personal apathy for creating more work for myself. However in the back of my mind the work was there (the seed was there, it just needed to be watered). Well, Ted did his part. The proof is in the current site. We intended to build a flyer and begin manual advertising (pamplets, etc.). That was my job. But I struggled with it. Reason being is that I wasn't comfortable with the tools for creating/designing pamplets, etc. Or I'm just lazy. But I honestly feel that drawing with a mouse is just wrong. So long story short, I bought a Tablet PC recently (Mar 19, 2009).





I picked up a IBM/Lenovo X41 Tablet. It's got a Pentium M Centrino 1.5 GHz processor, 1.56 GB of RAM, 60 GB Hard drive, Intel GMA 905 graphics. I love it already. Keyboard is excellent. Writing, via the stylus, is fun but more importantly it's natural. The stylus is a natural aparatus to use to create images. Even better would be my finger but a stylus is the next best thing. Someday, with the new multi-touch technology coming out on new tablets, one day I'll be able to use my finger.





Which leads me to the topic of the day. Computer graphics. I find this field funny because the reading material for it is so disparate that it's hard for me to find my way around. Eg. Computer graphics in universities start from math. We learn algebra and geometry and talk about vostroms and translation, etc. That's all fine. Another eg. Image processing. An image in essence can be nothing more than an array of pixels or a description of a vector path. However, the primary tools to manipulate images is Adobe Photoshop and vectors with Adobe Illustrator. The problem I find is that there is a gap between the theory and the practice. So theoretically I can learn Computer graphics in school and then take it to a Game design shop or a 3D graphics design shop. No problem there. But if I want to learn about theory about image processing what we get is alot of math or click here and click there as in Photoshop/Illustrator classes/online tutorials, etc. In other words, there isn't a decent theoretical treatment to the workflow and tools available for the image processing applications, i.e. Photoshop/Illustrator.





So I'd like to write my own. Some of my goals are as follows. I'd like to create images/scenes from photographs captured from a digital camera. Since capturing an image is easier than creating images from scratch I think it would be good to have an efficient workflow to design images. I want to convert these raster images (from the camera) into vector images. I've learned that Illustrator (CS2) contains a trace function which will build vectors images from raster images. And then through editing, storing, retrieving objects I learn the fine art of designing images. So whereas a course in Photoshop/Illustrator for instance would start the same way that the tutorial books would, i.e. here is the workspace, canvas, tools, colour, etc., I want to start documenting from a theoretical standpoint. Eg. Graphics programming with Perl describes it from a logical point of view (from a programmer's point of view). And image is a canvas. A canvas can be made up of one or more canvas. A primitive is a point. Elements are polygons, circles, squares, curves. By building it up this way it's easier to grasp that a layer (in Photoshop speak) is nothing more than a canvas. A point/handle is an anchor. A filter should be a function over a 2d object. All primitives are objects. A layer is an object. So as you can see by my description, when we use a filter or a channel I will know what it means (from a logical point of view).

More later.

Beijing - May 27 - May 31 - Pg1

When we got to Hong Kong and settled in, the first thing we did was travel to Kowloon to look for travel agencies that could book us a tour of Beijing. Since I'll be here for 3 weeks and travelling with my mom it would be good to take a tour of at least one other destination in China (other than Hong Kong).

Asian companies are kinda funny. We located a building that contained at least 5 travel agencies all on 2 floors. Why companies would choose to locate in an area that is in such close proximity is perplexing. You realize later that all companies do this because everything (or just about everything) is a commodity here. Here in Hong Kong the customer rules. The customer will shop for the best price. Price being principally the single most important factor in most transactions. Since an offering is never unique you can well find a better price somewhere else. To me this is kind of strange since time itself is limited. Why would I spend alot of time on something that might only be worth at least 5% (up to 45%) of the value of the commodity? The problem is in location. Here in Hong Kong an address (a street address) doesn't mean a whole lot. I searched for a book store at 299 Castle Peak Rd in Tsuen Wan. It is shop 46A-47A in a shopping centre called Tsuen Nam Plaza (?). I couldn't find it. I walked back and forth on Castle Peak Rd and couldn't find it. How do people find anything here? After spending almost 3 weeks here in Hong Kong I realized that people in most cases don't have a favourite shop and buy exclusively from them. And if your wandering around looking for a good price, what you find may be the best price at that moment, but you won't find that shop again after you make a comparison with someone else's price. That is why shopkeepers are all so pushy to make a sale. They know that once the customer leaves they are gone. So if you like something and you ask them what the price is ... you have consented to the protocol of full negotiation for the product. They get angry if your "just looking" and you ask them the price. And I can see why given what I wrote earlier. They wouldn't be able to make a living if every price for every commodity was somehow published openly for people to compare prices.

We woke up at 4 on Tuesday May 27th and got on a taxi to take us to a bus terminal to grab a bus to take us to the Airport. There are many ways of getting to the airport the bus being relatively cheap. We grabbed the N31 bus at 5 AM arriving at the airport around 5:50 AM ($20 HK). We boarded the plane at 8:15 AM and it took off at around 8:45 AM.

The plane trip took 3 hours and for most of the trip we flew above a haze of smog. I could be wrong but it didn't look like clouds as there was no definition in the view. As we neared Beijing the plane descended below the smog and we could see farmland. It was nice. Good ol' acres of squares lining the terrain. As we neared Beijing I could see buildings and markets and people quite clearly. I was amazed at how close the roads and people were to my vantage point. I wondered how close we were flying as I could distinctly see a road across my view and a bicycle riding up to the intersection. I was thinking this must be how someone arriving in the old Hong Kong airport must have been like. We landed fine and exited the plane onto the airstrip proper where there were waiting buses to take us to the terminal. We got to a terminal and then jumped onto a subway-like train which transported us to another terminal. Here we went through immigration and then jumped onto a waiting bus which started our tour.

Our first stop was to a shopping centre in Beijing. It was a building of 3-4 levels each selling different merchandise. The shops in the centre were organized like many Chinese shops are organized with square units each side by side with at least 2 retail reps waiting to serve you. We were there to grab some food. After a brief lunch the real tour began.

Over 5 days we visited countless sites full of history. These include: Forbidden City, Temple of Heaven, Ming Tombs, Great Wall, Beijing Hutong, Summer Palace, Tiananmen Square, etc. Each one containing artefacts from it's long history as the capital of China. Our first stop being the Forbidden Palace you realize that this isn't an ordinary modern city. The scope is so different from modern cities. In Beijing, roads were not built for pedestrians. In the centre of Beijing was the emperor's palace and all the roads encircle it. Meaning major roads are rings which form squared circles around the palace as it's centre. Each ring is numbered and there are at least 6 rings. Ring 1 being the palace itself and Ring 2 being the first ring outside the palace. Ring 2 is a single road which contains 3-5 lanes in one direction (at most 6-10 lanes in total). The road has 3 lanes dedicated for cars and trucks and 2 lanes dedicated for bicycles and motorbikes. If the road described above also has what is like a collector's lane which allows a vehicle to exit the road (onto a collector) to make a right turn. All the major roads have these collectors. And all of these roads require you to exit through the collector to make a left turn. I'm not describing it very well here but it's similar to the pedestrian walking routes throughout the city of Hong Kong. If you've been to HK you know that pedestrians don't have the right of way. And since land is so scarce walking across the street means that you have to walk up a staircase onto a bridge which lets you cross the road. In like manner our bus in Beijing was required to exit the road onto the collector, turn right, make a U-turn, and then reenter the main road again. Logistically allowing cars to make right turns would slow traffic down to a crawl.

More later.

End of the trip - Flying back on Friday

Just a quick note. I'll be flying back to Toronto on Friday. I would have preferred to keep a travel log for all the places I've been and seen (my first trip outside of North America). Unfortunately I wasn't able to update this log frequently enough. I expected to be able to access the internet more readily here in Hong Kong but I should have been more realistic. I'm here for only 3 weeks and there isn't enough time to spend an hour a day on a computer writing a log.

I've spent alot of time on buses and trains and planes while overseas. Public transportation is excellent over here. The problem is that 20% of my trip (3 weeks) has been on these vehicles. I'm actually quite tired of it and I'm looking forward to having a normal routine again.


I haven't seen it all and I don't expect to. I'm here for 2 more days and I look forward to seeing everybody soon.

Take care (from Tsuen Wan Public Library, New Territories, Hong Kong),
Dennis.