chuma.org

Category: things i’ve learned

  • Real mobile Wikipedia

    I’ve seen lots of talk recently about ‘mobile’ versions of Wikipedia. I’m a big fan of Wikipedia and I use it all the time. I’ve looked at all of the “solutions” for so-called mobile versions of Wikipedia and they all suck. First of all, many of them require full-on ‘net access or a cell phone, or they only let you load a subset of the data with limitations. Pffff. What’s the point then? I want everything, all the time, and I want it to be fast. Never mind that the english Wikipedia is huge – an XML dump of just the page contents is 7GB uncompressed.

    Hrm. I seem to have these two little handheld computers here, and they run Linux and have 20GB hard disks you say? Well geez, what are we waiting for! All we need to run MediaWiki – the free software that powers Wikipedia – is Linux (check), Apache (web server), MySQL (database), and PHP (web scripting language) – often called LAMP for short. Oh, and we need a dump of Wikipedia, of course – those are generated regularly. Off we go:

    B1B

    The Pad 3, with its Fedora Core underpinnings, makes getting LAMP install ridiculously easy. Open an xterm (Ctrl-Shift-1). Issue this command:

    yum install httpd php-mysql mysql-server

    Let it resolve all dependencies and install all that stuff. Great.

    B2B

    in contrast to its younger brother, the Pad 2 is based on MontaVista Linux CEE and Professional 3.1. This makes a LAMP stack harder to setup for two reasons:

    • the included apache-dev package is broken six ways from sunday
    • there are no MySQL or PHP packages included.

    I manually patched the busted files from apache-dev: in short, one of the config files in /usr/share/apache-2.0/build/ has to have all of its paths fixed, and the same for the apr-config script. I used the generic MySQL 4.1.21 source RPMs to build some halfway decent RPMs for the Pad 2: you still have to manually setup your my.cf and a startup script though. PHP I compiled and installed from a source tarball. Let it be known that MediaWiki and PHP should also work with thttpd, which is also provided by MontaVista and might not be so broken.

    For MySQL, you will have to set the max_allowed_packet setting to 4M in /etc/my.cnf otherwise your import process could fail.

    B3B

    Follow the MediaWiki setup insructions. In a nutshell: extract the tarball, rename the directory to something useful (like ‘wikipedia’), move the directory to your webroot (/var/www/html), and run the config page to generate LocalSettings.php. You probably want to disable all of the e-mail options, since you won’t be editing this wiki.

    Now, use mysql and delete all the records from the page, revision, and text tables.

    B4B

    Wikipedia provides database dumps as compressed XML files. However we need to import this into the MySQL database. A Java tool called mwdumper is the best way to go about this: it reads the compressed XML and outputs SQL statements, or will even directly connect to a MySQL server and insert them for you. But, this will take FOREVER (at least 24 hours) and if you have to interrupt it, you have to start from scratch. Provided you have another Linux box on which to do some preprocessing though, you can make things easier on yourself. Here’s what to do.

    1. Get a Wikipedia dump file – the one you want is called pages-articles.xml.bz2. Download it to your Linux box. It’ll be big.
    2. Get mwdumper as well.
    3. Get Sun Java 1.4 or 5.0. You can use mwdumper with GNU GCJ, but in my test it’s about 10x slower.
    4. Put both files in a directory and run java -jar mwdumper --format=sql:1.5 pages-articles.bz2 | sed -e 's/^INSERT/INSERT IGNORE/' | split - wikidump- and make sure you have about 7GB of free disk space.
    5. This process will chug along for a while and the result will be a bunch of files called wikidump-aa, wikidump-ab, etc. They contain the SQL statements that inject the data, but we’ve split it up into smaller chunks. That way we can interrupt things in the middle. We changed the INSERT command to INSERT IGNORE so that if we do interrupt things and start importing a file again, MySQL won’t complain about duplicate keys.
    6. Feed the data to the MySQL database on your Pepper: mysql -h pepper -u root -p'password' < wikidump-aa substituting your Pad’s IP, MySQL username and password (the ones you used in the MediaWiki setup, remember?) Or use a shell script to automate this, printing the filename before it starts loading so that you know where you are. We’re talking about 3.8 million wiki pages here. As of this very moment I’m somewhere in file ‘ah’ and I’ve done 784,000 pages.

  • Guess what? The router is crap.

    So apparently, the Linksys BEFSR81 router that I scored isn’t all it’s cracked up to be. Two major drawbacks are going to force me to get rid of it, possibly with extreme prejudice:

    1. The QoS features of this router are useless. You can do port-based QoS, but that only deals with transfers WITHIN THE LAN. Useless. You can also do ‘application-based’ QoS, but that just marks the outgoing packets with a ToS priority flag, so if your ISP doesn’t respect the flags, it’s useless. Also you have to enter one port at a time, and with Vonage the voice packets could use any UDP port from 10000-20000. ARGH. So no traffic prioritization has actually been happening for the past week, which explains a lot: we’ve been getting occasional echoing and stuttering on Vonage calls. This article at PracticallyNetworked has the skinny on the QoS.

    2. This router seems to have a problem with either UDP streaming or something else, because every so often it just goes batshit crazy and drops ALL LAN TRAFFIC for periods of time – anywhere from 5-15 seconds to minutes. The WAN side stays up, but the LAN just drops 100% of packets. It sometimes comes back, and after resetting the router it’s OK for a while, but this is purely unacceptable. Once when it did this it took 2-3 minutes for Vonage to reconnect and during that time Sandy was trying to get the voicemail. FFS! She was pissed and now I’m pissed. I did a bit of research and found lots of other people having the same issues.
      Particularly this guy’s page was informative.

    So it looks like I’ll have to either get a new router or use my Linux box. I didn’t want to spend any more money but a Linksys WRT54G or WRT54GS should do the trick with a minimum of fuss. I really don’t want to have to rely on my Linux box for routing because it’ll suck a ton of juice out of the UPS if the power goes out. With just the DSL modem, router, and VoIP box on the UPS it’ll last for hours and hours.

    Update my friend Jon writes below that he’s been using the same router, with the same issues. Bleah. I’m going out and getting a WRT54G tonight 🙁

  • Getting rid of Bell: still a head-spinning exercise

    The debate at home about going VoIP with Vonage for our main phone line heated back up at home yesterday. It’s something we’ve been thinking about for months, and according to our calculations we’d save $40 a month – but it has two main impediments:

    1. We use DSL for our internet connection and are not willing to switch to cable. “Dry” DSL is now an option (DSL service without local phone service on the same line) but it’s so new no one seems to know about it.
    2. Our security system requires that we have phone line monitoring. Nevermind the fact that we get reamed out the ass paying for it. And we’re locked into our contract until the summer of 2007.

    (1) is no longer an issue, now that “dry” DSL is available – DSL without Bell phone service. #2 is still an issue, though it might be possible to get it to work over VoIP.

    I called up my ISP to ask them abou dry DSL and they didn’t have a lot of answers for me unfortunately. They said to switch my line I’d have to fight Bell tooth and nail, and also that Bell still requires that you pay for using their wires even when they’re not providing service on it.

    I poked around online and found another provider – TekSavvy out of Chatham, Ontario. They mention dry DSL right on their website so I figured I’d call them up to talk about it. Well, I spent about 15 minutes on the phone with a fellow named Bill who answered all my questions and generally impressed the hell out of me with his candor and straightforward facts. Here’s what I learned:

    1. Yes, Bell still charges your DSL provider for using their copper when Bell isn’t charging you for phone service. In most places, that’s $16.99 a month. HOWEVER, that fee is currently unregulated (Bell set it themselves) and the CRTC is expected to set this fee in the next few months. It’s likely that the CRTC will force Bell to lower it. Right now, for my dry DSL, it would be $16.99/month on top of the DSL service.
    2. It is really hard to get Bell to convert a currently active phone line with DSL to “dry” DSL. They will fight and bitch and complain. Also you might be without any service for two or three weeks. BUT, if you have another free line running to your house (and you probably do – they always run lots of extra copper for future service), it’s much easier to get them to bring up dry DSL on that unallocated line. The wait time is still 2-3 weeks but since they’re not losing any money out of the workorder, there’s less crap to cut through. Once you get DSL up on the other pair, you can cancel your phone service and the other DSL service. You might have some overlap for a while though, but it sounds like less of a mess.
    3. If I wanted Vonage to keep my existing phone number, I’d have to ask Vonage to take it over when I activate their service. Yet another reason to do point #2 (get a second DSL service temporarially) first, because if I cancelled my phone number without Vonage taking it over, I’d never get it back.

    4. I even picked the guy’s brain about running my security system over VoIP. I’ve read about some people doing it but he said there are two main problems:

      the reason the security companies are so dead-set against internet monitoring is because of insurance and possible outages. If your internet went down say four or five times a year, that’s considered a lot of times. They are very slow to change their tune because they’re afraid of not being able to provide reliable enough service, and of losing their insurance coverage. And insurance policies take forever to catch up to technological advances.

      the reason that you will have problems trying to trick your security system into running over VoIP is because VoIP is asynchronous – only one side transmits at a time. Normal phone service it, by definition, synchronous – both ends of the line can talk and hear each other simultaneously. Security systems talk like this and need a synchronous connection. Your VoIP provider would have to be able to switch very fast between the two ends of the link while your security system is talking in order for the connection to work. I think this is what I have read about where people phone up Vonage and get them to change some setting on their line which improves the security system communication

    What I would kill for is for my security provider to just install and support a dialer capture module like this one. It tricks your existing system into thinking it’s talking to a phone line, while retransmitting the data over the internet using TCP/IP. Boy, would I.

    So that’s a lot to think about, but it’s also more information than I’ve ever gotten on the subject. Many thanks to TekSavvy, I might just be calling them up for my new DSL service if I can solve issue #2.

  • That Axe Commercial

    If you live in Canada, and you watch television, you’ve probably seen a commercial for Axe Shower Gel. In this particular spot, there’s a naked man being chased by a pack of dogs through backyards, etc (ostensibly because he was cheating with the dog owner’s wife/girlfriend). In any case Sandy and I loved the music in the background, and as Meatwad would say “the beat is awesome.”

    So I e-mailed the people at Unilever (parent company) to find out what the song is. Unfortunately, this is what they wrote:

    From: Spcl-ConsumerCentre.CA-LP-Tor@unilever.com
Subject: RE: Home and Personal Care related enquiry
Date: 29/03/2005 09:27

Hi Victor,
Thanks for writing!
The song was created for the commercial by contracted composers, which is often
the case for advertising.  Unfortunately it is not available anywhere else.

    Crap.

  • wedding bells…

    rungs

    Tomorrow, Sandy and I are getting married. That’s a really bad photo of our wedding rings. They were designed by my sister and made by an artist here in Toronto.

    If I’ve been incommunicado as of late, well, this is why 🙂 After the wedding we’ll be jetting off for two weeks of sun and no internet access in the Dominican Republic. If you need me, don’t e-mail me…

    See you around all.

  • Adventures in electronics disassembly

    I’ve got microelectronics to fix. Camera’s done, next will be a pair of eMates…

    Last night, I replaced the rear case cover on Sandy’s Kodak DX4530 digital camera. It must have taken a nasty bounce at some point a few weeks ago, because the clear plastic overtop of the LCD was cracked. It wasn’t in the middle of the screen (and thank goodness the LCD was OK!) but she wanted it fixed.

    Amazingly, Kodak will actually send you service parts! Yay! They sent me the rear case piece (part 3f2130 I believe), which includes all of the buttons on the back and the dial and shutter on the top. Disassembly was basicially like this (following instructions from memory, standard disclaimers apply):

    • Remove the battery and SD card.
    • Remove all the screws on the bottom of the camera plus the one under the rubber cable cover, carefully pry up the bottom piece, the rubber cable cover (which is now loose), and the EasyShare dock cover. There’s two red and black wires underneath there – take note of how they are routed and make sure you put them back where they came from! They power the LCD backlight. Theoretically you could carefully unplug the cable with some tweezers but I didn’t.
    • Be careful with the battery compartment lid – it’s not part of the bottom cover but it can now be easily dislodged and fall out. Once it’s out it’s a little difficult to put back, so just make sure it’s closed and locked at all times.
    • Look on the bottom and notice the flexible cable that’s peeking out… that connects the LCD to the camera. The LCD is attached to the rear case, so be careful in the next step…
    • BE CAREFUL HERE. Remove all the screws on the rear case cover (look at the new one to figure out where they are) and carefully pry it up from the non-card-slot side. Carefully put the camera lens-down and open the rear case up like a book, since it is still attached via the LCD cable!
    • You should be looking at a PCB on the back of the camera. There’s a connector with a wide flexible cable attached to it. There’s also the backlight cable that we talked about before. Take note of how the backlight cable is routed again, it goes behind a metal tab to ensure that it doesn’t get squished.
    • On either side of the big connector are some black locking clips. Take a very small jeweler’s screwdriver or a toothpick and carefully push these clips towards the cable. This will unlock the display cable and it should slide out easily. Don’t touch the contacts!
    • Now, on the inside of the rear case is the LCD, behind a metal shield. Peel back the piece of black tape to reveal a hidden screw. Remove it and the other two screws, then remove the metal shield.
    • Take your new rear case, remove the plastic insert inside. Pop the SD card slot cover out of the old rear case and put it into the new rear case.
    • Slide the LCD into the new rear case. Take care with the backlight cable. Replace the metal shield, screw it in as before.
    • CAREFULLY slide the display cable back into the connector until it won’t go any farther. While holding the two halves of your camera down so the cable doesn’t wiggle, push those locking tabs back so that the cable locks into place. Ensure that they’re locked all the way otherwise the cable won’t make contact and your display won’t work properly or at all.
    • Now you can put the rear case back on. Fold it back over the camera, and pop it over the video out connector first. Once that side is on it should easily snap into place. Don’t force anything!
    • Don’t screw the rear case on yet. Flip the camera over and look at the bottom. Re-route the backlight cable the way it was. Take the bottom cover and ensure that the cable fill fit properly and not be squished!
    • Replace the EasyDock connector cover, and rubber port cover, put the bottom cover on, and screw into place EXCEPT for the screw next to the rubber port cover. You might have to squeeze the two camera halves together to get a good fit. If it won’t go together easily then you’re probably crushing the backlight cable. STOP and check – it’s rather fragile and I almost severed mine.
    • Once the bottom cover is on, you can put batteries in and make sure the camera still works and (importantly) the display works. Try all of the buttons to make sure they work and are aligned.
    • If it’s all good then put the screws back into the rear cover… and you’re done!

    Yeah, that took me a good hour and a half last night to figure out.

  • Adventures in Babysitting

    Not that one. Sandy and I took Raven off of Mike and Nicole’s hands for a night, to give them some time off. Here are some photos – cute little bugger, ain’t she? Sandy kept on saying that she is an extremely well-behaved baby, and at only two months she’s already making lots of trying-to-talk noises. We even got a semi-decent amount of sleep, for watching a baby at least. And yes, I did change a diaper.

    Still, it’s nice to be able to return the rental at the end of the day…

  • When someone inconveniences you, hit ’em right bacl

    I feel great. I just had someone’s car towed.

    I tried to give them the benefit of the doubt, but I’m starting to realize that living in the city teaches you to be harsh and swift.

    Last night we got home from seeing Erin Smith Band, and there was someone else’s car parked in our spot. It’s not the first time this has happened: the time before, it was a simple mix-up between permit holders, so it wasn’t a big huge deal. However, this car did not have a parking permit for our building, so this was a problem. We waited an hour and the car was still there.

    I should point out that I’ve never been in this position before: I was faced with either calling the police and having the offending hunk of metal tagged and towed or waiting for the people. Sandy and I talked/argued about it a bit and I decided that we would wait until the morning, simply because I didn’t think that parking enforcement would respond very quickly and I wanted to get some sleep (Sandy needed some rest too).

    So this morning (10:45am), lo and behold, the bloody car is still in our spot. Furthermore, our car, which was parked in the visitor’s parking, has a parking ticket. So I’m screaming blue bloody murder in the middle of the parking lot for a few seconds (this is Toronto, so it’s a regular occurance anyways) and then I drive Sandy to work and race home. Car is still there. Smoke comes out of Victor’s ears. Victor calls Toronto Police Services and they dispatch a parking officer.

    The fun ensues! 20 minutes later the lady officer shows up, I’m waiting outside because I really don’t want these people to leave. I show her the parking tag inside our car with our parking space # on it, that’s all the proof she needs. Writes the offending car a ticket, asks me if I’d like it towed (“Hell yeah!”), I sign the indemnity that this is my spot and this car shouldn’t be here, she calls in a tow truck.

    I will admit that I have never actually seen the entire process of a car being towed performed for me before, so I sat around and waited. I can now safely say that I will never put my car in danger of being towed, ever, EVER, EVER again. Tow truck operators are not gentle.

    All said and done it only took about an hour… I should have done it last night but I was just too forgiving. Not any more!

    PS: Sandy was right all along.

  • If you are running Windows right now…

    … I urge you to do these simple things right now to protect your machine:

    • RUN WINDOWS UPDATE. Install any and all critical updates. If necessary, install the latest service pack, reboot and re-run Windows Update. It’s so darn easy to do and yet people don’t know about it.
    • Get a decent piece of anti-virus software. Whatever came with your computer is probably crap, or its automatic update licence may have expired by now. Get yourself a copy of Norton AntiVirus or Mcafee Virusscan. Like Coke vs. Pepsi, the choice is yours.
    • Don’t use Outlook Express for e-mail. Outlook Express is like drinking through a straw with holes in it. Use Mozilla for browsing, mail, news, IRC, or if you don’t want to leave your beloved exploit-filled Internet Explorer behind, just install Mozilla Thunderbird for e-mail. Built-in free and effective spam filtering will be your reward.

    I’m writing this because I’m amazed at how many bloody unpatched machines there are out there. It just happens that I was up until 2am last night de-worming, cleaning, updating, and patching Sandy’s co-worker’s laptop. It had three different worms on it, all of them expoiting the DCOM RPC exploit from August. One of them was an IRC bot, so likely someone was remote-controlling their machine to do distributed denial-of-service attacks. Sigh. And of course they were running plain-vanilla XP Home connected directly to their DSL provider, with some very outdated virus software that came with their computer and had never been updated.

    On the upside, I now know how to defeat nefarious worms that frickin’ close windows while I’m trying to use them.

  • CDO follies continue…

    …and because of this, my application still doesn’t work. I want to know who wrote this library, I want to know their address, and I want a plane ticket to the closest city. And a baseball bat. Grrrrrrrrrr.

    (addenum: and as I find the open SMTP server on the internal net, I want to cry as I’ve just wasted 2.5 days of work.)