Force Firefox to not create urlclassifier3.sqlite when Safebrowsing is disabled

I choose to disable Safebrowsing, mainly because I don't want to pay the bandwidth/disk space/processing overhead when I feel I can make a good judgement of the sites I visit. BTW, I have it enabled for my wife's profile. This is turned off by unticking both "Block reported attack sites" & "Block reported web forgeries" options under the Security tab of Firefox's options.

Guesswork & ideas

Until recently, urlclassifier3.sqlite could be deleted & wouldn't be recreated, but for some reason it now is, maybe since Firefox 8??. It comes in a 5MB, which I think is a pre-allocate strategy to avoid fragmentation of the SQLite database, I don't believe it downloads Safebrowsing data when the above 2 options are unticked. Probably need submit a bugzilla bug report to see if it can be resolved.
I actually thought urlclassifier3.sqlite was replaced by a new custom binary storage file designed to make it possible for Firefox Mobile to use Safebrowsing (due to limited storage/memory). Ideally the Safebrowsing datastore would be outside the Firefox profile, and accessible by the user account so multiple profiles could share it. Even better if there was one datastore for the whole computer, updated with the new Mozilla Update Service that is being developed to avoid UAC on Vista+ for silent updates.
There are many reasons to avoid creating unnecessary files (particularly large ones), the ones I can think of off the top of my head are:

1. Takes up excess room on the HDD.
2. Takes up extra space on capacity/speed limited Flash drives with portable Firefox profiles (and may slow down startup & browsing if the file is read in - hopefully not).
3. Causes profile backups to be larger, therefore taking longer to backup/synchronise.

Solution

1. Navigate to you Firefox profile (which can be opened from about:support).
2. Make urlclassifier3.sqlite 0 bytes. Easiest way to do that in Windows is:
   
   1. delete urlclassifier3.sqlite
   2. create a blank text file, by right-clicking in folder, select New->Text Document
   3. rename New Text Document.txt to urlclassifier3.sqlite
3. Make the urlclassifier3.sqlite file readonly
   ->Right-click on file, select Properties, tick Read-only & click OK
4. Done!!!

Firefox 4.0 released

Mozilla released Firefox 4.0 today, with many UI changes, faster everything & improvements to open web technologies.

It appears a lot of early adopters have jumped onto Chrome, together with normal users from massive marketing efforts & other programs which install Chrome during their setup. I'm personally sticking to Firefox because: non-profit organisation with a open web vision, takes privacy seriously (the way Sync works means they never see my data, unlike Google which can see Chrome data - don't think this includes passwords though). Also, 99% of their decisions & planning is in the open, which is incredibly brave (since it opens them to continual criticism), shows how actual transparency works & lets normal people view how these processes/decisions are made.

The things that interest me for this release include:

• New UI which should allow more space for the page (will take a while to used to the changes)
• New Add-ons Manager
• Restartless Add-ons
• Inbuilt Sync
• Quicker startup times
• Hardware accelerated graphic layers drawing for video & pages
• Improved page load
• Faster JavaScript by using JägerMonkey
• Upgrades to DOM and Style Resolution performance
• Do Not Track feature

I won't be upgrading to Firefox 4.0 just yet. I'll wait a little while so my Add-ons are updated (is their an easy way to check?). Also, I have been using the same profile since ~Phoenix v0.3, so plan on doing some research about only transferring over needed user data & setting options (& Add-ons) from scratch. I have done a fair bit of customising, so it will take a bit of effort.

Update (2011-03-29): It took a far bit of yesterday to do it, but I'm now on Firefox 4.0. Used the list of important profile files from Mozilla Support: Recovering important data from an old profile to copy across to the new profile. Cookies wouldn't work at all, so had to export them from the Firefox 3.6 profile to cookies.txt (original Netscape/Mozilla format) using Cookies Exporter and import into the Firefox 4.0 profile using Cookie Importer, worked like a charm!! (better than my attempts to do in low level with sqlite3.exe and/or SQLite Manager).
All Add-ons had to be reinstalled, which is probably a good thing, but took a long time. The Lazarus form recovery database failed to transfer, had to reset. Greasemonkey & Stylish worked well once their data files/database was copied over.
Finally, I made sure preferences for Add-ons were transferred (manually) & I checked the profile prefs.js file to transfer any important hidden settings I had in my old profile.

Download Firefox 4.0 now.

Upgrading to Firefox 3.5 issues

Having recently upgraded to Firefox 3.5 from 3.1b3, I stumbled onto a few issues. In the 2 profiles we use, both showed a empty (blank) Options dialog box with only Ok & Cancel buttons. Also tabs behave strangely and won't close + web pages seemed strange, most had scrollbars on the left hand side.

Solution:

Tried a few different things, but deleting the localstore.rdf file from the profile directory (see How to find your profile) fixed everything. A bit like the Firefox Support article Toolbar keeps resetting and it's solution, Method 2: Delete localstore.rdf manually

Happy now :-)

Actual Solution:

Whoops, turns out my localstore.rdf solution was wrong :-(
It was because I forced SwitchProxy v1.4.1 (homepage & Mozilla Addons page) to be compatible with Firefox 3.5 when it is only claimed to be up to 3.0. Turns out it really was incompatible, who would of thought??

Fixing wrong date/times within a photo and setting the correct file time

I keep forgetting to correct the camera's time when changing back & forth from daylight savings. So a stack of my photo's are 1 hour behind. This is how I did it and present some of my research findings

Goals

My goal is to keep the photo byte-for-byte the same, except changing the embedded dates. This means I can easily verify that the process hasn't changed anything else (by doing a byte comparison). It also means backups are easier because a tiny binary change is required for each photo, not needing to do a full copy. Versioning systems that handle binary data like SVN, Git & Mercurial will find these changes very compact (making everyone happy :-).

1. Modify all the embedded date/times so they are moved forward 1 hour.
2. Change the file time of the photo to match the embedded Date Taken.

Programs used

• Exifier v2.1.5 (build 263) Graphical image manage & metadata editor. No updates since 2002!!
• Exiv2 v0.18 Console app, easy to use (nice syntax/manual)
• ExifTool v7.62 Console app, Perl based but compiled for Windows. Has a million options, but hard to find all the right options (a bit of a rabbits warren).

Fixing the EXIF date/time within a photo

Most photo's taken with a digital camera will have 3 date/time's in them.

• Date Taken
• Date Digitised
• Date Modified

Date Taken is the most important one, as that is what Windows Explorer and most Photo editors/viewers use. Usually they are all the same, unless the photo has been edited in some way.

Therefore, I'm trying to change all the embedded date/times by 1hour, relative to their original time. All these programs let you specify the relative time to alter.

This comes in handy if you notice that the camera is the wrong time, but not just a simple Daylight Savings difference of 1 hour.
Before you change the camera's time to the correct time, take a photo of a clock with the correct time. Ideally a GPS (which has to always have the correct time & sync's it with satellites) or something that shows seconds and has an accurate time. Then on the computer, look at the photo's time & the clock's time to work out the relative difference. You can then use this when fixing the time of photos.

Program	version	Byte for byte?	Time (min)
Exifer	2.1.5	Yes	2:121 (1:532)
Exiv2	0.18	Yes	3:27.31
	0.17	Yes	4:05.25
ExifTool	7.62	No3	3:52.22
	7.25	No3	3:34.58

Notes:
1. Wall clock time
2. Exifer preloads all the EXIF data when selecting a folder, which means a lot of the Images are in the OS filesystem cache (cheating, kind of :-).
3. Looks like there are sections of the EXIF header removed, as well as random bytes have changed (looks like it might still have the same information). But it makes it very hard to verify proper changes, and stuff's up byte diffs for backup/version control.

You can see that the Exiv2 guys have been working hard on performance, making some serious improvements between v0.17 & v0.18.

Setting the photo's file time to match the EXIF date/time

Since we have changed the embedded date/time, the date/time of the file will not match, and it makes it much more convenient if the photo's file time is correct.

Program	version		Time (seconds)
Exifer	2.1.54		531 (22)
Exiv2	0.18		7.02
	0.17.1		13.81
ExifTool	7.62		14.64
	7.25		15.16

Notes: 4. For some reason, Exifer creates a hidden "descript.ion" file that is empty in the folder. Can't find an option to turn that off, and there aren't many programs that use descript.ion anymore (used to be used in DOS days to enter a description for files within a directory, since images didn't always support some kind of embedded metadata).

Warning: NTFS formatted drives will show a different time when you change into Daylight Savings time, as NTFS saves file times in UTC (GMT) which is universal time and has not timezone offset. When your region moves into Daylight Savings time, your timezone offset from UTC changes, therefore Windows displays a different time. FAT32 stores only the local time, so it doesn't change with Daylight Savings changes. Camera memory cards (SD, Memory Stick, Compact Flash etc.) are Fat16 or Fat32 formatted, and some external backup drives are also FAT32 formatted. Assuming the file time has been set to the Date Taken time, these will always match.

Recommendations

1. For normal usage, use Exifer to altering EXIF date/times & make file times match.
   If wanting to automate (using batch files etc.), use Exiv2.
2. Keep all your photo's & any backups on the same formatted drive. I have everything on FAT32 formatted drives, but if you are going to be on NTFS for one, use NTFS for all (not typical with camera memory cards & USB falsh drives).

Test Specifications

• Acer Travelmate 3220 Notebook
• Windows XP SP3
• Intel Pentium M 2.26Ghz
• 2.26Ghz 1Gb
• Toshiba MK8025GAS HDD
  • 2.5" 80GB
  • 4,200rpm
  • 8MB Buffer
  • ATA-6 Interface
  • 12ms Average Seek Time
  • NTFS formatted
• Photo's taken with a Panasonic Lumix TZ-11 8Mpixel camera
  • All contained within a single folder
  • 251 photo's
  • Total size of 769MB (806,767,153 bytes)
  • Average size of 3.07MB (3,214,212 bytes)
• All devices deactivated where possible
  • Wireless LAN turned off
  • Bluetooth turned off
  • CardBus turned off
  • LAN turned off
  • Firewire (IEEE1394) turned off
  • Extrenal drives disconnected
• All programs shutdown where possible
  • Anti-virus turned off
  • Programs in system tray terminated
  • Explorer windows closed Had to retest when realised I had explorer opened on test directory, meaning Windows was updating the view during the test
• Between test, Operating System File cache cleared by copying very large files (is there a better way/utility to do this for you?

Test commands

Exifer

Fix wrong time

• Select images as needed (can show sub-folders as well)
• "Edit EXIF/IPTC data" (Ctrl+E)
• Select "EXIF data" tab
• Select "Date" sub-tab
• Tick [x] Apply to Date modified and Date digitized
• Adjust "Date/Time offset" settings as needed

Set file date/time to match EXIF date

• Select images as needed (can show sub-folders as well)
• "Rename/redate images" (Ctrl+N)
• Untick [ ] Rename
• Tick [x] Redate (by EXIF date fields)

Note: Sets Fat32 2sec accuracy time on NTFS

Exiv2

Note: For testing Exiv2 & ExifTool, I wrapped the console commands in echo %time% to find out the elapsed time to run (I'm sure there's a nice chuck of DOS batch code that even does the duration time calculation for you).

Fix wrong time

exiv2.exe ad -a 1 -k *.jpg
Note:

• ad -a 1 adds 1 hour to all 3 EXIF times (can use 1:03; -0:34.57 etc.)
• -k keeps current file time

Set file date/time to match EXIF date

exiv2.exe mv -T *.jpg
Note: Sets 1sec accuracy time on NTFS
Manual talks about having to set TZ enviroment variable, but seems to work fine without...

ExifTool

Fix wrong time

exiftool.exe -overwrite_original -P -AllDates+=1 ./*.jpg
Note:

• -AllDates+=1 adds 1 hour to all 3 EXIF times
• -P keeps current file time
• -overwrite_original doesn't create a backup copy of the image

Set file date/time to match EXIF date

exiftool "-DateTimeOriginal>FileModifyDate" ./*.jpg
Note: Sets 1sec accuracy time on NTFS

Please suggest any other programs or configurations to try...

Performance of copying files to an external drive

Did some tests to see if the claim of the "fast copying" programs are true. Below is the results I got from copying a Firefox profile from a laptop hard drive (NTFS) to an external USB2 hard drive (WD Mybook Home 1TB as FAT32) on Windows XP SP3. The Firefox profile was 18.6MB (19,552,413 bytes), 630 files & 174 folders. This contains lots of small files, which usually hurt performance when copying & deleting. I copied 5x100MB files from the laptop to the external drive to make sure there is nothing in the drive buffer (which only takes about 10 seconds), to make the results fair. The WD Mybook Home has Firewire400 & eSATA ports as well, but I didn't try those (but I have done some informal testing in the past, and USB2 & Firewire seemed to be pretty equal.)

The Results:

Program	version	Time	Maintain folder date	Free?	Notes
Beyond Compare	v2.4.3	209.8s	Yes	No
FastCopy	v1.91	96.1s	Yes	Yes	default 32MB buffer
TeraCopy	v1.22	426s	No	Yes	default (to another HD)256KB buffer
WinXP explorer copy		97s (stopwatch)	No	Yes	Ext. HD still spinned for a while afterwards

Based on the results, FastCopy is my new "copy bulk amount of files" program of choice.
Please suggest any other programs or configurations to try...

First update on mapping OpenStreetMap (long overdue)

Australia » Victoria » Melbourne
I signed up for OpenStreetMap about a year ago, and there were plenty of warnings that it's very addictive, but little did I know!!!
I do mapping in spits & spurts (like a lot of other people I'd imagine).

Progress

Turns out I have done a fair bit since I signed up:

Looks like I'm the biggest contributor in greater Melbourne, scary stuff.

I've traced streets, added a lot of names and added some POI's like parks, foot/bike paths, pubs, schools etc.

Rough order of Suburbs I've done:

• Essendon
• Moonee Ponds
• Aberfeldie
• Strathmore
• Ascot Vale
• Flemington
• Travancore
• Kensington
• West Melbourne (industrial area)
• Strathmore Heights
• Gowanbrae
• Airport West
• Keilor East
• Avondale Heights
• Maribyrnong
• Maidstone
• West Footscray
• Footscray
• Braybrook
• North Sunshine
• Sunshine
• Keilor Park
• Tullamarine

Since I don't have a GPS, I trace from Yahoo! imagery and fill in as many details as possible.

My usual steps:

1. Trace streets, parks, foot/bike paths from Yahoo! imagery
   • Add as many tags as is obvious, leave alone if in doubt
     • eg. highway=residential, source=yahoo
2. Ride along "main" roads to pick up all the side street names, scribbling on my Palm T3 Notepad application (exports to PNG)
   • Tag as name=blah, source:name=survey
   • Later on use Maplint or NoName layers to spot missing street names
     • Plan another ride that swings past all remaining streets
3. Walk or ride down *every* street, after printing off a Mapnik layer map of the area
   • Note down:
     • All street names/references/lanes/bike lanes/speed limits
     • Schools, park, shopping centre names and shapes (to refine the rough tracing from Yahoo!
   • This takes a long time, and I've only done a little bit so far, but it does make the area almost 100% complete

Wishlist

1. Either Yahoo! updates their Melbourne satellite photo's (~5 years old) and provide higher zoom levels (currently level 16) or Google provides their photo's to OSM
2. Potlatch is my editor of choice, and it has improved a lot since I started using it
   • Zoom further in by scaling the Yahoo! tiles when hitting the highest zoom Y! provides
     • Potlatch used to do this, which made it much faster to trace, but a Flash bug that occurs when the auto-maximise feature was added means this had to be turned off :(
   • More tag "autocomplete" entries (especially source:name=survey/photo etc.)
   • When crossing a way with another (either as a T-intersection or cross), I think the snapping feature of Potlatch means the created node is not where the two ways mathematically cross, but the nearest snapped point
     • So the angles of ways change a bit, needing zooming in to fine tune
       • But snapping is very obvious at z18, which becomes dodgy when ways are close to (but not) orthogonal
     • I usually layout a grid of streets with a little overlap, then come through and join all crossing ways, crop T-intersections overhang and fine tune
3. Get a GPS and:
   • Circumnavigate local parks etc. which are hard to see the boundry from Yahoo! imagery because of trees
   • Enter POI points like postboxes, speedhumps, changes of speedlimits
   • Take photos as I go of all the street signs, amenity names etc. which, with a GPS, can be geotagged (and upgrade name's to name:source=photo)

Ownership

I've noticed that there is a fair amount of pride/possessiveness around the contribution that OSM users have made, and I'm no different.

I get annoyed when other users come through and add inappropriate tags (not what was on the ground, "for the renderer", not "best-practise" etc.), move carefully smoothed ways around/offset. I guess the "pride" of one's contribution is a double edged sword.

Good tools for letting me know when someone has editing "my areas" would be fantastic, and I've just signed up for Itoworld's osmmapper (RSS feeds of changes to specific areas). But I think it's hard to communicate what has changed, how do you convey changes in a concise/meaningful way?

• eg. Mary St is now oneway & nodes moved; Frank Rd deleted; Railway Hotel pub added+tagged opening hours; etc.

Crossposted at OSM diary

Palm T3 Documents To Go problem

I sync my Palm T3 on 2 computers, which sometimes cause problems after I've had a hard reset (a complete data wipe). This time it was because it ran out of battery juice (I suspect - a button was held pressed when packed in a bag).

I mostly fixed all the issues (after having to manually hard reset again after not entering the correct time/date - stuffs up the forced sync for some reason), but Documents To Go (v6.008) sometimes gets confused about syncing existing documents.

The error message I see is:

Another file exists with the same name as this one...

But there is no documents with the same name!!!
DataViz support had nothing in their Knowledge Base. I have seen this with both "Sheets to Go" and "Microsoft Excel" formats.

I tried everything to fix this, but was left with the poor option of just changing the name of the document to something unique. But today I found a solution

Using either Filez or Resco Explorer, I deleted all of the Documents To Go Palm databases. I think that DocsToGoDB.DATA is the one that solved the problem, hard to tell. It did cause all the categories and their assignments to disappear, but easy for me to add again. So I deleted:

• DocsToGoDB.DATA
• DxtgTempDB.TDAT
• (in Resco Explorer) all Documents [DTGP,xx].pref "files"

I was then magically able to see the "existing" files of the same name, delete them, rename each document to their "proper" name, assign a category and sync properly!!

Firefox 3.0 released, download now!!

The Mozilla corporation/community have released Firefox 3.0 today, with many improvements (both UI wise and backend changes).

Similar to Firefox 2.0 & 1.5, this release has had some serious amount of QA pounding on it, I would say it is their most planned & executed release management to date.

The things that interest me for this release include:

• Awesome bar (smart location bar)
• one-click bookmarking
• Tagging history/bookmarks
• Better password manager (not getting in your face)
• New EV certificates (better identification of secure sites)
• Add-ons Manager with inbuilt Addons listing
• Download Manager improvements (mainly resumable across sessions, more robust backend)
• Full Zoom (images & text zoom together)
• Improved Memory Management (cycle collector, reduced leaks, jemalloc allocator to use less memory)
• Faster JavaScript
• Faster Page Load
• Advanced Graphics (reflow rewrite, better SVG/Canvas support, better graphics backend (Cairo))
• Future Weave sync'ing

I only started using 3.0 with the release candidates (RC1), because of the "addons not yet updated" problem. But I found updates for my day-today addons during RC1, so upgraded.

Finally Firefox 3.0 has places support, which I think will make a real difference to users web browsing experience (mainly through the "Awesome bar"). Here's my anticipation from earlier:

When Firefox 1.5 came out, I was already excited about 2.0, what with Places & the SQLite backends, but that didn't really pan out, should be in Firefox 3.0 though...

Firefox 1.5 released today
... Already waiting for Firefox 2.0 (from the 1.8.1 Gecko branch) with its "Places" support (redoing bookmarks, history, feeds etc. into a new user interface and storing in a super fast SQLite DB).

Firefox 3.0 will come off the 1.9 branch, with it's massive Gecko improvements (rewrite rendering using Cairo on all platforms, reflow rewritten, complete SVG support??) and the aforementioned Places with tagging system for bookmarks/history etc.

Download Firefox 3.0 now.
Hopefully download before 3am, 19th June AEST to create a new Guinness world record for the most downloads of a software program in a day!!

Be careful of wet newspaper!

Riding into work today, I got about half way in before noticing that my rear shifting wasn't working very well. Looking down, it looked like a rag had got caught in the jockey wheels/cassette.

So I stopped on the Moonee Ponds Creek bridge and had a look. It turned out to be a single sheet of damp newspaper, all caught up and mashed around the jockey wheels & cassette & under the chain.

Cleaning it out was difficult because it was caught up under the chain, and that it disintegrated when picking it out. Took about 5 minutes to get it out but then I noticed that the cassette, the paper had somehow loosened the locknut on the cassette and the cassette & the separate sprockets were very loose. Tried finger tightening the locknut, but that only worked a little bit, it was still loose.

Seems that the paper must of compressed on each rotation, in the gap between the locknut & chainstays. And since the chainstays are "going backwards" in relation to the clockwise drive of the cassette when pedalling, it unscrewed the locknut!! Who would of guessed, I'm going to make it nice & tight when I get back home to my cassette locknut spanner.

PS. I've heard some horror stories of plastic shopping bags wrecking the rear derailleur, so I got away pretty cheaply, especially since I just replaced my Shimano Deore 2001 rear derailleur with a SRAM X.9 2007 one.

Firefox Mouse Gestures (Optimoz extension): Snapback gesture

It seems the Optimoz gesture exchange page is down. But the project page on Mozdev seems to be up!!

matt added a comment (well 3) to the Firefox Extensions I use (2007-08-09) post, asking for it since the page is down and googling didn't find anything, so here it is:

1. In Firefox, goto Add-ons -> Mouse Gestures -> Options -> "Edit Gestures" button (at the bottom of the General tab, and click New.
2. In the Gesture code textfield, enter: LR
3. For Function Type, select: Custom
4. In Gesture Name textfield, enter: Go to first page
5. In Custom JavaScript code textarea, enter: if (gBrowser.sessionHistory.index > 0) gBrowser.gotoIndex(0);

Now, you only have to hold down the Right mouse key, and flick the mouse to the Left and then back to the Right!!