This post is really a note-to-self for when I next have to remember how to deal with missing photo and QuickTime movie metadata. Nevertheless, since it took me a little while to work out, maybe someone else searching for the same issue will benefit from it. If you’re short on time the ExifTool command that helped me is right at the end – skip down there and copy and paste.
For anyone else who can’t sleep, it might be useful, but I won’t be offended if you skip reading this.
The photo management problem
Like many people, my photo libraries have grown to many gigabytes over the years, encrusted with cruft from various photo management apps. Multiple versions of iPhoto and Lightroom, not to mention a few corrupt libraries and recoveries have left their scars. Since I had an iPhone, everything got worse. Do I use iCloud, Dropbox or iPhoto? I used to religiously use Lightroom, but it was always a drag importing photos from my iPhone into Lightroom and sorting them. Because it was a drag, I didn’t do it all that often and I lost track of which photos were where.
Of all things that would be hard to replace if all my devices caught fire and all my backups failed, the photos would be the one I care about most. Almost everything else is either replaceable, recoverable or possible to take a Zen-like attitude to letting go of. But I would be very sad to lose the photos of my daughter when she was tiny.
I tried various automatic services, such as Dropbox’s automated Camera Upload feature and I also tried Everpix before they shut down and then moved to Loom, but then were acquired by Dropbox. Back then, the whole point of using another service was because I didn’t have enough space on Dropbox for over 72GB of images and videos. Since Dropbox upgraded their plans, I know have the opposite problem, which is that my 1.1TB of cloud space is larger than my laptop’s hard drive.
Having tried all these services on and off, I could no longer remember which photos I had backed up from my iPhone and where. My iPhone was getting too full and I was nervous to delete older images in case that was the only copy.
I decided to follow most of Federico Viticci’s photo workflow and go for near Camera Roll Zero. I use Camera Sync on the iPhone to upload all my photos to Dropbox and use Hazel to rename and sort them based on their metadata. Once uploaded I keep a handful of photos of my family on my iPhone, but otherwise delete all the others.
Similar to Federico, my Hazel rules delete screenshots, since I generally don’t want to keep them, but I also have Camera Sync set up not to upload screenshots anyway:
Then I have a rule to sort the photos into date based folders, by year and then simply date. The files are also renamed with date and time, such as “2015-01-02 3.08.29.jpg”:
Videos are renamed and sorted into a video sub-folder of each year:
Deadly exciting, I know, but I’ve wasted so much time searching for photos and so much drive space with duplicates that the time spent getting this to work has been fantastic for me. I usually have a pretty good memory of roughly when important events happened in life and there are plenty of ways to view folders of images as thumbnails (see Federico’s article for a round-up of these). Duplicates are now easily spotted, because they are in the same folder with the same filename (a number is appended if filenames are identical).
The stripped EXIF data problem
This approach has one major drawback, enforced upon it by other apps. It relies on accurate file creation dates and EXIF data – the metadata stored in all image files. Sometimes this data is missing or incorrect.
My Hazel rules sort files based on their creation dates. One of the previous photo storage cloud services – either Loom or Everpix (the culprit, I think) – re-stamped the file creation dates. This meant I had a bunch of different images all with the same creation dates and, thus, all the same filename if I ran my Hazel rules on them.
I thought I could probably recover the proper dates by using the EXIF data. There are apps to view and extract this and in fact Hazel can examine some EXIF data in its “other” settings:
But I needed something more powerful and Phil Harvey’s excellent ExifTool came to the rescue. With it, recreating the correct file-creation date based on the EXIF data is trivial.
The command:
exiftool -a /Users/andypolaine/Dropbox/Camera\ Uploads/2015/2015-01-27/2015-01-27\ 9.38.57.jpg
gives you this enormous output:
ExifTool Version Number : 9.70
File Name : 2015-01-27 9.38.57.jpg
Directory : /Users/andypolaine/Dropbox/Camera Uploads/2015/2015-01-27
File Size : 1167 kB
File Modification Date/Time : 2015:01:27 09:38:57+01:00
File Access Date/Time : 2015:01:30 11:41:54+01:00
File Inode Change Date/Time : 2015:01:27 15:22:41+01:00
File Permissions : rw-r--r--
File Type : JPEG
MIME Type : image/jpeg
Exif Byte Order : Big-endian (Motorola, MM)
Make : Apple
Camera Model Name : iPhone 6
Orientation : Horizontal (normal)
X Resolution : 72
Y Resolution : 72
Resolution Unit : inches
Software : 8.1.2
Modify Date : 2015:01:27 09:38:57
Y Cb Cr Positioning : Centered
Exposure Time : 1/618
F Number : 2.2
Exposure Program : Program AE
ISO : 32
Exif Version : 0221
Date/Time Original : 2015:01:27 09:38:57
Create Date : 2015:01:27 09:38:57
Components Configuration : Y, Cb, Cr, -
Shutter Speed Value : 1/618
Aperture Value : 2.2
Brightness Value : 9.203691496
Exposure Compensation : 0
Metering Mode : Multi-segment
Flash : Auto, Did not fire
Focal Length : 4.2 mm
Subject Area : 1631 1223 1795 1077
Run Time Flags : Valid
Run Time Value : 169926077986458
Run Time Epoch : 0
Run Time Scale : 1000000000
Sub Sec Time Original : 210
Sub Sec Time Digitized : 210
Flashpix Version : 0100
Color Space : sRGB
Exif Image Width : 3264
Exif Image Height : 2448
Sensing Method : One-chip color area
Scene Type : Directly photographed
Exposure Mode : Auto
White Balance : Auto
Focal Length In 35mm Format : 29 mm
Scene Capture Type : Standard
Lens Info : 4.15mm f/2.2
Lens Make : Apple
Lens Model : iPhone 6 back camera 4.15mm f/2.2
GPS Latitude Ref : North
GPS Latitude : 47 deg 17' 16.46"
GPS Longitude Ref : East
GPS Longitude : 7 deg 56' 36.27"
GPS Altitude Ref : Above Sea Level
GPS Altitude : 432.9546539 m
GPS Time Stamp : 08:38:46.72
GPS Speed Ref : km/h
GPS Speed : 0
GPS Img Direction Ref : True North
GPS Img Direction : 81.40054496
GPS Dest Bearing Ref : True North
GPS Dest Bearing : 261.400545
GPS Date Stamp : 2015:01:27
Compression : JPEG (old-style)
X Resolution : 72
Y Resolution : 72
Resolution Unit : inches
Thumbnail Offset : 1980
Thumbnail Length : 5024
Image Width : 3264
Image Height : 2448
Encoding Process : Baseline DCT, Huffman coding
Bits Per Sample : 8
Color Components : 3
Y Cb Cr Sub Sampling : YCbCr4:2:0 (2 2)
Aperture : 2.2
GPS Altitude : 432.9 m Above Sea Level
GPS Date/Time : 2015:01:27 08:38:46.72Z
GPS Latitude : 47 deg 17' 16.46" N
GPS Longitude : 7 deg 56' 36.27" E
GPS Position : 47 deg 17' 16.46" N, 7 deg 56' 36.27" E
Image Size : 3264x2448
Run Time Since Power Up : 1 days 23:12:06
Scale Factor To 35 mm Equivalent: 7.0
Shutter Speed : 1/618
Create Date : 2015:01:27 09:38:57.210
Date/Time Original : 2015:01:27 09:38:57.210
Thumbnail Image : (Binary data 5024 bytes, use -b option to extract)
Circle Of Confusion : 0.004 mm
Field Of View : 63.7 deg
Focal Length : 4.2 mm (35 mm equivalent: 29.0 mm)
Hyperfocal Distance : 1.82 m
Light Value : 13.2
If you pop that GPS data into Google Maps, you’ll see I took this photo
when I was on the train in Switzerland:
(Actually the GPS data seems slightly off, because I was already about 200m past that position when I took the photo. I can only assume a laggy GPS location, since the signal is pretty bad and the train is moving fast. You can also see why snooping governments’ claims of “we only want to look at the metadata” is such a load of nonsense.)
You will also notice there are several tags that have a date stamp. There is a Date/Time Original or the GPS Date/Time (they’re different because of Daylight Savings time) plus the File Modification Date/Time stamps. (My favourite EXIF tag is “Circle Of Confusion” – sounds like my life.)
I ran ExifTool on my problem images and restamped all the File Creation Dates using the EXIF data.
WhatsApp with your EXIF data?
This was all good until I found a whole load of photos all time-stamped with 1999-11-30 12.00.00 and heading into a November 1999 folder. These turned out to be mostly images saved from people’s tweets or from WhatsApp. I’m not sure about Twitter, but WhatsApp deliberately strips the EXIF data from images presumably as a privacy measure, unless it’s just a bug/feature in the app or iOS. iOS 8’s new photo editing extension that lets third-party apps edit photos directly also strips the EXIF data.
An image sent via WhatsApp yields only this:
File Name : 1999-11-30 12.00.00-110.jpg
Directory : /Users/andypolaine/Dropbox/Camera Uploads/1999/1999-11-30
File Size : 101 kB
File Modification Date/Time : 1999:11:30 00:00:00+01:00
File Access Date/Time : 2015:01:30 12:01:04+01:00
File Inode Change Date/Time : 2015:01:28 09:18:22+01:00
File Permissions : rw-r--r--
File Type : JPEG
MIME Type : image/jpeg
Exif Byte Order : Big-endian (Motorola, MM)
Orientation : Horizontal (normal)
X Resolution : 72
Y Resolution : 72
Resolution Unit : inches
Y Cb Cr Positioning : Centered
Exif Version : 0221
Components Configuration : Y, Cb, Cr, -
Flashpix Version : 0100
Color Space : sRGB
Exif Image Width : 960
Exif Image Height : 637
Scene Capture Type : Standard
Compression : JPEG (old-style)
X Resolution : 72
Y Resolution : 72
Resolution Unit : inches
Thumbnail Offset : 298
Thumbnail Length : 8676
Current IPTC Digest : d41d8cd98f00b204e9800998ecf8427e
IPTC Digest : d41d8cd98f00b204e9800998ecf8427e
Image Width : 960
Image Height : 637
Encoding Process : Baseline DCT, Huffman coding
Bits Per Sample : 8
Color Components : 3
Y Cb Cr Sub Sampling : YCbCr4:2:0 (2 2)
Image Size : 960x637
Thumbnail Image : (Binary data 8676 bytes, use -b option to extract)
(This is post-Hazel renaming, so the filename and File Modification and Inode dates are based on when I uploaded the files to Dropbox, not the real ones of when the image was originally created.)
The lack of this EXIF data is an enormous pain, since a common practice when getting together with friends is to set up a WhatsApp group and share all the photos with each other that way. Of course my friends look at me blankly if I ask them to send the photos another way “because I need the EXIF data.” So, unfortunately all those image end up in 1999 and I have to manually sort them out of there again.
The only salvation would be to upload them to Dropbox on the day of the event and use that date, but that defeats the point of the whole exercise. If anyone knows a way around this, please let me know.
The video metadata problem
The other problem I had was with a bunch of QuickTime movies I had shot with my iPhone. I don’t know which app (Everpix?) screwed these file creation dates up. This time the Creation Date was 1st January 1970 on all of them.
I often used QuickTime Player 7 for transcribing because it has better controls (jog shuttle, playback speed, etc.) that Apple removed from later versions. I have an old QuickTime Pro licence too, which allows you to see individual track data and manipulate them:
I noticed there was a Track Creation Date annotation in there, which hadn’t been stripped out and had the correct capture date. Fortunately, ExifTool can handle videos too and extracting the EXIF data on the video gave me these tags (among many others):
Track Create Date : 2011:02:27 12:11:11
Track Modify Date : 2011:02:27 12:11:24
I could use one of these to set the rest of the metadata of the file. The following command extracts the Track Creation Date and writes it as the File Creation Date and also renames the file to match (and appends a lowercase file extension):
exiftool '-FileModifyDate<TrackCreateDate' '-FileName<TrackCreateDate' -d %Y-%m-%d_%H.%M.%S.%%e
This left me with a movie file that Hazel could correctly sort. I think “2011-02-27 1.11.11.mov” might possibly have the wrong time (or miscalculate the GMT offset), but the date is certainly correct. It’s good enough for me.
You can run ExifTool commands on individual files (essential when testing), but also on directories of files and it all happens pretty quick.
Now if only I could do the same thing to my brain.