Archive for January, 2012

Jan 21 2012

RDF Nuts and Bolts or Get me a “LASER”

Published by under Uncategorized

I’m obsessing about a talk I’m giving at one of my favorite conferences, code4lib.  My talk proposal is about how we deal with whatever metadata comes our way.  For those of you not inside my head at this moment, “we” is where I work, part of which is in developing software for and maintaining a Digital Asset Management System (DAMS).  A digit asset is just a computer file, or set of files – often a picture, sound file, PDF, or video – that you have some desire to promote beyond just sitting on one person’s computer, unmanaged.  We all have computer files coming out of our ears, but we know there are some that are more “valuable” than others that we’d like to give special treatment.  So we call them digital assets and get then moved into some sort of management system beyond the random file systems on our desktops.  This system is a DAMS.

I see a DAMS as a secure, reliable file system set up with good organization rules, and a goal of making the assets easy to find.  Here are some of the rules I like to see followed:

Good Organization

  • Describe the assets as thoroughly as possible and/or practicable.  This is your metadata.
  • Save this metadata and keep it associated with the assets.
  • Keep the metadata in a system that is flexible enough to be able to accomodate attributes that you didn’t know about when you started – people change their minds about what they want all the time.  We chose RDF for this and this is what my talk will be about.
  • Create unique identifiers for the assets.  We chose the ARK spec for this.

Reliability

  • Store the assets as simply as possible – don’t create a new file system because no one is going to understand it 30 years from now.  We chose the Pairtree spec for this.
  • Back your assets up – think like an IT person and have a data lifecycle plan.
  • Preserve your assets and metadata – think like a Librarian/Archivist and store the assets in at least three, geographically separated places.  We use Chronopolis for this.
  • Export your metadata to the file system on a regular basis – this way the metadata becomes a digital asset/computer file too.

Discoverability

  • Put all that lovely metadata into a full text search index.  We use solr for this.
  • Tie all of the metadata to the unique identifier for the assets.

 

As I noted above, I’m only talking about RDF and metadata at code4lib.  What I’m obsessing about is that the talk is only 20 minutes.  I usually talk about our DAMS in about an hour, and I’m only getting warmed up in the first 20 minutes.  So I’ve got to empty my head of all this other DAMS stuff and laser down on just the RDF and metadata part.

We didn’t choose RDF because of the newish Linked Open Data (LOD) movement. Our (now retired) architect, Chris Frymann, was aware of the possibility but this was nearly ten year ago and LOD was barely a twinkle on the horizon.  Previous to this job, I had been working in industry for years, so this approach looked silly and academic.  Once Chris had me drink the RDF Kool-Aid, we envisioned a system that embraced flexibility from the start.

RDF is so simple and so terrifyingly different from the fixed database world that I was used to.  Instead of a well defined table, or tables, we had millions of triples.  We didn’t even have a triple store, just three columns in an SQL database.

What is wonderful about this approach was that each triple is somewhat self documenting.  A triple is made up of a Subject, a Predicate, and an Object:

 

RDF Triple

 

We use our asset’s unique identifier, the ARK, as our Subject.  Now we needed to describe the assets with their metadata – so we started creating Predicates that could hold types of metadata.  Three years later…. no really.  This was probably one of the hardest things to do, and I’m not sure we’ll ever stop doing it.  Some of our original assets had MARC records, and there were ways to convert MARC to RDF.  Lots of deep discussions among metadata librarians, asset owner librarians, and the tech folks came to the conclusion that we wanted to cast our metadata into specific namespaces, namely MODS, PREMIS, and MIX.  This was way beyond the Dublin Core defaults that other products were using, but we knew RDF was flexible enough to accomodate just about anything, so we just did it.

Guided by the head of our Metadata Analysis and Specification Unit (MASU – lots of great detail at that link), Brad Westbrook, we started specing out what the metadata needs were for each asset.  Ok, that’s a lie… We did it per “collection” which was how we actually received assets from the librarians.  Our DAMS works at the asset level, but our librarians normally think at the collection level.  This was just another layer of translation that the MASU group stepped up to play a liaison role in getting the assets ingested.  Over time, this became a workflow where:

  1. Collections are identified/approved for ingestion into the DAMS.  (The project management and institutional buy in on this process is another talk!)
  2. MASU creates an Assembly Plan that maps the assets into collections and specifies what namespaces the metadata pieces are placed in and hand it off to IT Development
  3. IT Development creates mapping scripts from what the Assembly Plan calls for into RDF.  This is done in XSL.
  4. IT Development ingests the assets into the storage system and parses the metadata into RDF in the triple store.
  5. Profit.

Ok, someone tell me how to get all that into 20 minutes… 😉  The Assemble Plan alone is an intense spreadsheet and text document that explains what is needed.  Then the translation scripts are another challenge to present without everyone going cross eyed.  Not to mention this thing of beauty!

DAMS RDF Graph

That’s all of the metadata and relationships of one asset.  Maybe I’ll just put that on the screen and take questions for 20 minutes… 😉

5 responses so far

Jan 10 2012

The Missing/Stolen/Returned iPad Saga

Saturday 12/31/11

So, I have an iPad – 64G, 3G, ATT.  It’s a work device that I use all the time, especially for travel and in the gym so I have something to distract me from 40 minutes of hell on the elliptical.  Over the holiday break, I committed to getting into the gym every other day.  Our YMCA was open until 4p on New Year’s Eve, and I was sweating my fat butt off right up until the place closed.  I took a quick stop in the men’s room, and hustled out the door as they were closing up.

Sunday 1/1/12

It was later on the next day that I realized that I couldn’t find my iPad.  I looked in all the usual places, scoured the house and car, and tried out the “Find My iPad” app on icloud.  Once it didn’t show up, I realized that I must have left it in the men’s room at the Y!    Well, the Y was closed all day, so I left voicemail there and told myself to get to bed early so I could hit their door at the 5:30am opening on 1/2.  I also changed all my passwords, and deactivated the device from all the important services.

Monday 1/2/12

I actually went to bed at 1am (damn Skyrim), but was up and out the door at 5:20a and at the Y at 5:30a.  My buddy Damon was also just pulling up for a very early swim class, because he is insane.

We walked in and I asked at the front desk if the iPad had been turned in, but nope.  I wasn’t surprised, as they’d been closed up solid since I left it, so I walked over to check the men’s room.  Uh oh, no iPad…  I went back to the front desk and I filled out a slip saying what I’d lost and the nice young lady said they’d talk to more staff as they came in later.  In fact it might take until tomorrow as there were people still out on the the holiday break.  I was upset, but mostly at myself for leaving it behind. I kept trying the “Find My iPad” app, but nothing.  I saw there was a way to send a message to the iPad, so I put my phone number in there.  I also noticed a switch that would send me an email when it was found – so I hit that too.  I printed some LOST IPAD signs, and went back and stuck them up around the Y.  Worried, but feeling like I’d done all I could, I went about my day, keeping my eye on my email and occasionally trying the app again.

Then at 9:29p:

Woohoo!  Someone at the Y found it and turned it on, right?!?  I ran to the computer and hit the location link, expecting to see a map of my neighborhood and the Y.  Instead, the device is showing up about 24.4 miles to the south, in a town called El Cajon!

I have a screen grab of the map with the green dot locating the iPad right on a specific address, but I’m not sure of I should post that.  I’ll get into that more later.  Basically, I KNEW where my iPad was, and it was nowhere NEAR the place I’d lost it.  Suddenly I had a theft situation rather than a dummy-lost-his-iPad situation.  So I did what a normal person would do and called the police.

I called the San Diego Police non-emergency line and explained the story so far to what sounded like a very overworked lady.  She asked if there was a police report, and I said no because I just now discovered that it was stolen.  She directed me to a web site to fill out a form to get the police report started.  I got a little panicked and said “but I can see it now!  Why can’t we go get it?!” to which I got more of a push toward the web site to fill out a report.  I’m sure those of you who know I run an IT shop and highly promote a central ticketing system are just laughing your heads off at this delay… but, to my mind, time was of the essence!  I pressed harder and she said she could give me the El Cajon Police Dispatch number, but wasn’t sure what they might do for me.

So I called them and they listened to the story, then asked for a police report.  Le Sigh – I don’t have one because… blah blah – go fill one out and call us back.  Ok, back to the web site, filled it out, grumbled about UI design, got the confirmation email and number, and called back El Cajon.  They said come on down and we’ll have a squad car meet you by the address and do a “Peace Call.”  It’s 10:30p by now, and I’m all amped up from the situation, so Elaine says she’s going with me.  We hop in the car and head south for the 20-30 minute drive.

On the drive, Elaine has the “Find My iPad” app going on my iPhone, but it goes red, meaning that the device is no longer on.  We’re committed tho, so we keep going – at or below the speed limit, of course.  We get to the arranged meeting spot at 11p, on the same block as the address that showed up on my map, and wait.  In a little bit of a sketchy looking place.  Surrounded by suspected iPad stealers!  At about 11:40p I call back to dispatch and they say no one is available, but they know we’re waiting.  About five minutes later, 2 squad cars pull up and I get to tell the story again.

The cops were great, and offered to go knocking on doors, but that address is a 50 unit apartment complex.  Given that GPS can be +/- 25m, the dot on my map could encompass 6-8 apartment units in the 2 story building.  Added to that, the building is “known cop-unfriendly” and not a nice place.  As soon as they started asking questions, everyone would know about it, and I’d probably never see my device again.  And since the device was no longer responding to the app, I couldn’t make it sound off to make it easier to find.

At this point, I was frustrated to know that I was within a block of my iPad, but could do nothing about it.  The cops offered a few suggestions, such as sending a message to the unit offering a reward, and to watch Craigslist, – something I should have thought of myself!  They were both adamant that I NOT meet with anyone from Craigslist without the police.  They told me of someone who’d been killed in San Diego not long ago after confronting a Craigslist person.  They also suggested sharing the address with the people at the Y to see if an employee or member matched.  We drove back home, defeated and deflated.  I sent a lock command to the device, setting a password and with the reward message,  then we headed to bed.

Tuesday 1/3/12

On the way to work on Tuesday, we stopped into the Y and talked with a manager, telling her the whole story.  She happened to also be the HR person and said they had no one working in El Cajon, but that they contracted out their cleaning and they’d follow up with them.  I then headed into work and one of my guys suggested that I send a wipe command from our email server.  Even though I’d changed my email password, I thought it wouldn’t be a bad idea to also wipe my work email from it, so I did.  More on that later…

Wednesday 1/4/12

In the meantime, I’d been watching Craigslist, and got a match!  64G iPad, 3G, ATT!  So I set up a fake gmail account and mailed the Craigslist email address.  I also mailed one of the cops for advice about how to deal with the mails I would send, and he got back to me with some thoughts.  I got a response back soon after I went to bed from the Craigslist person, asking how close I was to his asking price, and if I could come get it now.

The facilities manager at the Y called me on Wednesday, very upset about the whole situation, and said he was following up with the cleaning people.  I later ran into him when I was working out and he was very nice and concerned that the cleaning company hadn’t gotten back to him.  I could really tell he cared personally and as a representative of the Y.

Thursday 1/5/12

The Craigslist response back was from a hotmail account with a person’s name, so I passed that name on to the cops in email and with a call to the Y facilities manager.  It was probably a made up account, but it was worth trying.

I had to run out of town for a day for work, so I didn’t reply back to the Craigslist person until tonight, asking for more details about the iPad he was selling – as advised by the cop.   I got back a very short reply saying “Too bad, I sold it for $200” and that was it.  I mailed back thanking him for the info, thinking that I might be able to squeeze more info later by being nice.

Friday 1/6/12

So, now we’re up to Friday, and nothing seems to be happening.

Saturday 1/7/12

On Saturday morning the phone rings at 9am.  It was a lady from the Y asking if I’d lost an iPad, because they had it at the front desk!  I said “very interesting!” and that I’d be in to pick it up.  I went right over and it was my iPad, grey cover and all!  I asked the ladies at the desk if they knew more about where it came from.  One of them said that she’d closed the night before and that it wasn’t there, but when she’d opened this morning, there it was on the front desk!

The device was all out of power, so I got home and plugged it in, trying hard not to touch it too much because I had some thoughts of trying to lift the fingerprints I’d see on it (this is a lot harder than you’d think… 😉 ).  I’d fully expected the device to be wiped to factory default – especially if it was being sold off – but it popped up with my background screen (which includes my name) and all my apps installed.  It also started syncing with my Mac over wifi, so I left it alone to charge up and get caught up.  I was a little worried that maybe it had some phone home software installed or something, then I remembered that I’m not that interesting.

I did not have a password on the device when I lost it, mainly because it’s a pain in the neck to use it all the time, and because my wife and kid use the device sometimes and I didn’t want to keep reminding them of the password.  I thought I’d dealt with this by sending the lock and password command when it was gone, but I was surprised that the device asked for no password when I turned it on.  In fact, I was able to play around with it for about 20 minutes with no problem.  I also noticed that my reward message wasn’t on it.  It wasn’t until I ran the “Find My iPad” app again from icloud that the lock came on and the message appeared.  I don’t know why the lock didn’t happen as soon as the device got power.

Another lovely feature was that work email wipe command I’d sent.  I assumed that it would wipe my email – but nope, it wiped the whole device!  The iPad had been prompting me for my email password, since I’d changed it days ago.  I went ahead and put it in, looked away, and when I looked back, the device was wiped and asking for a setup language!  So, no worries about spy software, but that was a surprise.  I ran a restore from my computer and it all came back in about 30 mins.

Sunday 1/8/12

Imagine my surprise this morning when it was wiped again!  I guess I needed to totally delete the device from my work email server’s awareness for it to stop trying to wipe it.  I’m afraid to put my work email password into it now tho!  😉

By now the device is in full working order and I’m back on track!

Monday 1/9/12

But I’m left with a LOT of questions!  I called and left a voicemail with the Y facilites manager.  He called back, unaware that I’d gotten it back!  He was thrilled and I asked, “What did you do?!?”  He said he still hadn’t heard back from the cleaning company, but he was even more interested in talking to them now.  I am dying to hear what he learns!

Tuesday 1/10/12

I’m gonna go ahead an publish, even though there are a lot of questions unanswered.  I’ll add more later as I know it.  I’ll also tack some other lessons learned on here, and more as I think of them:

  • I should have put a password on the device.  Even if it’s a pain in the neck.  I could have reduced the stress of the situation by knowing that the data were safe.
  • I thought that having a map with a green dot on an address would ensure I got my iPad back pronto.  Thank goodness, cops follow the Constitution and won’t kick doors in based on a map I printed up.  I REALLY wanted them to, but it would have been dumb.  Now, had the device been active and I could have made it make a sound, the cops would have done a lot more.
  • Fill out a police report as soon as you can.  No one will do anything until you have that magic report number.  I learned during filling it out that I could have filed under “Lost Property” when I knew the device was missing.  I didn’t have to wait until I thought it was a theft.
  • Don’t hold your breath waiting for the cops to do anything.  I have never heard from the San Diego police.  I just filled out a web form and it went into a black hole.
  • The San Diego area has a lot of police forces.  El Cajon is a totally different jurisdiction.  I remarked to the cops I met how it seemed that they cared a ton more than the police closer to my house.  They said that while El Cajon is certainly underfunded for police (hence my 45 minute wait in the parking lot), SD City is decimated.
  • Wipe means wipe!  Well at least from the Outlook web client.  And don’t forget to take the device out of the known list and re-add it later, or you’ll get wiped each time the devices talk.
  • Customize your device if you can.  I had a known dent on mine, but think about the Craigslist picture and how you could help convince the cops that it’s your device that’s listed.  No, don’t dent the device, work people… 😉 But silly stickers don’t hurt.
  • The folks at my Y are great and take member concerns very seriously.  Well, except for widening the entrance turnstile for my fat butt!  😉
  • Tweeting vague references to a big story you are going to write up really aggravates people.  Do it often!  😉

 

Enhanced by Zemanta

One response so far