Posted by: tommybrennan | February 9, 2019

The Miracle of the Database

database-schema-1895779When we are discussing the world of the miraculous, certain stories come to mind. An unexpected provision of money, a healing, or the safe return of a loved one from the battlefield. I daresay that repair and correction of a damaged database is usually not in the top ten, but here we are.

I was a Computer Aided Facility Management (CAFM) Consultant at CFI of Detroit in the mid 90s. We essentially helped large corporations manage and track their company’s square footage and assets. This position brought me into client’s work sites to review, document, install and tweak CAFM software solutions. I had a pretty solid background in CAFM, as I had worked on such a system as a user at Beaumont Hospital in the early 90s.

CFI was a great place to work for many reasons. I worked with the best AutoCAD and CAFM professionals in the country. We were the innovators, and it was not unusual for us to break new ground with various Commercial Off The Shelf (COTS) CAFM programs. We literally wrote the book, and discovered what would work, and what couldn’t work in the real world. We were, without a doubt, on the cutting edge. It was exhilarating, interesting, fun, and challenging.

Most of the time. And then there were those times where the old tech adage “The cutting edge is also the bleeding edge”, came to bite us with its nasty, sharp, pointy teeth. And I have some scars. The following is a true account, and left me wiser, and convinced of God’s providence and attention to little old me.

I was providing follow up for a full implementation of CAFM at the Owens Corning World Headquarters in Toledo Ohio. It was a beautiful, state of the art office complex, designed by renowned architect Cesar Pelli. The implementation went fairly smoothly, and we were ironing out the kinks in the Building Operations module, where work orders were generated for various maintenance tasks throughout the facility. This is when the trouble started.

It was not unusual to perform minor modifications on site. We would pull up the appropriate file, type in a new name, variable, or number, and this allowed us to customize the system to our client’s particular needs. I was adding some new work request types, such as filter replacements, and belt adjustments when I noticed that the database was displaying error messages. Again, this was not unusual, as the database often had to be refreshed to accept new entries. Often, a simple closing of a module and opening it back up would resolve the issue.

Not this time. The errors persisted. New errors appeared. The database became locked up. This is never what you want to see, particularly when you were the last person working on the schema of the system. I persisted in trying various workarounds to access the system and resolve the errors. Nothing. The staff began to ask why they couldn’t access their drawings. “Just a minor issue. I’ll have it resolved in a minute” I said. But a low level panic started to set in. I tried to act like it was no big deal. Probably not very convincing.

After at least an hour of trying to fix this, I decided that we would have to replace the current database with the backup. Now, restoring from a backup is a more involved process than minor modifications, and I would need to ask their IT people to provide the back up file. I nonchalantly asked my contact for the IT person’s number. Why? Oh, no particular reason.

I was able to get ahold of the IT person for our department that day, and they told me that yes, there was a back up file. I breathed a sigh of relief, and asked them to meet me in the Data Center. Once we were in the refrigerated, raised floor room filled with the whirring of fans, and blinking amber and green lights, she navigated out to the backup file and selected it. I noticed that the date was over a month old. Back ups are done every night. At least they were supposed to be. I asked if there was any more recent back up. She said no, that was the most recent one. This was bad. Real bad. Over a month of data was lost, and this reflected maintenance records, inventory, purchase orders and other critical data to the organization. I was now pretty much in panic mode. Again trying not to show it.

I was to learn later that this particular application needed to be stopped on the server, and backed up explicitly. Automatic backups wouldn’t capture any of the live data that was active in the system. It wasn’t my responsibility to back up the system, but the fact that I had made changes to the system without first confirming the existence of a good back up was my responsibility. I had made an assumption that proved to be false, and I was to pay the price for that assumption.

I went back to trying to salvage the database, consulting an associate at another company, but he had no good suggestions. I was stuck with a damaged database, and a bruised reputation. There was an overwhelming sense of dread. How was I going to explain this to my boss? What would he think of me? Would I get fired? I had given up any possibility that the situation could be remedied. I had to accept this worst case scenario.  Or so I thought.

I copied the damaged database onto a Zip Drive and brought it back to our Detroit office. I talked to several of our best database people, and they looked at me like “Dude, you’re screwed”, but not without sympathy. I was ready to face the worst.

It was at this point that one of my team members, Dave Czezewski, showed up and asked how things went in Toledo. I told him what happened. He sympathized, but asked if I had the file. I told him I did and gave it to him. He asked if I would mind if he tried to fix it.  I said sure, but I had asked some of our best, and they were not hopeful. Give it a shot, it couldn’t hurt.

It was late, and I left the office. It was a Friday night, so I would have all weekend to imagine what would happen when the boss found out. I spent that weekend obsessing over my fault, and trying to keep my perspective healthy. I didn’t do too well, and I got little sleep. I prayed about it as well, but I thought there was little hope in God coming to the rescue when this whole ordeal could have been prevented if I had asked about a back. I braced myself to get hammered.

A funny thing happened on Monday morning. I got to the office, and Jim Vazquez, my mentor in all things CAFM asked if I had spoken with Dave C. this morning. I said no, and he told me he had something to tell me.  I walked down the hall to our CAFM office, expecting Dave to just hand the file back to me. But to my surprise, delight, and answer to prayer, he said he had been able to repair it. It couldn’t be possible.  He didn’t know what the data looked like. He is probably just seeing 30 day old information and thinks he fixed it. I went to his cubicle and looked at his screen. Sure enough, I was looking at 30 day old work requests. He then said “and here is last Friday’s data”, and it popped up on the screen. I couldn’t believe it! I asked to sit down and, I scrolled through the data. It was all there! Work Orders, inventory, materials, bills, all of it. I was dumbstruck. I never felt like kissing a man in my life, and I didn’t, but man, I was elated. Relieved. Thankful. A quick whispered prayer to God “You came through for me, God. Thanks!”

I asked Dave how he did it, and he explained that he ran a routine that automatically fixed errors. The same routine I had run with no luck. Somehow, it worked for him. I asked him a bunch of questions, but the end result was the same.  The database was repaired, and I was saved.

I went back to Owens Corning, repaired database on a Zip Drive. I asked for the IT person again. I told her about the need to stop the database to get a full backup. She was grateful. I installed the good database in place of the damaged one and tested the system. It was perfect. What a beautiful sight.

I share this story, because it is important for us to rehearse the great things that God has done for us.  Remind ourselves of how God came through when there didn’t seem to be a way. To declare the faithfulness of God and the power of prayer. I share this with you today that you may have hope. He hears you. Dare to hope that He will do more than you can ask or think.

And bring to mind The Miracle of the Database.

 


Responses

  1. Thank you for this personal example of the power of “asking for help”. All too often, out of ego or pride, I try and do it alone, go to loan, thinking I know better than the collective. I too have examples of this kind of thing. It’s refreshing to know others deal with it to

    • Thanks John. I appreciate your comments.

  2. Wow Tom!!! This is SO ON TIME!!!!!! TY for sending this!!!!!!!!! Proverbs 13:12 English Standard Version (ESV) 12 Hope deferred makes the heart sick, but a desire fulfilled is a tree of life.

    Blessings, Ellie Hurst

    Sent from my iPhone

    >

    • Thanks Ellie. I appreciate your kind words.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Categories

%d bloggers like this: