I've been through the most distressing work week of my life...well, at least since the days in the oilpatch when I would occasionally end up staying awake for 30 to 40 hours at a time.
This week, the primary data server for all web applications crashed and burned a week ago Friday, followed closely by its backup on Sunday. The servers were over eight years old, the software components even older, and of course there were no rebuild instructions or disaster recovery plan. Lets just say that it has taken several attempts to get things going down the right path and it was only this weekend -- seven days after the initial incident -- that things have started to make remarkable progress.
Today, the servers are back up as is the customer portal and 90% of the applications. We are working to deal with partially complete source code bases to try and get the remaining 10% back online.
Needless to say, since last Sunday I have logged over 110 hours on this issue. No cycling, no socializing, nothing but sitting in a room and figuring things out. My brain is tired in addition to my eyes and body.
I'm not the only one. There has essentially been a team of 20-40 people working on this issue at any given time over the past week, including direct involvement by my company's CEO and our customer company's CEO and CIO. Makes for interesting decisions and political posturing which we probably could have not been involved with, but no matter. That crap follows you wherever you go. It's the same reason why this problem occurred in the first place.
There is a group of people closer to the level of the technologies and software that have been saying for four years now that this infrastructure needed to be upgraded; the risks of not doing so were high, but the customer seemed okay to accept that risk profile. You would think that because of that, a proper disaster recovery plan would have been put in place, but it didn't. Needless to say, now fully knowing what the issues and problems were, even with a proper recovery plan, the outages maybe would have been four days instead of seven days, but even so, millions of dolloars have been lost every day while this has been going on.
I'm very sick and tired and sore and stressed and depressed. It is particularly stuff like this that makes me hate my job; I think this may be the straw that breaks the camel's back. I don't think I can continue this position for much longer. I did like having the leverage of experience in an environment where everyone else has left, but after days like Wednesday that were 16 hours straight of being asked a question every 30 seconds, I'm not sure I want to assume the responsibility anymore. Is that wrong of me?
"In a world that has begun to believe that financial profit is the only religion, sometimes not wanting money is more frightening to capitalist society than acts of terrorism." Arundhati Ray
28 July 2008
My favorite quote
Progress, far from consisting in change, depends on retentiveness. When change is absolute there remains no being to improve and no direction is set for possible improvement: and when experience is not retained, as among savages, infancy is perpetual. Those who cannot remember the past are condemned to repeat it. In the first stage of life the mind is frivolous and easily distracted, it misses progress by failing in consecutiveness and persistence. This is the condition of children and barbarians, in which instinct has learned nothing from experience.
George Santayana, The Life of Reason, Volume 1, 1905
23 July 2008
Owwee
My friend Felix is the first one down in this spectacular crash at the Alpenrose Six Day in Portland. Every...single...racer in the Keiren final goes down. I heard that Felix is pretty banged up and heading home prior to the Redmond races this weekend. It's been a bad luck year for Synergy.....hope you're feeling better, Felix.
16 July 2008
Blinded when you're down
Ravers lose sight at laser show
Mon Jul 14, 8:57 AM
By Chris Baldwin
MOSCOW (Reuters) - Dozens of partygoers at an outdoor rave near Moscow last week have lost partial vision after a laser light show burned their retinas, Russian health officials said on Monday.
Moscow city health department officials confirmed 12 cases of laser-blindness at the Central Ophthalmological Clinic, and daily newspaper Kommersant said another 17 were registered at City Hospital 32 in the centre of the capital.
Attendees at the July 5 Aquamarine Open Air Festival in Kirzhach, 80 km (50 miles) northeast of Moscow, began seeking medical help days after the show, complaining of eye and vision problems, health officials told Reuters.
"They all have retinal burns, scarring is visible on them. Loss of vision in individual cases is as high as 80 percent, and regaining it is already impossible," Kommersant quoted a treating ophthalmologist as saying.
Attendees said heavy rains forced organizers to erect massive tents for the all-night dance party, and lasers that normally illuminate upwards into the sky were instead partially refracted into the ravers' eyes.
"I immediately had a spot like when you stare into the sun," rave-attendee Dmitry told Kommersant.
"After three days I decided to go to the hospital. They examined me, asked if I had been at Open Air, and then put me straight in the hospital. I didn't even get to go home and get my stuff," he said.
Cosmic Connection, promoters of the Aquamarine rave, were unreachable and did not list contact numbers on their Web site.
Industry Web site www.laserfx.com said focused laser light can cause eye damage almost instantly.
The owner of a Moscow laser rental company told Reuters the accidental blindings were due to "illiteracy on the part of technicians."
"It was partly the rain, but also partly the size of the laser. Somebody set up an extremely powerful laser for such a small space," said Valentin Vasiliev, who said his company did not provide the Aquamarine lasers.
(Additional reporting by Tatiana Ustinova; Editing by Giles Elgood)
Wow. That really sucks. It's one of those things you always laughed about when you were in such a situation, but who knew it would actually happen?
Mon Jul 14, 8:57 AM
By Chris Baldwin
MOSCOW (Reuters) - Dozens of partygoers at an outdoor rave near Moscow last week have lost partial vision after a laser light show burned their retinas, Russian health officials said on Monday.
Moscow city health department officials confirmed 12 cases of laser-blindness at the Central Ophthalmological Clinic, and daily newspaper Kommersant said another 17 were registered at City Hospital 32 in the centre of the capital.
Attendees at the July 5 Aquamarine Open Air Festival in Kirzhach, 80 km (50 miles) northeast of Moscow, began seeking medical help days after the show, complaining of eye and vision problems, health officials told Reuters.
"They all have retinal burns, scarring is visible on them. Loss of vision in individual cases is as high as 80 percent, and regaining it is already impossible," Kommersant quoted a treating ophthalmologist as saying.
Attendees said heavy rains forced organizers to erect massive tents for the all-night dance party, and lasers that normally illuminate upwards into the sky were instead partially refracted into the ravers' eyes.
"I immediately had a spot like when you stare into the sun," rave-attendee Dmitry told Kommersant.
"After three days I decided to go to the hospital. They examined me, asked if I had been at Open Air, and then put me straight in the hospital. I didn't even get to go home and get my stuff," he said.
Cosmic Connection, promoters of the Aquamarine rave, were unreachable and did not list contact numbers on their Web site.
Industry Web site www.laserfx.com said focused laser light can cause eye damage almost instantly.
The owner of a Moscow laser rental company told Reuters the accidental blindings were due to "illiteracy on the part of technicians."
"It was partly the rain, but also partly the size of the laser. Somebody set up an extremely powerful laser for such a small space," said Valentin Vasiliev, who said his company did not provide the Aquamarine lasers.
(Additional reporting by Tatiana Ustinova; Editing by Giles Elgood)
Wow. That really sucks. It's one of those things you always laughed about when you were in such a situation, but who knew it would actually happen?
14 July 2008
If you wait long enough...nothing changes
Court Rewards Exxon for Valdez Oil Spill
by Greg Palast
Chicago Tribune (revised)
[Thursday, June 26, 2008] Twenty years after Exxon Valdez slimed over one thousand miles of Alaskan beaches, the company has yet to pay the $5 billion in punitive damages awarded by the jury. And now they won't have to. The Supreme Court today cut Exxon's liability by 90% to half a billion. It's so cheap, it's like a permit to spill.
Exxon knew this would happen. Right after the spill, I was brought to Alaska by the Natives whose Prince William Sound islands, livelihoods, and their food source was contaminated by Exxon crude. My assignment: to investigate oil company frauds that led to to the disaster. There were plenty.
But before we brought charges, the Natives hoped to settle with the oil company, to receive just enough compensation to buy some boats and rebuild their island villages to withstand what would be a decade of trying to survive in a polluted ecological death zone.
In San Diego, I met with Exxon's US production chief, Otto Harrison, who said, "Admit it; the oil spill's the best thing to happen" to the Natives.
His company offered the Natives pennies on the dollar. The oil men added a cruel threat: take it or leave it and wait twenty years to get even the pennies. Exxon is immortal - but Natives die.
And they did. A third of the Native fishermen and seal hunters I worked with are dead. Now their families will collect one tenth of their award, two decades too late.
In today's ruling, Supreme Court Justice David Souter wrote that Exxon's recklessness was ''profitless'' - so the company shouldn't have to pay punitive damages. Profitless, Mr. Souter? Exxon and its oil shipping partners saved billions - BILLIONS - by operating for sixteen years without the oil spill safety equipment they promised, in writing, under oath and by contract.
The official story is, "Drunken Skipper Hits Reef." But don't believe it, Mr. Souter. Alaska's Native lands and coastline were destroyed by a systematic fraud motivated by profit-crazed penny-pinching.
Here's the unreported story, the one you won't get tonight on the Petroleum Broadcast System:
It begins in 1969 when big shots from Humble Oil and ARCO (now known as Exxon and British Petroleum) met with the Chugach Natives, owners of the most valuable parcel of land on the planet: Valdez Port, the only conceivable terminus for a pipeline that would handle a trillion dollars in crude oil.
These Alaskan natives ultimately agreed to sell the Exxon consortium this astronomically valuable patch of land -- for a single dollar.
The Natives refused cash. Rather, in 1969, they asked only that the oil companies promise to protect their Prince William Sound fishing and seal hunting grounds from oil.
In 1971, Exxon and partners agreed to place the Natives' specific list of safeguards into federal law. These commitments to safety reassured enough Congressmen for the oil group to win, by one vote, the right to ship oil from Valdez.
The oil companies repeated their promises under oath to the US Congress.
The spill disaster was the result of Exxon and partners breaking every one of those promises - cynically, systematically, disastrously, in the fifteen years leading up to the spill.
Forget the drunken skipper fable. As to Captain Joe Hazelwood, he was below decks, sleeping off his bender. At the helm, the third mate would never have collided with Bligh Reef had he looked at his Raycas radar. But the radar was not turned on. In fact, the tanker's radar was left broken and disasbled for more than a year before the disaster, and Exxon management knew it. It was just too expensive to fix and operate.
For the Chugach, this discovery was poignantly ironic. On their list of safety demands in return for Valdez was "state-of-the-art" on-ship radar.
We discovered more, but because of the labyrinthine ways of litigation, little became public, especially about the reckless acts of the industry consortium, Alyeska, which controls the Alaska Pipeline.
Several smaller oil spills before the Exxon Valdez could have warned of a system breakdown. But a former Senior Lab Technician with Alyeska, Erlene Blake, told our investigators that management routinely ordered her to toss out test samples of water evidencing spilled oil. She was ordered to refill the test tubes with a bucket of clean sea water called, "The Miracle Barrel."
In a secret meeting in April 1988, Alyeska Vice-President T.L. Polasek confidentially warned the oil group executives that, because Alyeska had never purchased promised safety equipment, it was simply "not possible" to contain an oil spill past the Valdez Narrows -- exactly where the Exxon Valdez ran aground 10 months later.
The Natives demanded (and law requires) that the shippers maintain round- the-clock oil spill response teams. Alyeska hired the Natives, especiallly qualified by their generations-old knowledge of the Sound, for this emergency work. They trained to drop from helicopters into the water with special equipment to contain an oil slick at a moments notice. But in 1979, quietly, Alyeska fired them all. To deflect inquisitive state inspectors, the oil consortium created sham teams, listing names of oil terminal workers who had not the foggiest idea how to use spill equipment which, in any event, was missing, broken or existed only on paper.
In 1989, when the oil poured from the tanker, there was no Native response team, only chaos.
Today, twenty years after the oil washed over the Chugach beaches, you can kick over a rock and it will smell like an old gas station.
The cover story of the Drunken Captain serves the oil industry well. It falsely presents America's greatest environmental disaster as a tale of human frailty, a one-time accident. But broken radar, missing equipment, phantom spill teams, faked tests -- the profit-driven disregard of the law -- made the spill an inevitability, not an accident.
Yet Big Oil tells us, as they plead to drill in the Arctic National Wildlife Reserve, as Senator John McCain calls for drilling off the shores of the Lower 48, it can't happen again.
They promise.
Greg Palast is a Puffin Foundation Writing Fellow for Investigative Reporting at the Nation Institute, New York. Read and view his investigations for BBC Television at www.GregPalast.com. An earlier version of this report originally appeared in the Chicago Tribune. Photos by James Macalpine (1993).
by Greg Palast
Chicago Tribune (revised)
[Thursday, June 26, 2008] Twenty years after Exxon Valdez slimed over one thousand miles of Alaskan beaches, the company has yet to pay the $5 billion in punitive damages awarded by the jury. And now they won't have to. The Supreme Court today cut Exxon's liability by 90% to half a billion. It's so cheap, it's like a permit to spill.
Exxon knew this would happen. Right after the spill, I was brought to Alaska by the Natives whose Prince William Sound islands, livelihoods, and their food source was contaminated by Exxon crude. My assignment: to investigate oil company frauds that led to to the disaster. There were plenty.
But before we brought charges, the Natives hoped to settle with the oil company, to receive just enough compensation to buy some boats and rebuild their island villages to withstand what would be a decade of trying to survive in a polluted ecological death zone.
In San Diego, I met with Exxon's US production chief, Otto Harrison, who said, "Admit it; the oil spill's the best thing to happen" to the Natives.
His company offered the Natives pennies on the dollar. The oil men added a cruel threat: take it or leave it and wait twenty years to get even the pennies. Exxon is immortal - but Natives die.
And they did. A third of the Native fishermen and seal hunters I worked with are dead. Now their families will collect one tenth of their award, two decades too late.
In today's ruling, Supreme Court Justice David Souter wrote that Exxon's recklessness was ''profitless'' - so the company shouldn't have to pay punitive damages. Profitless, Mr. Souter? Exxon and its oil shipping partners saved billions - BILLIONS - by operating for sixteen years without the oil spill safety equipment they promised, in writing, under oath and by contract.
The official story is, "Drunken Skipper Hits Reef." But don't believe it, Mr. Souter. Alaska's Native lands and coastline were destroyed by a systematic fraud motivated by profit-crazed penny-pinching.
Here's the unreported story, the one you won't get tonight on the Petroleum Broadcast System:
It begins in 1969 when big shots from Humble Oil and ARCO (now known as Exxon and British Petroleum) met with the Chugach Natives, owners of the most valuable parcel of land on the planet: Valdez Port, the only conceivable terminus for a pipeline that would handle a trillion dollars in crude oil.
These Alaskan natives ultimately agreed to sell the Exxon consortium this astronomically valuable patch of land -- for a single dollar.
The Natives refused cash. Rather, in 1969, they asked only that the oil companies promise to protect their Prince William Sound fishing and seal hunting grounds from oil.
In 1971, Exxon and partners agreed to place the Natives' specific list of safeguards into federal law. These commitments to safety reassured enough Congressmen for the oil group to win, by one vote, the right to ship oil from Valdez.
The oil companies repeated their promises under oath to the US Congress.
The spill disaster was the result of Exxon and partners breaking every one of those promises - cynically, systematically, disastrously, in the fifteen years leading up to the spill.
Forget the drunken skipper fable. As to Captain Joe Hazelwood, he was below decks, sleeping off his bender. At the helm, the third mate would never have collided with Bligh Reef had he looked at his Raycas radar. But the radar was not turned on. In fact, the tanker's radar was left broken and disasbled for more than a year before the disaster, and Exxon management knew it. It was just too expensive to fix and operate.
For the Chugach, this discovery was poignantly ironic. On their list of safety demands in return for Valdez was "state-of-the-art" on-ship radar.
We discovered more, but because of the labyrinthine ways of litigation, little became public, especially about the reckless acts of the industry consortium, Alyeska, which controls the Alaska Pipeline.
Several smaller oil spills before the Exxon Valdez could have warned of a system breakdown. But a former Senior Lab Technician with Alyeska, Erlene Blake, told our investigators that management routinely ordered her to toss out test samples of water evidencing spilled oil. She was ordered to refill the test tubes with a bucket of clean sea water called, "The Miracle Barrel."
In a secret meeting in April 1988, Alyeska Vice-President T.L. Polasek confidentially warned the oil group executives that, because Alyeska had never purchased promised safety equipment, it was simply "not possible" to contain an oil spill past the Valdez Narrows -- exactly where the Exxon Valdez ran aground 10 months later.
The Natives demanded (and law requires) that the shippers maintain round- the-clock oil spill response teams. Alyeska hired the Natives, especiallly qualified by their generations-old knowledge of the Sound, for this emergency work. They trained to drop from helicopters into the water with special equipment to contain an oil slick at a moments notice. But in 1979, quietly, Alyeska fired them all. To deflect inquisitive state inspectors, the oil consortium created sham teams, listing names of oil terminal workers who had not the foggiest idea how to use spill equipment which, in any event, was missing, broken or existed only on paper.
In 1989, when the oil poured from the tanker, there was no Native response team, only chaos.
Today, twenty years after the oil washed over the Chugach beaches, you can kick over a rock and it will smell like an old gas station.
The cover story of the Drunken Captain serves the oil industry well. It falsely presents America's greatest environmental disaster as a tale of human frailty, a one-time accident. But broken radar, missing equipment, phantom spill teams, faked tests -- the profit-driven disregard of the law -- made the spill an inevitability, not an accident.
Yet Big Oil tells us, as they plead to drill in the Arctic National Wildlife Reserve, as Senator John McCain calls for drilling off the shores of the Lower 48, it can't happen again.
They promise.
Greg Palast is a Puffin Foundation Writing Fellow for Investigative Reporting at the Nation Institute, New York. Read and view his investigations for BBC Television at www.GregPalast.com. An earlier version of this report originally appeared in the Chicago Tribune. Photos by James Macalpine (1993).
08 July 2008
Give It 2 Me
Paul Oakenfold is up to his classic goodness with his remix of Madonna and Pharell's new song, 'Give It 2 Me' (which incidentally is one of my favorite songs on 'Hard Candy'). Oakenfold's mix just gives it that extra oomph that takes the song from the HipHop club into the Trance club. I love it and have been listening to this mix for days now. I can't get enough!
We went to the Kaskade show at FlamesCentral on Saturday. Best DJ show I've been to in ages. And now that I've been to FlamesCentral and experienced the visual overload and audio superiority of the sound system and acoustics there, it will be hard to go to another show in the city at another venue again!
We went to the Kaskade show at FlamesCentral on Saturday. Best DJ show I've been to in ages. And now that I've been to FlamesCentral and experienced the visual overload and audio superiority of the sound system and acoustics there, it will be hard to go to another show in the city at another venue again!
02 July 2008
Pass The Buck
IEA says world in the grips of a 'third oil shock'
The world economy is staggering under a “third oil shock” and can expect little respite from sky-high crude prices over the next five years, the International Energy Agency said yesterday. “Record oil prices in the past several months have become a threat to the global economy and the social welfare of millions of people,” Nobuo Tanaka, executive director of the Paris-based energy watchdog, said as he released its newest five-year forecast. But those high prices will not spur enough new supply, nor force enough reductions in demand to provide significant relief to hard-pressed consumers and businesses. “While we have seen some weakening demand in the [developed world], supply constraints, refinery limitations and continued demand growth in key emerging markets will maintain pressure in the market over the medium term,” Tanaka told a Madrid press conference.
Existing oil fields worldwide are being depleted faster than had been believed, the agency reported, and it now requires an additional 3.5 million barrels a day of new production each year just to replace that decline. The IEA also rejected the widely held view that speculators are driving crude oil markets to new heights, though it applauded efforts by US regulators to improve transparency in futures markets. While investors have plowed some US$260-billion into oil-related futures – up from just $15B five years ago – the agency said there is no evidence that that avalanche of funds caused the stunning price runup. Both the Organization of Petroleum Exporting Countries and some members of the US Congress have blamed excessive speculation for the doubling of crude prices over the past year. But the IEA warned against a tendency to “scapegoat” speculators rather than face tough decisions. “Blaming speculation is an easy solution which avoids taking the necessary steps to improve supply-side access and investment, or to implement measures to improve energy efficiency,” the report said.
(Globe and Mail 080702)
The IEA is the 'watchdog'? They obviously haven't been doing their job very well if this is all coming as a big surprise. I'm not very surprised. I've been wondering why everything was deemed perfectly fine until the last six months or so, then whammo! Suddenly everything's changed. Yeesh.
The world economy is staggering under a “third oil shock” and can expect little respite from sky-high crude prices over the next five years, the International Energy Agency said yesterday. “Record oil prices in the past several months have become a threat to the global economy and the social welfare of millions of people,” Nobuo Tanaka, executive director of the Paris-based energy watchdog, said as he released its newest five-year forecast. But those high prices will not spur enough new supply, nor force enough reductions in demand to provide significant relief to hard-pressed consumers and businesses. “While we have seen some weakening demand in the [developed world], supply constraints, refinery limitations and continued demand growth in key emerging markets will maintain pressure in the market over the medium term,” Tanaka told a Madrid press conference.
Existing oil fields worldwide are being depleted faster than had been believed, the agency reported, and it now requires an additional 3.5 million barrels a day of new production each year just to replace that decline. The IEA also rejected the widely held view that speculators are driving crude oil markets to new heights, though it applauded efforts by US regulators to improve transparency in futures markets. While investors have plowed some US$260-billion into oil-related futures – up from just $15B five years ago – the agency said there is no evidence that that avalanche of funds caused the stunning price runup. Both the Organization of Petroleum Exporting Countries and some members of the US Congress have blamed excessive speculation for the doubling of crude prices over the past year. But the IEA warned against a tendency to “scapegoat” speculators rather than face tough decisions. “Blaming speculation is an easy solution which avoids taking the necessary steps to improve supply-side access and investment, or to implement measures to improve energy efficiency,” the report said.
(Globe and Mail 080702)
The IEA is the 'watchdog'? They obviously haven't been doing their job very well if this is all coming as a big surprise. I'm not very surprised. I've been wondering why everything was deemed perfectly fine until the last six months or so, then whammo! Suddenly everything's changed. Yeesh.
Subscribe to:
Posts (Atom)