Scenario: Concerns about
Safety.
- published in "Urgency of ethical
standards intensifies in computer community."
IEEE Computer, 23:77-81. Michael McFarland.
reprinted in Computers, Ethics &
Social Values. Deborah G. Johnson and Helen Nissenbaum.
The past
several months, George, an electrical engineer working for an aerospace contractor,
has been the quality control manager on a project to develop a computerized
control system for a new military aircraft. Early simulations of the software
for the control system showed that, under certain conditions, instabilities
would arise that would cause the plane to crash. The software was subsequently
patched to eliminate the specific problems uncovered by the tests. After
the repairs were made, the system passed all of the required simulation tests.
George is convinced,
however, that those problems were symptomatic of a fundamental design flaw
that could only be eliminated by an extensive redesign of the system. Yet,
when he brought his concern to this superiors, they assured him that the
problems had been respolved, as shown by the tests. Anyway, to reevaluate
and possibly redesign the system would introduce delays that would cause
the company to miss the delivery date specified in the contract, and that
would be very costly.
Now, there's a gread
deal of pressure on George to sign off on the system and allow it to be flight
tested. It has even been hinted that, if he persists in delaying release
of the system, the responsibility will be taken away from him and given to
someone who is more compliant. . . .
What make the situation
so difficult for George is that he must choose between conflicting duties:
loyalty to self, family, employer, and superiors versus the obligation to
tell the truth and to protect others from harm. . . .