Things that go wrong with systems

Problem

Most code in software systems controls against or recovers
from exceptions or handles special cases, rather than supporting
what the system is "really" about. In the ScenIC project, we're
exploring principles for decoupling the goals for a system from
these obstacles at an early stage in the design process.

These three projects will contribute to this research effort:

(a) Quickly review the existing literature on ScenIC and a small
portion of the relevant research literature on specifying systems
in terms of goals. Then develop a typology of things that go
wrong. For example

Actor failures
        Human actor failures
                User not there
                User performs wrong action at right time
                ...
        Device failure
                ... various failure modes
Resource obstacles
        Resource contention
                Two objects can't be associated because of
                the unavailablity of one of them.
                ...
        Replicate confusion
                Lots of instances of a type that get mixed up
        ...
Channel failures
        Non-delivery
        Garbling
        ...

This project provides opportunity for a lot of creativity. If you
have some experience in systems thinking, human factors or applied
psychology, reliability or safety engineering, you should be able
to apply your existing knowledge in a new way. This project could
be relevant to your interests in software engineering, artificial
intelligence (i.e. reasoning about goals and actions), HCI (i.e.
user errors), etc.


(b) Take a fragment from the previous typology (no need to wait for
someone else to do project (a)) and develop a set of heuristics for
developing defensive and mitigation strategies. A defensive
strategy is a collection of secondary goals and actions that avoid
or reduce the likelihood of an obstacle occurring. A mitigation
strategy is a collection of secondary goals and actions that repair
the damage caused by an obstacle, perhaps by undoing the incorrect
actions or by gracefully degrading functionality while a user takes
over.

What I'm looking for here is a very general typology of defensive
and mitigation strategies that you can derive from everyday analogies
of human action. For example, a defensive strategy to overcome
non-reception on a channel is "talk louder." Another is "talk more
slowly." To avoid garbling, you can use the "tell me what I just
said" strategy. To mitigate a resource contention, you can "wait
until it's available" or "make do with something else."

The heuristics should be in the form of a list of if-then rules
stated clearly and precisely in plain English.

There's an opportunity here for more than one project. Again, it
should appeal to those of you with an interest in SE, AI and HCI.


(c) A standard technique in ScenIC (from which it gets its name)
is to explore planned system behavior using scenarios. Scenarios
are "stories" about the usage of the system that illustrate goals
being achieved, blocked by obstacles and overcome using mitigation
strategy. The idea is to walk through scenarios with customers and
other experts to ascertain whether the system as concretely portrayed
in the scenarios will fit the bill.

But the big problem -- and this is where you come in -- is knowing
which and how many scenarios to explore. It's like software testing:
you can't exhaustively test a program to prove that it's correct;
you have to sample the input space and give the program those test
cases. So, we're investigating the notion of scenario "coverage",
the degree to which a given set of scenarios covers something or
other. The coverage of a set of test cases is usually assessed with
respect to such objective properties as lines of code, branches
through the program's flow graph, or uniquely identified requirements.
But what do you compare the coverage of a set of scenarios with?
Here are some ideas: Does each obstacle get illustrated in at least
one scenario? Is each communication between an actor (a user, device,
external system, or software architectural component) contained in
at least one scenario? Are all the interesting combinations of
goals and obstacles (difficult one, this one) treated somewhere?

So the goal is to be able to name and produce a limited number of
scenarios and be confident that the system's behavior is being
"tested" (albeit by manual inspection) as thoroughly as is reasonable.
But on the other hand, we don't want to snow the users with a huge
number of similar scenarios.

This project will involve proposing a couple of coverage measures
and applying them to a simple example of a system. The questions that
arise from walking through the resulting scenarios will be catalogued
and analyzed.


(d) The notion of scenarios as stories has led us to propose a
simple scenario schema that helps structure scenarios when writing
them. A scenario consists of a setting, a number of chronologically
sequenced episodes, and an optional conclusion. Each episode
details a set of concrete actions performed by actors, actions that
either accomplish some of the goals for the system or create
obstacles that thwart them.

Again, the challenge is balancing expressiveness with convenience. An
expressive schema would be built on a complex theory of purposive
behavior. (AI research from the 1970s is full of these.) But this
would likely be too rich to be useful when designing complex systems.
On the other hand, overemphasizing convenience leads to scenarios that
are just linear traces of events, which doesn't help us assess what's
wrong with the current view of the system.

In this project you will propose three schemas, expressed as context-
free grammars. Each grammar should generate an outline of a scenario.
One should be simple, one highly structured, and one somewhere in
the middle. Having proposed the schemas, you will then manually
"parse" some existing scenarios that are written just as text
paragraphs. And, of course, you need to think about the results: what
types of insights about the system arise from structuring the scenarios
differently? And is the overhead worth it?