Good test automation that checks nothing

To:          Eric Jacobson
Date:        May 25, 2012
From:        Ilya Kurnosov
Subject:     RE: The Most Important Part Of Test Automation

Define your problem as “unemployment”, for example, and you may restrict your answers to new jobs – and not consider retraining leave or the desirability of leisure and study-time.

— “The learning revolution”, chapter 5 “How to think for great ideas

A lot of people seem obsessed with automated checks these days. Or with proving that “automated testing” is really nothing more than “mere automated checking”. Let us pause for a moment.

  • Do you use ProcessExplorer or ProcessMonitor during your troubleshooting adventures?
  • Do you use DebugView for leveraging instrumentation built into your product?
  • Do you use (God forbid!) debugger for capturing crash dumps?
  • Do you use WireShark to see if something weird goes over the network?
  • Do you use Perl script that beeps on otherwise invisible events to help you reveal them?
  • Do you use bug tracker or something as simple as Google Docs for communicating with your teammates?
  • Do you use SnagIt or Wink or Problem Steps Recorder to make your problem reports more clear?
  • Do you use Git for keeping your artifacts?
  • Do you use SQL scripts for test data preparation?
  • Do you use PowerShell scripts to deploy fresh builds of your product to test environment at night?
  • Do you use simple HTML page (with sortable JavaScript table) to publish testing progress on your intranet?
  • Do you use Pomodoro technique with kitchen timer to get the most out of your test sessions?
  • Do you use XMind for capturing your test ideas?
  • Do you use Compendium for capturing your notes while exploring the product?

All these examples are not about checking anything. If you ask me I’ll say it is test automation nevertheless. And I merely scratched the surface with these examples. I see purpose of test automation in supporting and enhancing testing. Automation should not necessarily be complicated, but it has to be effective.

Shy, Inc.

“Well, ya see, sir, I understand you’re lookin’ for sparrin’ partners for Apollo, and I jus’ want ta let ya know that I am very available.”

— “Rocky”, 1976 movie

I’ve just finished reading Seth Godin’s “The dip” where he advocates “quitting as an intelligent strategy”. Seth goes as long as suggesting that “(t)he only position you can count on now is best in the world”. One of examples in the book that particularly impressed me was Jack Welch, true to his “fix it, sell it or close it” principle. How many businesses are there that will dare to “sell a billion-dollar division that’s making a profit quite happily while ranking #4 in market share”? GE did exactly that.

Pause for a moment and look around. You’ll see a lot of companies that even do not dare dreaming about being the best in their world. Chances are all the companies you’ll see are of this very kind. No wonder. It’s hard to find those shooting for best in the world even going through mission statements of Fortune 500 companies. (Yes, I do understand that all these so-called mission statements can be, as Tom Fishburne suggests, just “laughable reading”.)

You may argue that dominant number of shy enterprises is not a problem. Thankfully, businesses are free to die (which is rarely the case) or to sink into mediocrity. After all, Drucker hit the bull’s eye suggesting that “(k)nowledge workers outlive organizations, and they are mobile”. The problem is that organisation must embrace boldness as one of its core values in order to go for best in the world. And boldness is not fit for everyone.

This suggests that our society is misdesigned for everyone’s wellbeing. Now, it’s not a problem. It is disaster. Shall most of us change our value systems to include boldness into them then? Or shall we change the world so it permits ways to survival other than being best in the world? As humanity, are we facing the Dip, the Cul-de-Sac, or maybe the Cliff?

Give up the “sticks”

“Why not whip the teacher when the pupil misbehaves?”

— Diogenes of Sinope

It’s tempting to punish people when a problem happens. It’s tempting because it’s easy to assume the link between cause and effect. “I punished him, he got the message, the problem is fixed.” This is a good ol’ way of reasoning familiar to everyone since childhood. The only problem is that it is wrong. In case of malfunctions you have to punish a system, not people, in order to fix a problem forever and for everyone. Ever punished a system? I never heard about successful cases.

You can’t punish systems. You have to change them instead. You can’t change people. You have to take them as they are. Let’s make the “sticks” exclusive privilege of a state as most stupid things are already its privilege.

There’s a quote from an old email from my boss explaining his management approach:

They [people] (1) are responsible for the declared results. If they (2) did smth wrong they (3) will be punished.

There is one strange thing about this email. I never saw my boss use a “stick” in the years I work for him. No, your guess is wrong. We had a lot of situations calling for it. We just work differently, trying to never blame neither ourselves nor our colleagues. Peculiarity is that my boss never openly promotes his real management style around the organisation. But I deflected.

In the end I have a question for you. If we give up “sticks”, should the “carrots” follow? I’m sure they should. What do you think?

Wolves in agile clothing

There is little point in investigating the culture of any org you are considering joining. Crucial, however is to understand their mindset.

— Bob Marshall aka @flowchainsensei

Wolf in sheep's clothing“(A)gile minds, able to see beyond the obvious and act effectively in an ever-changing global business landscape. At every level agile thinking is nurtured. And at every level agile minds are rewarded with competitive pay, support and opportunities to excel. So if you’re ready to join us, apply now,” says job advertising of one of large investment banks. The ad uses the a-word three times in a single paragraph. Sounds like a  buzzword great place to work at, right?

One thing bothers me each time folks from banks approach me with job interview offers. I read Jim Highsmith‘s “Agile software development ecosystems” book for the first time more than five years ago, but I can still recall mixed feeling of disgust and disbelief I had reading this passage:

Managers in the process-centric community value people, but they still consider process more important. For those who might argue with this assessment, I offer the following paragraph from a ComputerWorld article about Ashutosh Gupta, CEO of Deutsche Software India, a subsidiary of Deutsche Bank AG. “There is this myth that software development is a creative effort that relies heavily on individual effort,” says Gupta from his air-conditioned office high above the din of traffic-clogged Mahatma Gandhi Road. “It is not. It is just very labor-intensive, mechanical work once the initial project definition and specification stage is past” (Vijayan and Anthes 2001).

Will you consider doing knowledge work for organization that denies its very essence? I shall not. Could Deutsche Bank shift to agile in past 11 years? It could not without a reason. As Jim observes in the same book, banks have no such reasons:

I am reminded of the comments by Lloyd Darlington of the Bank of Montreal, who observed that the new information economy defines the most pervasive change in banking in 300 years… His message was clear: Banks must adapt, but they don’t have to remake their entire structure overnight. Darlington’s comments were made at the height of the dotcom frenzy, during which a number of Internet-only banks were established. They are all gone.

You make the difficulty that has always been there visible, please keep up

“After having devoted a considerable number of years of my scientific life to clarifying the programmer’s task, with the aim of making it intellectually better manageable, I found this effort at clarification to my amazement (and annoyance) repeatedly rewarded by the accusation that “I had made programming difficult”. But the difficulty has always been there, and only by making it visible can we hope to become able to design programs with a high confidence level, rather that “smearing code”, i.e., producing texts with the status of hardly supported conjectures that wait to be killed by the first counterexample.”

— Edsger W. Dijkstra “A discipline of programming”

I struggle to never write comments that can be drilled down to “Great stuff, dude. Write more on the topic!” Instead I prefer keeping silence if I have nothing to add or ask but nitpicking. In part, because I do not want to look as one “leaving inane comments on blog postings written by people who care about the craft”. Better to remain silent and appear a fool than to open your mouth and remove all doubt.

This time I gonna make an exception though. As you could notice, Michael Bolton had posted a beautiful argument against using passed vs. failed ratios “as a means of estimating the state of a development project”. You can easily imagine how popular such metric is. My team, for one, used it until our latest project. We weren’t happy with it. It took us quite an effort, certain dose of persistence, and a fair amount of guerilla tactics to quit this metric. If you ever walked the similar path, you can easily imagine how thorny it is. (“Trying to solve the problem on your own seems the only way in which you can assess how difficult the problem is”, yeah?)

Part of the problem, of course, was that we didn’t have our arguments worded as beautifully as Michael did in the post. Thus, one can expect that publication of Michael’s work inevitably makes life for all other practitioners more simple. There’s another factor. As a rule, ideas expressed by renowned thought leaders have more influence on managers than the very same ideas expressed by managers’ peers. I’m OK with that. This again increases importance of Michael’s contribution.

I hope, at this point you can understand why I was shocked by Adi’s comment that he (she?) got “tired reading more and more test articles that fall in the philosophical side”. Please, don’t get me wrong. I’m not an enemy of good case-studies and real-life examples. I do know that “all experts… operate on the basis of intimate knowledge of several thousand concrete cases in their areas of expertise.” I’m a big fan of war stories, after all. But dismissing something on the basis it is “philosophical”, seems strange to me. What shall we do then with ontology and epistemology? Never spell their names?

My formal education was some hybrid of mathematician and physicist. Philosophy course was mandatory part of the curriculum. I remember that everyone expected it to be dull waste. Prior school experience suggested such expectations. Believe it or not, but philosophy was the funniest course I took that year. Believe it or not, it is philosophy course that I find the most useful for my tester-self.

I wonder, do most people fail to accept that software testing is social science (note both “science” and “social”) or just do not bother to even think about it? It is year 2012, how can it look like most people didn’t even hear about qualitative research?

PS. And yes… Great stuff, Michael. Please, write more on the topic! I take my hat off before you and all the other testers advancing our craft.

My attempt of twitterview on Agile Testers with Bob Marshall

It looks like there is a new beast in software development taxonomy, namely Agile Tester. Somehow I missed its emergence, I was sure they are not separate species. Maybe they are. I’m going to investigate this phenomenon, as soon as I’m done with my intellectual debt to people who were so kind to answer my comments on their blogs (Brent, Eric I’m speaking about you here).

My current understanding of “Agile Tester” is pretty much summed up in the following Twitter conversation with Bob Marshall (@flowchainsensei). Apparently, my interviewing skills are lame, and Mr. Marshall got bored pretty quickly. Still, I believe that the chat does touch a couple of important points. By the way, if you didn’t hear about Bob Marshall, you may be missing a lot of fresh ideas that are going to shape future of software development. As Alan Cooper put it: “In the meantime, @flowchainsensei is tweeting up a storm of good stuff. Highly recommended.”

@kurniliya: How’s that? ->“If you have testers, then you’re not Agile. Calling them Agile Testers does not change this fact.”
@flowchainsensei: I am unsure as to the nature of your question.
@kurniliya: I mean, it’s not an obvious fact, is it? I’d appreciate if you provide arguments, explanation. I.e. go deeper into it.
@flowchainsensei: For example, what does Scrum say about team membership / roles / job titles?
@kurniliya: I see your point, but. 1. Scrum is Agile, Agile is not Scrum. 2. Scrum has distinct ScrumMaster role, why not have Tester?
@flowchainsensei: Agile, as a subset of the Synergistic mindset, invites folks to take a whole-system view of (tech) business. 1/2
@flowchainsensei: Why, then, perpetual the Analytic mindset [cf Ackoff] with siloed roles and responsibilities, and narrow specialisms? 2/2
@kurniliya: Can I say then “If you have programmers/designers/analysts/you-name-it, then you’re not Agile”?
@flowchainsensei: Yes
@kurniliya: Is it direct conclusion from Agile Manifesto, or we deal with evolved definition of Agile? (No probs, just to get context.)
@flowchainsensei: Strongly implicit in Manifesto (and esp. 12 practices), but not explicit.
@kurniliya: “I’m tester” doesn’t mean I can’t code, design, take business POV. Rather “I can and love testing more than anything else”
@flowchainsensei: I’m sure many buggy-whip makers loved making buggy whips, too. Didn’t stop them becoming extinct.
@kurniliya: The car made them extinct, did it? What is “the car” for coding, testing, design?
@flowchainsensei: Lean Product Development (probably), FlowChain. :)
@kurniliya: Does Lean let one produce software w/o coding? If not then buggy-whip makers case is *essentially* different.
@flowchainsensei: How so, different?
@kurniliya: No need for horse-stuff-makers, if there are no horses. We need ‘em otherwise. Software is still here => we need coders etc
@flowchainsensei: Or AI. Or sw that users can “code”. Or…
@kurniliya: I thought we speak about *present* days :-) Or nearest, possible, and desirable future.
@flowchainsensei: Who can tell when/what the next innovation might be?
@kurniliya: If some magic makes everyone ultimately happy tomorrow, I’m ok with that. But can we base arguments on what we know today?
@flowchainsensei: Sure, but erm, that’s not very *agile* is it? ;)
@kurniliya: “Yesterday’s weather” approach by Martin Fowler (can be wrong with ref) is considered agile, AFAIK. Note *yesterday* ;-)
@kurniliya: Tiny sidestep: Who can tell *if* the next innovation will be? Strange, how deep we got used to constant innovation.
@flowchainsensei: Nothing in life is certain :)

Carl Menger’s method of research

There are fascinating passages from preface to “Principles of economics” by Carl Menger. What makes them fascinating is that they have been true for 140 years. What makes these passages even more fascinating is that one could change economics to software development, and they still would be true.

The impartial observer can have no doubt about the reason our generation pays general and enthusiastic tribute to progress in the field of the natural sciences, while economic science receives little attention and its value is seriously questioned by the very men in society to whom it should provide a guide for practical action.

Never was there an age that placed economic interests higher than does our own. Never was the need of a scientific foundation for economic affairs felt more generally or more acutely. And never was the ability of practical men to utilize the achievements of science, in all fields of human activity, greater than in our day. If practical men, therefore, rely wholly on their own experience, and disregard our science in its present state of development, it cannot be due to a lack of serious interest or ability on their part. Nor can their disregard be the result of a haughty rejection of the deeper insight a true science would give into the circumstances and relationships determining the outcome of their activity. The cause of such remarkable indifference must not be sought elsewhere than in the present state of our science itself, in the sterility of all past endeavors to find its empirical foundations.

Every new attempt in this direction, however modest the effort, contains its own justification. To aim at the discovery of the fundamentals of our science is to devote one’s abilities to the solution of a problem that is directly related to human welfare, to serve a public interest of the highest importance, and to enter a path where even error is not entirely without merit.

In order to avoid any justifiable doubts on the part of experts, we must not, in such an enterprise, neglect to pay careful attention to past work in all the fields of our science thus far explored. Nor can we abstain from applying criticism, with full independence of judgment, to the opinions of our predecessors, and even to doctrines until now considered definitive attainments of our science. Were we to fail in the first task, we would abandon lightly the whole sum of experience collected by the many excellent minds of all peoples and of all times who have attempted to achieve the same end. Should we fail in the second, we would renounce from the beginning any hope of a fundamental reform of the foundations of our science. These dangers can be evaded by making the views of our predecessors our own, though only after an unhesitating examination, and by appealing from doctrine to experience, from the thoughts of men to the nature of things.

This is the ground on which I stand. In what follows I have endeavored to reduce the complex phenomena of human economic activity to the simplest elements that can still be subjected to accurate observation, to apply to these elements the measure corresponding to their nature, and constantly adhering to this measure, to investigate the manner in which the more complex economic phenomena evolve from their elements according to definite principles.

This method of research, attaining universal acceptance in the natural sciences, led to very great results, and on this account came mistakenly to be called the natural-scientific method. It is, in reality, a method common to all fields of empirical knowledge, and should properly be called the empirical method. The distinction is important because every method of investigation acquires its own specific character from the nature of the field of knowledge to which it is applied. It would be improper, accordingly, to attempt a natural-scientific orientation of our science.

Past attempts to carry over the peculiarities of the natural-scientific method of investigation uncritically into economics have led to most serious methodological errors, and to idle play with external analogies between the phenomena of economics and those of nature. Bacon said of scholars of this description: “Similitudes and sympathies of things that have no reality,  . . . they describe and sometimes invent with great vanity and folly,” a statement which, strangely enough, is still true today of precisely those writers on economic subjects who continue to call themselves disciples of Bacon while they completely misunderstand the spirit of his method.

If it is stated, in justification of these efforts, that the task of our age is to establish the interconnections between all fields of science and to unify their most important principles, I should like to question seriously the qualifications of our contemporaries to solve this problem. I believe that scholars in the various fields of science can never lose sight of this common goal of their endeavors without damage to their research. But the solution of this problem can be taken up successfully only when the several fields of knowledge have been examined most carefully, and when the laws peculiar to each field have been discovered.

It is now the task of the reader to judge to what results the method of investigation I have adopted has led, and whether I have been able to demonstrate successfully that the phenomena of economic life, like those of nature, are ordered strictly in accordance with definite laws.

Follow

Get every new post delivered to your Inbox.