NORBERT WIENER

"GOD AND GOLEM, Inc.,A Comment on Certain Points where

Cybernetics Impinges on Religion

THE M.I.T. PRESSMassachusetts Institute of Technology

Cambridge, Massachusetts

Preface

Some years ago, in The Human Use of HumanBeings,* I gave an account of some of the ethicaland sociological implications of my previous workCybernetics,^ the study of control and communi-cation in machines and living beings. At thatperiod, cybernetics was a relatively new idea, andneither the scientific nor the social implicationshad become fully clear. Now—some fifteen yearslater—cybernetics has made a certain social andscientific impact, and enough has happened tojustify a new book in a related field.

The problem of unemployment arising fromautomatization is no longer conjectural, but hasbecome a very vital difficulty of modern society.The cybernetic circle of ideas, from being a pro-gram for the future and a pious hope, is now a

* Wiener, N., The Human Use of Human Beings; Cy-bernetics and Society, Houghton Mifflin Company, Boston,1950.

f Wiener, N., Cybernetics, or Control and Communicationin the Animal and the Machine, The Technology Press andJohn Wiley & Sons, Inc., New York, 1948.

[ V i i ]

working technique in engineering, in biology,in medicine, and in sociology, and has undergonea great internal development.

I have given more than one series of lecturestrying to outline the impingement of this circleof ideas on society, ethics, and religion, and Ithink the time has come to attempt a synthesis ofmy' ideas in this direction, to consider more indetail the social consequences of cybernetics. Thisbook is devoted to certain aspects of these conse-quences, in the discussion of which, although Iretain the ideas and many of the comments whichI made in the Human Use of Human Beings, Ican consider the matter more in detail and morecompletely.

In this undertaking, I wish to acknowledge thegreat help I have received from the criticism ofmany friends on both sides of the Atlantic, es-pecially from Mr. Piet Hein of Rungsted Kystin Denmark, from Dr. Lawrence Frank of Bel-mont, Massachusetts, and from Professor KarlDeutsch of Yale University, as well as from manyothers. In addition, I wish to thank my secretary,Mrs. Eva-Maria Ritter, for her assistance in thepreparation of this material.

I found an opportunity to elaborate my ideasin a course of lectures which I gave in January of

[ i x ]

1962 at Yale University, and in a seminar thatwas held in the summer of 1962 at the ColloquesPhilosophiques Internationaux de Royaumontnear Paris. However, this book, though containingmaterial from my talks at both these places, hasbeen completely rewritten and reorganized.

With gratitude to the many who have helpedin this effort.

Norbert WienerSandwich, New HampshireAugust 30, 1963

e»^^<*w?*a^«W>iB«^^

I

It is here my intention to discuss not religionand science as a whole but certain points in thosesciences in which I have been interested—thecommunication and control sciences—which seemto me to be near that frontier on which science im-pinges upon religion. I wish to avoid those logicalparadoxes that are bound to accompany the ex-treme (but usual) claims of religion to deal withabsolutes. If we are to treat knowledge only interms of Omniscience, power only in terms ofOmnipotence, worship only in terms of the OneGodhead, we shall find ourselves entangled inmetaphysical subtleties before we shall have reallyembarked upon our study of the relations betweenreligion and science.

Nevertheless, there are many questions con-

[ i 3

[ 2 ]

cerning knowledge, power, and worship which doimpinge upon some of the more recent develop-ments of science, and which we may well discusswithout entering upon these absolute notions,which are surrounded with so much emotion andreverence that it is quite impossible to enter uponthem in a detached way. Knowledge is a fact,power is a fact, worship is a fact, and these factsare subject to human investigation quite apartfrom an accepted theology. As facts, these mattersare subject to study, and in this study we may ad-duce our observations of knowledge, power, andworship in other fields, more accessible to themethods of the natural sciences, without at oncedemanding of the student a complete acceptanceof the "credo quia incredible est" attitude.

It may be said that by starting in this way out-side religion, I have already removed this discus-sion from being one of the relations between sci-ence and religion, which is suggested by thegeneral trend of this essay. Therefore, I had betterdefine my theme at the beginning, specify thecorner of my subject in which I intend to remain,and disclaim those purposes that are alien to my

>-, specific task. As I have said, I have been workingy\. for several years on problems of communication

v and control, whether in machines or in living or-

vfo\

ganisms; on the new engineering and physiologicaltechniques attaching to these notions; and on thestudy of the consequences of these techniques forthe achievement of human purposes. Knowledgeis inextricably intertwined with communication,power with control, and the evaluation of humanpurposes with ethics and the whole normativeside of religion. It is hence germane to a revisedstudy of the relations between science and religionthat we should re-examine our ideas of these mat-ters in terms of the latest developments of theoryand practical technique. This may not itself con-stitute a study of science and its relations to reli-gion in the full sense, but it certainly constitutesan indispensable prolegomenon to such a study.

In a.study of this sort, if it is to lead anywhere,we must disencumber ourselves of the superim-posed layers of prejudice that we use nominallyto protect the homage which we pay dignified andholy things but in fact, as often as not to re-lieve ourselves from the sense of unworthinesswhich we feel in looking unpleasant realities anddangerous comparisons in the face.

If this essay is to mean anything, it must be areal probing of real questions. The spirit in whichit is to be undertaken is that of the operatingroom, not of the ceremonial feast of weeping about

mmmmmmm

[ 4 ]

a corpse. Squeamishness is out of place here—it iseven a blasphemy, like the bedside manners of thefashionable physician of the last century, with hisblack frock coat and the surgical needles hidingunder the silk lapel of his coat.

Religion, whatever else it contains, has oftensomething in itself of the closed front parlor of aNew England farmhouse, with drawn blinds, waxflowers under a bell jar on the mantelpiece, gildedbulrushes surrounding grandfather's portrait onan easel, and a harmonium in black walnut, neverplayed except at weddings or funerals. Or again,it is the moral counterpart of a Neapolitan hearse,one of those black plate-glass-windowed royal car-riages, with their black-plumed stallions carryingeven into death the assertion of status, or at anyrate of the aspiration to status. Religion is a seri-ous matter that we must separate sharply from anyconsideration of personal values of less significancethan religion itself.

I have spoken of the layers of prejudice whichencumber our approach to those problems in thevital common ground where science and religioncome together: we must avoid discussing Godand man in the same breath—that is blasphemy.Like Descartes, we must maintain the dignity ofMan by treating him on a basis entirely different

[ 5 ]

from that on which we treat the lower animals.Evolution and the origin of species are a desecra-tion of human values; and as the earlier Darwin-ians found, to entertain these ideas is very danger-ous for the scientist in a world fundamentallysuspicious of science.

But even in the field of science, it is perilous torun counter to the accepted tables of precedence.On no account is it permissible to mention livingbeings and machines in the same breath. Livingbeings are living beings in all their parts; whilemachines are made of metals and other unorgan-ized substances, with no fine structure relevantto their purposive or quasi-purposive function.Physics—or so it is generally supposed—takes noaccount of purpose; and the emergence of life issomething totally new.

If we adhere to all these tabus, we may acquirea great reputation as conservative and sound think-ers, but we shall contribute very little to thefurther advance of knowledge. It is the part ofthe scientist—of the intelligent and honest man ofletters and of the intelligent and honest clergymanas well—to entertain heretical and forbidden opin-ions experimentally, even if he is finally to rejectthem. Moreover, this rejection must not be takenfor granted at the beginning and merely consti-

[ 6 ]

tute an empty spiritual exercise, understood fromthe start to be no more than a game, in which oneengages to show one's spiritual open-mindedness.It is a serious exercise, and should be undertakenin all earnestness: it is only when it involves ajeal risk of heresythat there is any point to_it; andif heresy involves a risk of spiritual damnation,then this risk must be undertaken honestly andcourageously. In the words of the Calvinist, "Areyou willing to be damned for the greater glory ofGod?"

It is in this light of honest and searching criti-cism that we must regard an attitude which wehave already mentioned, and which it is hard toavoid in discussions of religious matters—theevasion implied by the false superlative. I have al-ready mentioned the intellectual difficulties aris-ing out of the notions of omnipotence, omnisci-ence, and the like. These appear in their crudestform in the question often asked by the scofferwho turns up uninvited at religious meetings:"Can God make a stone so heavy that He cannotlift it?" If He cannot, there is a limit to His power,or at least there appears to be; and if He can, thisseems to constitute a limitation to His power too.

It is easy to dispose of this difficulty as a verbalquibble, but it is more. The paradox of this ques-

[ 7 ]

tion is one of the many paradoxes that centerabout the notion of infinity, in its many forms.On the one hand, the least manipulation of themathematically infinite introduces the notion ofzero over zero, or infinity over infinity, or in-finity times zero, or infinity minus infinity.These are called indeterminate forms, and thedifficulty they conceal lies fundamentally in thefact that infinity does not conform to the ordinaryconditions of a number or a quantity, so thatoo / oo only means for the mathematician the limitof x/y, as x and y both tend to infinity. This maybe 1 if y — x, 0 if y = x2, or <» if y — l/x, and

so on.Again, there is a different infinity which arises

in counting. It can be shown that this notion tooleads to paradoxes. How many numbers are inthe class of all numbers? It can be shown that thisis not a legitimate question, and that however onedefines number, the number of all numbers isgreater than any number. This is one of theFrege-Russell paradoxes and involves the com-plexities of the theory of types.

The fact is that the superlatives of Omnipotenceand Omniscience are not true superlatives butmerely loose ways of asserting very great powerand very great knowledge. They express an emo-

m

!'•' J[ 8 ]

tion of reverence and not a metaphysically defen-sible statement. If God surpasseth the humanintellect, and cannot be compassed by intellectualforms—and this is at least a defensibleposition—it is not intellectually honest tostultify the intellect itself by forcing God into in-tellectual forms which should have a very defi-nite intellectual meaning. Thus, when we findlimited situations that seem to cast light uponsome of the statements generally made in reli-gious books, it seems to me disingenuous to castthese aside because they do not have the absolute,infinite, and complete character which we arewont to attribute to religious utterances.

This statement gives the key to my purposes inthe present book. I wish to take certain situationswhich have been discussed in religious books,and have a religious aspect, but possess a closeanalogy to other situations which belong to sci-ence, and in particular to the new science ofcybernetics, the science of communication andcontrol, whether in machines or in living organ-isms. I propose to use the limited analogies ofcybernetic situations to cast a little light on thereligious situations.

In doing this, I certainly shall have to force thereligious situations somewhat into my cybernetic

[ 9 ]

frame. I am quite conscious of the violence I mustuse in doing so. My excuse is that it is onlythrough the knife of the anatomist that we havethe science of anatomy, and that the knife of theanatomist is also an instrument which exploresonly by doing violence.

\\

II

i

With these preliminary remarks, let me turn tothe real theme of this little book.

There are at least three points in cyberneticswhich appear to me to be relevant to religious is-sues. One of these concerns machines which learn;one concerns machines which reproduce J.hem-

jelves; and one, the coordination of machine andman. I may say that such machines are known toexist. A program has been written by Dr. A. L.Samuel of the International Business MachinesCorporation which allows a computer to play agame of checkers, and this computer learns, or atleast appears to learn, to improve its game by itsown experience.* There are certain statements

• Samuel, A. L., "Some Studies in Machine Learning, Usingthe Game of Checkers," IBM Journal of Research and Develop-ment, Vol. 3, 210-229 (July, 1959).

[ 12 ]

here which need confirmation, or at least clarifica-tion; and I shall devote one section of this book tothis clarification.

Learning is a property that we often attributeexclusively to self-conscious systems, and almostalways to living systems. It is a phenomenon thatoccurs in its most characteristic form in Man, andconstitutes one of those attributes of Man whichis most easily put in conjunction with those aspectsof Man which are easily associated with his reli-gious life. Indeed, it is hard to see how any non-learning being can be concerned with religion.

There is, however, another aspect of life whichis naturally associated with religion. God is sup-posed to have made man in His own image, andthe propagation of the race may also be inter-preted as a function in which one living beingmakes another in its own image. In our desire toglorify God with respect to man and Man withrespect to matter, it is thus natural to assume thatmachines cannot make other machines in theirown image; that this is something associated witha sharp dichotomy of systems into living and non-living; and that it is moreover associated with theother dichotomy between creator and creature.

Is this, however, so? I shall devote a sectionof this book to certain considerations which, in my

opinion, show that machines are very well ableto make other machines in their own image. Thesubject upon which I am entering here is at oncevery technical and very precise. It should not betaken too seriously as an actual model of the proc-ess of biological generation, and even less as acomplete model of divine creation; but neither isit negligible as to the light it throws upon bothconcepts.

These two parts of this book of lectures may beregarded as complementary the one to the other.The learning of the individual is a process thatoccurs in the life of the individual, inBiological reproduction is a phenomenon that oc-curs in the life of the race, in phytogeny, but therace learns even as the individual does.JDarwiniannatural^selection is a kind of racial learning, whichoperates within the conditions imposed by the re-production of the individual.

The third group of topics of this book is alsorelated to problems of learning. It is concernedwith the relations of the machine to the livingbeing, and with systems involving elements ofboth kinds. As such, it involves considerations ofa normative and, more specifically, of an (ethical^nature. It concerns some of the most importantmoral traps into which the present generation of

^

[ 1 4 ]

human beings is likely to fall. It is also closely con-nected with a great body of human tradition andhuman legend, concerning magic and the like.

To begin with learning machines: an organized;——J- system may.be said to be one which transforms aV certain j££ojniBS_Hei!2S? m t o a n outgoing mes-

CD ^S- e ' a c c o rding to some principle of transform^r%\ tion. If this principle of transformation is sub-

ject to a certain criterion of merit of pertormance,and if the method of transformation is adjustedso as to tend to improve the performance of thesystem according to this criterion, the system issaid to learn. A very simple type of system with aneasily interpreted criterion of performance is a

•—i—rr™ game, to be played according to fixed rules, where. —— the criterion of performance is the successful win-

ning of the game according to these rules.Among such games are games with a perfect

theory, which are not interesting. Nim, as definedby Bouton, and ticktacktoe are examples of suchgames. In these games, we not only can theoreti-cally find a best policy for the playing of the game,but this policy is known in all its details. Theplayer of such a game (either the first or thesecond) can always win, or at any rate draw, byfollowing the policy indicated. In theory, anygame can be brought to such a state—this is the

5"j

[ 1 5 ]

idea of the late John von Neumann—but once agame has been brought to this state, it loses allinterest, and need no longer be considered even as

an amusement.An omniscient being such as God would find

chess and checkers (or draughts in England, ordames on the continent) to be examples of suchvon Neumann games, but as yet their completetheory has not been humanly worked out, andthey still represent genuine contests of insight andintelligence. However, they are not played accord-ing to the manner suggested in the von Neumanntheory. That is, we do not play them by makingthe best possible move, on the assumption that anopponent will make the best possible move, onthe assumption that we shall make the best possi-ble move, and so on, until one of the players winsor the game repeats itself. Indeed, to be able toplay a game in the von Neumann manner is tanta-mount to possessing a complete theory of the gameand to having reduced the game to a triviality.

The subject of learning, and in particular ofmachines that learn to play games, may seem some-what remote from religion. Nevertheless, there isa religious problem to which those notions arerelevant. This is the problem of the game be-tween the Creator and a creature. This is the

.^...^Mt/V'V^

[ 1 6 ]

theme of the Book of Job, and of Paradise Lost

as well.In both these religious works the Devil is con-

ceived as J^laying; a_game with God, for the soulof Job, or the souls of mankind in general. Now,according to orthodox Jewish and Christian views,the Devil is one of God's creatures. Any othersupposition would lead to a moral dualism, savor-ing of Zoroastrianism and of that bastard offshootof Zoroastrianism and Christianity which we callManicheanism.

But if the Devil is one of God's creatures, thegame that furnishes the content of the Book ofJob and of Paradise Lost is a game between Godand one of his creatures. Such a game seems at firstsight a pitifully unequal contest. To play a gamewith an omnipotent, omniscient God is the act ofa fool; and, as we are told, the Devil is a master ofsubtlety. Any uprising of the rebel angels is fore-doomed to failure. It is not worth the Manfred-like rebellion of Satan to prove this point. Orelse that omnipotence which needs to establish it-self by celestial bombardments of thunderbolts isno omnipotence at all but merely a very greatstrength, and the Battle of the Angels might haveended with Satan on the celestial throne, and Godcast down into eternal damnation.

[ 17 ]

Thus, if we do not lose ourselves in the dogmasof omnipotence and omniscience, the conflict be-tween God and the Devil is a real conflict, andGod is something less than absolutely omnipotent.He is actually engaged in a conflict with his crea-ture, in which he may very well lose the game.And yet his creature is made by him according tohis own free will, and would seem to derive allits possibility of action from God himself. CanGod play a significant game with his own creature?Can any creator, even a limited one, play a signifi-cant game with his own creature?

In constructing machines with which he playsgames, the inventor has arrogated to himself thefunction of a limited creator, whatever the natureof the game-playing device that he has constructed. .This is in particular true in the case of game-playing machines that learn by experience. As Ihave already mentioned, such machines exist.How do these machines function? What degreeof success have they had?

Instead of functioning after the pattern of thevon Neumann game theory, they act in a mannermuch more closely analogous to the proceedingof the ordinary human game player. At each stage,they are subject to constraints that restrict thechoice of the next move to one which is legal ac-

J

t< ;£' : ' • ' " • •n

^^

[ 18 ]

cording to the rules of the game. One of thesemoves must be selected according to some norma-tive criterion of good play.

Here, the experience of the human player ofthe game furnishes a number of clues to be usedin picking out this criterion. In checkers or chessit is generally disadvantageous to lose pieces andgenerally advantageous to take an opponent'spiece. The player who retains his mobility andright of choice, as well as the player who securesthe command of a large number of squares, isusually better off than his opponent who has beencareless in these respects.

These criteria of good play hold throughoutthe game, but there are other criteria that belongto a particular stage of the game. At the end ofthe game, when the pieces are sparse on the board,it becomes more difficult to close with the op-ponent for the kill. At the beginning of the game—and this is a far more important factor in chessthan in checkers—the pieces are arranged in away that tends to make them immobile and im-potent, and a development is needed that will getthem out of one another's way, both for offensiveand defensive purposes. Furthermore, with thegreat variety of pieces in chess as compared withthe poverty of checkers in this regard, there are

[ 19 ]

in chess a large number of special criteria of goodplay, the importance of which has been proved bycenturies of experience.

These considerations may be combined (eitheradditively or in some more complicated way) togive a figure of merit for the next move to beplayed by the machine. This may be done in asomewhat arbitrary manner. Then the machinecompares the figures of merit of the moves legallypossible and chooses that move with the largestfigure of merit. This gives one way of automatizingthe next move.

This automatization of the next move is notnecessarily, or even usually, an optimum choice,but it is a choice, and the machine can go on play-ing. To judge the merit of this way of mechaniz-ing a game, one should divest oneself of all theimages of mechanization belonging to the techni-cal devices used, or the physical image of human-ity as displayed by the ordinary game player.Luckily this is easy, for it is what we always do incorrespondence chess.

In correspondence chess, one player sends themoves by mail to the other, so that the only con-nection between the two players is a written docu-ment. Even in this sort of chess, a skilled playersoon develops an image of the personality of his

^

[ 2 0 ]

opponent—of his chess personality, that is. Hewill learn if his opponent is hasty or careful; ifhe is easily tricked or subtle; if he learns thetricks of the other player, or can be fooled againand again by the same elementary strategy. Allthis is done, I repeat, with no further communi-cation than the playing of the game itself.

From this point of view, the player, be he a manor a machine, who plays by a simple table of merit,chosen once for all and unalterable, will give theimpression of a rigid chess personality. Once youhave found out his weak point, you have found itout for all time. If a strategem has worked onceagainst him, it will always work. A very smallnumber of plays are enough to establish his tech-nique.

So much for the mechanized player who does notlearn. However, there is nothing to prevent amechanized player from playing in a more intel-ligent way. For this he must keep a record of pastgames and past plays. Then, at the end of eachgame or each sequence of games of a determinedsort, his mechanism is put to a totally differentsort of use.

In building up the figure of merit, certainconstants are introduced which might have beenchosen differently. The relative importance of thecommand constant, the mobility constant, and the

[21 ]

number-of-pieces constant might have been10:3:2, instead of 9:4:4. The new use of the regu-lating machine is to examine games already playedand, in view of the outcome of these, to give afigure of merit; not to the plays already made, butto the weighting chosen for the evaluation of theseplays.

In this way, the figure of merit is continuallybeing re-evaluated, in such a manner as to give ahigher figure of merit for configurations occurringchiefly in winning games, and a lower figure ofmerit for situations occurring chiefly in losinggames. The play will continue with this newfigure of merit, which may be established in manyways differing in detail. The result will be thatthe game-playing machine will continually trans-form itself into a different machine, in accordancewith the history of the actual play. In this, theexperience and success, both of the machine andof its human opponent, will play a role.

In playing against such a machine, which ab-sorbs part of its playing personality from its op-ponent, this playing personality will not be ab-solutely rigid. The opponent may find thatstrategems which have worked in the past, willfail to work in the future. The machine maydevelop an uncanny canniness.

It may be said that all this unexpected intelli-

[ 2 2 ]

gence of the machine has been built into it by itsdesigner and programmer. This is true in onesense, but it need not be true that all of the newhabits of the machine have been explicitly fore-seen by him. If this were the case, he should haveno difficulty in defeating his own creation. Thisis not in accordance with the actual history ofSamuel's machine.

As a matter of fact, for a considerable periodSamuel's machine was able to defeat him ratherconsistently, after a day or so of working in.It must be said that Samuel, by his own statement,was no checker expert to begin with, and that witha little further instruction and practice he wasable to win over his own creation. It will not do,however, to belittle the fact that there was a periodwhen the machine was rather a consistent victor.It did win, and it did learn to win; and the methodof its learning was no different in principle fromthat of the human being who learns to playcheckers.

It is ttue that the choice o£ policies open to thechecker-playing machine is almost certainly nar-rower than that open to the human checker player;but it is also true that the choice o£ policies e£-iecuveVy open to the "human checker player isnot unlimited. He may be restrained from, a -wider

[ 2 3 ]

choice only by the bounds of his intelligence andimagination, but those are very real bounds in-deed and not of a sort essentially different fromthe bounds of the machine.

Thus the checker-playing machine already playsa reasonably good game, which with a little furtherstudy of the end game and a little more skill inapplying the coup de grace may begin to approachmaster level. If it were not for the fact that theinterest in checker championship has already beengreatly diminished by the cut-and-dried nature ofnormal human play, the checker-playing machinemight already be said to have destroyed the inter-est in checkers as a game. It is not surprising thatpeople are already beginning to ask, will chess gothe same way? And, when is this castastrophe tobe expected?

Chess-playing machines, or machines to play atleast an appreciable part of a chess game, are al-ready in existence, but they are comparativelypoor things. They do not, at their best, go beyondthe level of a competent game between amateurswith no pretense to chess mastership, and theyvery seldom reach that level. This is largely dueto the far greater complexity of chess than ofcheckers, both as to pieces and moves, and as tothe greater discrimination between the policies

•••A"

[ 2 4 ]

suitable for the different stages of the game. Therelatively small number of considerations neces-sary for mechanizing a checker game and the lowdegree of discrimination needed between its dif-ferent stages are totally inadequate for chess.

Nevertheless, I find it to be the general opinionof those of my friends who are reasonably profi-cient chess players that the days of chess as aninteresting human occupation are numbered.They expect that within from ten to twenty-fiveyears, chess machines will have reached the masterclass, and then, if the efficient but somewhat raa-chinelike methods of the Russian school have al-lowed chess to survive so long, it will cease tointerest human players.

Be this as it may, there will be many othergames that will continue to offer a challenge tothe games engineer. Among these is Go, that FarEastern game in which there are seven or moredifferent levels of recognized mastery. Moreover,war and business are conflicts resembling games,and as such, they may be so formalized as to consti-tute games with definite rules. Indeed, I have noreason to suppose that such formalized versions ofthem are not already being established as modelsto determine the policies for pressing the GreatPush Button and burning the earth clean for a

[ 2 5 ]

new and less humanly undependable order ofthings.

In general, a game-playing machine may beused to secure the automatic performance of anyfunction if the performance of this function issubject to a clear-cut, objective criterion of merit.In checkers and chess, this merit consists of thewinning of the game according to the acceptedrules of permissible play. These rules, whichare totally different from the accepted maxims ofgood play, are simple and inexorable. Not evenan intelligent child can be in doubt concerningthem for longer than it takes to read them whilefacing a board. There may be great doubt as tohow to win a game, but no doubt whatever as towhether it has been won or lost.

The chief criterion as to whether a line ofhuman effort can be embodied in a game iswhether there is some objectively recognizablecriterion of the merit of the performance of thiseffort. Otherwise the game assumes the formless-ness of the croquet game in Alice in Wonderland,where the balls were hedgehogs and kept unroll-ing themselves, the mallets were flamingoes, thearches cardboard soldiers who kept marching a-bout the field, and the umpire the Queen ofHearts, who kept changing the rules and sending

[ 2 6 ]

the players to the Headsman to be beheaded. Un-der these circumstances, to win has no meaning,and a successful policy cannot be learned, becausethere is no criterion of success.

However, given an objective criterion of suc-cess, the learning game may certainly be played,and is much closer to the way in which we learn toplay games than the image of a game given in. thevon Neumann theory. Unquestionably the tech-nique of the learning game is certain to be em-ployed in many fields of human effort which havenot yet been subjected to it. Nevertheless, as weshall see later, the determination of a sharp testfor good performance raises many problems con-cerning learning games.

Ill

The learning to which we have been alluding sofar is the learning of the individual, which occurswithin the time course of his individual privatelife. There is another type of learning of equalimportance—-phylogenetic learning, or learning inthe history of the race. It is this type of learningfor which one type of basis has been laid down byDarwin in his theory of natural selection.

The basis of natural selection lies in three facts.The first of these is that there is such a phenome-non as heredity: that an individual plant or ani-mal produces offspring after its own image. Thesecond is that these offspring are not completelyafter its own image but may differ from it in waysalso subject to heredity. This is the fact of varia-tion and by no means implies the very doubtful

[ 2 7 ]

[ 2 8 ]

inheritance of acquired characteristics. The thirdelement of Darwinian evolution is that the over-rich pattern of spontaneous variation is trimmedby the difference in the viability of different var-iations, most of which tend to diminish the proba-bility of continued racial existence, although some,perhaps a very few, tend to increase it.

The basis of racial survival and racial change—of evolution, as we call it—may be much morecomplicated than this, and probably is. For ex-ample, one very important type of variation isvariation of higher order—the variation of varia-bility. Again, the mechanism of heredity and var-iation generally involves the processes describedfunctionally by Mendel and structurally by thephenomenon of mitosis: the processes of the du-plication of genes and their separation, of theiraggregation into chromosomes, of linkage, and therest of it.

Nevertheless, behind all this fantastically com-plex concatenation of processes lies one verysimple fact: that in the presence of a suitable nu-tritive medium of nucleic acids and amino acids,a molecule of a gene, consisting itself of a highlyspecific combination of amino acids and nucleicacids, can cause the medium to lay itself down intoother molecules which either are molecules of the

[ 2 9 ]

same gene or of other genes differing from it byrelatively slight variations. It has been thoughtindeed that this process is strictly analogous to thatby which a molecule of a virus, a sort of molecu-lar parasite of a host, can draw together from thetissues of the host, which act as a nutrient medium,other molecules of the same virus. It is this act ofmolecular multiplication, whether of gene or ofvirus, which seems to represent a late stage of theanalysis of the vast and complicated process ofreproduction.

Man makes man in his own image. This seemsto be the echo or the prototype of the act of crea-tion, by which God is supposed to have made manin His image. Can something similar occur in theless complicated (and perhaps more understand-able) case of the nonliving systems that we callmachines?

What is the image of a machine? Can this image,as embodied in one machine, bring a machine ofa general sort, not yet committed to a particularspecific identity, to reproduce the original ma-chine, either absolutely or under some changethat may be construed as a variation? Can the newand varied machine itself act as an archetype, evenas to its own departures from its own archetypalpattern?

ir

sx

^

V5

NT^

v<P

i; u

v-\ ;'

[ 3 0 ]

It is the purpose of the present section to an-s w e r t n e s e questions, and to answer them by "yes."The value of what I shall say here, or rather ofwhat I have said elsewhere in a more technicalmanner,* and what I shall sketch here, is that ofwhat the mathematician calls an existence proof.I shall give one method in accordance with whichmachines can reproduce themselves. I do not saythat this is the only method in which this repro-duction can take place, for it is not; nor even thatit is the manner in which biological reproductiontakes place, for that it also is certainly not. How-ever, different as the mechanical and the biologicalreproduction may be, they are parallel processes,achieving similar results; and an account of theone may well produce relevant suggestions in the

study of the other."j7

In order to discuss intelligently the problem ofa machine constructing another after its own im-age, we must make the notion of image more pre-cise. Here we must be aware that there are imagesand images. Pygmalion made the statue of Gala-

* Cybernetics, The M.I.T. Press and John Wiley 8c Sons,Inc., New York • London, 2nd ed., Chapter IX, 1961.

•f The pattern of reproduction of genes by the splitting o£a double spiral of DNA needs to be supplemented by an ade-quate dynamics to be complete.

[ S I ]

tea in the image of his ideal beloved, b u t afterthe gods b rough t it to life, it became an image ofhis beloved in a m u c h more real sense. I t was nolonger merely a pictorial image b u t an operative

image.A reproducing lathe can make an image of a

gunstock model , which can be used for a gun-stock, b u t this is merely because the purposewhich a gunstock fulfills is very simple. O n theother hand, an electric circuit may fulfill a rela-tively complicated function, and its image, as re-produced by a pr in t ing press using metall ic inks,may itself function as the circuit it represents.These pr in ted circuits have obta ined a consider-able vogue in the techniques of mode rn electricalengineering.

T h u s , besides pictorial images, we may haveoperative images. These operative images, whichperform the functions of their original, may ormay not bear a pictorial likeness to it. W h e t h e rthey do or not , they may replace the original inits action, and this is a m u c h deeper similarity. I tis from the s tandpoint of operative similarity thatwe shall study the possible reproduc t ion of ma-chines. "~~~~

But what is a machine? F r o m one standpoint , >we may consider a machine as a p r ime mover, a

,vX

y

r

[ 3 2 ]

source o£ energy. This is not the standpoint whichwe shall take in this book. For us, a machine is adevice for converting incoming messages into out-going messages. A message, from this point ofview, is a sequence of quantities that represent sig-nals in the message. Such quantities may be elec-trical currents or potentials, but are not confinedto these, and may indeed be of a very differentnature. Moreover, the component signals may bedistributed continuously or discretely in time. Amachine transforms a number of such input mes-sages into a number of output messages, each out-put message at any moment depending on theinput messages up to this moment. As the engineerwould say in his jargon, a machine is a multiple-input, multiple-output transducer.

Most of the problems that we shall consider hereare not very different or very much more differentfrom those arising in single-input, single-outputtransducers. This might suggest to the engineersthat we are dealing with a problem which he al-ready knows very well: the classical problem ofthe electric circuit and its impedance or admit-tance or its voltage ratio.

This, however, is not exactly so. Impedance andadmittance and voltage ratio are notions whichcan be used with any degree of precision in only

[ 3 3 ]

the case of linear circuits: this is of circuits forwhich the addition of inputs as series in the timecorresponds to the addition of the correspondingoutputs. This will be the case for pure resistances,pure capacitances, and pure inductances, and forcircuits composed exclusively of these elements,connected according to Kirchhoff's laws. For these,the appropriate input on which to test the circuitis a trigonometrically oscillating input potentialthat can be varied in frequency and can be deter-mined in phase and amplitude. The output willthen also be a sequence of oscillations of the samefrequency, and by comparing it with the input inamplitude and phase, the circuit or transducer canbe completely characterized.

If a circuit is nonlinear, if, for example, it con-tains rectifiers or voltage limiters or other similardevices, the trigonometric input is not an ade-quate test input. In this case, a trigonometric in-put will not in general produce a trigonometricoutput. Moreover, strictly speaking, there are nolinear circuits, but only circuits with a better orworse approach to linearity.

The test input that we choose for the examina-tion of nonlinear circuits—and it can be used forlinear circuits, too—is of a statistical nature. The-oretically, unlike the trigonometric input, which

[ 3 4 ]

must be varied over the entire range of frequen-cies, it is a single statistical ensemble of inputsthat can be used for all transducers. It is knownas the shot effect. Shot-effect generators are well-defined pieces of apparatus with a physical exist-ence as instruments, and may be ordered from thecatalogues of several houses of electrical-instru-ment makers.*

The output of a transducer excited by a giveninput message is a message that depends at thesame time on the input message and on the trans-ducer itself. Under the most usual circumstances,a transducer is a mode of transforming messages,and our attention is drawn to the output messageas a transformation of the input message. However,there are circumstances, and these chiefly arisewhen the input message carries a minimum of

• Let me explain here what a shot-effect flow of electricityis. Electricity does not flow continuously but in a flow ofcharged particles, each with the same charge. In general, thesedo not flow at fixed intervals but with a random distributionin time, which superimposes on the steady flow fluctuationsthat are independent for nonoverlapping intervals of time.This produces a noise with a uniform distribution over fre-quency. This is often a disadvantage and limits the message-carrying power of the line. There are, however, cases such asthe present, where these irregularities are just what we wishto produce, and there are commercial devices for producingthem, These are known as shot-effect generators.

[ 35 ]

information, when we may conceive the informa-tion of the output message as arising chiefly fromthe transducer itself. No input message may beconceived as containing less information than therandom flow of electrons constituting the shot ef-fect. Thus the output of a transducer stimulatedby a random shot effect may be conceived as amessage embodying the action of the transducer.

As a matter of fact, it embodies the action of thetransducer for any possible input message. Thisis owing to the fact that over a finite time, thereis a finite (though small) possibility that the shoteffect will simulate any possible message withinany given finite degree of accuracy. Thus the sta-tistics of the message arising from a given trans-ducer under a given standardized statistical shot-effect input constitute an operative image of thetransducer, and it is quite conceivable that theymay be used for reconstituting an equivalent trans-ducer, in another physical embodiment. That is,if we know how a transducer will respond to ashot-effect input, we know ipso facto how it willrespond to any input.

The transducer—the machine, as instrumentand as message—thus suggests the sort of dualitywhich is so dear to the physicist, and is exempli-fied by duality between wave and particle. Again,

[ 36 ]

it suggests that biological alternation of genera-tions which is expressed by the bon mot—I do notremember whether it was Bernard Shaw's orSamuel Butler's—that a hen is merely an egg'sway of making another egg. The liver fluke in theliver of the sheep is but another phase of a raceof parasites that infects certain pond snails. Thusthe machine may generate the message, and themessage may generate another machine.

This is an idea with which I have toyed before—that it is conceptually possible for a humanbeing to be sent over a telegraph line. Let me sayat once that the difficulties far exceed my ingenui-ty to overcome them, and that I have no intentionto add to the present embarrassment of the rail-roads by calling in the American Telegraph andTelephone Company as a new competitor. At pres-ent, and perhaps for the whole existence of thehuman race, the idea is impracticable, but it isnot on that account inconceivable.

Quite apart from the difficulties of bringingthis notion into practice in the case of man, it isa thoroughly realizable concept in the case of theman-made machines of a lower degree of complex-ity. For this is precisely what I am proposing as amethod by which nonlinear transducers may re-produce themselves. The messages in which the

[ 3 7 ]

function of a given transducer may be embodiedwill also embody all those many embodiments ofa transducer with the same operative image. A-mong these there is at least one embodiment witha certain special sort of mechanical structure, andit is this embodiment that I am proposing to re-construct from the message carrying the opera-tional image of the machine.

In describing the particular embodiment that Ishall choose for the operational pattern of the ma-chine to be reproduced, I also describe the formalcharacter of the pattern. For this description to beanything more than a vague fantasy, it must beexpressed in mathematical terms, and mathematicsis not a language to be understood by the generalreader for whom this book is destined. Thus Imust forego precision at this place. I have alreadyexpressed these ideas in mathematical language,*so that I have fulfilled my duty to the specialist.If I leave the matter at that, I shall have done lessthan my duty for the reader for whose eyes thisbook is intended. I shall appear to have assertedonly some possibly empty claims. On the otherhand, a full presentation of my ideas here wouldbe utterly futile. Therefore, I shall confine my-

• Cybernetics, The M.I.T. Press and John Wiley & Sons,Inc., New York, 2nd ed., Chapter IX, 1961.

[ 3 8 ]

self in this book to as good a paraphrase as I canmake of the mathematics that is the real heart of

the matter.Even at this I am afraid that the following pages

will be rough going. For those who wish to avoidrough going at any cost, I must warn them to skipthis part of the text. I am writing them only forthose whose curiosity is sufficiently intense to in-duce them to read on despite such warnings.

IV

Reader, you have received the statutory warning,and anything you say in derogation of the follow-ing text may be used against youl

It is possible to multiply a machine, say a lineartransducer, by a constant and to add two machines.Remember that we take the output of a machineto be an electric potential, which we may supposeto be read on open circuit, if we take advantageof the modern devices which are known as cathodefollowers. By the use of potentiometers, and/ortransformers, we can multiply the output of atransducer by any constant, positive or negative.If we have two or more separate transducers, wemay add their output potentials for the sameinput by arranging them in series, and thus obtaina compound device with an output any sum of

[ 3 9 ]

[ 4 0 ]

the outputs of its component parts, each with anappropriate positive or negative coefficient.

We thus can introduce into the analysis andsynthesis of machines the familiar notions of poly-nomial developments and of series. These notionsare familiar in the case of trigonometric develop-ments and of Fourier series. It remains to give anappropriate repertory of competent transducersfor the formation of such a series, and we shallhave given a standard form for the realizationand, consequently, the duplication of an operativeimage.

Such a standard repertory of elementary ma-chines for the approximate representation of allmachines to what is, in an appropriate sense, anydegree of accuracy is known to exist. To describethis in .mathematical form is a matter of somedegree of complexity; but for the benefit of thestray mathematician who may happen to perusethese pages, I shall say that for any input messagethese devices yield products of the Hermite poly-nomials in the Laguerre coefficients of the pastof the input. This is really quite as specific andquite as complicated as it sounds.

Where can one obtain these devices? Not atpresent, I am afraid, as made-up devices in anelectrician's supply house; however, they can be

[41 ]

put together according to precise specifications.The components of these devices will be on the onehand resistances, capacitances, and inductances,familiar components of linear apparatus. To-gether with these, in order to obtain linearity, weneed multipliers which take two potentials asinputs and yield a potential which is the productof the two. Such devices are for sale on the mar-ket; and if they are somewhat costlier than wouldbe desirable in view of the number of themneeded, the development of invention may bringthe price down; and at any rate, expense is nota consideration of the same order as possibility.An extremely interesting device of this sort, work-ing on piezoelectric principles, has been made inthe laboratory of Professor Dennis Gabor* of theImperial College of Science and Technology. Heuses it for a device which is different in manyways from that which I have indicated but whichis also used for the analysis and synthesis of arbi-trary machines.

To return to the particular devices I have men-tioned, they have three properties that make them

• Gabor, D., "Electronic Inventions and Their Impact onCivilization," Inaugural Lecture, March S, 1959, Imperial Col-lege of Science and Technology, University of London, Eng-land.

• • • * • - , . .

[ 4 2 ]

suitable for the analysis and synthesis of the gen-eral machine. To begin with, they are a closedset of machines. That is, by combining them withappropriate coefficients, we may approximate toany machine whatever. Then they can be so pro-portioned as to be normal, in the sense that for arandom impulse of unit-statistical strength theywill give outputs of unit-statistical strength.Lastly, they are orthogonal. This means that ifwe take any two, give them the same standardizedshot-effect input, and multiply their outputs, theproduct of these outputs, averaged over the shot-effect statistics of all the inputs, will be zero.

In a development of a machine in this form,analysis is as easy as synthesis. Suppose that wehave a machine in the form of a "black box," thatis, a machine performing a definite stable opera-tion (one which does not go into spontaneousoscillation) but with an internal structure inac-cessible to us and which we do not know. Letus also have a "white box," or a machine withknown structure, representing one of the termsin the development of the black box. If then thetwo boxes have their input terminals attachedto the same shot-effect generator, and their outputterminals are attached to a multiplier that multi-plies their outputs, the product of their outputs,

[ 4 3 ]

averaged over the entire shot-effect distributionof their common input, will be the coefficients ofthe white box in the development of the blackbox as a sum of all the white boxes with appro-priate coefficients.

To obtain this is seemingly impossible, as itwould appear to involve the study of the systemfor the entire statistical range of shot-effect inputs.However, there is an important accident thatenables us to circumvent this difficulty. Thereis a theorem in mathematical physics which en-ables us in certain cases to replace averages overdistributions with time averages, not in everysingle case, but in a set of cases with the probabil-ity 1. In the particular case of the shot effect, itmay be proved rigorously that the conditions forthe validity of this theorem are fulfilled. Thuswe may replace the average over the entire ensem-ble of possible shot effects, necessary to obtainthe coefficient of the white box in the develop-ment of the black box, by an average over time,and we shall get the right coefficient with theprobability 1. This, though not theoretically acertainty, is in practice equivalent to a certainty.

For this we need to be able to take a timeaverage of a potential. Luckily, apparatus for theobtaining of such time averages is well known and

J . ; * . .

[ 4 4 ]

easy to procure. It consists only of resistances,capacitances, and devices for measuring poten-tials. Thus our type of system is equally usefulfor the analysis and the synthesis of machines. Ifwe use it for the analysis of machines, and thenuse the same apparatus for the synthesis of amachine according to this analysis, we shall havereproduced the operative image of the machine.

This would seem at first sight to involve ahuman intervention. However, it is easy—mucheasier than the analysis and synthesis themselves—to cause the readings of the analysis to appear notas measurements on a scale but as the settings ofa number of potentiometers. Thus, as far as thenumber of terms available and the precision ofthe engineering technique permit, we have madean unknown black box, by its own operation,transfer its pattern of action to a complex whitebox initially capable of assuming any pattern ofaction. This is in fact very similar to what occursin the fundamental act of reproduction of livingmatter. Here, too, a substrate capable of assuminga large number of forms, molecular structures inthis case, is caused to assume a particular formby the presence of a structure—a molecule—thatalready possesses this form.

When I have presented this discussion of sell-

[ 4 5 ]

multiplying systems to philosophers and biochem-ists, I have been met with the statement, "But thetwo processes are entirely different! Any analogybetween life and the nonliving must be purelysuperficial. Certainly the detail of the processof biological multiplication is understood, and hasnothing to do with the process which you invokefor the multiplication of machines.

"On the one hand, machines are made of ironand brass, the finer chemical structure of whichhas nothing to do with their functions as partsof a machine. Living matter, however, is livingdown to the finest parts which characterize it asthe same sort, of matter—the molecules. Then,too, the multiplication of living matter occurs bya well-described template process, in which thenucleic acids determine the laying down of thechain of the amino acids, and this chain is double,consisting of a pair of complementary spirals.When these separate, each gathers to itself themolecular residues needed to reconstitute thedouble spiral of the original chain."

It is clear that the process of reproduction ofliving matter is different in its details from theprocess of the reproduction of machines which Ihave sketched. As is indicated by the work ofGabor, which I have already mentioned, there are

I1:.,

I'll

[46]

other ways of making a machine reproduce itself;and these, which are less rigid than the one I havegiven, are more likely to bear a resemblance tothe multiplication phenomenon in life. Livingmatter certainly has a fine structure more relevantto its function and multiplication than that ofthe parts of a nonliving machine, though thismay not be equally the case for those newermachines which operate according to the prin-ciples of solid-state physics.

However, even living systems are not (in allprobability) living below the molecular level.Furthermore, with all the differences betweenliving systems and the usual mechanical ones, itis presumptuous to deny that systems of the onesort may throw some light upon systems of theother. One respect in which this may well be thecase is that of the mutual convertibility of spatialand functional structure, on the one hand, andof messages in time, on the other. The templateaccount of reproduction is manifestly not thewhole story. There must be some communicationbetween the molecules of genes and the residuesto be found in the nutrient fluid, and this com-munication must have a dynamics. It is quite inthe spirit of modern physics to suppose that fieldphenomena of a radiative nature mediate the

[ 4 7 ]

dynamics of such communication. It will not doto state categorically that the processes of re-production in the machine and in the living beinghave nothing in common.

Pronouncements of this kind often seem tocautious and conservative minds to be less riskythan rash statements of analogy. However, if it isdangerous to assert an analogy on insufficientevidence, it is equally dangerous to reject onewithout proof of its inconsequentialness. Intel-lectual honesty is not the same thing as the refusalto assume an intellectual risk, and the refusaleven to consider the new and emotionally disturb-ing has no particular ethical merit.

For the idea that God's supposed creation ofman and the animals, the begetting of livingbeings according to their kind, and the possiblereproduction of machines are all part of the sameorder of phenomena is emotionally disturbing,just as Darwin's speculations on evolution andthe descent of man were disturbing. If it is anoffense against our self-pride to be compared toan ape, we have now got pretty well over it; andit is an even greater offense to be compared to amachine. To each suggestion in its own age thereattaches something of the reprobation that at-tached in earlier ages to the sin of sorcery.

f I

[ 4 8 ]

I have already mentioned the heredity of themachine and Darwin's evolution through naturalselection. For the genetics that we have attachedto the machine to be the basis of a kind of evolu-tion through natural selection, we must accountfor it by variation and the inheritance of varia-tions. However, the type of machine geneticswhich we suppose has room for both. Variationoccurs in the inaccuracy of the realization of thecopying process that we have discussed, while thecopied machine exemplified in our white box isitself available as an archetype for further copy-ing. Indeed, whereas in the original one-stagecopying the copy resembles its original in opera-tive image, but not in appearance, in the nextstage of copying the spatial structure is preserved,and the replica is a replica in that as well.

It is clear that the process of copying may usethe former copy as a new original. That is, varia-tions in the heredity are preserved, though theyare subject to a further variation.

V

I have said that the reprobation attaching informer ages to the sin of sorcery attaches now inmany minds to the speculations of modern cy-bernetics. For make no mistake, if but two hun-dred years ago a scholar had pretended to makemachines that should learn to play games or thatshould propagate themselves, he would surelyhave been made to assume the sanbenito, thegown worn by the victims of the Inquisition, andhave been handed over to the secular arm, withthe injunction that there be no shedding of blood;surely, that is, unless he could convince somegreat patron that he could transmute the basemetals into gold, as Rabbi Low of Prague, whoclaimed that his incantations blew breath of lifeinto the Golem of clay, had persuaded the Em-

[ 4 9 ]

[ 5 0 ]

peror Rudolf. For even now, i£ an inventor couldprove to a computing-machine company that hismagic could be of service to them, he could castblack spells from now till doomsday, without the

least personal risk.What is sorcery, and why is it condemned as a

sin? Why is the foolish mummery of the Black

Mass so frowned upon?The Black Mass must be understood from the

point of view of the orthpdox believer. For othersit is a meaningless if obscene ceremony. Thosewho participate in it are far nearer to orthodoxythan most of us realize. The principal element inthe Black Mass is the normal Christian dogmathat the priest performs a real miracle, and thatthe Element of the Host becomes the very Bloodand Body of Christ.

The orthodox Christian and the sorcerer agTeethat after the miracle of the consecration of theHost is performed, the Divine Elements are capa-ble of performing further miracles. They agreemoreover that the miracle of transubstantiationcan be performed only by a duly ordained priest.Furthermore, they agree that such a priest canneveT lose the power to perform the miracle,though if he is unfrocked he performs it at thesure peril of damnation.

[51 ]

Under these postulates, what is more naturalthan that some soul, damned but ingenious,should have hit upon the idea of laying his holdon the magic Host and using its powers for hispersonal advantage. It is here, and not in anyungodly orgies, that the central sin of the BlackMass consists. The magic of the Host is intrinsi-cally good: its perversion to other ends than theGreater Glory of God is a deadly sin.

This was the sin which the Bible attributes toSimon Magus, for bargaining with Saint Peter forthe miraculous powers of the Christians. I canwell imagine the puzzled aggrievement of thepoor man when he discovered that these powerswere not for sale, and that Peter refused to acceptwhat was, in Simon's mind, an honorable, accept-able, and natural bargain. It is an attitude thatmost of us have encountered when we have de-clined to sell an invention at the really flatteringterms offered us by a modern captain of industry.

Be that as it may, Christianity has always con-sidered simony as a sin, that is, the buying andselling of the offices of the Church and the super-natural powers implied therein. Dante indeedplaces it among the worst of sins, and consigns tothe bottom of his Hell some of the most notoriouspractitioners of simony of his own times. How-

SSJf

ever, simony was a besetting sin of the highlyecclesiastical world in which Dante lived, and isof course extinct in the more rationalistic and ra-tional world of the present day.

It is extinct! It is extinct. It is extinct? Perhapsthe powers of the age of the machine are not trulysupernatural, but at least they seem beyond theordinary course of nature to the man in thestreet. Perhaps we no longer interpret our dutyas obliging us to devote these great powers to thegreater glory of God, but it still seems improperto us to devote them to vain or selfish purposes.There is a sin, which consists of using the magicof modern automatization to further personalprofit or let loose the apocalyptic terrors of nu-clear warfare. If this sin is to have a name, letthat name be Simony or Sorcery.

For whether we believe or not in God and hisgreater glory, not all things are equally permittedto us. The late Mr. Adolf Hitler to the contrary,we have not yet arrived at that pinnacle of sub-lime moral indifference which puts us beyondGood and Evil. And just so long as we retain onetrace of ethical discrimination, the use of greatpowers for base purposes will constitute the fullmoral equivalent of Sorcery and Simony.

As long as automata can be made, whether in

the metal or merely in principle, the study oftheir making and their theory is a legitimatephase of human curiosity, and human intelligenceis stultified when man sets fixed bounds to hiscuriosity. Yet there are aspects of the motives toautomatization that go beyond a legitimate curios-ity and are sinful in themselves. These are to beexemplified in the particular type of engineer andorganizer of engineering which I shall designateby the name of gadget worshiper.

I am most familiar with gadget worshipers inmy own world, with its slogans of free enterpriseand the profit-motive economy. They can anddo exist in that through-the-looking-glass worldwhere the slogans are the dictatorship of theproletariat and Marxism and communism. Powerand the search for power are unfortunately reali-ties that can assume many garbs. Of the devotedpriests of power, there are many who regard withimpatience the limitations of mankind, and inparticular the limitation consisting in man's un-dependability and unpredictability. You mayknow a mastermind of this type by the subor-dinates whom he chooses. They are meek, self-effacing, and wholly at his disposal; and on ac-count of this, are generally ineffective when theyonce cease to be limbs at the disposal of his

: • ' • i

I;

t,

[ 5 4 ]

brain. They are capable of great industry but oflittle independent initiative—the chamberlains ofthe harem of ideas to which their Sultan is

wedded.In addition to the motive which the gadget

worshiper finds for his admiration of the ma-chine in its freedom from the human limitationsof speed and accuracy, there is one motive whichit is harder to establish in any concrete case, butwhich must play a very considerable role never-theless. It is the desire to avoid the personalresponsibility for a dangerous or disastrous deci-sion by placing the responsibility elsewhere: onchance, on human superiors and their policieswhich one cannot question, or on a mechanicaldevice which one cannot fully understand butwhich has a presumed objectivity. It is this thatleads shipwrecked castaways to draw lots to deter-mine which of them shall first be eaten. It is thisto which the late Mr. Eichmann entrusted hisable defense. It is this that leads to the issue ofsome blank cartridges among the ball cartridgesfurnished to a firing squad. This will unquestion-ably be the manner in which the official whopushes the button in the next (and last) atomicwar, whatever side he represents, will salve hisconscience. And it is an old trick in magic—one,

[ 5 5 ]

however, rich in tragic consequences—to sacrificeto a vow the first living creature that one seesafter safe return from a perilous undertaking.

Once such a master becomes aware that someof the supposedly human functions of his slavesmay be transferred to machines, he is delighted.At last he has found the new subordinate—efficient, subservient, dependable in his action,never talking back, swift, and not demanding asingle thought of personal consideration.

Such subordinates are contemplated in Capek'splay R.U.R. The Slave of the Lamp makes nodemands. He does not ask for a day off each weekor a television set in his servant's quarters. In fact,he demands no quarters at all but appears outof nowhere when the lamp is rubbed. If your pur-poses involve you in a course sailing pretty close-hauled to the moral wind, your slave will neverreprove you, even to the extent of a questioningglance. Now you are free, to dree your weirdwhere destiny may lead you!

This type of mastermind is the mind of the*sorcerer in the full sense of the word. To thissort of sorcerer, not only the doctrines of theChurch give a warning but the accumulated com-mon sense of humanity, as accumulated in leg-ends, in myths, and in the writings of the con-

fVI

i l l

[ 5 6 ]

scious literary man. All of these insist that notonly is sorcery a sin leading to Hell but it is apersonal peril in this life. It is a two-edged sword,and sooner or later it will cut you deep.

In the Thousand Nights and a Night, the taleof the "Fisherman and the Jinni" is well to thepoint. A fisherman, casting his nets off the coastof Palestine, pulls up an earthen jar sealed withthe Seal of Solomon. He breaks the seal, smokeboils out of the jar and takes the figure of anenormous Jinni. The Being tells him that he isone of those rebellious beings imprisoned by thegreat King Solomon; that at first he had intendedto reward anyone who liberated him with powerand riches; but that in the course of ages, he hadcome to the decision to slay the first mortal hemight meet, and above all the man who shouldbring him freedom.

Fortunately for himself, the fisherman seemsto have been an ingenious fellow, with a rich lineof blarney. He plays on the vanity of the Jinni*and persuades him to show how such a greatBeing could have been confined in such a smallvessel by going back again into the jar. He clapsthe sealed lid on again, throws the vessel backinto the sea, congratulates himself on his narrowescape, and lives happily ever after.

[ 57 ]

In other tales, the chief character does not haveso accidental an encounter with magic and eithercomes even closer to the edge of catastrophe orincurs utter ruin. In Goethe's poem, The Sor-cerer's Apprentice, the young factotum whocleans the master's magic garments, sweeps hisfloors, and fetches his water is left alone by thesorcerer, with the command to fill his water butt.Having a full portion of that laziness which is thetrue mother of invention—it led the boy whotended Newcomen's engine to fasten the valvestring which he was to pull to the crosshead, andso led to the idea of the automatic valve gear—the lad remembers some fragments of an incanta-tion which he has heard from his master and putsthe broom to work fetching water. This task thebroom carries out with promptness and efficiency.When the water begins to overflow the top ofthe water butt, the boy finds that he does notremember the incantation that the magician hasused to stop the broom. The boy is well on theway to be drowned when the magician comesback, recites the words of power, and gives theapprentice a good wholesome scolding.

Even here the final catastrophe is avertedthrough a deus ex machina. W. W. Jacobs, anEnglish writer of the beginning of the present

1

[ 5 8 ]

century, has carried the principle to its stark logi-cal conclusion in a tale called "The Monkey'sPaw,"* which is one of the classics of the literature

o£ horror.In this tale, an English working family is sitting

down to dinner in its kitchen. The son leaves towork at a factory, and the old parents listen to thetales of their guest, a sergeant-major back fromservice in the Indian army. He tells them ofIndian magic and shows them a dried monkey'spaw, which, he tells them, is a talisman which hasbeen endowed by an Indian holy man with thevirtue of giving three wishes to each of three suc-cessive owners. This, he says, was to prove thefolly of defying fate.

He says that he does not know what were thefirst two wishes of the first owner, but that the lastone was for death. He himself was the secondowner, but his experiences were too terrible torelate. He is about to cast the paw on the coalfire, when his host retrieves it, and despite allthe sergeant-major can do, wishes for £200.

Shortly thereafter there is a knock at the door.

• Jacobs, W. W.( "The Monkey's Paw," in The Lady ofthe Barge, Dodd, Mead, and Company; also in Modern ShortStories, Ashmun, Margaret, Ed., The Macmillan Co., NewYork, 1915.

[ 5 9 ]

A very solemn gentleman is there from the com-pany which has employed his son. As gently as hecan, he breaks the news that the son has beenkilled in an accident at the factory. Withoutrecognizing any responsibility in the matter, thecompany offers its sympathy and £200 as a sola-tium.

The parents are distracted, and at the mother'ssuggestion, they wish the son back again. By nowit is dark without, a dark windy night. Againthere is a knocking at the door. Somehow theparents know that it is their son, but not in theflesh. The story ends with the third wish, that theghost should go away.

The theme of all these tales is the danger ofmagic. This seems to lie in the fact that the opera-tion of magic is singularly literal-minded, andthat if it grants you anything at all it grants whatyou ask for, not what you should have asked foror what you intend. If you ask for £200, and donot express the condition that you do not wishit at the cost of the life of your son, £200 youwill get, whether your son lives or dies.

The magic of automation, and in particular themagic of an automatization in which the deviceslearn, may be expected to be similarly literal-minded. If you are playing a game according to

[ 6 0 ]

certain rules and set the playing-machine to playfor victory, you will get victory if you get any-thing at all, and the machine will not pay theslightest attention to any consideration exceptvictory according to the rules. If you are playinga war game with a certain conventional inter-pretation of victory, victory will be the goal at anycost, even that of the extermination of your ownside, unless this condition of survival is explicitlycontained in the definition of victory accordingto which you program the machine.

This is more than a purely innocent verbalparadox. I certainly know nothing to contradictthe assumption that Russia and the United States,either or both of them, are toying with the ideaof using machines, learning machines at that, todetermine the moment of pushing the atomic-bomb button which is the ultima ratio of thispresent world of ours.

For many years all armies have played wargames, and these games have always been behindthe times. It has been said that in every war,the good generals fight the last war, the bad onesthe war before the last. That is, the rules of thewar game never catch up with the facts of thereal situation.

This has always been true, though in periods

[61 ]

of much war, there has always been a body ofseasoned warriors who have experienced warunder conditions that have not varied veryrapidly. These experienced men are the only "warexperts," in the true sense of the word. At present,there are no experts in atomic warfare: no men,that is, who have any experience of a conflict inwhich both sides have had atomic weapons attheir disposal and have used them. The destruc-tion of our cities in an atomic war, the demoral-ization of our people, the hunger and disease, andthe incidental destruction (which well may befar greater than the number of deaths from explo-sion and immediate fallout) are known only byconjecture.

Here those who conjecture the least amount ofsecondary damage, the greatest possibility of thesurvival, of the nations under the new type ofcatastrophe, can and do draw about themselvesthe proud garment of patriotism. If war is utterlyself-destructive, if a military operation has lostall possible sense, why then the Army and Navyhave lost much of their purpose, and the poorloyal generals and admirals will be thrown outof work. The missile companies will no longerhave the ideal market where all the goods canbe used only once and do not remain to compete

[ 6 2 ]

with other goods yet to be made. The clergy willbe cheated of the enthusiasm and exultationwhich go with a crusade. In short, when there isa war game to program such a campaign, therewill be many to forget its consequences, to askfor the £200 and to forget to mention that theson should survive.

While it is always possible to ask for somethingother than we really want, this possibility is mostserious when the process by which we are toobtain our wish is indirect, and the degree towhich we have obtained our wish is not clear untilthe very end. Usually we realize our wishes, inso-far as we do actually realize them, by a feedbackprocess, in which we compare the degree ofattainment of intermediate goals with our antic-ipation of them. In this process, the feedback goesthrough us, and we can turn back before it istoo late. If the feedback is built into a machinethat cannot be inspected until the final goal isattained, the possibilities for catastrophe aregreatly increased. I should very much hate to rideon the first trial of an automobile regulated byphotoelectric feedback devices, unless there weresomewhere a handle by which I could take overcontrol if I found myself driving smack into atree.

[ 6 3 ]

The gadget-minded people often have the illu-sion that a highly automatized world will makesmaller claims on human ingenuity than does thepresent one and will take over from us our needfor difficult thinking, as a Roman slave who wasalso a Greek philosopher might have done for hismaster. This is palpably false. A goal-seekingmechanism will not necessarily seek our goalsunless we design it for that purpose, and in thatdesigning we must foresee all steps of the processfor which it is designed, instead of exercising atentative foresight which goes up to a certainpoint, and can be continued from that point onas new difficulties arise. The penalties for errorsof foresight, great as they are now, will be enor-mously increased as automatization comes intoits full use.

At present, there is a great vogue for the ideaof avoiding some of the dangers, and in particularthe dangers accompanying atomic war, by so-called "failsafe" devices. The notion behind thisis that even if a device does not perform properly,it is possible to direct the mode of its failure ina harmless way. For example, if a pump is tobreak down, it is often much better that it doso by emptying itself of water than by explodingunder pressure. When we are facing a particular

[ 6 4 ]

understood danger, the failsafe technique is legiti-mate and useful. However, it is of very little valueagainst a danger whose nature has not been al-ready recognized. If, for example, the danger isa remote but terminal one to the human race,involving extermination, only a very careful studyof society will exhibit it as a danger until it isupon us. Dangerous contingencies of this sort donot bear a label on their face. Thus the failsafetechnique, while it may be necessary to avoid ahuman catastrophe, can most emphatically not beregarded as a sufficient precaution.

As engineering technique becomes more andmore able to achieve human purposes, it mustbecome more and more accustomed to formulatehuman purposes. In the past, a partial and inade-quate view of human purpose has been relativelyinnocuous only because it has been accompaniedby technical limitations that made it difficult forus to perform operations involving a carefulevaluation of human purpose. This is only oneof the many places where human impotence hashitherto shielded us from the full destructiveimpact of human folly.

In other words, while in the past humanity hasfaced many dangers, these have been much easierto handle, because in many cases peril offered

[ 6 5 ]

itself from one side only. In an age where hungeris the great threat, • there is safety in an'increasedproduction of food, and not much danger from it.With a higher death rate (and above all, a highinfant death rate) and a medicine of very littleeffectiveness, the individual human life was ofthe greatest value, and it was appropriate to en-join upon us to be fruitful and multiply. Thepressure of the threat of hunger was like thepressure of gravity, to which our muscles, bones,and tendons are always attuned.

The change in the tensions of modern life,which results both from the rise of new strainsand the disappearance of old ones, is ratheranalogous to the new problems of space travel. Inthe weightlessness that is imposed upon us in aspace vehicle, this one-directional constant force,upon which we count so much in our daily life,is no longer present. The traveler in such a spacevehicle must have handles to which to cling,squeeze bottles for his food and drink, variousdirectional auxiliaries from which he can judgehis position, and even at that, though it nowappears that his physiology will not be too seri-ously affected, he may scarcely be as comfortableas he would like. Gravity is our friend at least asmuch as it is our enemy.

".

ft Si '

m

[ 6 6 ]

Similarly, in the absence of hunger, overpro-duction of food, purposelessness, and an attitudeof waste and squandering become serious prob-lems. Improved medicine is one factor contribut-ing to overpopulation, which is by far the mostserious danger confronting mankind at the mo-ment. The old maxims by which humanity haslived so long—such as "a penny saved is a pennygained"—are no longer to be taken as valid with-out question.

I have been to dinner with a group of doctors—they were talking freely among themselves, andthey were sufficiently self-confident not to beafraid of saying unconventional things—whenthey began to discuss the possibility of a radicalattack upon the degenerative disease known asold age. They did not consider it as beyond allpossibility of medical attack, but rather lookedforward to the day—perhaps not too far in thefuture—when the time of inevitable death shouldbe rolled back, perhaps into the indefinite future,and death would be accidental, as it seems to bewith giant sequoias and perhaps some fish.

I am not saying that they were right in thisconjecture (and I am quite sure that they wouldnot claim it to be more than a conjecture), but theweight of the names supporting it—there was a

^

[ 6 7 ]

Nobel laureate present—was too great to allowme to reject the suggestion out of hand. Consol-ing as the suggestion may seem at first sight, it isin reality very terrifying, and above all for thedoctors. For if one thing is clear, it is that human-ity as such could not long survive the indefiniteprolongation of all lives which come into being.Not only would the nonselfsupporting part ofhumanity come to outweigh the part on whichits continued existence depends, but we shouldbe under such a perpetual debt to the men of thepast that we should be totally unprepared to facethe new problems of the future.

It is unthinkable that all lives should be pro-longed in an indiscriminate way. If, however, thereexists the possibility of indefinite prolongation,the termination of a life or even the refusal orneglect to prolong it involves a moral decision ofthe doctors. What will then become of the tradi-tional prestige of the medical profession as priestsof the battle against death and as ministers ofmercy? I will grant that there are cases even atpresent when doctors qualify this mission of theirsand decide not to prolong a useless and miserablelife. They will often refuse to tie the umbilicalcord of a monster; or when an old man sufferingfrom an inoperable cancer falls victim to the "old

K

[ 6 8 ]

man's friend," hypostatic pneumonia, they willgrant him the easier death rather than exact fromhim the last measure of pain to which survival willcondemn him. Most often this is done quietly anddecently, and it is only when some incontinentfool blabs the secret that the courts and the papersare full of the talk of "euthanasia."

But what if such decisions, instead of being rareand unmentioned, will have to be made, not ina few special cases, but in the case of almost everydeath? What if every patient comes to regardevery. doctor, not only as his savior but his ulti-mate executioner? Can the doctor survive thispower of good and evil that will be thrust uponhim? Can mankind itself survive this new orderof things?

It is relatively easy to promote good and tofight evil when evil and good are arranged againstone another in two clear lines, and when those onthe other side are our unquestioned enemies,those on our side our trusted allies. What, how-ever, if we must ask, each time in every situation,where is the friend and where the enemy? What,moreover, when we have put the decision in thehands of an inexorable magic or an inexorablemachine of which we must ask the right questionsin advance, without fully understanding the

f'[ 69 ]

operations of the process by which they will beanswered? Can we then be confident in the actionof the Monkey's Paw from which we have re-quested the grant of the £200?

No, the future offers very little hope for thosewho expect that our new mechanical slaves willoffer us a world in which we may rest from think-ing. Help us they may, but at the cost of su-preme demands upon our honesty and our intel-ligence. The world of the future will be an evermore demanding struggle against the limitationsof our intelligence, not a comfortable hammockin which we can lie down to be waited upon byour robot slaves.

j ^

VI

ivHV* Thus one of the great future problems which

we must face is that of the relation between manand the machine, of the functions which shouldproperly be assigned to these two agencies. On thesurface, the machine has certain clear advantages.It is faster in its action and more uniform, or atleast it can be made to have these properties ifit is well designed. A digital computing machinecan accomplish in a day a body of work thatwould take the full efforts of a team of computersfor a year, and it will accomplish this work witha minimum of blots and blunders.

On the other hand, the human being has cer-tain nonnegligible advantages. Apart from thefact that any sensible man would consider the pur-poses of man as paramount in the relations be-

: --.N

X

[ 7 1 ]

n

[ 7 2 ]

tween man and the machine, the machine is farless complicated than man and has far less scopein the variety of its actions. If we consider theneuron of the gray matter of the brain as ofthe order 1/1,000,000 of a cubic millimeter, andthe smallest transistor obtainable at present asof the order of a cubic millimeter, we shall nothave judged the situation too unfavorably from thepoint of view of the advantage of the neuron inthe matter of smaller bulk. If the white matterof the brain is considered equivalent to the wiringof a computer circuit, and if we take each neuronas the functional equivalent of a transistor, thecomputer equivalent to a brain should occupya sphere of something like thirty feet in diameter.Actually, it would be impossible to construct acomputer with anything like the relative closenessof the texture of the brain, and any computerwith powers comparable with the brain wouldhave to occupy a fair-sized office building, if nota skyscraper. It is hard to believe that, as com-pared with existing computing machines, thebrain does not have some advantages correspond-ing to its enormous operational size, which is in-comparably greater than what we might expectof its physical size.

Chief among these advantages would seem to

[ 7 3 ]

be the ability of the brain to handle vague ideas,as yet imperfectly defined. In dealing with these,mechanical computers, or at least the mechanicalcomputers of the present day, are very nearlyincapable of programming themselves. Yet inpoems, in novels, in paintings, the brain seems tofind itself able to work very well with materialthat any computer would have to reject as form-less.

Render unto man the things which are man'sand unto the computer the things which are thecomputer's. This would seem the intelligent ^J

policy to adopt when we employ men and com-puters together in common undertakings. It is apolicy as far removed from that of the gadgetworshiper as it is from the man who sees onlyblasphemy and the degradation of man in theuse of any mechanical adjuvants whatever tothoughts. What we now need is an independent'study of systems involving both human andmechanical elements. This system should not beprejudiced either by a mechanical or antimechani-cal bias. I think that such a study is already underway and that it will promise a much better com- Vprehension of automatization. \

One place where we can and do use such mixedsystems is in the design of prostheses, of devices \

,w

\

r.

<** >

^

\ « ^ '

v|

[ 7 4 ]

that replace limbs or damaged sense organs. Awooden leg is a mechanical replacement for a lostleg of flesh and blood, and a man with a woodenleg represents a system composed both of mechani-cal and human parts.

Perhaps the classical peg leg is not interesting, asit replaces the lost limb only in the most elemen-tary way, nor is the limb-shaped wooden leg muchmore interesting. However, there is some workbeing done on artificial limbs in Russia, in theUnited States, and elsewhere by a group to whichI belong. This work is much more interesting inprinciple and really makes use of cybernetical

ideas.Let us suppose that a man has lost a hand at

the wrist. He has lost a few muscles that servechiefly to spread the fingers and to bring themtogether again, but the greater part of the musclesthat normally move the hand and the fingers arestill intact in the stump of the forearm. Whenthey are contracted, they move no hand andfingers, but they do produce certain electricaleffects known as action potentials. These can bepicked up by appropriate electrodes and can beamplified and combined by transistor circuits.They can be made to control the motions of anartificial hand through electric motors, which

[ 7 5 ]

derive their power through appropriate electricbatteries or accumulators, but the signals control-ling them are sent through transistor circuits. Thecentral nervous part of the control apparatus isgenerally almost intact and should be used.

Such artificial hands have already been madein Russia, and they have even permitted somehand amputees to go back to effective work. Thisresult is facilitated by the circumstance that thesame nervous signal which was effective in produc-ing a muscular contraction before the amputationwill still be effective in controlling the motormoving the artificial hand. Thus the learning ofthe use of these hands is made much easier andmore natural.

However, as such, an artificial hand cannot feel,and the hand is as much an organ of touch as ofmotion. But wait, why can an artificial hand notfeel? It is easy to put pressure gauges into theartificial fingers, and these can communicate elec-tric impulses to a suitable circuit. This can in itsterm activate devices acting on the living skin,say, the skin of the stump. For example, thesedevices may be vibrators. Thereby we can pro-duce a vicarious sensation of touch, and we maylearn to use this to replace the missing naturaltactile sensation. Moreover, there are still sensory

[ 7 6 ]

kinesthetic elements in the mutilated muscles,and these can be turned to good account./

Thus there is a new engineering of prosthesespossible, and it will involve the construction ofsystems of a mixed nature, involving both humanand mechanical parts. However, this type ofengineering need not be confined to the replace-ment of parts that we have lost. There is a pros-thesis of parts which we do not have and whichwe never have had. The dolphin propels itselfthrough the water by its flukes, and avoids ob-stacles by listening for the reflections of soundswhich it itself emits. What is the propeller of aship but an artificial pair of flukes, or the depth-sounding apparatus but a vicarious sound-detect-ing and sound-emitting apparatus like that of thedolphin? The wings and jet engines of an airplanereplace the wings of the eagle, and the radar itseyes, while the nervous system that combinesthem is eked out by the automatic pilot and othersuch navigation devices.

Thus human-mechanical systems have a largepractical field in which they are useful, but insome situations they are indispensable. We havealready seen that learning machines must actaccording to some norm of good performance. Inthe case of game-playing machines, where the

!.

[ 7 7 ]

permissible moves are arbitrarily established inadvance, and the object of the game is to win bya series of permissible rules according to a strictconvention that determines winning or losing,this norm creates no problem. However, there aremany activities that we should like to improve bylearning processes in which the success of theactivity is itself to be judged by a criterion involv-ing human beings, and in which the problem ofthe reduction of this criterion to formal rulesis far from easy.

A field in which there is a great demand forautomatization, and a great possible demand forlearning automatization, is that of mechanicaltranslation. In view of the present metastablestate of international tension, the United Statesand Russia are filled with an equal and oppositenecessity for each to find out what the other isthinking and saying. Since there is a limited num-ber of competent human translators on both sides,each side is exploring the possibilities of mechani-cal translation. This has been achieved after afashion, but neither the literary qualities nor theintelligibility of the products of these translationshas been sufficient to excite any great enthusiasmon either part. None of the mechanical devices fortranslation has proved itself deserving of trust

[ 7 8 ]

when momentous issues depend on the aoof the translation.

Perhaps the most promising way of medi^translation is through a learning machine,^Fsuch a machine to function, we must havecriterion of a good translation. This will involveone of two things: either a complete set of otjectively applicable rules determining whentranslation is good, or some agency that is cap-iblof applying a criterion of good performance affrom such rules.

The normal criterion of good translation is irtelligibility. The people who read the languaginto which the translation is made must obtainsame impression of the text as that obtained h(the original by people understanding theguage of the original. If this criterion maylittle difficult to apply, we can give one that,necessary if not sufficient. Let us suppose that •have two independent translating machines, sayone from English into Danish and the other from iDanish into English. When a text in English hggibeen translated into Danish by the first mlet the second translate it back into English. TKethe final translation must be recognizably eq-uivjKJlent to the original, by a person acquainted'English.

It is conceivable that a set of formal rules be!

[ 79 ]

"given for such a translation so definite that theycan be entrusted to a machine, and so perfect

f that it will be sufficient for a translation to accordwith these rules to be satisfactory as to the cri-terion which we have given. I do not believe thatlinguistic science is so far advanced as to make aset of rules of this sort practicable, nor that thereis any prospect of its being so advanced in the pre-dictable future. Short of this state of affairs, atranslating machine will have a chance of error.If any important consideration of action or policy

"is to be determined by the use of a translation^machine, a small error or even a small chance of[error may have disproportionally large and serious•consequences.

It seems to me that the best hope of a reason-f ably satisfactory mechanical translation is to re-

place a pure mechanism, at least at first, by amechanicohuman system, involving as critic an

[expert human translator, to teach it by exercisesas a schoolteacher instructs human pupils. Per-haps at some later stage the memory of themachine may have absorbed enough human in-struction to dispense with later human participa-tion, except perhaps for a refresher course now

' and then. In this way, the machine would developlinguistic maturity.

Such a scheme would not eliminate the need

[ 8 0 ]

for a translation office to have attached to it anexpert linguist whose ability and judgment couldbe trusted. It would, or at least it might, enablehim to handle a considerably larger body of trans-lation than he could without mechanical assist-ance. This, in my mind, is the best that we canhope of mechanical translation.

Up to this point we have discussed the need ofa critic sensitive to human values, such as, for ex-ample, in a translating system where all but thecritic is mechanical. However, if the human ele-ment is to come in as the critic, it is quite reason-able to introduce the human element in otherstages, too. In a translation machine it is by nomeans essential that the mechanical element ofthe machine give us a single complete translation.It can give us a large number of alternative transla-tions for individual sentences that lie within thegrammatical and lexicographical rules and leaveto the critic the highly responsible task of censor-ship and selection of the mechanical translationthat best fits the sense. There is no need whateverwhy the use of the machine in translation shouldleave the formation of a complete closed transla-tion to the machine even in the sense that thistranslation is to be improved by a criticism as awhole. Criticism may begin at a much earlierstage.

[81 ]

What I have said about translating machineswill apply with equal or even greater force to ma-chines that are to perform medical diagnoses. Suchmachines are very much in vogue in plans for themedicine of the future. They may help pick outelements that the doctor will use in diagnosis,but there is no need whatever for them to com-plete the diagnosis without the doctor. Such aclosed, permanent policy in a medical machine issooner or later likely to produce much ill healthand many deaths.

A related problem requiring the joint consider-ation of mechanical and human elements is theoperational problem of invention, which has beendiscussed with me by Dr. Gordon Raisbeck ofArthur D. Little, Inc. Operationally, we mustconsider an invention not only with regard towhat we can invent but also as to how the inven-tion can be used and will be used in a human con-text. The second part of the problem is often moredifficult than the first and has a less closed method-ology. Thus we are confronted with a problem \ ' ^of development which is essentially a jlearning t ^ ^^Sproblem, not purely in the mechanical system but ^\ ,^ } ^in the mechanical system conjoined with society. vj\F ,y-VThis is definitely a case requiring a consideration ,. A.of the problem of the best joint use of machineand man.

A

V1XV

[ 8 2 ]

A similar problem and also a very pressing one, |is that of the use and development of militar\ de- \vices in conjunction with the evolution of tactics ^and strategy. Here, too, the operational probU.mcannot be separated from the automatization piob- _

lem.Not only is the problem of adapting the

chine to the present conditions by the propelof the intelligence of the translator or the docior the inventor one that must be faced now biit is one that must be faced again and again. 1 hegrowing state of the arts and sciences means thatwe cannot be content to assume the all-wiMlonof any single epoch. This is perhaps most clearlytrue in social controls and the organization oL thtlearning systems of politics. In a period of rela- _,tive stability, if not in the philosophy of life, then t

in the actual circumstances that we have produced,in the world about us, we can safely ignore new «dangers such as have arisen in the present genera-1..tion in connection with the population explosion,.the atomic bomb, the presence of a widely ex- *tended medicine, and so on. Nevertheless, in -'the course of time we must reconsider our oldoptimization, and a new and revised one v, ill jneed to take these phenomena into account. «jHomeostasis, whether for the individual or ihe -'

[ 8 3 ]

race, is something of which the very basis mustsooner or later be reconsidered. This means, forexample, as I have said in an article for theVoprosy Filosofii in Moscow,* that although sci-ence is an important contribution to the homeosta-sis of the community, it is a contribution thebasis of which must be assessed anew every genera-tion or so. Here let me remark that both theEastern and Western homeostasis of the presentday is being made with the intention of fixingpermanently the concepts of a period now longpast. Marx lived in the middle of the first indus-trial revolution, and we are now well into thesecond one. Adam Smith belongs to a still earlierand more obsolete phase of the first industrialrevolution. Permanent homeostasis of society can-not be made on a rigid assumption of a complete

k permanence of Marxianism, nor can it be made| t on a similar assumption concerning a standardizedI concept of free enterprise and the profit motive. It?>, is not the form of rigidity that is particularlyW' deadly so much as rigidity itself, whatever the

I form.It seemed to me important to say something in

that article which would emphasize the homeo-

* Wiener, N., "Science and Society," Voprosy Filosofii, No. 7

(1961).

[ 84 ] \

static function of science and would at the sametime protest against the rigidity of the social ap-plication of science both in Russia and elsewhere.When I sent this article to Voprosy Filosofii, I ;?!anticipated that there would be a strong reaction *:|

to my attitude toward rigidity; in fact, my article !was accompanied by a considerably longer articlepointing out the defects of my position from astrictly Marxist standpoint. I have no doubt thatif my original paper had been first published overhere, I would have had a similar and almost equalreaction from the standpoint of our own preju-dices, which if not as rigidly and formally ex* jpressed are also very strong. The thesis which 11wish to maintain is neither pro- nor anticom-munist but antirigidity. Therefore, I am express-ing my ideas here in a form that is not too closelyconnected with an evaluation of the difference be-tween the dangers lying in these parallel but op-posed rigidities. The moral I have wished tostress is that the difficulties of establishing a reallyhomeostatic regulation of society are not to beovercome by replacing one set pattern which is notsubject to continual reconsideration by an equaland opposed set pattern of the same sort.

But there are other learning machines besidesthe translation machine and the checker-playing

vt)

[ 8 5 ]

machine. Some of these may be programmed in acompletely mechanical way, and others, like thetranslation machine, need the intervention of ahuman expert as arbiter. It seems to me that theuses for the latter sort greatly exceed those forthe former sort. Moreover, remember that in thegame of atomic warfare, there are no experts.

VII

We have accomplished the task of showing manyvalid analogies between certain religious state-ments and the phenomena studied by cybernetics,and we have gone reasonably far in showing howcybernetic ideas may be relevant to the moralproblems of the individual. There remains an-other field in which cybernetic ideas may be ap-plied to problems with an ethical aspect: the cyber-netics of society and the race.

From the very beginning of my interest incybernetics, I have been well aware that the con-siderations of control and of communicationwhich I have found applicable in engineeringand in physiology were also applicable in so-ciology and in economics. However, I havedeliberately refrained from emphasizing these

[ 8 7 ]

[ 8 8 ]

fields as much as the others, and here are myreasons for this course. Cybernetics is nothingif it is not mathematical, if not in esse then inposse. I have found mathematical sociologyand mathematical economics or econometricssuffering under a misapprehension of what isthe proper use of mathematics in the socialsciences and of what is to be expected frommathematical techniques, and I have deliberatelyrefrained from giving advice that, as I was con-vinced, would be bound to lead to a flood of super-ficial and ill-considered work.

Mathematical physics has come to be one of thegreat triumphs of modern times. It is only duringthis century, however, that the task of the mathe-matical physicist has come to be properly under-stood, more especially in its relation to the taskof the experimental physicist. Until the criticalyears from 1900 to 1905, it was generally con-sidered that the main repertory of the ideas ofmathematical physics had been completed with thework of Newton; that time and space, mass andmomentum, force and energy were ideas groundedonce for all; and that the future task of physicswould consist in making models in terms of thesenotions for phenomena which had not yet beenreduced to these terms.

[ 8 9 ]

With the work of Planck and of Einstein, itbecame clear that the task of the physicist was notso simple. The categories of physics were seen notto have been laid down once for all at the begin-ning of the eighteenth century, and the task ofthe physicist now has to be placed back of theNewtonian concepts, to bring our quantitativeobservations of the world into an order thatshould start with the experiments themselves andend with new predictions of observations andapplied engineering techniques. The observer hasceased to be an innocent registrar of his objectiveobservations but has, rather, come to take anactive participation in the experiment. Both inrelativity and in quantum theory, his role inmodifying the observations is to be regarded asfar from negligible. This has led to the birth ofthe logical positivism of the present day.

The success of mathematical physics led thesocial scientist to be jealous of its power withoutquite understanding the intellectual attitudesthat had contributed to this power. The use ofmathematical formulae had accompanied thedevelopment of the natural sciences and becomethe mode in the social sciences. Just as primitivepeoples adopt the Western modes of denation-alized clothing and of parliamentarism out of a

t V

\ ) v

[ 90 ]

vague feeling that these magic rites and vestmentswill at once put them abreast of modern cultureand technique, so the economists have developedthe habit of dressing up their rather impreciseideas in the language of the infinitesimal calculus.

In doing this, they show scarcely more dis-crimination than some of the emerging African

, i • nations in the assertion of their rights. The mathe-matics that the social scientists employ and themathematical physics that they use as their modelare the mathematics and the mathematical physicsof 1850. An econometrician will develop an elab-orate and ingenious theory of demand and supply,inventories and unemployment, and the like, witha relative or total indifference to the methods bywhich these elusive quantities are observed ormeasured. Their quantitative theories are treatedwith the unquestioning respect with which thephysicists of a less sophisticated age treated theconcepts of the Newtonian physics. Very feweconometricians are aware that if they are toimitate the procedure of modern physics and notits mere appearances, a mathematical economicsmust begin with a critical account of these quanti-tative notions and the means adopted for collect-ing and measuring them.

Difficult as it is to collect good physical data,

[91 ]

it is far more difficult to collect long runs of eco-nomic or social data so that the whole of the runshall have a uniform significance. The data of theproduction of steel, for instance, change theirsignificance not only with every invention thatchanges the technique of the steelmaker but withevery social and economic change affecting busi-ness and industry at large, and in particular, withevery technique changing the demand for steelor the supply and nature of the competing mate-rials. For example, even the first skyscraper madeof aluminum instead of steel will turn out toaffect the whole future demand for structuralsteel, as the first diesel ship did the unquestioneddominance of the steamship.

Thus the economic game is a game where therules are subject to important revisions, say, everyten years, and bears an uncomfortable resem-blance to the Queen's croquet game in Alice inWonderland, which I have already mentioned.Under the circumstances, it is hopeless to give tooprecise a measurement to the quantities occurringin it. To assign what purports to be precise valuesto such essentially vague quantities is neither use-ful nor honest, and any pretense of applyingprecise formulae to these loosely defined quanti-ties is a sham and a waste of time.

c fS-J

( V

[ 9 2 ]

Here some recent work of Mandelbrot is muchto the point. He has shown that the intimate wayin which the commodity market is both theoreti-cally and practically subject to random fluctua-tions arriving from the very contemplation of itsown irregularities is something much wilder andmuch deeper than has been supposed, and thatthe usual continuous approximations to thedynamics of the market must be applied withmuch more caution than has usually been thecase, or not at all.

Thus the social sciences are a bad provingground for the ideas of cybernetics—far worsethan the biological sciences, where the runs aremade under conditions that are far more uniformon their own proper scale of time. For humanbeings as physiological structures, unlike societyas a whole, have changed very little since theStone Age, and the life of an individual containsmany years over which the physiological con-ditions change slowly and predictably, all in all.This does not mean, however, that the ideas ofcybernetics are not applicable to sociology andeconomics. It means rather that these ideas shouldbe tested in engineering and in biology beforethey are applied to so formless a field.

j Under these cautions, the familiar analogy of the

[ 9 3 ]

body politic to the body of the individual is ajustifiable and a useful one. It is to the bodypolitic that many considerations of ethics mustapply, and to that part of religion which is essen-tially a paraphrase of ethics.

VIII

I have now run through a number of essays thatare united by their covering the entire theme ofcreative activity, from God to the machine, underone set of concepts. The machine, as I have al-ready said, is the modern counterpart of theGolem of the Rabbi of Prague. Since I have in-sisted upon discussing creative activity under oneheading, and in not parceling it out into separatepieces belonging to God, to man, and to themachine, I do not consider that I have taken morethan an author's normal liberty in calling thisbook

GOD AND GOLEM, Inc.

v<

:«

-< '{