It’s complicated

Human ingenuity has created a world that the mind cannot master. Have we finally reached our limits?

by 2000 2,000 words
  • Read later or Kindle
    • KindleKindle
Photo by nobiann/flickr/Getty

Photo by nobiann/flickr/Getty

Samuel Arbesman is a senior scholar at the Ewing Marion Kauffman Foundation, and a fellow at Harvard. His work has appeared in the New York Times and others. He is the author of The Half-Life of Facts (2012).

Despite the vastness of the sky, airplanes occasionally crash into each other. To avoid these catastrophes, the Traffic Alert and Collision Avoidance System (TCAS) was developed. TCAS alerts pilots to potential hazards, and tells them how to respond by using a series of complicated rules. In fact, this set of rules — developed over decades — is so complex, perhaps only a handful of individuals alive even understand it anymore. When a TCAS is developed, humans are pushed to the sidelines and, instead, simulation is used. If the system responds as expected after a number of test cases, it receives the engineer’s seal of approval and goes into use.

While the problem of avoiding collisions is itself a complex question, the system we’ve built to handle this problem has essentially become too complicated for us to understand, and even experts sometimes react with surprise to its behaviour. This escalating complexity points to a larger phenomenon in modern life. When the systems designed to save our lives are hard to grasp, we have reached a technological threshold that bears examining.

For centuries, humans have been creating ever-more complicated systems, from the machines we live with to the informational systems and laws that keep our global civilisation stitched together. Technology continues its fantastic pace of accelerating complexity — offering efficiencies and benefits that previous generations could not have imagined — but with this increasing sophistication and interconnectedness come complicated and messy effects that we can’t always anticipate. It’s one thing to recognise that technology continues to grow more complex, making the task of the experts who build and maintain our systems more complicated still, but it’s quite another to recognise that many of these systems are actually no longer completely understandable.  We now live in a world filled with incomprehensible glitches and bugs. When we find a bug in a video game, it’s intriguing, but when we are surprised by the very infrastructure of our society, that should give us pause.

One of the earliest signs of technology complicating human life was the advent of the railroads, which necessitated the development of standardised time zones in the United States, to co-ordinate the dozens of new trains that were criss-crossing the continent. And things have gotten orders of magnitude more complex since then in the realm of transportation. Automobiles have gone from mechanical contraptions of limited complexity to computational engines on wheels. Indeed, it’s estimated that the US has more than 300,000 intersections with traffic signals in its road system.  And it’s not just the systems and networks these machines inhabit. During the past 200 years, the number of individual parts in our complicated machines — from airplanes to calculators — has increased exponentially.

The encroachment of technological complication through increased computerisation has affected every aspect of our lives, from kitchen appliances to workout equipment.  We are now living with the unintended consequences: a world we have created for ourselves that is too complicated for our humble human brains to handle. The nightmare scenario is not Skynet — a self-aware network declaring war on humanity — but messy systems so convoluted that nearly any glitch you can think of can happen. And they actually happen far more often than we would like.

machines are interacting with each other in rich ways, essentially as algorithms trading among themselves, with humans on the sidelines

We already see hints of the endpoint toward which we seem to be hurtling: a world where nearly self-contained technological ecosystems operate outside of human knowledge and understanding. As a scientific paper in Nature in September 2013 put it, there is a complete ‘machine ecology beyond human response time’ in the financial world, where stocks are traded in an eyeblink, and mini-crashes and spikes can occur on the order of a second or less. When we try to push our financial trades to the limits of the speed of light, it is time to recognise that machines are interacting with each other in rich ways, essentially as algorithms trading among themselves, with humans on the sidelines.

It used to be taken for granted that there would be knowledge that no human could possibly attain. In his book The Guide for the Perplexed, the medieval scholar Moses Maimonides opined that ‘man’s intellect indubitably has a limit at which it stops’ and even enumerated several concepts he thought we would never grasp, including ‘the number of the stars of heaven’ and ‘whether that number is even or odd’. But then the Scientific Revolution happened, and with it, a triumphalism of understanding. Hundreds of years later, we now know the exact number of objects in the night sky visible to the naked eye — it’s 9,110 (an even number).

But ever since the Enlightenment, we have moved steadily toward the ‘Entanglement’, a term coined by the American computer scientist Danny Hillis. The Entanglement is the trend towards more interconnected and less comprehensible technological surroundings. Hillis argues that our machines, while subject to rational rules, are now too complicated to understand. Whether it’s the entirety of the internet or other large pieces of our infrastructure, understanding the whole — keeping it in your head — is no longer even close to possible.

One example of this trend is our software’s increasing complexity, as measured by the number of lines of code it takes to write it. According to some estimates, the source code for the Windows operating system increased by an order of magnitude over the course of a decade, making it impossible for a single person to understand all the different parts at once. And remember Y2K? It’s true that the so-called Millennium Bug passed without serious complications, but the startling fact was that we couldn’t be sure what would happen on 1 January 2000 because the systems involved were too complex.

Even our legal systems have grown irreconcilably messy. The US Code, itself a kind of technology, is more than 22 million words long and contains more than 80,000 links within it, between one section and another. This vast legal network is profoundly complicated, the functionality of which no person could understand in its entirety. Michael Mandel and Diana Carew, of the Progressive Policy Institute in WashingtonDC, have referred to this growth of legal systems as ‘regulatory accumulation’, wherein we keep adding more and more rules and regulations. Each law individually might make sense, but taken together they can be debilitating, and even interact in surprising and unexpected ways. We even see the interplay between legal complexity and computational complexity in the problematic rollout of a website for Obamacare. The glitches in this technological system can affect each of us.

And this trend is accelerating. For instance, we now have 3D printers, vast machinery to help construct tunnels and bridges, and even software that helps with the design of new products and infrastructure, such as sophisticated computer-aided design (CAD) programs. One computational realm, evolutionary programming, even allows software to ‘evolve’ solutions to problems, while being agnostic on what shape that final solution could take. Need an equation to fit some data? Evolutionary programming can do that — even if you can’t understand the answer it comes up with.

A number of years ago, a team of research scientists tried to improve the design of a certain kind of computer circuit. They created a simple task that the circuit needed to solve and then tried to evolve a potential solution. After many generations, the team eventually found a successful circuit design. But here’s the interesting part: there were parts of it that were disconnected from the main part of the circuit, but were essential for its function. Essentially, the evolutionary program took advantage of weird physical and electromagnetic phenomena that no engineer would ever think of using in order to make the circuit complete its task. In the words of the researchers: ‘Evolution was able to exploit this physical behaviour, even though it would be difficult to analyse.’

This evolutionary technique yielded a novel technological system, one that we have difficulty understanding, because we would never have come up with something like this on our own. In chess, a realm where computers are more powerful than humans and have the ability to win in ways that the human mind can’t always understand, these types of solutions are known as ‘computer moves’ — the moves that no human would ever do, the ones that are ugly but still get results. As the American economist Tyler Cowen noted in his book Average Is Over (2013), these types of moves often seem wrong, but they are very effective. Computers have exposed the fact that chess, at least when played at the highest levels, is too complicated, with too many moving parts for a person — even a grandmaster — to understand.

intellectual surrender in the face of increasing complexity seems too extreme and even a bit cowardly, but what should we replace it with if we can’t understand our creations any more?

So how do we respond to all of this technological impenetrability? One response is to simply give up, much like the comic strip character Calvin (friend to a philosophical tiger) who declared that everything from light bulbs to vacuum cleaners works via ‘magic’. Rather than confront the complicated truth of how wind works, Calvin resorts to calling it ‘trees sneezing’. This intellectual surrender in the face of increasing complexity seems too extreme and even a bit cowardly, but what should we replace it with if we can’t understand our creations any more?

Perhaps we can replace it with the same kind of attitude we have towards weather. While we can’t actually control the weather or understand it in all of its nonlinear details, we can predict it reasonably well, adapt to it, and even prepare for it. And when the elements deliver us something unexpected, we muddle through as best as we can. So, just as we have weather models, we can begin to make models of our technological systems, even somewhat simplified ones. Playing with a simulation of the system we’re interested in — testing its limits and fiddling with its parameters, rather than understanding it completely — can be a powerful path to insight, and is a skill that needs cultivation.

For example, the computer game SimCity, a model of sorts, gives its users insights into how a city works. Before SimCity, few outside the realm of urban planning and civil engineering had a clear mental model of how cities worked, and none were able to twiddle the knobs of urban life to produce counterfactual outcomes. We probably still can’t do that at the level of complexity of an actual city, but those who play these types of games do have a better understanding of the general effects of their actions. We need to get better at ‘playing’ simulations of the technological world more generally. This could conceivably be geared towards the direction our educational system needs to move, teaching students how to play with something, examining its bounds and how it works, at least ‘sort of’.

We also need interpreters of what’s going on in these systems, a bit like TV meteorologists. Near the end of Average Is Over, Cowen speculates about these future interpreters. He says they ‘will hone their skills of seeking out, absorbing, and evaluating this information… They will be translators of the truths coming out of our networks of machines… At least for a while, they will be the only people left who will have a clear notion of what is going on.’

And when things get too complicated and we end up being surprised by the workings of the structures humanity has created? At that point, we will have to take a cue from those who turn up their collars to the unexpected wintry mix and sigh as they proceed outdoors: we will have to become a bit more humble. Those like Maimonides, who lived before the Enlightenment, recognised that there were bounds to what we could know, and it might be time to return to that way of thinking. Of course, we shouldn’t throw our hands up and say that just because we can’t understand something, there is nothing else to learn. But at the same time, it might be time to get reacquainted with our limits.

Read more essays on , and

Comments

  • James

    The bigger the systems,the bigger the failures.

  • The Monk

    The Glass Bead Game

  • riverrat37

    Every new solution brings new problems. Always.

    • Agga

      And that solution brings new problems. Always.

      Solutionist thinking is the greatest scourge of the planet.

      • G

        Better to just be a critic. That way you're never under the pressure of actually building something that works. Right.

        • Agga

          It seems you don't understand what is meant by solutionism.

          The world is overwhelmed with people "doing"and "creating" and "building". Most of these people who strive to make more and faster and cheaper and bigger, do so for economic gain, and are operating with total disregard for the processes of life on earth. They leave wide-spreading, fundamental, systemic problems in their wake. Then others, using exactly the same logic that created the problem in the first place, try to "fix" things by making more stuff, causing more disrupting into systems we don't even understand.

          Yes, we should criticise. And yes, it is often better NOT to try to build something. Especially when you are motivated and informed by a mechanistic, capitalist, undemocratic culture.

          • G

            The world is overwhelmed by people multiplying like mice, consuming like locusts, and pooping in their nests. This they do, and would do, with or without technology.

            So tell us: do you go to work every day as an act of charity, with no expectation of being paid?

  • balthasar999

    Bring on the brain augmentation, then.

  • Sean Smith

    I would argue that systems that cannot be completely understood by their creators are simply poorly designed systems. Sure, not everyone at IBM knows and understands all the tiny details of how a computer processor works but when their individual knowledge bases are combined, they understand the system as a whole. That is the direction humanity is going with technology. Human tasks in the development of extremely complicated systems will become more and more specialized so the number of experts working on a system will increase.

    • skanik

      Sean,

      How are you going to gather all the "experts" at the same time and
      get a coherent and complete answer from them whether the computer system has flaw - be it via a program or hardware - if you need to make
      an immediate decision based on what the computer is telling you ?
      And what if the experts disagree ?

    • Rich S.

      Spend a few years in the aircraft industry, or any industry building machines of similar complexity. You are greatly oversimplifying the matter.

  • Bigg

    That is why we work in abstractions - things are too complicated. We can understand why things work, but we choose to deal in abstractions to develop more and more complex systems. http://www.ece.cmu.edu/~koopman/abs_chng/abstract.html

  • Rich Linville

    As a Biology teacher, I can assure you that technology is no where near as complex as nature. In the early 1900's someone is supposed to have said that everything has been invented so we can close down the patent office.

    • http://netdost.com/ NetDost Social Network

      but biology is based on evolution, while technology has human brains with limitations behind it

      • Sam

        What limitations? Technology evolves too and it's doing so exponentially

    • doug

      As a fan of Animal House, it looks like the average person aspires to be more like Bluto ... Fat, lazy, and stupid.

      • Robert Walther

        In AH emerging techno world, Bluto became a wealthy, powerful senator and married the really hot 'straight' chick. ;)

    • Jim Clark

      I have to agree with Rich ... by using reason and empiricism researchers have collectively come to a deep understanding of and control over extremely complex natural systems. Heart transplants, humans on the moon, ... all reflect that deep understanding and not simply some technological achievement. And surely our understanding of even human-made systems is far better today than in the past. At least I'm glad that the internet is more reliable and widely used than its precursors were when I started using e-mail 40 years ago. The reality is that we do not know the limits of our understanding (e.g., the human brain?) until we exert all our efforts toward the goal of understanding something and fail (or more likely, not).

    • Nico Anderson

      Yeah I agree. Until we can understand the question of where consciousness comes from (the ultimate question), technology will not hold a candle to nature.

      • JenJen10

        "where consciousness comes from" is the theme of Asimov's famous short-story. I think that was written back in the 1950's.

  • Ed Aaron Goering

    This article is blathering nonsense. Corporations and government regulations have been stifling true innovation for decades. Things aren't too complicated, our system of education is a joke and most of us are left scratching our heads as user interfaces are continuously being "dumbed down" for the masses. Technological advancement hasn't even begun yet.

  • Andre

    This is Neo-Luddist proselytism at its best. For the author of this article ignorance might be bliss but, for us to survive, the quest to master technology is the only way
    forward. Thanks to our technology, we are now aware that planet earth might be
    a very special place for life to thrive but, there were many massive life extinctions.
    If you think humanity is special and that we have to assure its survival, it
    would be very ill advised to count on Mother Earth or God to protect us. For
    one thing, because of the sun, in around 700 000 years there will not be
    life on the surface of this planet; might as well prepare ourselves to go
    beyond.
    Fear of complex systems is irrational. Mr Arbesman idealise simpler time when human brain was the most powerful tool. But human brain is a system that is too complex for its constituents (neurons) to comprehend. Still it works.

    • Agga

      The greatest mass extinction ever is happening right now, and it was created by our technologies. So what is the purpose of it again?

      Fear of complex systems is totally rational. Hence the universal obsession with weather. It affects our lives profoundly and we just have to play by its rules. For one class of people to create systems that reshape the conditions for live on earth, and then threaten life on earth, is a violation of democracy and human rights. Most people have no control whatsoever over this, and so they rightly fear it. That the people in charge speak with contempt about their fear kinda proves that it was justified in the first place.

      • Andre

        I will reply to your comment. But let me first express my surprise with the reactions that this short essay provoked. My first comment was simply a gut reaction, and so it seem were those of most other commentators. The few more cerebral expressed their condescending thoughts by arguing that peoples here might be to intellectually challenge to comprehend what the author was talking
        about; that is a lack of respect to Aeon Magazine readers. Incidentally, it is probably because of the quality of the readers and debates in this webmag that Mr Arbesman published here. Anyway, what is there to debate; our intellect is finite there is no arguing to that! The interesting debate is about the implications of this humbling problem.

        I find the argument about complex systems weak; man has never been good at grasping complexity, anyway that is not where our strength is for sure. Might it be a natural system like climate, or one of our own making like the economy, we study those by
        putting them into pieces. Extracting the most significant information first, and after we go for the subtleties. Then we synthesize by making models. I understand very well that the sum of the parts of something is not the essence of that thing (that is my way to paraphrase Cantor’s theorem) and because of that we will never understand everything. But bit by bit our knowledge is inching
        closer to the truth. Someday soon I hope, with the help of quantum computing and its vast capacity of factorisation, we will be better equipped to master complexity.

        For you Mr Agga, I can only say that technology does not make our life better or worse, its main goal is to make our world more efficient. So there will be good and bad consequences. I prefer to look at the positive side and see how it makes my life interesting, than be in your shoes and dread everything that challenges my beliefs and values. For what it’s worth, we are on the same side concerning climate changes, so I wish you well.

    • G

      Wrong interpretation. If anything, the author is advocating the supine position: passivity and even submission to the new Gods of our own creation. The Luddite attitude is nonselective rejection. Its opposite, uncritical cheering for every new bauble, is if anything worse because it has strong support from the makers of baubles and their captive media (e.g. Wired). But the supine position is equally bad, as a kind of fatalism that is the antithesis of the Enlightenment attitude of deliberate engagement.

      Speaking here as an engineer, what's needed is a clear-headed outlook of open-minded skepticism: ask questions, raise counterpoints, get informed, be a selective adopter and an active citizen. Technology is our creation, for the purpose of serving our needs. Every want isn't a need, and every bauble isn't a gem.

      • Andre

        All right Mr Howard Wolowitz! You realize that your title here, genuine or fantasized, do not add any weight to your comments. Their values have to stand on their own. But the response you managed to pull from MrKamikaze sent me rolling on the floor.
        So I will generously grant you this reply.

        Mr Arbesman is a senior scholar. His gibbering is not without a purpose; if his esoteric rhetoric filled with half truth conveys a dreadful feeling, it is because he wants it that way. With the help of the wisdom of Maimonides he seems to try to pose himself as a prophet of a dystopian future. I would really like to tell him that exacerbating the Ted Kaminski’s of this world is unwise. I will end this rant with a more cheerful and down to earth citation; predictions are hard... especially about the future (Yogi Berra).

        • G

          I had to look up Howard Wolowitz to discover he's a character on TV. BTW, it's "Ted Kaczynski", whose awful gibberish I once attempted to read when the FBI were begging anyone connected to any ecology group to help them catch him before he killed anyone else. The rest of your rant is indecipherable, but I'm not a cryptanalyst nor do I play one on the telly.

          Other: yes, my title is genuine, and if you work in an office but frequently have to receive telephone calls on the office line whilst away from your desk, there's a decent probability that you use something I designed. But the point isn't to go credential-dropping, it's to demonstrate that I'm professionally engaged with technology and not a Luddite (despite having GPO rotary dial phones around the house including a 232 on my desk;-)

  • http://provegodtome.blogspot.com/ BT Richards

    The Renaissance man is no more. There can be no 1 person who understands every system to its fullest. We do have people who understand individual systems to their fullest – or micro experts. On a macro level we need to understand the flows between each micro system while knowing who the micro system experts are.

    • G

      There are still Renaissance men and women. Look up Jaron Lanier. Computer scientist, first inventor of virtual reality, and accomplished musician including classical piano and a range of obscure and ancient instruments. He also apparently has a wide scope of knowledge in the social sciences.

      There are more where he came from.

    • Ari BenDavid

      I have been called a renaissance man. It's not a matter of modern complexity. You can be a generalist (a term I prefer) by working at it. But of course no one has both breadth and depth in all things. It's just a matter of gaining perspective.

  • MrKamikaze

    As long as the older more laborous ways of doing things are not lost then we will be ok. Good example is US ARMY Artillery where fire direction soldiers are trained to compute firing data using nothing but maps, protractors and a little trig at the same time they are trained to use the fire direction computers. 100% Dependency on technology is a bad thing.

    • skanik

      Are fighter pilots/navigators still taught how to steer by the stars ?

      If the Chinese take out all of our GPS satellites right before beginning

      a war with us - where will that leave the US of A ?

      • MrKamikaze

        Nukes are not guided by GPS.

    • G

      The military are also at the forefront of recognising the implications of climate change for global security.

      Thus we come to another paradox of modern times, which is that military institutions, traditionally very conservative by nature, are today among the most progressive elements in advanced democratic societies.

      And the reason for this is pretty obvious: when your daily business is concerned with matters of life and death on a large scale, you don't muck about or indulge in ideology-driven fantasies: you start from solid facts and reasoning, and seek sound conclusions.

      • MrKamikaze

        I agree, they recognize that 99% of all AGW arguments are politically motivated and are a tool of leftist anti-sovereignty advocates that want a one world government and an end to capitalism. AGW advocates are ultimate enemies of the free world.

        • G

          Oh b----y hell, are you one of those anti-nukers?

          If I'm wrong and the military are wrong and 99% of climate scientists are wrong, and AWG is false, we still get huge investment in 21st century energy sources (or rather I should say, 20th century until fusion is feasible; but that's a century or two better than fossil fuels!), record profits for investors, high-skill jobs throughout the economy, and independence from unstable tyrannical regimes run by people who tithe to Al Qaeda.

          If you're wrong and AGW is true, what we get is an economic catastrophe that'll make other historical and recent ones seem like a picnic by comparison, and global turmoil including an upsurge in terrorist events and regional resource wars.

          Care to bet on that?

          Hint: conspiracy theories about "leftist anti-sovereignty advocates that want a one world government and an end to capitalism" makes you sound like the bizarre species of leftist who conspiracy-theorise about Dick Cheney flying the planes into the buildings by remote-control from his undisclosed location. Have you had your teeth checked for secret radio receivers lately?

  • ASSASSYN

    The whole point of technology is to allow humans to exceed their own limitations and improve on on our limited natural ability. Complex becomes simple, simple becomes human independent. That is the course of technology. Hand washed dishes, dishwasher.

    • Agga

      In what sense is a dishwasher an improvement over hand washed dishes? It isn't. It is just a product created for people to buy, to free up labour so they can do waged work instead, to enable them to buy products such as a newer dishwasher.

      All of this improvement and complexity is designed for one purpose: shifting value up the pyramid.

  • Wang Chung

    A.I. will take care of everything. We just need to sit back and take a nap.

  • Itchy_Robot

    Hell, if you think can't comprehend our own technology now, just wait until artificial intelligence starts creating its own technology. Without the human minds constraints, there is no telling what future AI's will dream up for us to use.

  • Ken

    I'd be interested in a reference to the researcher's "evolutionary circuit" design.

    • nemo

      http://scienceblogs.com/goodmath/2008/11/11/evolution-produces-better-ante/

      I will say that a trail and error approach in engineering is nothing new, but computers are probably able to do it a bit faster. Also there is nothing wrong with division of labor in a society and nobody needs to understand everything to enjoy the fruits of collective cooperation. Understanding and testing the principles is the fun part.

  • Joe

    Article completely disregards the research being done to "upload/download" information directly to the brain. Call it Matrix magic or whatever, they're attempting to do it.

    • G

      "Upload" is religion disguised as technology. If you can get a mind outside of a human brain, what you have there is a soul by any other name. If you can transplant a mind into a computer, what you have is reincarnation.

      The fact that some otherwise-smart people are attempting to do it, is not surprising: history is full of people, smart and not-so-smart, who were just plain wrong.

  • Lexxvs

    Most of the comments don’t seem to grasp what the author tries to communicate, even if he is pretty well at it. Becoming an irony that further proves that he is –in fact- right.

  • Cyber Dactyl

    This article basically says nothing.

  • BDewnorkin

    Bits of Arbesman's article conjure a looming sense of danger, some great threat of complexity that, presumably, the article intends to help us manage. But I'm struggling to actually identify what this threat is. Is the problem of complexity the "incomprehensible glitches and bugs"? The humanly-inaccessible engineering or chess solutions?

    My best guess for what Arbesman is really up to – whether he knows it or not – is the expression of a kind of dread at the idea that we've created, collectively, something we can't understand. After all, this is the only identified problem that the proposed solutions – the SimCity-esque simulations or the "complexity interpreters" – help us solve.

    If I am right, then I think the only sensible response to Arbesman is reproach. The fear of complexity, contrary to what this article suggests, is not new. It's motivated as much the early American critics of science-based medicine as the TED Talk fanatics. There is no reason to think that we've, only now, reached a "technological threshold." And the management of complexity requires not analgesic half-measures – the collective dumbing-down of our society's managed population – but the development of institutions that sustain a Democratic way of life. As the relationship between Wall St. and the financial regulatory institutions show, at least one aspect of these institutions will have to be the maintenance – via the proper allocation of resources and incentives – of a group of expert glitch-fixers as capable as the glitch-makers.

    • skanik

      Is there anyone who can fully understand Microsoft's computing program.
      Or the Four Colour Theorem
      On whether/why we should buy or sell stocks/derivatives/derivatives of
      derivatives because the latest financial program tells us to ?
      Can anyone say whether Obama care will save money or not ?
      We end up trusting programs/intertwining tax laws that no one understands.
      Soon you will be told to take all sorts of medicines that no one fully
      understands how they work and what the consequences will be -
      but your Insurance company will mandate that you take them because
      the Medical Studies tell them so...

      • G

        The difference though, is that the medicines we take have been subjected to thorough empirical testing for safety & efficacy.

        Contrast to the Internet.

        Seriously.

        • danwalter

          "Soon you will be told to take all sorts of medicines that no one fully understands how they work and what the consequences will be."

          Soon? That is already happening.

          "... the medicines we take have been subjected to thorough empirical testing for safety & efficacy."

          Hah! Who told you that?

          Google: "Institutional Corruption of Pharmaceuticals and the Myth of Safe and Effective Drugs" -
          Journal of Law, Medicine & Ethics Fall 2013

  • silverfox

    It's completely insane to think we've reached some kind of limit with technology or anything to do with out mind and ingenuity. What has become of Digg (face in hands).. This is sad. The title alone of this article should have been enough for it to not have been posted. Madness is all I can say, utter and complete madness.

  • Neo

    I don't agree with this article at all. I find the premise of this quite shaky and on weak grounds.

    The article begins with a tone of disappointment that engineers rely on simulation for the collision detecting software and towards the end, the author prefers a solution to this complexity with simulation like SimCity.

    Rather than be scared of complexity, it would be wise to question the true cause of complexity - It's human behavior. We are the only creatures on the planet that strive for improvement. We strive for efficiency and innovate on status quo.

    Why is the TCAS code so complex? It's probably because of years of refinement of the code. Why the refinements? I assume they are to make it better. Just like Adam Smith's pin factory, division of labor leads to increasing granularity of labor. Humans will innovate, regardless of the granularity and innovation at each level leads to an efficient output, but probably a increase in the overall complexity.

    This isn't something to be feared. As I see it, it's a by-product of human nature and innovation. Yes, the world is very complex and people need to spend time to understand it. So, let's focus on the solution that Adam Smith outlined right along the pin factory example - Education. Let's focus on bringing everyone up to speed with the complexity, while also striving hard to make our lives easier and better.

    Let's not expend resources to build a system and then build a simulation of the very system for us to experiment with.

    • Agga

      It likely isn't a by-product of human nature, but of culture. Those who behave in this way are a staggering minority, and not everyone would agree that improvement is the motivation behind it.

  • skanik

    Rather shocked to see the off-handed dismissal of the essay.

    I used to think no one was more arrogant than MBA students

    but now I wonder if the Computer Scientists are not even more so.

    When you have to make life or death decisions on what a computer tells

    you and you have no idea whether the computer is right or not -

    and no one else can tell you - not even all the programmers and engineers

    who designed and implemented the system and those creators of the system

    just tell you to trust in technology...it only brings back Dow Chemical's advertisement

    when I was a kid: A Better Future through Chemistry - Napalm...Agent Orange -

    Dioxin...

    • JenJen10

      The next time your doctor tells you you need an operation/drug/implant, remember this. Our medical system is already this complex.

    • teddyknox

      We live in a world dicated by probability and cost/benefit analysis. Most of the time, "good enough" suffices. Cars generally run for 130K miles or so before breaking down, and that's fine, because the economics say so. Maybe the engineering cost of producing a car that wouldn't eventually break would drive the car's price up, and since no one would buy this more reliable car, the company would lose money.

      Every one of these overly complex technological systems has been vetted as "good enough", and not just by experts. All of us, the people that use these systems, have collectively decided (whether we intended to or not) that these overly-complex systems are the best option we have for now. If someone came up with a air traffic control program that was completely understandable and performed perfectly, then we would obviously begin to use it. Until then, we will be stuck with the next best option, which is hard to understand, and occasionally fails. But this is OK! Unless there is some ulterior force that wants to slow the progress of air traffic control technology, have faith that if the need is dire enough (as measured through voting with dollars and government activism), manpower will go towards a better solution.

      This article is not about all that, however. The author acknowledges that the systems in place today are flawed. That ideally, they would be perfect from the start. You seem a bit disappointed that economics have allowed such low standards of quality. Take heart, brother, another promising approach to these seemingly-intractable problems awaits attempt. Black box solutions, driven by big data and big computers, have shown to work very well. The problem with these approaches is that their engineering paradigm is completely different from the paradigm behind hand-crafted designs, like say, a car. These approaches are made by designing machine-learning systems that are designed on the meta level.. systems where the engineering insight doesn't go into solving the core problem itself, as much as optimizing the model to best come up with a solution on its own. These systems are sort of unpredictable though, you know, since they're dealing with millions of variables which are combined to produce in ways controlled by those same variables. The feedback loops in these systems make following their logic as futile as trying to keep up with a computer at arithmetic.

      In the end, the author of this article suggests that we start to look at these systems like we look at the weather. He's not giving up on us or our engineering prowess. He's saying that we, as humans, should (A) embrace systems that are too complex for us understand if they are in fact better, AND (B) stick to what we do best, which is producing models of ultra complex systems. This time the system will be one we created.

  • http://netdost.com/ NetDost Social Network

    All future inventions are going to be a group effort, all things simple has already been invented...

  • CultofXoth_com

    Everything will be fine after the Great Old Ones return and destroy the world. Until then, we must continue to heed the words of the prophet Lovecraft, study the truths of the Necronomicon and do the will of the Outer Gods. Azathoth akbar!

    • Ronanfitz

      well said

  • Dustin Currie

    Isn't complication really the result of years of refusing to refactor systems though? Newer systems usually are easier to understand because years of Rube Goldberg layers haven't been added yet. So the solution isn't to give up, it's to simplify.

  • Alexander Pereira

    Awesome article, very thought provoking. I get the feeling that most people posting comments here havent really gotten their heads around what the author is telling us, calling him names and laying out their own one-paragraph solutions. Our way of thinking lies behind this denialism (is that a word?). The author mentions this and compares this to Maimonides way of thinking, which I partly share, and that is that there may be things outside of our realm of comprehension. Grounded as we are in the physical world, limited surely by the constraints of time if not the size of our brain or ability of our spirit (if the brain alone is too material for you), perhaps we cant know everything, even if we narrow it down to everything manmade. The 'we can understand and control everything' promise of modern civilization is going to take more than an article to dispell, but its a good place to start Samuel.

    Ive often thought about this unknowable complexity when dealing with laws (also touched upon in this article). Everyone is obliged to obey all the laws of his/her land. From the moment you are born, these laws apply to you. Even though no single lawyer, judge, lawmaker or other can read, remember and understand the exact ramifications of the entirety of all laws of any country. Thats why we have specialists in various fields. Yet we must all obey this unknowable total. Somethings wrong here. What you may ask? Well, who is to make sure we keep to the laws? The police? They dont know even know a tiny percentage of the law. Who will interpret the laws? The judges? What about when its a complex case (e.g. antitrust agaisnt global corporations)? I laughed when they did the Microsoft prosecution for bundling browsers etc into their OS, by the time the verdict came the world had moved on, it was totally moot, the system just couldn't handle the size of the case. Justice cannot be served under such circumstances. And so on.

    The exact and particular ramifications of such a environment in technological or legal fields can only be guessed at, but history shows that unworkable systems will eventually collapse. So either we figure this one out or well be starting from scratch again soon enough, with new and simplified technology, laws, you name it. Nothing like some dark ages to sort this out.

    • MSsusu

      Since you seem to be interested in the adherence in laws- the social sciences have got a few very compelling theories as to why we readily adhere. All other things being equal(!)*, things that play a big role is the desire to conform, the loss of utility when NOT adhering to the law, peer pressure and also "ideology" meaning the teaching of what is right and what is wrong. An interesting overview is provided in the book "Why societies need dissent", C. Sunstein.

      *a situation in which the assumptions break down is for example when the state has no monopoly on security; i.e. it cannot assure its citizens safety through its institutions. This changes the dynamic.

      A wider point I would like to make is that we humans and our social structures are just as complex as the things we create, and there is a science devoted to it.

  • rshewmaker

    Depressing and true. Such is life.

  • goatonastick

    Is the world not already a "world that the mind cannot master"? Even without the technology?

  • goatonastick

    I'm surprised that there was not even one mention of the Singularity!

  • bwbeeman

    This article is an astounding admission of ignorance fo the real world. Technology knows exactly what it is about, but the left-behinds like journalists and lawyers pretend to understand while crapping on perfectly logical and good systems. There may be billions of lines of code in the operating systems that run many of the mundane portions of our existence, but they are in SYSTEMS and MODULES that have been tested and connected by design. That's more than be claimed by practitioners of the law, and even medicine.

    t s important that over 80% of observational studes are wrong from a sentfc settng, and there les a problem n our ignorant press. They just don't know what s going on, therefore this ridiculous artcle.

  • Agga

    Yet another article in the series of articles on Aeon that tells us to just give in to what is happening. As if humans were no longer in control of the direction of our combined actions and decisions. And that is obviously so; most humans aren't, control is in the pockets of the Technopoly elite, and the little elf-nerds in their workshops, blissfully building their toys and counting their money and ignoring the ill effects on the rest of us.

    What this article suggests; that we should stop worrying and love the bomb, is non-democratic.

    I find the weather analogy interesting. The weather is growing more and more unpredictable, it kills more and more people, and this is probably because of our tech interfering in ways we are not really understanding. Two complex systems are interacting in more complex ways. We are seeing the same thing happening with life itself, as the very complex systems that are our bodies and their complex interactions with for example microorganisms are similarly interacting with our creations, causing new health issues.

    The weather is the weather, we didn't create it. When it kills or destroys, we perhaps should "love it", or adjust. But when things created by other people destroy or kill, why should we not stand up and say: "If you can't play nice with that toy, you can't bring it into the sandbox with us."

    The author, in comparing a system that someone has to decide to make, then others spend time designing, yet others work to implement it, to the weather, is hiding all the steps along that process when any number of persons could put a hand up and say; "Have we though about the effects on this system on the world? On people's lives?"

    Designing and building things without doing so shouldn't be allowed, not in this world where we are all dangling from the same thread. Technological development is becoming a force that should be limited and strictly controlled by us, just like the government should, and the corporations. Couple the attitudes of the people responsible here (solutionism, disregard, ignorance, greed and scientism) with developments such as the Trans-Pacific Partnership Agreement (the Corporate Death Star), and it is clear that democracy is a thing of the past.

    The world is a complex web of complex systems and to think that we can just inject new complex systems into this and predict what will happen is total folly. It is evident everywhere from medicine to biology to resource management to pollution. Species are dying, humans are suffering. Our complex tech is creating, then exacerbating, countless problems like this, and it is time to stop "loving it".

    Humans created these systems, and they should serve us all, not a wealthy elite of Technocrat investors or some lofty principle of "innovation" or "progress", but real flesh and blood people.

  • Bryan Burgess

    I think the answer is obvious. Build better brains.

    • http://www.facebook.com/mindbound Noetic Jun

      This.

  • Guest

    In the book, "The Singularity Is Near: When Humans Transcend Biology" by Ray Kurzweil proposes that computers will soon rival the full range of human intelligence at its best and there will be a union of human and machine in which the knowledge and skills embedded in our brains will be combined with the vastly greater capacity, speed, and knowledge-sharing ability of technology.

  • Rich Linville

    In the book, "The Singularity Is Near: When Humans Transcend Biology" the author Ray Kurzweil proposes that computers will soon rival the full range of human intelligence at its best and there will be a union of human and machine in which the knowledge and skills embedded in our brains will be combined with the vastly greater capacity, speed, and knowledge-sharing ability of technology.

    • G

      See my comment above, about the Church of Singularitology.

      And contemplate this:

      True AI necessarily includes self-aware consciousness. That is also the defining criterion for personhood. QED, building an AI entails the same moral considerations as having a baby. One doesn't have babies in order to provide oneself with extensions or spare parts.

  • Andre V

    This article misses the point.

    I am a software engineer since 14 years now and since I have started we always had "software that no single person masters". And older people told me they had this problem since the early eighties.

    You just forgot that all technology history is the history of engineers developping ways to cope with complexity. If we still had to code programs in assembly language (some early form of computer language that is very close to the real language machines speak) developping software like Windows, Firefox or anything that you use would be nearly impossible. The same is true with modern operating systems : Windows, and Linux are mostly there to shield application developpers from hardware complexity. If you can get some memory by a simple malloc() system call, be sure that there is some complex mechanism behind that does the dirty jobs. And most of the time you'll never have to care about it. When you do, you'll find someone who knows. And it continues with most API or libraries.

    We are now starting to use so called "Infrastructure as a Service" and "Platform as a Service" technology. This will ultimately shield most of us from complex system administration tasks. Yet operating systems and network will still exist in that future. But application developpers will be able to build very complex apps and just be confident that it will work. The apps that they'll write will be of a complexity that may astoish today's engineers but we are confident that this will happen.

    And you have to keep in mind that yes these new technologies to shield us from complexity create costs and problems in the first years of adoption. For example current adoption of ORM (Object relational mapping) is sometime seen as a nightmare for someone who was used to do it the old way. Since they master the old way of doing things, bugs and headache added by ORM seems not to be worth the benefit. But ORM technology is improving steadily and we can envision a world where most programmers won't have to learn SQL because it will most of the time NOT fail them.

    In 14 years, for example I had only one time I had to know what happen when I call malloc(). This was a very special case in a highly technical work. But most of the devs probably never have to put their fingers in this. And this is fortunate.

    Don't panic about complexity, it is there to be mastered.

    • http://www.zvodovodu.cz/ zuzkacab

      Hi Andre,

      it is always encouraging to see brave and optimistic people like yourself.
      Complexity indeed is a challenge for humans and many feel it as their mission to get closer to mastering it, as you say.

      Coming from an environmental science background, I worry, however, that the way we master nature's complexity is by creating another level of it. Engineers, mathematicians, physicists etc. develop artificial worlds that commonly serve to the benefit of society (internet, health care...). Their complexity is both a source of pride and anxiety though, because as so few or no individuals grasp the complexity it can be abused without 99,99% of people noticing.

      An example referring to the legal maze is the Trans-Pacific partnership which enables multi-national companies to sew governments and run court processes behind closed doors, building on armies of corporate lawyers which governments can barely match http://www.citizenstrade.org/ctc/trade-policies/tpp-potential-trade-policy-problems/.

      A more tech example is high-frequency trading, which has gone out of hands and has nothing to do with what rational beings consider as a market where value should be traded. https://www.youtube.com/watch?v=kFQJNeQDDHA

      What do you think about such aspects?

  • Michael Varian Daly

    We Transhumanists say this makes the case for Augmented Human
    Intelligence, the implantation of cybernetic interface elements directly
    into the human brain. The basics of that tech is already being applied
    and plenty more is in the pipeline.

    • G

      Ahh yes, the Church of Singularitology. Most of its premises are as untestable as the existence of traditional deities.

      But even if a silicon neuron-replacement becomes possible, then: replacing the 1-2 billion biological neurons in the human brain with silicon neurons, at the rate of one replacement per second (including all of its approx. 8,000 connexions to other neurons) translates to approx. 32 to 64 years in the hospital, 24/7/365 inpatient treatment, hooked up to something that can cross the blood/brain barrier.

      Then there's Upload, the belief that human minds can achieve eternal life by being transplanted to computers. This is nothing more than the latest version of reincarnation. If you can reincarnate into a computer, you can also reincarnate into a cat.

      As for "augmentation," we can make much better use of the brains we already have. Continuing education, or even voracious reading online, can increase knowledge, and regular practice can improve skills. Twenty minutes a day for plain-jane meditation exercises (concentration and mindfulness) can improve concentration, discernment, and capacity for objective thinking.

      Education + exercise = enhancement. No need to become a Dalek.

  • susan

    poor liberal artist--lost in a STEM world.
    technology is based on a few basic physics, math, chem principles.
    less art history--more differential equations.

    • Andre

      Ironically Mathematics and natural sciences are liberal arts. Realize
      this; the inspiration of the U.S. people went from being symbolize by Martin
      Luther king in the sixties to Steve Jobs more recently. But do not worry art is
      doing quite fine everywhere else in the world.

  • http://www.mazepath.com/uncleal/qz4.htm Uncle Al

    We have become Peter Griffin in a Spock world. Here comes the giant rooster....

    • Andre

      Exactly... an epic fight. But thankfully some of us understand that the giant rooster is just an irrational fear.

  • x

    Dear author of article, please read up on the concept of abstraction.

  • Max Bleyleben

    Regulatory accumulation is depressingly visible everywhere. How about this for a solution: an enlightened government commits itself to never growing the volume of regulation - for every new rule it writes, it has to retire an old rule of similar size. This would:
    - force us to really consider the merits of new rules, given that they come at a regulatory cost;
    - provide politicians with a mechanism and reason for getting rid of regulation that no longer makes sense, or has had unintended negative consequences ('political cover'); and,
    - force us to write a clearer, more concise regulation in order to minimise the volume of rules we have to remove to accommodate it.

    Whilst such an idea is not likely to be promoted by sitting politicians, we can always promote a grassroots idea that might squeak through on a reformist ticket... ;-)

  • Michael Hanlon

    Of course it may turn out that underlying all this apparent complexity is a Universe of beautiful simplicity. Let's hope so. And we should not be surprised that no one understands everything about, say, a large computer operating system. No one has read and understands all the books in the British Library either but that doesn't make it a scary or unapproachable resource.

  • logik22

    Interesting article, but it really does underestimate the human brain's abilities to acquire infinite knowledge and master infinite skills, even at these early stages of brain research and science. I refuse to believe that anything man designs, builds and maintains is beyond the ability of man to understand how and why it works. All the technological advances we are experiencing are technological advances designed, manufactured and tested by human engineers who have honed their knowledge and creativity into machines, lines of code and every other aspect of human life. There may not be many who could understand the theory of operation of many complex machines and software programs, but there are many who do. The fact that technology is obviously expanding at a rate that the human brain cannot keep up with, does not have to be looked upon as a negative thing, but rather a challenge. A challenge for the designer, engineer, manufacturer, software engineer, et al to always be aware of, but never to shy away from. The advantages to humanity of ever advancing technology will always have more benefit to society than drawbacks, challenging as it may be to comprehend and understand them.

  • cfromke

    If it crashes, can we reboot? Will we want to?

  • bicwood

    Does anyone have a reference with more information on the project described in the article where evolutionary design was used to create a circuit the engineers could not understand?

  • lemonfunn

    Have you guys ever heard of the term "eugenics"? by advancing the intellectual level of the next generation to make the increasingly complex system relatively simpler to mind

  • shrinivas_nm

    maybe god too is a human creation, a system to regulate the world, but this system over the period of time became too complex for humans to handle or even understand

  • Ari BenDavid

    A rather silly article. There have always been unintended consequences of actions that someone was not able to predict. If you want to really examine a complexity out of control, it is the patent system. Neither man nor machine is reliably able to determine what is patentable. Our corporations are spending much more on intellectual property litigation than on on R&D -- and the lawyers so employed make more money than the engineers that would be able to do something constructive.

  • Jasmine

    I'd like to think that as an entire population, over time, there aren't any limits.

  • Neuromancer

    On the one hand, I am reminded of caption to illustrations in Warhammer 40,000 (paraphrased..) "One takes the Holy Wrench of Bahamut and rotates the mechnaism is a direction following that of the passing Sun." Or some such malarky. The point was that despite technology having evolved so much in 38,000 years, humans in effect became less intelligent and less capable than anytime previous. Religion was used to educate people without actually teaching them anything but route maintenance and adherance to doctrine (Looking at you MSAA's)

    On the other hand the truth is, as alluded to both in the article is that specialization comes at a cost. I found myself working with computer Science post grads that really have no clue how a computer works. (Which I found frustrating as a high school dropout with a ken for Computer Engineering).

  • simon

    "what is matter, never mind... what is mind, no matter" quoted from simpsons.

    Thought provoking article which i really agree with. I think the easiest example of humans creating something that has evolved to be far more complex than what we initially envisioned would be money.

    A simple trade system that replaced the barter system has now evolved into something so complex, no one comprehends now the multiple ways in which an economy is affected. Are recessions a consequence of the dis-equilibrium in the market economy, are fiscal or monetary policy more effective at stabilizing market fluctuations?

    These are just some examples of many in the real of economics. To think that something which started out so easy to understand has evolved into something so complex really boggles the mind.

  • Kingtut

    There is too much knowledge in the world for a single human being to ever understand. Everything is complicated these days. Everything. Its insane. Cars are complicated, weather is complicated, schools are complicated, life is complicated, problems, problems, problems, wars ..while schools try to educate people for a complicated world, people are starting to hate each other and take revenge on one another for atrocities happening throughout history. Nobody cares about careers anymore (at least in north america) - sometimes the complexity is too much that nobody can handle it and they just quit. There are limits. There are financial limits. Resource limits. Humans have infinite dreams in a finite world.

    • Kingtut

      Man should go back to remembering God and just leave all knowledge to God rather than becoming God. Man can never win. Your head will explode. You will go crazy. SOMETHING WILL GO WRONG.

    • Kingtut

      Even doctors do not understand everything in the universe. NOBODY DOES. Only the creator can!!!!! ITS OBVIOUS. THERE HAS TO BE A GOD. LOOK AROUND YOU. THINGS ARE THE WAY THEY ARE BECAUSE ITS KNOWLEDGE ON TOP OF KNOWLEDGE ON TOP OF KNOWLEDGE ON TOP OF KNOWLEDGE.

      "IF I HAVE SEEN FURTHER THAN ANYONE , IT IS BY STANDING ON THE SHOULDERS OF GIANTS" -- Sir Isaac Newton.

  • JenJen10

    What this didn’t suggest but could have is that maybe instead of just becoming more “humble”, we should take steps to make our world more simple, one which we do understand. I think we need to live in a world that we can understand. Maybe not to the extent the Amish have, but there's a lot of space inbetween technology & the Amish.

  • teddyknox

    I think a lot of the misunderstandings about this article would be cleared up if more precise vocabulary was used in the place of words like "complex" and "understanding".

    As I mentioned in another threaded comment, "understanding" complex black box machine-learning systems, in the conventional sense of the word, is a futile endeavor for humans, just as competing with a computer to solve arithmetic problems would be a waste of time. We don't feel bad about being worse than a computer at math, and so we shouldn't feel bad about being worse than a computer at understanding the contents of their models. I want to place emphasis on the fact that we fully understand why these models generate answers to these problems (that's obviously how they were invented in the first place). But we don't understand the internal mechanics of an instance of the model that produce the answers.

    I'll make another analogy to try and illustrate this point. Imagine we as scientists understood the human neuron perfectly, so that we could simulate is 100% accurately. Imagine too that we had the 3D map of a person's brain, with all the neurons inside it. With a powerful-enough computer, we would be able to simulate that person's brain using only our knowledge of the neuron. We would also still have absolutely no idea whatsoever how it worked collectively, how 113 billion stupid neurons turns into an intelligent personality. Most of the amazing behaviour is probably an emergent property of relatively mechanical and logical work on the part of the individual neuron (if we encounter this kind of mental labor in our modern life, we usually delegate it to a computer; the beauty of the brain is that its grassroots computation occurs on the molecular level). So the brain is not capable of understanding the brain, at least in the sense of understanding how a car's engine works. The brain is great at making models, noticing patterns, making inferences. We love our brains for this. I'd rather write poetry then add numbers all day. So this extended analogy is to say that the word "understand" should never be used anywhere near the words "model", unless to say that we understand the rules that created it in the first place.

    Since we love our brains and what they do, why not start using principles of its design to solve problems that the brain of a single person can't solve? Just like a professional, it would occasionally make mistakes, and just like the mistakes of a professional, we wouldn't get hung up on the mental processes that resulted in those mistakes. What we get is a machine that's really good at doing things, better than a human in fact, but thinks more like a human than a machine, and comes with the drawbacks associated. It's sort of a no-brainer, we'll have fewer mistakes in the end, and we'll have a simpler set of responsibilities to manage.

  • Sarthak Kulkarni

    Hello and thank you very much for thus post
    I truly agree with what you said . I believe that the ancients civilizations were much more wiser than us because they had only that technology which made life peaceful and simple and convenient and NATURAL .Today I was seeing some videos on 3dfood replicator which produces edible delicacies and how a device can induce lucid dreaming and provide control over our dreams like in the movie inception.Moreover the information burst is not good. I am a Hindu but as Jesus said if only you'll believe everything is possible .So simple life is yet stupid people are not using their subconscious mind in the right way complicating life unnecessarily.
    Perhaps the next big thing in the world would be to help people simplify their lives and make it natural.