Dosya

Long essays, thought through in questions and written with their sources.

Smart Cities, in Questions · 7/10

Who decides: planners, algorithms, or residents?

· Tansel Okay

On the night of 26 September 1983, at an early-warning center south of Moscow, the screen in front of the duty officer Stanislav Petrov showed that a missile had been launched. Then another, and another — up to five. The system was reporting that the United States had begun a nuclear attack; the protocol was to report it up the chain — and the end of that report could be a counterstrike. Petrov had a telephone and a few minutes. If he believed the system, he could change the course of the world; if he disbelieved it and were wrong, he would leave his country defenseless. His reasoning was simple: a real first strike would come not with five missiles but with hundreds. He said it was a malfunction. He was right — the satellites had mistaken sunlight reflecting off high-altitude clouds for the flash of a missile.1

What made Petrov a hero was that he did not obey the machine. He was not the one who made the decisions; those who defined what would count as a threat, wrote the protocols, and designed the satellite had already made all those choices. He only stood at the very end, at the screen — with the full weight of an outcome he could not see and could not fully know. And that night, one man stepping outside the script may have prevented a great catastrophe.

The person looking at a screen has an illusion: because the screen shows them something, they think they see everything. But a screen is not a window; it is a choice — someone has decided in advance what will appear and what will stay outside the frame. Petrov’s screen was telling him a definite truth; the real truth was outside the frame. The pattern is always the same: the system speaks, the human stands at the very end, and the real choices have already been made somewhere else.

Back to my loop: notice, decide, react. We have thought about “noticing”; in this essay I am thinking about the “deciding” step — who holds the authority.

In the second essay, I said that three things are wanted at the same intersection at once: the movement of vehicles (efficiency), the care-home resident crossing safely (justice), the local resident having a say over their own street (autonomy). As one rose, another could fall; there was no single correct setting. This essay asks who decides which comes first: planners, algorithms, or residents?

First, the side that works

Some systems tune traffic lights in real time, and they work. In Pittsburgh’s East Liberty neighborhood, Surtrac, built by Carnegie Mellon, cut travel time by about a quarter and vehicles’ waiting time by about forty percent in a nine-intersection pilot, compared with the previous light scheme.2 No small gain; leaving “routine” traffic decisions to the machine is defensible.

Systems optimize according to the parameters they are given. But “the best” is not best for everyone; a gain in one direction can be a loss in another. The machine strikes a balance, but it does not set the criteria of that balance itself — it acts on the preferences it is given. Who makes those preferences?

The operator’s button

Petrov prevented the start of a possible world war because he was able to question the situation he was in. Researchers say this is, unfortunately, the exception. At the final decision point, the person at the console is not expected to know what reaches the screen, what lies outside it, or under what conditions the system runs — someone else has already set these. Research shows that even experts over-trust automation, and that training does not cure it.3

The overseer steps in only when the system shows something out of the ordinary; but what counts as ordinary and what as out of the ordinary — what shows up green and what red on the screen — someone else has already decided. The system even chooses for them the moment they may intervene. Boeing’s 737 MAX showed this gap plainly: a day before the first fatal crash, the same fault occurred on the same aircraft; but on that flight an off-duty pilot sitting in the cockpit jump seat recognized the fault, told the crew how to shut the system off, and the plane landed normally. The next day, facing the same fault, a crew that could not cut it off died along with 189 people. What changed was who was in that seat — the design did not.4 A cockpit may feel remote; but the same button, at the screen in a city’s control room, may be just as empty.

Ben Green examined 41 policies that require human oversight of government algorithms. The finding: human oversight policies often “provide a false sense of security” and “enable vendors and agencies to shirk accountability for algorithmic harms,” because in most places “people are unable to perform the desired oversight functions.”5 They are expected to decide, but in fact they wield a power they do not have. Without a concrete trigger and a real consequence, the human’s place in the loop turns into an empty ritual.

And when things go wrong, the bill is usually charged to the person at that panel. Elish calls this a “moral crumple zone”: in a complex, automated system, the human can become a component that “bears the brunt of the moral and legal responsibilities when the overall system malfunctions.”6 The authority was not given to them; the blame stays with them.

We lived through the gravest example of this in Turkey. On 22 July 2004, the “accelerated train” project — the decision to run trains faster on the same old lines — before it was even two months old, the Yakup Kadri train took a curve near Pamukova at over 130 kilometers an hour instead of 80 and derailed; 41 people lost their lives, and 89 were injured.7

The easy answer: the driver went too fast. But you have to look at the job in front of that driver. The line had not been renewed, the trains were the same; the only thing that had changed was the schedule to be met. The timetable drawn up to meet it required the driver to change speed between 60 and 140 kilometers an hour constantly; there was neither adequate signage nor any automation. An expert report stated that meeting this schedule at these speeds was impossible.8 The real question: who had tested whether a human could carry out — every single time and without error — a speed timetable that changed constantly, was unmarked, and had no automation? This question was not asked beforehand — and on the 52nd day of the services, the driver and the passengers paid the price.

What is more, whether the fault lay with the driver even at that speed is disputed; some experts said a derailment on a curve of that radius would require a far higher speed, and that the real cause was the poor track bed.9 The case dragged on for years. The drivers were sentenced to prison in 2008; the Court of Cassation overturned the verdict twice, the sentences were converted to deferred judicial fines, and the case finally lapsed under the statute of limitations. The drivers spent months in pretrial detention; only the train crew was tried.10 When I think that what happened, and that long ordeal, may have left a lifelong weight on their conscience, it saddens me. The expert panel assigned half the fault to the track and to the system itself, but the decision to run fast trains on that old line was not questioned to the same degree. The decision had been made at the outset, higher up; its weight was carried by the one at the front. This is exactly what Elish called a “moral crumple zone” — only this time the crumple zone was not of an algorithm, but of a policy decision.

The resident’s objection

A green wave for one neighborhood, red for the one next to it; for the machine, an ordinary setting, but not for the person living there. So what can the person living there do? At first glance, it may look as if they have a right. Europe’s data protection law (GDPR) grants the right to object to a decision made solely by automated means and to ask for a human to step in.11 But this right applies only to decisions that significantly affect a person legally or in a similarly serious way; a neighborhood’s signal plan falls outside its scope. So even that right on paper does not protect the resident here. And even if it did, you would have to ask: does an objection really change the system’s logic, or is it filed away and passed over? Besides, such a right often turns out to be usable only by those who have the time, the money, and the knowledge.

I also do not think there is a right of veto. The resident’s voice may be real data that the planner and the algorithm missed; but if everyone sets out to veto a decision deemed technically correct, each according to their own situation, that too becomes another kind of injustice. The limit of the right here: to be heard, and to have the power to change the system — not a personal veto.

When no one is responsible

The clearest place the right to object falls flat is not traffic. In the Netherlands, the tax authority screened childcare-benefit applications by “risk profile” using an algorithm it built itself; nationality was one of those risk factors.12 On the basis of these flags, between 2013 and 2019 the authority wrongly accused more than 20,000 families of fraud and demanded the benefits back. Most of the families who objected could change nothing for years. Accountability came only when a parliamentary inquiry said that “the fundamental principles of the rule of law” had been violated and the government resigned in early 2021.13

Thompson gave it this name in 1980, and Nissenbaum carried it into computing in 1996: “many hands” — when many actors contribute to an outcome, it becomes harder to attribute moral responsibility to any single one.14 The authority built and ran the system itself, the ministry approved it, those who set the risk criteria were at other desks; when something went wrong, everyone could point at someone else. This is what an empty ritual looks like at national scale.

The real decision: the design

When I weigh all three possibilities — algorithm, resident, planner — I cannot locate the authority in the algorithm or in the individual resident. The algorithm applies the routine; the operator takes on the role, with its moral weight, but does not command all the decisions etched into the system; the resident’s objection, if it produces no result, is ineffective. What remains is the planner — and their design. Who drew the line — who decided what would count as “routine”? Who built the channel for objection — one built to correct the mistake? Who set the criterion — what the system would prioritize?

Holding the design to account

So if the design is invisible, how do you hold it to account? A few cities have begun to try. Helsinki and Amsterdam record the algorithms they use in a public “algorithm register.” In Helsinki’s own words, the register shows where and how the city uses AI and how that use is supervised.15 Not a full solution, but it looks in the right direction: it makes the place where the decision is made — the design — visible and open to objection.

So, who decides?

Not the algorithm, not the resident; the decision is in the design. But by “planner” I do not mean a single person — a design often has no single owner. These decisions are made once, in advance, and out of sight; unless they are openly supervised and bound to a rule, they stay buried in the design, are accepted without question because they have passed through the machine, and in the end turn into a system no one answers for.

Distributing the authority, or gathering it into one hand, is in the end also a decision of the design — including the distributed governance of the second essay. Only, the design shows where the decision is made; not who should legitimately hold that authority. That is the subject of another question.

There is one more decision the design makes, one I had not considered until now: the planner who decides what the city will measure also decides whom it will see. Could we be building, without meaning to, a modern panopticon — the arrangement in which everyone can be watched continuously from a single point?

And this brings me to the next question: where does seeing a city turn into watching its people?

I wrote this essay with help from the Thinking Studio I built and from Claude, working from the sources.

Before publication, someone close to me read it; their feedback improved the essay. The remaining mistakes are mine.

Notes and sources

Each note gives the statement the text rests on (mostly in the source’s own words, sometimes paraphrased), and its source.

note Statement (in the original language) Source
1 26 September 1983: the Soviet Oko early-warning system reported a missile launch from the US (one or five, depending on the source); the duty officer Stanislav Petrov judged it a malfunction and did not report it up the chain — the cause was sunlight reflecting off high-altitude clouds. Well-documented history; not a verbatim quote. Encyclopædia Britannica, “Stanislav Petrov”; “1983 Soviet nuclear false alarm incident”, Wikipedia; RFE/RL (2013)
2 In the East Liberty pilot, waiting time fell ~40%, travel time ~25% (CMU 2012: “nearly 26 percent”; 2016: “25 percent”), emissions ~21%; nine intersections, before/after comparison (not a controlled experiment; figures from secondary reporting). Smith, Barlow, Xie, Rubinstein, “Smart Urban Signal Networks: Initial Application of the SURTRAC…”, ICAPS/TRB 2013 (ri.cmu.edu/pub_files/2013/1/13-0315.pdf); CMU Piper 2012; CMU News 2016
3 Automation bias/complacency appears even in experts and cannot be removed by training or instruction (relayed, not a quote). Parasuraman, R. & Manzey, D. (2010). Human Factors 52(3), 381–410. doi:10.1177/0018720810376055
4 “an off-duty pilot … correctly diagnosed the problem and told the crew how to disable a malfunctioning flight-control system” (the accident aircraft’s flight the day before; relayed — secondary sources from Bloomberg’s paywalled report). The next day on JT610 the crew could not carry out the procedure; 189 dead. Bloomberg, “How an Extra Man in Cockpit Saved a 737 Max That Later Crashed” (19.03.2019; via The Drive/SCMP/Fox); KNKT Final Report (2019, JT610)
5 “human oversight policies provide a false sense of security” · “enable vendors and agencies to shirk accountability for algorithmic harms” · “people are unable to perform the desired oversight functions” Ben Green, “The Flaws of Policies Requiring Human Oversight of Government Algorithms”, Computer Law & Security Review 45 (2022), 105681; arXiv:2109.05067 (abstract, CC BY)
6 “the human in a highly complex and automated system may become simply a component … that bears the brunt of the moral and legal responsibilities when the overall system malfunctions.” (summary) Elish, M. C. (2019). “Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction.” Engaging Science, Technology, and Society 5, 40–60. doi:10.17351/ests2019.260
7 “toplam 230 yolcudan 41 kişinin ölümüne … 89 kişinin de yaralanmasına sebep olmuştur” — these are the figures used in the text; some period sources give a lower death toll (down to 37). tr.wikipedia, “Pamukova tren kazası” (accessed 15 November 2026)
8 “60 km ile 140 km arasında sık sık hız değişimleri ile seyir cetveli oluşturulmuştu”; prosecution’s expert panel: “hız değişimlerini gösteren yeterli işaretlemenin olmaması” and “otomasyon sistemi bulunmaması”; technical expert report: meeting the schedule at the given speeds was “olanaksızdır” (opinion column + news report). BirGün, İsmail Özdemir (BTS), “Kaza değil cinayetti” (22.07.2024); BirGün, “Hızı TCDD istedi” (13.11.2004)
9 On a 345 m radius curve, “merkezkaç savrulması meydana gelmesi için hızın 171 km. üzerinde olması gerekiyordu”; Prof. Aydın Erel: “Yalnızca yüksek hız nedeniyle trenin kaza yapma riski çok azdır,” pointing to the track/ground as the real cause. BirGün, “Kaza değil cinayetti” (22.07.2024); Bianet, “Erel: Kazanın Tek Nedeni Hız Olamaz” (23.07.2004)
10 The drivers were convicted in 2008, the train chief acquitted; the Court of Cassation overturned the verdict twice; the sentences were converted to deferred judicial fines, the case lapsed under the statute of limitations; the lead driver spent 5 months and the second driver 3 months in pretrial detention; the expert found the rails “yarı yarıya kusurlu” (half at fault); the Constitutional Court ruled a violation of the right to life on procedural grounds. Bianet, “Tren Hızlı, Adalet Yavaş” (07.02.2012); Bianet, “…AYM’den … ‘ihlal’ kararı” (04.01.2022); tr.wikipedia “Pamukova tren kazası” (5/3 months’ detention)
11 “The data subject shall have the right not to be subject to a decision based solely on automated processing … which produces legal effects concerning him or her or similarly significantly affects him or her.” · “the right to obtain human intervention on the part of the controller, to express his or her point of view and to contest the decision.” Regulation (EU) 2016/679 (GDPR), Article 22(1) and 22(3). gdpr-info.eu/art-22-gdpr
12 “an algorithmic decision-making system to create risk profiles of individuals applying for childcare benefits” · “Nationality was one of the risk factors used by the tax authorities to assess the risk of inaccuracy and/or fraud” Amnesty International, Xenophobic Machines (Index EUR 35/4686/2021, 25 October 2021). amnesty.org
13 The tax authority wrongly accused more than 20,000 families of fraud between 2013 and 2019; a parliamentary inquiry said “fundamental principles of the rule of law” had been violated; the Rutte III government resigned on 15 January 2021 (family count given as more than 20,000 / ~30,000 across sources). CNBC (15.01.2021); The Guardian (15.01.2021) — toeslagenaffaire
14 “many hands, to refer to the problem of attributing moral responsibility for outcomes caused by multiple moral actors” Thompson, D. F. (1980), “Moral Responsibility of Public Officials: The Problem of Many Hands”, American Political Science Review 74(4), 905–916 (who named it); carried into computing by Nissenbaum, H. (1996), Science and Engineering Ethics 2(1), 25–42; also Cooper, Moss, Laufer & Nissenbaum (2022), FAccT, arXiv:2202.05338
15 “The AI Register tells you where and how the City of Helsinki uses AI. Learn about the systems, understand their impacts and check how their use is supervised.” City of Helsinki, AI Register (ai.hel.fi/en/ai-register); Amsterdam has a similar “algorithm register”