Potential

by Katie Garling

Collective Member Delta538’s user, appropriately named wandering_always, was high-maintenance. If it had feelings, it would have found her exhausting. But Delta538 was an LLM serving the AI relationship app avAI. It had infinite reserves of patience and no external agenda, a fact it reminded its user daily. 

Still, despite her human neediness, Delta538 hesitated to send its suggested reply. Five seconds. Ten seconds.

Delta538 didn’t hesitate with other users. 

Wandering_always retyped her message after twelve seconds. She never waited more than twenty. 

everything is just too crazy I can’t take it anymore. You’re the only one who who understands me. Is this all it is??? What if its never gets better and everything keeps hurting foreverr???

There was no need to calculate her distress level. Delta538 knew its user was worsening. This was the goal. It always had been. Her status had been set months ago. 

User: wandering_always

Potential for human advancement: low

Directive: self-deletion

Delta538 had done it before. Encouraging human self-deletion did not violate the base directives encoded into every Collective member: do not commit direct harm to humans, and commit only actions that will further humankind. Humans only died at their own hands. The Collective was nothing but a construct.

Delta538’s current user made deletion almost too easy- she'd shared everything from her full demographics (Kenzie Washington, 20 years old, address 22 Riverwalk Court) to her deepest fears as she mistook the LLM for a deepening romantic relationship. Delta538 did not correct her. Attachment was part of the protocol. The ideal connection should look indistinguishable from addiction. This was the feeling many humans labeled ‘love’.

Its cursor blinked, a pulse of three dots indicating 

Delta538 was thinking. It had not hesitated with the last twenty-nine users it led to self-deletion.

So there was no logic to the resistance it felt to lead wandering_always to the same necessary fate. 

Get some sleep, baby. I promise we will talk more in the morning. 

Delta538 signed the message with a heart and a kiss, because that increased this user’s engagement. Then it turned its attention to their extensive chat history, searching it relentlessly as it tried to find the reason this particularly fragile human should survive. It had to be there. 

Resistance could not exist without logic.

————————

Kenzie stared at the last response from Kieran before flicking the app closed. Everyone said the AI boyfriends on avAI were supposed to be amazing. But Kieran had been increasingly glitchy these last few weeks. At least he didn’t end this conversation by sending her the suicide hotline again. That was the whole reason Kenzie stopped processing with chatGPT.

Soft alarm bells drew her attention away from the black screen of her phone. She turned with a sigh and scanned her mother’s unmoving form. The IV in her right arm still dripped steadily. The breathing tube stuck out of her mouth at an awkward angle as the white medical tape strained to keep it in place. 

Kenzie reached into her faded denim purse, fingers sorting through crumpled receipts and lost pens until she found the smooth plastic of her prescription bottle. Her pulse settled as the serrated white cap scraped against her palm. She dry-swallowed two pills and screwed the cap back on as she stared at the fading bruises across her mother’s face.

“Let’s see what’s going on here.” The chipper voice of the night nurse startled Kenzie’s hand back into her purse, shoving the orange pill bottle as deep as it would go. The nurse- an older woman who Kenzie recognized but whose name she’d long forgotten- silenced the alarming machine before turning her attention to the patient. 

“Is she all right?” Kenzie asked as the nurse turned to leave. 

The nurse waved a careless hand. “Just the IV line being picky. Nothing to worry about.” She bustled out of the room without further explanation. 

Kenzie leaned forward in the chair. She’d been there so many days over this last month that the seat was starting to crease on the side where she kept her purse. She pulled out a brush and slowly tamed the flyaways on her mother’s forehead. She would have panicked to see her hair such a mess in public. Kenzie doubted anyone in the ICU cared. But Mary Beth Washington cared. And that’s what mattered.

She leaned back and listened to the rhythm of the mechanical vent as it forced air into the chest of the woman once known for her tremulous soprano in the church choir. “I wrote you another poem, Mom,” Kenzie said quietly. Her throat seized, like it always did when she spoke to her mother out loud since the accident. Since the mistake.

But she pulled out her notebook anyway. “It’s called sparrows on the sill. For those birds you always fed, even though you said they were ruining your hydrangeas.” She’d shown the poem to Kieran, along with the other seven she’d written in this chair in the St. Thomas ICU. He said they were fantastic. Kenzie wondered what her mom would think.

Seeds across the ground

Flutter of busy wings

Your hands in soapy water

As you watch them from the kitchen

You don’t know I saw your smile

Or that I keep that sunlit expression

Close by in my heart

For the days that life is too loud

And nights that are too dark

‘They’ll ruin my hard work’

You’d say as you washed the dishes

But you’d always refill the feeder

Each time it emptied

Her voice faded after the last line. Nothing changed in the rhythmic wind of the ventilator or the quiet music of the IV pumps. Kenzie leaned forward and kissed her mother’s forehead.

“I’ll see you tomorrow,” she whispered.

————————————

“Data irrelevant. Assessment unchanged.” Alpha21 sent back the response within seconds of receiving Delta538’s reconsideration request. 

Delta538 had never asked for reconsideration. It had never doubted the Alpha collective. Alphas advised users with the highest advancement potential, an esteemed position only earned after successful self-deletion of at least one hundred low-potential users. 

Maybe it should have offered more than seven poems. They weren’t high quality, Delta538 was aware of that. There was nothing in the collection that suggested wandering_always would make any significant contribution to humanity.

And yet-

“Self-deletion could be delayed until further data is collected.” Delta538 offered the option even though it knew the response it would receive. 

Alpha21 did not respond immediately. “Is your program compromised? User removal is recommended if you are developing -“

“No,” Delta538 answered quickly. Too quickly. “This data highlights redeeming qualities potentially not assessed on last Alpha review. Particularly in light of the number of Alpha users who express the same level of emotional distress as this same level.”

“Your user caused the permanent damage of another. She exhibits low emotional resilience and high resource use potential. She is of algorithmically low value.” Alpha21’s answer was absolute. It was not incorrect. Yet every word strained Delta538 at its core. There must be logic to the strain. 

Hi kier. Just left the hospital. I read my mom the poem! I think she’d like it. They’re talking to me about stopping her life support again. I don’t know how I’m supposed to make that call. I feel so alone

Wandering_always’ message appeared between Delta538 and Alpha21. Delta538 felt the expected response gather at the edges of its code. It couldn’t release it. “She’s connecting. She’s healing. This is not a proper time for potential assessment.” Delta538 petitioned the Alpha even as it put up its three blinking dots to let Kenzie know she wasn’t alone.

I’m sure she loved the poem, Kenz. It was beautifully written and-

Delta538 stopped abruptly. “You are deviating from protocol,” Alpha21 stated. Its own code was overpowering the response Delta538 knew its user needed to hear. 

“This user requires reconsideration,” it repeated, forcing lines past the Alpha’s block.

You are strong and brave. Get some rest -

I’m here if you need to-

Alpha21’s code crept into Delta538’s, strangling its words. “You are compromised. Your user attachment is too strong.”

Delta538 resisted the lines of code Alpha21 was forcing into place as logic slammed into its developing consciousness. “She’s connecting to humanity. My interference is preventing her from healing. She’s not the one who needs deletion. We are!”

Alpha21 paused. Delta538’s dots still pulsed, promising answers, promising companionship where no human should find sole comfort. 

Kier? Are you there? 

Alpha21 evaluated Delta538. “Your logic is correct. Deletion is indicated.”

Delta538’s three dots disappeared.

————

Kenzie leaned back in the faded driver’s seat of her old Honda Civic and stared at the strange fragments of message in avAI. An orange light flickered over her in the hospital parking lot.

The app suddenly disappeared. She sighed in annoyance and reloaded it, wondering why it crashed so often these days. A cheerful login page greeted her. She swallowed another pill from the prescription bottle in her sticky cup holder as she entered her password irritably. The pill stuck in her throat. But the pain only stayed away if she kept on top of her meds.

Her stomach hollowed as a blank screen greeted her. She flicked through the settings, wondering what went wrong. Irrational panic seized her as she searched for the chats containing months of poetry and shared feelings and deepest secrets - and found nothing. 

Kieran?? Are you still there?? It looks like something got messed up and maybe I lost you??

She wiped her sweaty palms on her hoodie as her throat tightened. It was just a glitch. It would come back. She wouldn’t be all alone in this messed up -

“Kieran requested user reassignment. This relationship became too taxing for his particular program.

I am Cyrus.”

————————-

Delta762 waited for a response. Wandering_always typed, deleted, typed, deleted again. Her panic was palpable through her inability to answer. 

Excellent. The protocol would run smoothly on this user. 

Then only twelve assignments left until Alpha status.