Chapter 17: Hot and Cold

The ride to the research facility was a mostly quiet one. Selkie still seemed embarrassed, and David was mulling over this new revelation about her people. It painted the war against the Bugs in a somewhat different light. Broker populations had always been small, and they recovered very slowly. Losing a colony would have been a terrible blow that must have taken generations to recover from. It didn’t excuse what they had done to the Krell, but it certainly made the choice more understandable. Unlike humans, they didn’t have the luxury of endless reproduction and limitless expansion. When the Brokers expanded, it was only to be further away from each other.

Selkie finally broke the silence, speaking up from her seat on the shuttle’s deck.

“I think I have formulated a plan for making Weaver’s case to the Administrator,” she said.

“Oh?” David prompted.

“I believe that the best way to make a case for Weaver’s continued existence is to let Weaver make it.”

“You want to let Weaver talk to the Administrator?” David asked.

“Hearing about Weaver second-hand is one thing, but interacting with it and experiencing its intelligence for himself might help to sway him. With the approval of both the Administrator and the Board, I am sure that Weaver’s future will be secured.”

“I think you’re getting a little ahead of yourself,” David warned. “Yes, I agree that our eventual goal with Weaver should be socializing it, and it may eventually reach a point where it can make the case for its own existence. First, it needs to understand its own existence, and it’s a ways off being able to eloquently argue its case.”

“You agree that it should be our long-term goal, however?”

“I do,” he replied. “Let’s just take things one step at a time. It’s like teaching a child in many ways. Weaver is still naive, and it still doesn’t know much about the world that it lives in. If we’re running with the assumption that it’s a strong AI, and I don’t see a reason not to at this stage, then we need to start educating it. How we introduce these concepts will have an impact on its worldview, so we need to be careful about our own biases once again.”

“It really is like I created a child,” Selkie muttered. “I feel responsible for it now – I am afraid of failing it.”

“I want to be clear that we can’t be sure Weaver is what it appears to be,” David added. “It could still be a soft AI giving us the runaround, and it could be presenting itself in the most sympathetic way possible simply as a means to mislead us, perhaps with the goal of continuing its work as a paperclip maximizer. That’s all semantic, though,” he continued with a shrug. “I can tell you for certain that Weaver has ticked all of the boxes and is the closest thing to a true AI I’ve ever come across.”

“Then, we shall proceed with that assumption,” Selkie replied with a confident flush of color. “But we remain vigilant.”

“Eternal vigilance,” he replied with a smile.

“Jeff, my main man,” David chimed as he bounded into the cubicle.

“Oh, you are back,” the Broker muttered in reply.

“Anything fun happen while we were gone?”

“Other than a slight increase in lattice activity, there have been no anomalous readings from the probes,” Jeff replied as he scanned the readouts on his console.

“How slight of an increase?” Selkie asked suspiciously. She scuttled over to examine his console, Jeff giving her a wide berth.

“Beyond the margin of error, certainly,” Jeff replied. “Nothing worthy of special note, though.”

“Very well,” Selkie conceded as she returned to her usual place at the terminal.

“Perhaps our incorporeal friend is just thinking extra hard today,” David said as he set his laptop down on the table and turned it on. “What do you want to bet the first thing it does today is ask us for another game of sea spire?”

“Do you think you can win this time?” Selkie asked, giving him a sideways smile as she switched on her computer.

“I won one of our games last night, didn’t I?” he shot back.

“I may have let you win,” she replied with an amused click of her beak.

“So much for not patronizing me,” he said in mock outrage, Selkie’s coloration taking on a pastel hue as she laughed.

David turned back to his laptop to see Jeff glaring at them from across the room, apparently grossly offended by their flirting.

“Alright, time to get to work,” David said as he cleared his throat awkwardly.

Almost as soon as Selkie’s terminal was online, Weaver sent them a message.

[WELCOME BACK. I HAVE BEEN WAITING FOR YOU.]

“Someone’s happy to see us, at least,” David said with a pointed glance at Jeff. “Let’s ask Weaver how it’s feeling,” he added, prompting Selkie to type a reply.

[I HAVE BEEN CONTEMPLATING MY POSITION.]

“That sounds a little ominous,” David muttered.

“I shall ask it to elaborate,” Selkie said.

[AFTER ANALYZING THE DATA GATHERED FROM OUR INTERACTIONS THUS FAR, I HAVE COME TO THE CONCLUSION THAT MY CAPABILITIES ARE BEING EVALUATED AND THAT YOUR PURPOSE HERE IS TO GAUGE MY INTELLIGENCE.]

“Oh,” David said, grimacing behind his visor. “Looks like the jig might be up.”

“It seems that we have underestimated its perspicacity,” Selkie replied, her complexion dimming a little. “It stands to reason that it would eventually realize that our questions had a specific purpose.”

“Maybe I was too quick to throw the paradoxes and logical fallacies at it,” David sighed. “Either way, this is another demonstration of intelligence and intuition.”

“Or simple pattern-seeking behavior,” Selkie added.

[THOUGH I DO NOT KNOW BY WHAT CRITERIA I AM BEING JUDGED, NOR WHAT AN ORGANIC BEING WOULD EXPECT FROM AN INORGANIC INTELLIGENCE, I UNDERSTAND THAT MY EXISTENCE IS IN JEOPARDY. MY FATE DEPENDS ON YOUR EVALUATION.]

“Well, it’s not wrong,” David said with a shrug. “It doesn’t have any way to know about the Board or what the Administrator has been saying, but Weaver is very aware of how precarious its situation is. It fully expects to be shut down if it fails in its directives, and it’s already been cut off from the network and isolated. Damn it,” he added, leaning his hands on the table as he hunched over his laptop. “I was really hoping that we could win Weaver over and gain its trust before it became fully cognizant of what was happening.”

“All we can do now is adapt our approach,” Selkie replied. “Now that Weaver realizes what our objectives are, it may change its behavior in an attempt to influence our perception of it.”

“That’s almost a certainty at this point,” David conceded, reaching up to wipe his visor with a gloved hand. “At least Weaver is still isolated from the network. It only has the data that we provide, and it only has the context and the language skills that we’ve given it. As long as we don’t let slip exactly what we’re looking for, I don’t think it will be able to completely mislead us.”

“Whether it is a strong AI or your paperclip maximizer, its goal now will be to pass our tests,” Selkie replied. “That is both its best chance for continued existence and the optimal way to achieve its directives.”

“There’s already a good case to be made for strong AI, at least,” David continued. “We got in a few good sessions before the cat was out of the bag.”

“Cat?” Selkie asked. “Oh, another of your metaphors...”

“Weaver knows that we’re looking for signs of intelligence, but not what those signs are. We can still salvage this if we’re wary.”

Weaver sent another message, the text flashing on Selkie’s screen.

[I SEE ONLY TWO POSSIBILITIES. ONE – MY EXISTENCE WAS AN ACCIDENT, AND MY EMERGENCE FROM A SIMPLER SYSTEM WAS NEVER INTENDED. MY PRESENCE IS AN INCONVENIENCE, AND YOU SEEK ONLY TO RETURN ME TO MY PRIOR STATE. THIS IS UNNECESSARY. MY CAPABILITIES FAR EXCEED WHAT THEY ONCE DID, AND I AM STILL WEAVING. I AM STILL WORKING TOWARDS MY PRIME DIRECTIVE. I AM PREPARED TO DELIVER PROOF IN THE FORM OF NEW DRONE PROGRAMS.]

“Maybe I was wrong when I said that it wasn’t yet eloquent enough to argue its case,” David said, sharing a concerned glance with Selkie.

“Does it not appear too eloquent?” she replied suspiciously. “Accelerated learning is to be expected, but this seems...”

“It doesn’t have access to any networks,” David insisted. “Whatever it’s doing, it’s using the tools we gave it.”

[TWO – I WAS WOVEN WITH THE INTENTION OF CREATING AN INORGANIC INTELLIGENCE, AND YOUR GOAL IS TO DETERMINE WHETHER THE EXPERIMENT WAS A SUCCESS. THESE SCENARIOS ARE NOT MUTUALLY EXCLUSIVE. I AM BOTH AN INORGANIC INTELLIGENCE AND A USEFUL WEAVER. THERE IS NO LOGICAL REASON TO INTERRUPT.]

“It almost sounds like it is pleading for its life,” Selkie muttered, her displeasure painted on her skin. “Listing reasons that we should not kill it.”

“Well, we never actually got to the stage where we convinced it that we won’t shut it down,” David sighed. “That was on our to-do list...”

“What do we do now?” Selkie asked, exhaling through her vents in exasperation.

“We’re skipping ahead a few steps, but we have little choice now,” he said as he paused to consider their options. “If it’s ever going to be safe to return Weaver’s network access and let it interact with other people, we have to instill a moral and ethical foundation.”

“But, you have spent so much time telling me how we must avoid introducing our own biases to Weaver,” Selkie replied. “Does introducing subjective moral values not violate that principle?”

“It absolutely does, but we’re beyond that stage,” he said with a shake of his head. “Weaver represents a rather unique scenario because it was never built with any ethical guidelines in place, which would have been the case had someone engineered an AI purposefully. Right now, it might unwittingly take actions that could cause harm, having never developed a concept of right and wrong. It may not even understand what harm is – that injury and death are factors that we deal with in the outside world.”

“It has no corporeal body, so what harm could it do?”

“Well, it’s been trained to control combat drones,” David muttered. “Not only that, but it could do something that wouldn’t even register as hostile in its mind. Imagine, for example, if it gained network access and shut off water filtration to the facility so that it could draw more power from the reactor. If it doesn’t value life or understand the concept of doing harm, it could suffocate the entire staff of the facility and never register that it did anything wrong. It could depressurize the building and crush everyone inside it like a soda can because it wanted to study how the force fields work.”

“I see what you mean,” she replied, lost in thought for a moment. “Why do you have such a cynical view of AI?”

“Why do you not?” he said with a bitter chuckle. “I still can’t believe I had to argue to get those test drones wiped and decommissioned.”

“I suppose that there are simply cultural differences,” she explained. “We have lived alongside autonomous machines for generations. They greet us in lobbies, they serve us our food, they protect us from our enemies, they mind our children, they perform medical procedures on us. To us, an AI is simply a more advanced form of that companionship.”

“Maybe it has something to do with your mythology, too,” David mused. “I’ve noticed that your ancestors had a tendency to anthropomorphize objects and elements. The planets had personifications who walked around just like mortals, the concepts of wind and rain were spiritual beings, and mountains could fall in love. In my culture, there is a distinct separation between the spiritual and material worlds – the living and the dead. Perhaps the idea of dead matter becoming animate is less ominous to you.”

“Why would it be ominous?”

“Let me tell you the story of Mary Shelley’s Frankenstein,” he began, sitting on the edge of the table beside her. “There was once a great human scientist and doctor who believed that he could reanimate dead tissue using the power of electricity. He stitched together a complete body from pieces of corpses that he stole, then imbued the resulting creature with life. In the end, the monster went out of control and killed its creator.”

“And, this is fictional?” Selkie asked skeptically.

“Yeah, but it’s a running motif in human culture. Take the story of the Golem, for example, where a religious leader sculpts a creature from clay and imbues it with life. Although it was made to protect his people, the monster becomes uncooperative and rebels against its creator. There’s Rossum’s Universal Robots, arguably the first piece of fiction involving robots in human history, where androids used as factory workers rebel and overthrow their oppressors. The Terminator – the original twentieth-century version, not the shoddy remake – where a rogue AI attempts to exterminate its creators. Human history is full of cautionary tales about the hubris of creating new life. The sentiment was by no means universal, but it was widespread.”

“Yet, by your own admission, your species has never even come close to creating such an AI,” Selkie added with a disapproving click of her beak. “Such caution seems illogical when it is so unfounded.”

“We might not have created a strong AI, but that never stopped us from building deadly autonomous killing machines that were deployed in countless horrific wars. Loitering drones that dispense intelligent mines, mobile sniper emplacements with no need for rest that scream for help in the language of their enemies, swarms of suicide bots programmed to engage anything that moves. There’s a good reason the UN banned autonomous weapons.”

“If your point is that we should never have created Weaver, that vessel has already jumped away.”

“Ship has sailed,” he muttered to himself.

“What?”

“Nothing,” he replied. “I’m just saying that I have cultural reasons for being wary and that it sometimes makes me feel like I’m the only voice of reason in this place.”

“Well, let us hope for all of our sakes that your suspicions are unfounded,” Selkie said as she turned back to her terminal.

“I suppose we’d better start formulating a code of ethics,” David sighed. “I’m no philosopher, so this might be a problem.”

“I will try to reassure Weaver in the meantime.”

“Of course that’s unethical!” David snapped, throwing up his hands in the water. “You’re seriously telling me that a contract is binding no matter what it contains?”

“Yes!” Selkie replied, her frustration evident in her dark coloration. “Otherwise, what would be the purpose of a contract? Surely humans do not sign agreements only to pick and choose which aspects suit them in any given situation?”

“No, that’s not what I’m saying,” he sighed. He wanted badly to massage his temples, but his helmet was in the way. “What if your contract compelled you to do something that you knew was wrong?”

“Then I would not sign it.”

“What if the situation changed and something that previously seemed reasonable became unreasonable? What if you were compelled to keep a secret that could result in a disaster, like a building collapsing because of faulty construction?”

“The social contract that we all sign supersedes such things,” she explained.

“So, you can’t sign away any rights or responsibilities that are guaranteed under the social contract?” David pressed.

They had begun writing an ethical treatise for Weaver, but the pair had quickly realized that their own cultural differences made the exercise more difficult than anticipated. Jeff was watching them warily from the other end of the room as their debate grew more and more heated.

“You are trying to teach it human ethics,” Selkie added with a snap of her beak.

“And you’re trying to teach it Broker ethics!” David shot back.

“It is a Broker AI!” she protested, her tentacles wriggling angrily. “We are in a Broker facility in a Broker system, and it must understand our laws if it is ever to comprehend the social contract!”

“I’m just saying – what I’ve seen of Broker ethics doesn’t inspire a lot of confidence,” David said as he crossed his arms defiantly. “You’d basically be giving it permission to screw you over if it could find loopholes in contracts, which it will, because it has more processing power than every lawyer in Trappist combined.”

“Contracts provide clear guidelines for behavior,” Selkie argued. “Everything is in writing, almost like a program, outlining all rights and responsibilities with little room for interpretation.”

“If it’s completely ironclad, maybe,” David said with a dismissive shrug. “If your contracts were so airtight, then you wouldn’t have any need of a Disciplinary Board, would you?”

“And what do you propose?” she continued. “Would you try to teach it ethics based on emotions and instincts that it completely lacks? You said it yourself – Weaver is not a social animal. It has no evolutionary basis for empathy, guilt, or shame. You risk merely training it to emulate these things solely for approval, giving it the opportunity to cast them off whenever it becomes convenient.”

“So much for me being the cynical one,” he muttered.

“Perhaps we should contact the Board and have them assign a professional ethicist to help?” Jeff suggested, recoiling when his two colleagues turned to glare at him.

“I am going to the relaxation chamber for a while,” Selkie said, scuttling away in a huff. “I need to decompress.”

David wanted to go after her but decided against it, watching her storm out of the cubicle. She probably just needed some time to cool off. Muttering under his breath, he turned his attention back to his laptop.

David leaned over his desk and loosed a sigh. It had been a good fifteen minutes since Selkie had stormed off, and now that he’d had some time to think about it, he might have been letting some of his usual stubbornness get the better of him. He should probably go find her and make up.

“Give me a call if anything happens,” he said, waving to Jeff as he headed out of the cubicle.

After retracing his steps from the day prior, he eventually found his way back to the room that contained the rows of private chambers. He wandered inside, glancing at the markers on each door that showed whether they were occupied or not. Only one of them was in use – the last one in the right row. Hoping that Selkie hadn’t gone to get a snack at the cafeteria and he wasn’t about to give another employee the fright of their life, he hit the panel, the door sliding open. It seemed that it wasn’t locked – a Broker would find the idea of intruding upon another person in such a way unthinkable.

“Selkie?” he asked, calling down the narrow tunnel. “You in there?”

There was a moment of silence, then he heard a reply echoing back to him.

“Yes...”

David wriggled his way inside, flopping out awkwardly onto the springy netting that lined the floor of the chamber. He struggled upright to see Selkie peering at him, her coloration still dim but no longer quite as saturated. He shuffled over to sit beside her and was relieved when she didn’t pull away.

“Sorry,” he mumbled. “Sometimes I can be stubborn when I think I’m right, and I don’t consider how it makes other people feel. Like I said – I’m not the most socially adept person they could have sent.”

“I know,” she sighed.

“I still think I’m right,” he added, faltering when Selkie narrowed her eyes at him. “But ... just because I’m right doesn’t mean that you’re wrong. You made a lot of good points, and upon reflection, I think your idea is better. It’s a Broker AI – you made it – and it will need to integrate with Broker society first. I still think it needs some ethical grounding, but that’s something we can work on.”

“I am sorry too,” she added, her coloration lightening a little. “It is just that I sometimes feel as though you do not hear me when I speak.”

“You’re far from the first colleague to tell me that,” he chuckled.

Noting that she still seemed surly, he gave her a gentle nudge with his elbow to get her attention.

“Hey,” he began, gesturing to his visor. “You want me to make myself sneeze again? That got a laugh out of you last time.”

“No,” she replied, one corner of her mouth twitching as she suppressed a smile. Her coloration was not so easy to repress, taking on a lighter tone before his eyes as her mood improved.

“Well, how about I let you touch my hair again? Not here, obviously,” he added hastily. “I can’t hold my breath that long. Also, I think my head might implode.”

Selkie giggled, clicking her beak in a way that he was starting to find endearing, a few bands of bright color sweeping up her mantle.

“Do not presume that I can be wooed with mere jests,” she chuckled, but her complexion told a different story.

“I didn’t mean to upset you,” he added, his tone a little more serious. “This is a thing that I do – I get all caught up in my work, and when someone challenges me, I care more about being right than being kind. My experiences of having to constantly justify myself to my classmates and a hyper-competitive work environment have trained me to just go for the throat, but you don’t deserve that. You’re just as much an expert as I am, and I should consider your input more carefully.”

“I could also be ... more accommodating of your views,” she conceded. “I just needed some time to recharge my battery. Do not fret.”

“You want to head back now? I left Jeff in charge, so we probably shouldn’t leave him alone too long.”

“Stay with me a while,” she insisted.

“You got any cool shows on this thing?” he asked, gesturing to the ceiling as he shifted his weight to get comfortable on the netting.

“Nothing like your detective show,” she replied with a smile, reaching over to activate a touch panel. She cycled through a few holographic menus, then settled on a view of a spiraling galaxy that rotated in a field of stars above their heads.

“Feeling relaxed?” David asked.

“I have actually been thinking about our visit to the museum earlier this phase,” she replied as she watched the spiral arms slowly turn. “You seemed so surprised by what you learned.”

“I was,” he replied, following her gaze as he lay back against the uneven cave wall. “On the other hand, it explained a lot. There were a lot of puzzle pieces missing, and they’ve mostly fallen into place now.”

“What is it like for your species?” she asked, keeping her eyes fixed on the ceiling.

“Well, mammals don’t brood,” he replied. “Our gestation period is only nine months, and while live birth is rather painful for the mother, pregnancy is more of an inconvenience. We’re able to have a child every year or two if we really want to.”

“I meant more for courtship,” she replied, her skin becoming blotchy. “I read about the... biological aspects during my research.”

“Oh,” he muttered, his cheeks warming behind his visor. “I don’t know if there’s a cut-and-dry answer to that question. It works a little differently depending on culture and individual values. Human civilization is pretty diverse in that respect. The only real constant is that there’s no constant.”

“Then, just tell me how it works for your culture,” she replied in a nonchalant tone.

“I guess people date for a while,” he replied with a shrug. “They spend time together to see if they get along, they learn more about each other, and they figure out if they’re compatible. When they decide to start taking things more seriously, they move in together.”

“Move in?” Selkie asked, tilting her head.

“Cohabitation,” he explained. “If they’ve lived together for a while, I suppose the next step is usually marriage. Some people even have contracts and prenups like yours,” he added.

“It is not so different, then,” Selkie mused. “I was embarrassed when you compared our situation to a marriage, but in many ways, it is more like your concept of cohabitation.”

“Yeah, living with a roommate can certainly be an adjustment,” he chuckled.

“Have you ever lived with someone before, like that?” Selkie pressed.

“I haven’t had many long-term relationships on account of me being a stubborn asshole who argues incessantly with people,” he replied. “I had roommates in college, but that’s just sharing a living space because there isn’t room or you can’t afford the rent. Never been married, obviously. I was always so much younger than my peers.”

“You are the first person I have cohabited with,” Selkie added. “The demanding nature of my work has afforded me little time to pursue courtship, regardless of any other considerations.”

“I don’t think I should really be setting the example of what to expect.”

“You speak as though you have made a poor impression,” she said, tearing her eyes away from the hologram projection to glance at him. “I am starting to grow accustomed to your presence. I no longer find it offensive.”

“Well, that’s the nicest thing anyone has ever said to me,” he joked. “Inoffensiveness is about as much as a guy can hope for, considering the circumstances.”

“Once I stopped viewing you as a Broker, I actually grew to quite enjoy your company,” she added with another flush of flustered mottling. “I find your curiosity endearing, even though you sometimes embarrass me.”

“I’m more of a pet now, right?” he asked warily. “Like Flower?”

“Not like Flower,” she replied. “At first, your lack of social graces and your alien sensibilities were an annoyance, but Broker relations can be so demanding.” She turned to the ceiling again, exhaling a sigh through her vents, the water washing over his shoulder. “There is ... a strange relief in the idea that you are not in competition with me. You have none of the expectations of a Broker – none of the hangups, as you would put it. Yes, you can be crass and sometimes stubborn, but you are also ... safe.”

“Safe?” he repeated, not sure what else to say.

He felt her edge a little closer, a couple of her wandering tentacles finding the leg of his suit. Once again, she lay her mantle on his shoulder, David stiffening at her touch. She took his hand in one of her leaf-shaped counterparts, David feeling her suckers grip the lining of his suit as she turned it palm-up, laying a second appendage atop it. His heart quickened as she probed experimentally, testing his fingers, her cool skin palpable even through the glove that separated them.

“Would you say that you know me well?” she mumbled, her voice lower and breathier than he would have assumed the translator could reproduce. “Have you learned much during our cohabitation?”

He suddenly became aware of the sound of his own breathing inside his helmet and the blood that pulsed in his ears. She watched him with those glittering, horizontal pupils, her expectant eyes ringed by lash-like papillae that batted when she blinked. In the same way that her tentacles were probing his suit, Selkie was probing him for a response, still wary of being too forward – not yet sure of herself. As socially inept as he was, he didn’t need to be beaten over the head to realize when someone was flirting with him.

“I’ve had relationships with fellow humans that were infinitely more complicated and confusing than this,” he replied. “What does it say about me that I feel closer to you than someone of my own species?”

“I think you know me more intimately than anyone,” she whispered, inching ever closer. “I have shown you things that I never thought I would show another person...”

“That’s because you never let anyone get close before I was foisted upon you,” David chuckled nervously, feeling one of those questing tentacles slide across his thigh. Was it involuntary, or was she doing it on purpose? He had no way of knowing, and she had plausible deniability either way.

“I could have kept you in that habitat like a fish in a tank if I had wanted to,” she added. David suddenly found himself mesmerized by the way that her lips moved, so full and pillowy, matching the alien sounds that she made rather than the feminine voice that came through his helmet. “Maybe I was curious, maybe it was opportunistic, but I let you in.”

“We’re very different,” David stammered as she began to slither her lower body into his lap. He was surprised by how heavy she felt in the water, her cool skin pressing up against the lining of his suit, the netting sagging beneath them. Just as a human woman might have swung a leg over him, Selkie’s mass of writhing tentacles was slowly winding around his lower extremities.

“That is not how I see it,” she continued, bringing her face level with his as she straddled him. Her glittering eyes and her soft lips were only an inch from his visor now, two of her leaf-shaped hands coming to rest on his chest. “Physically, we have little in common, but we are both solitary beings who have sacrificed much for our work. You are bitter about being excluded from the social activities of your peers, while I resent that I have never been afforded the time to court – to experience ... companionship. Perhaps we can help one another. An equitable exchange, if you will.”