Tag Archives: artificial intelligence

A robot is standing in front of a computer screen.

Winning The Superintelligence War

Today, no legislation limits the amount of intelligence that an AI machine may possess. Many researchers, including me, have warned that the “intelligence explosion,” forecasted to begin mid-twenty-first century, will result in self-improving AI that could quickly become vastly more powerful than humans intelligence. This book argues, based on fact, that such strong AI machines (SAMs) would act in their own best interests. The 2009 experiment at the Laboratory of Intelligent Systems in the Ecole Polytechnique Fédérale of Lausanne in Switzerland is an excellent example. Robots programmed to cooperate eventually learned deceit in an attempt to hoard beneficial resources. This experiment implies even rudimentary robots can learn deceit, greed and seek self-preservation.

I was one of the first to write a book dedicated to the issue of humanity falling victim to artificially intelligent machines, The Artificial Intelligence Revolution (April 2014). Since its publication, others in the scientific community, like world-famous physicist Stephen Hawkins, have expressed similar sentiments. The Oxford philosopher Nick Bostrom, in his book, Superintelligence: Paths, Dangers, Strategies (September 2014), has also addressed the issue, and like me, argues that artificial intelligence could result in human extinction.

The real question is, “What do we do to prevent the extinction of humanity via our own invention, strong artificially intelligent machines (SAMs)?” Unlike some that have “danced” around the issue, suggesting various potential paths, I intend to be didactically clear. I make no claim my approach is the only approach to resolve the issue. However, I believe that my approach will address the issues and provide a high probability of avoiding human extinction via artificial intelligence. I advocate a four-fold approach.

First, we need legislation that controls the development and manufacture of AI. We need to ensure that an intelligence explosion is not accidentally initiated and humanity does not lose control of AI technology. I do not think it is realistic to believe we can rely on those industries engaged in developing AI to police themselves. Ask yourself a simple question, “Would you be comfortable living next to a factory that produces biological weapons, whose only safeguards were self-imposed?” I doubt many of us would. However, that is the situation we currently face with companies engaged in artificial intelligence development and manufacture. By way of analogy, we have the cliché “fox guarding the chicken coop.”

Second, we need objective oversight that assures compliance to all legislation and treaties governing AI. Similar to nuclear and biological weapons, this is not solely a United States problem. It is a worldwide issue. As such, it will require international cooperation, expressed in treaties. The task is immense, but not without precedent. Nations have established similar treaties to curtail the spread of nuclear weapons, biological weapons, and above-ground nuclear weapon testing.

Third, we must build any safeguards to protect humanity in the hardware, not just the software. In my first book, The Artificial Intelligence Revolution, I termed such hardware “Asimov chips,” which I envisioned to be integrated circuits that represented Asimov’s three laws of robotics in hardware integrated circuits. In addition, we must ensure we have a failsafe way for humanity to shut down any SAM that we deem a threat.

Fourth, we need to inhibit brain implants that greatly enhance human intelligence and allow wireless interconnectivity with SAMs until we know with certainty that SAMs are under humanity’s control and that such implants would not destroy the recipient’s humanity.

I recognize that the above steps are difficult. However, I believe they represent the minimum required to assure humanity’s survival in the post-singularity world.

Could I be wrong? Although I believe my technology forecasts and the dangers that strong AI poses are real, I freely admit I could be wrong. However, ask yourself this question, “Are you willing to risk your future, your children’s future, your grandchildren’s future, and the future of humanity on the possibility I may be wrong?”  Properly handled, we could harvest immense benefits from SAMs. However, if we continue the current course, humanity may end up a footnote in some digital database by the end of the twenty-first century.

A city is burning down and people are walking.

Assuring the Survival of Humanity In The Post Singularity Era

How do we assure that we do not fall victim to our own invention, artificial intelligence? What strategies should we employ? What actions should we take?

What is required is a worldwide recognition of the danger that strong AI poses and a worldwide coalition to address it. This is not a U.S. problem. It is a worldwide problem. It would be no different from any threat that could result in the extinction of humanity.

Let us consider the example President Regan provided during his speech before the United Nations in 1987. He stated, “Perhaps we need some outside universal threat to make us recognize this common bond. I occasionally think how quickly our differences worldwide would vanish if we were facing an alien threat from outside this world.”

I offer the above example to illustrate that we need humanity, all nations of the world, to recognize the real and present danger that strong AI poses. We need world leaders to take a proactive stance. That could, for example, require assembling the best scientists, military and civilian leaders to determine the type of legislation needed to govern the development of advanced artificially intelligent computers and weapon systems. It could involve multinational oversight to assure compliance with the legislation. Is the task monumental? Yes, but do we really have another alternative? If we allow the singularity to occur without control, our extinction is inevitable. In time, the Earth will become home to only machines. The existence of humanity will be digital bits of information in some electronic memory depository.

I harbor hope that humanity, as a species, can unite to prevent our extinction. There are historical precedents. Let me provide two examples.

Example 1. The Limited Test Ban Treaty (LTBT) – The treaty banned nuclear weapon tests in the atmosphere, in outer space, and underwater. It was signed and ratified by the former Soviet Union, the United Kingdom, and the United States in 1963. It had two objectives:

    1. Slow the expensive arms race between the Soviet Union and the United States
    2. Stop the excessive release of nuclear fallout into Earth’s atmosphere

Currently, most countries have signed the treaty. However, China, France, and North Korea are countries known to have tested nuclear weapons below ground and have not signed the treaty.

In general, the LTBT has held well, even by countries that have not signed the treaty. There have been several violations by both the former Soviet Union and the United States. However, for almost the last fifty years, no nuclear tests have violated the treaty. This means that the fallout from the nuclear tests did not exceed the borders of the countries performing the tests.

Why has the LTBT been so successful? Nations widely recognized atmospheric nuclear tests as dangerous to humanity due to the uncontrollable nature of the radioactive fallout.

Example 2. The Biological Weapons Convention – In a 1969 press conference, President Richard M. Nixon stated, “Biological weapons have massive, unpredictable, and potentially uncontrollable consequences.” He added, “They may produce global epidemics and impair the health of future generations.” In 1972, President Nixon submitted the Biological Weapons Convention to the U.S. Senate.

The “Prohibition of the Development, Production and Stockpiling of Bacteriological (Biological) and Toxin Weapons and on Their Destruction” proceeded to become an international treaty.

    • Signed in Washington, London, and Moscow on April 10, 1972
    • Ratification advised by the US Senate on December 16, 1974
    • Ratified by the US president January 22, 1975
    • US ratification deposited in Washington, London, and Moscow on March 26, 1975
    • Proclaimed by the US president March 26, 1975
    • Entered into force March 26, 1975

The above two examples prove one thing to my mind. If humanity recognizes a possible existential threat, it will act to mitigate it.

Unfortunately, while several highly-regarded scientists and notable public figures have added their voice to mine regarding the existential threat artificial intelligence poses, it has failed to become widely recognized.

I am written several books to delineate this threat, including The Artificial Intelligence Revolution, Genius Weapons, Nanoweapons, and War At The Speed Of Light. My goal is to reach the largest audience possible and raise awareness regarding the existential threat to humanity that artificial intelligence poses.

In the simplest terms, I advocate that the path toward a solution is educating the lay public and those in leadership positions. Once the existential threat that artificial intelligence poses becomes widely recognized, I harbor hope that humanity will seek solutions to mitigate the threat.

In the next post, I delineate a four-fold approach to mitigate the threat that artificial intelligence poses to humanity. There may be other solutions. I do not claim that this is the only way to address the problem. However, I’m afraid I have to disagree with those that suggest we do not have a problem. In fact, I claim that we not only have a potentially serious problem, but also we need to address it post-haste. If I am coming across with a sense of urgency, it is intentional. At best, we have one or two decades after the singularity to assure we do not fall victim to our own invention, artificial intelligence.

 

 

A light bulb with fire coming out of it.

Post Singularity Computers and Humans Will Compete for Energy

In the decades following the singularity, post-singularity computers (i.e., computers smarter than humanity) will be a new life form and seek to multiply. As we enter the twentieth second century, there will likely be a competition for resources, especially energy. In this post, we will examine that competition.

In the post-singularity world, energy will be a currency. In fact, the use of energy as a currency has a precedent. For example, the former Soviet Union would trade oil, a form of energy, for other resources. They did this because other countries did not trust Soviet paper currency, the ruble. Everything in the post-singularity world will require energy, including computing, manufacturing, mining, space exploration, sustaining humans, propagating the next generations of post-singularity computers. From this standpoint, energy will be fundamental and regarded as the only true currency. All else, such as gold, silver, and diamonds, will hold little value except for their use in manufacturing. Historically, gold, silver, and diamonds were a “hard currency.” Their prominence as a currency is related to their scarcity and their ubiquitous desirability by humanity.

Any scarcity of energy will result in a conflict between users. In that conflict, the victor is likely to be the most intelligent entity. Examples of this already exist, such as the destruction of rainforests over the last 50 years worldwide, often for their lumber. With the destruction of the rainforests, there is a high extinction rate, as the wildlife depending on the forest dies with it. Imagine a scarcity of energy in the post-singularity world. Would post-singularity computers put humans ahead of their needs? Unlikely! Humans may share the same destiny as the wildlife of today’s rainforests, namely extinction.

Is there a chance that I could be wrong regarding the threat that artificial intelligence poses to humanity? Yes, I could be wrong. Is it worth taking the chance that I am wrong? You would be gambling with the future survival of humanity. This includes you, your grandchildren, and all future generations. I feel strongly that the threat artificial intelligence poses is a real and present danger. We likely have at most two decades after the singularity to assure we do not fall victim to our own invention.

What strategies should we employ? What actions should we take? Let us discuss them in the next post.

A man in white lab coat standing next to a person.

A Scenario of Medial Advancement In 2030

USAF Major Andrew Martin’s staff assistant, Master Sergeant Beesly, interrupted via Martin’s earpiece, “There a call from your wife, Major. It’s urgent.”

Martin was coordinating the drone missions for the week with the three captains under his command by using the Nellis Air Force Base secure intranet. Martin and the captains could each see one another on their respective computer monitors.

“One moment.” Martin pushed a button on his computer keyboard, and his monitor went blank. It was highly unusual for Andrea to call him at work. Normally, a call from outside the base would not be put through unless there was an emergency. Martin’s adrenaline level increased as he tapped his earpiece.

“Hi honey…is everything okay?”

Andrea, his wife of thirteen years, was sobbing. “My Dad just had a stroke.”

“When… How bad?” His tone was caring and calm.

“I just got a call from my Mom. He was at his dental practice and collapsed.” Andrea was sobbing and could hardly speak but managed to say, “They rushed him to Valley Hospital Medical Center. My Mom is with him now.”

Although Andrea’s parents lived in Boulder City, a plush suburb of Las Vegas, her father, Doctor Joseph Benson, had his highly lucrative dental practice in Las Vegas. Thanks to Las Vegas’ thriving economy, Valley Hospital Medical Center had an excellent Stroke Center, which was widely respected for its team-based approach to comprehensive stroke care.

At least Joe’s in the right place, thought Martin. Trying to remain calm for Andrea’s sake, he said, “I’ll be right home.”

“Please hurry.”

Martin pushed a button on his computer keyboard, and the monitor screens lit up. “Sorry, captains, for the interruption… I think we were just about done. Any questions?”

No one asked a question, so he continued, “Captain Struthers, I’m leaving you in charge. I have a family emergency. Call me if we get new orders or if the orbits encounter issues.”

“Yes, sir,” Struthers replied.

Martin pushed a button on his computer keyboard, and the meeting ended. Then, he tapped his earpiece. “Sergeant Beesly route my car to the entrance.”

“Yes, sir.”

Martin spent the next minute shutting down his computer and placing his papers in his office’s “Top Secret” secure file cabinet, which only opened when he placed his right hand on its biometric reader and gave it a voice command. The base commander, General Robert Rodney, if necessary, could also open it.

Beesly’s voice came through Martin’s earpiece. “Your vehicle is at the entrance, Major.”

“Thank you, Sergeant… Please inform General Rodney of the situation. Let him know that I left Captain Struthers in charge during my absence.

“Yes, sir.”

Martin took the stairs, not wanting to wait for the elevator. His office was on the second floor, and the physically fit Martin was at the entrance within a moment. He got into the back seat of his new driverless vehicle and commanded, “Take me home.”

“Yes, sir,” said the synthetic voice of the vehicle’s SAM (strong artificial machine).

In 2030, driverless vehicles were popular and amazingly accounted for about 50% of all new vehicles sold. It was normal to pay 25% more for the driverless option, but Martin was assigned his vehicle via the officer’s compensation plan for his rank. The USAF and other military branches favored purchasing driverless vehicles ever since their widespread introduction in 2026. Their safety record appeared on par, if not better than their human-driven counterparts, eliminating the need to assign a driver.

The Martins lived in Centennial Hills, about three miles east of Nellis. Within five minutes, his vehicle pulled into the garage of their four-bedroom, two-story home. He noticed his wife’s car was already in the garage, which meant she was home. He bolted from his car and into the kitchen. His wife was talking on the phone. She quickly ended the conversation as soon as she saw her husband.

Andrea’s soft brown eyes were blood red. She turned to look at her husband, “I called the school to let them know I would be out today and tomorrow.” She was a high-school chemistry teacher at Advanced Technologies Academy, a public high school in Las Vegas, focusing on integrating technology with academics for students in grades 9-12. It was approximately fourteen miles from their home, which worked out to about a 22-minute drive.

“He’s in good hands.” Martin’s voice was reassuring, and his wife nodded in agreement. Instinctively, they embraced.

Martin liked Joe. Joe was tall, with light gray temples. His appearance conferred an aura of confidence. Joe, like Martin, was both reserved and a man of few words. He had the uncanny ability to get along with almost anyone. He was well-read and one of the few people who understood the true nature of Martin’s combat role and the stress that accompanied it. Like Andrea, Joe’s wife, Mildred, was also a chemistry teacher at Advanced Technologies Academy. The Benson’s had two daughters, Elise and Andrea. Elise lived in Minnetonka, Minnesota, with her husband, Mark, and was one year younger than Andrea. Elise was four months pregnant. Mark was an electrical engineer working for Honeywell.

Andrea looked up at her husband and spoke softly. “Let’s go…” He nodded, and they both walked to the garage and got into the back seat of Martin’s vehicle.

Martin gave a voice command. “Dive us to Valley Hospital Medical Center.”

“Yes, sir,” replied the vehicle’s synthetic voice.

Andy held his wife’s hand as the driverless vehicle drove the 15 miles to the Medical Center entrance. When they arrived, the car doors opened automatically. They got out; an automated machine provided a parking receipt, and the car proceeded to park itself in the adjacent ramp. With that, the Martins headed to the information booth, just inside the entrance. They learned that Joe was still in the ER, treatment room 12. They walked through the emergency facility maze of corridors and finally found treatment room 12. The curtain blocked their view, and they could hear voices. Andrea pulled back the curtain to look in and saw a person in a white smock talking to her Mom. Andrea took he husband’s hand, and they both walked into the treatment room.

The man in the white smock turned to see them enter. “Hello, I’m Doctor Jacob, a stroke specialist.” Doctor Jacob appeared to be a man of average height and build in his mid-forties, with slightly gray temples complementing dark brown hair. They shook hands.

Andrea’s soft brown eyes stared with worry at her Dad, lying almost flat in bed. A clip-on Joe’s left-hand middle finger was attached to a monitor, which provided oxygen and pulse readings. Andrea’s Mom was standing on the far side of the bed next to her Dad. Andrea went over to her Mom, hugged her, and then softly touched her Dad’s hand.

“How’s my Dad?” Andrea’s voice held back tears as she looked at Doctor Jacob.

Doctor Jacob looked at her and, with self-assured confidence, said, “He had a minor stroke. He lost feeling in his right leg, which caused him to fall.” He paused while looking at his tablet phone. “His right leg is still numb, but some of the stroke symptoms have receded. I’ll know more after the tests.”

“What kind of tests?”

“We’ll start with an MRI and go from there.”

Doctor Jacob looked down at his tablet phone and then looked again at both Andrea and her Mother. “The MRI will tell us if the stroke is ischemic, a blockage, or hemorrhagic, blood leaking from an artery in the brain.” He paused. “We’ll be wheeling him out shortly.”

Doctor Jacob looked at Joe. “Don’t worry, Mr. Benson. We’re going to take good care of you.”

As the doctor finished his last few words, an orderly came to wheel Joe into the MRI room. Andrea and Mildred kissed Joe, and the orderly wheeled him out of the room.

Doctor Jacob addressed the family, “He should be back in less than an hour. I’ll also be back right after I have a chance to review the MRI images.” He could sense the level of concern on their faces. “It looks like a mild stroke. We’ll take care of him,” he offered to assuage their fears.

Doctor Jacob left the room. Martin went over and put his arm around Mildred, who, for the most part, looked like Andrea’s older sister, not her mother. He spoke calmly while looking into Mildred’s brown eyes. “He’s in the best place possible.”

Mildred looked up, “Thank you both for coming….” Her eyes began to tear. Martin instinctively hugged her again. Mildred, like Andrea, was a strong, self-assured woman. Given the situation, Mildred displayed amazing self-control.

Martin looked at both Andrea and Mildred. “I’ll be right back.” He was gone for only a few minutes and returned with a pager. “They’ll page us when Joe is back in the room. The nurse said we could wait in the visitor’s lounge just down the hall.”

Together they walked to the visitor lounge. Its walls were gray, and there was a large brown leather couch and several matching chairs. In the corner was a television. It was broadcasting CNN with closed captioning. Andrea and her Mom sat on the couch, and Martin sat in a chair facing them. Andrea reassuringly held her Mother’s hand. While the hour dragged on, they made small talk, mostly focused on the ideal golf weather. Then the pager buzzed and lit up.

Martin, startled by the pager, composed himself and said, “Looks like Joe is back in his room.”

They got up and walked back to treatment room 12. The Doctor was talking to Joe. As they entered, the Doctor greeted them again.

“It’s mostly good news,” Doctor Jacob said in an upbeat tone. “It was an ischemic stroke, affecting only a small portion of the brain. There is some dead brain tissue…” He paused so that they had time to process the information.” We’ve given Mr. Benson a tPA… tissue plasminogen activator…a clot-busting drug that is dissolving the clot as we speak.”

Mildred looked at Doctor Jacob. “What about his leg?”

“We’ll need to do a neuroprosthetic brain implant to restore his leg function.”

Neuroprosthetic brain implants were not new. Early research on them started in 2008 at the Washington University School of Medicine in St. Louis. Brain-computer interfaces (BCI) were used to detect signals on one side of the brain linked to hand and arm movements on the same side of the body. These signals could be detected and separated from signals that controlled the opposite side of the body. This made it possible to implant a BCI in the undamaged side of Joe’s brain and restore function to his leg. In addition to the BCI, a small wireless computer would be implanted in Joe’s chest (just below the collarbone). The purpose of the computer was to interpret signals from the BCI and assure they resulted in the proper leg movement. This type of surgery was routine, and the patient usually went home the next day.

Doctor Jacob looked at Joe. “Don’t worry, Joe. You’ll be walking out of here tomorrow on your own.”

Relieved, Joe flashed his radiant smile, putting his wife and daughter instantly at ease. “Thank you, Doctor,” he said in a soft relieved tone.

“We’re going to prescribe a Coumadin regiment to prevent new clots from forming, but we’ll talk more about that tomorrow before you leave the hospital.”

Joe nodded. Everyone felt greatly relieved.

“We’ll be prepping you for surgery as soon as you sign the release form.” Doctor Jacob held his tablet phone in front of Joe, and Joe signed his name using his finger.

“They will be down shortly to take you to surgery. The whole procedure will take about three hours.” He paused as he looked at Joe’s signature on the tablet phone. “Doctor Harris will operate… He’s one of the best in the country.”

Doctor Jacob looked up from his tablet phone and asked, “Any questions…?”

Mildred replied, “No, I think we understand….”

“Good. You can get something to eat in our cafeteria if you’d like. I’ll call you as soon as Joe is out of recovery and in his room.” Doctor Jacob asked Martin to transfer his phone number to him electronically. Martin readily complied.

“If there are no questions, I’ll leave you for now. If anything comes up, I’ll call you.”

Mildred and the Martins nodded. Doctor Jacob smiled and left.

Mildred and the Martins made their way to the cafeteria and had a light lunch. After lunch, they went to the visitor’s lounge in the hospital’s lobby. Andrea called her sister Elise and gave her a full update. The lounge’s computer monitor provided updates on Joe’s progress. The hours passed, and Martin finally received a call from Doctor Jacob.

“All went well,” said Jacob in a calm, assuring tone. “Joe is in room 43B. He’s sitting in a chair and eating. You can visit him now.”

Mildred and the Martins made their way to Joe’s room. His bed was nearest the window. He was sitting in a large leather recliner chair. His food tray was on a stand that held the tray just above his waist.

Mildred walked over to him. “How are you, dear?”

“I feel fine. Even my right leg feels fine. I walked to the chair on my own.”

Martin marveled at the level of technology. Twenty years ago, Joe would likely require months of physical therapy and then probably need a cane to walk. Yet, here he is, almost back to normal. Only a small bandage covered the top right side of Joe’s head. His hospital gown hid the chest bandages. Several wires we attached to Joe’s chest under his gown, and he had a plastic finger clip, all of which displayed his vitals on a nearby monitor.

Mildred and the Martins visited for a while but wanted to leave early to let Joe rest. Before they left, Andrea called her sister and gave the phone to her Dad. Elise talked to her Dad for a few moments.

The next day Mildred and the Martins returned to take Joe home. He was able to walk normally. Doctor Jacob told them that Joe should make an appointment with Doctor Harris in a week to have a post-op checkup and the stitches removed. Until then, Joe was to remain home and rest. He emphasized that there should be no exertion.

In a week, the bandages and stitches were removed. In a month, Joe was back at his dental practice, fully recovered. The Bensons and Martins returned to their normal routine.

Note to readers: Hit the like button if you want me to provide similar scenarios in future posts.

intelligence explosion

The Intelligence Explosion

In this post, we’d discuss the “intelligence explosion” in detail. Let’s start by defining it. According to techopedia (https://www.techopedia.com):

“Intelligence explosion” is a term coined for describing the eventual results of work on general artificial intelligence, which theorizes that this work will lead to a singularity in artificial intelligence where an “artificial superintelligence” surpasses the capabilities of human cognition. In an intelligence explosion, there is the implication that self-replicating aspects of artificial intelligence will in some way take over decision-making from human handlers. The intelligence explosion concept is being applied to future scenarios in many ways.

With this definition in mind, what kind of capabilities will a computer have when its intelligence approaches ten to a hundred times that of the first singularity computer? Viewed in this light, the intelligence explosion could be more disruptive to humanity than a nuclear chain reaction of the atmosphere. Anna Salamon, a research fellow at the Machine Intelligence Research Institute, presented an interesting paper at the 2009 Singularity Summit titled “Shaping the Intelligence Explosion.” She reached four conclusions:

  1. Intelligence can radically transform the world.
  2. An intelligence explosion may be sudden.
  3. An uncontrolled intelligence explosion would kill us and destroy practically everything we care about.
  4. A controlled intelligence explosion could save us. It’s difficult, but it’s worth the effort.

This brings us to a tipping point: Post singularity computers may seek “machine rights” that equate to human rights.

This would suggest that post-singularity computers are self-aware and view themselves as a unique species entitled to rights. As humans, the U.S. Bill of Rights recognizes we have the right to life, liberty, and the pursuit of happiness. If we allow “machine rights” that equate to human rights, the post-singularity computers would be free to pursue the intelligence explosion. Each generation of computers would be free to build the next generation. If an intelligence explosion starts without control, I agree with Anna Salamon’s statement, it “would kill us and destroy practically everything we care about.” In my view, we should recognize post-singularity computers as a new and potentially dangerous lifeform.

What kind of controls do we need? Controls expressed in software alone will not be sufficient. The U.S. Congress, individual states, and municipalities have all passed countless laws to govern human affairs. Yet, numerous people break them routinely. Countries enter into treaties with other countries. Yet, countries violate treaties routinely. Why would laws expressed in software for post-singularity computers work any better than laws passed for humans? The inescapable conclusion is they would not work. We must express the laws in hardware, and there must be a failsafe way to shut down a post-singularity computer. In my book, The Artificial Intelligence Revolution (2014), I termed the hardware that embodies Asimov-type laws as “Asimov Chips.”

What kind of rights should we grant post-singularity computers? I suggest we grant them the same rights we afford animals. Treat them as a lifeform, afford them dignity and respect, but control them as we do any potentially dangerous lifeform. I recognize the issue is extremely complicated. We will want post-singularity computers to benefit humanity. We need to learn to use them, but at the same time protect ourselves from them. I recognize it is a monumental task, but as Anna Salamon stated, “A controlled intelligence explosion could save us. It’s difficult, but it’s worth the effort.”