When I first became a doctor, I cared for an older man whom I’ll call Ted. He was so sick with pneumonia that he was struggling to breathe. His primary-care physician had prescribed one antibiotic after another, but his symptoms had only worsened; by the time I saw him in the hospital, he had a high fever and was coughing up blood. His lungs seemed to be infected with methicillin-resistant Staphylococcus aureus (MRSA), a bacterium so hardy that few drugs can kill it. I placed an oxygen tube in his nostrils, and one of my colleagues inserted an I.V. into his arm. We decided to give him vancomycin, a last line of defense against otherwise untreatable infections.

Ted recovered with astonishing speed. When I stopped by the next morning, he smiled and removed the oxygen tube, letting it dangle near his neck like a pendant. Then he pointed to the I.V. pole near his bed, where a clear liquid was dripping from a bag and into his veins.

“Where did that stuff come from?” Ted asked.

“The pharmacy,” I said.

“No, I mean, where did it come from?”

At the time, I could barely pronounce the names of medications, let alone hold forth on their provenance. “I’ll have to get back to you,” I told Ted. He was discharged before I could. But, in the years that followed, I often thought about his question. Every day, I administer medicines whose origins are a mystery to me. I occasionally meet a patient for whom I have no effective treatment to offer, and Ted’s inquiry starts to seem existential. Where do drugs come from, and how can we get more of them?

On a wet morning in May, I walked to the stately gates of the Rockefeller University, on the Upper East Side, and met Sean Brady, a bearded and bespectacled chemical biologist who discovers medicines for a living. Wearing jeans, a button-down shirt, and chunky sneakers, Brady looked like a nerdy hiker. He had promised to take me prospecting for antibiotics. “Ready?” he asked, his glasses spattered with rain.

Brady oversees a program called Drugs from Dirt, which sifts through soil samples from around the world in search of antibiotics. On our way into his lab, we passed a bench piled with bags of dirt—one from the Chihuahuan Desert, another from the Sonoran Desert. “My parents sent me those during the pandemic,” Brady told me. He has been given soil from Saudi Arabia and the Serengeti; his collaborators have gathered samples in Mexican sinkholes and Australian grasslands. Inside the lab, we found shovels and black buckets amid flasks and mass-spectrometry machines. I took off my blazer and picked up a bucket, and Brady grabbed a shovel. We set off into the drizzle.

Many of the world’s leading drugs originated in the natural world. Ancient Egyptians soothed their wounds with aloe-vera gel; morphine and codeine came from the opium poppy; Ozempic was inspired by a peptide in lizard venom. Dirt is one of the richest sources of medicine, because its microbes have been waging a war with one another for millions of years. Vancomycin—essentially a biochemical weapon that one bacterium uses to kill others—was discovered in soil samples from India and Indonesia in 1953. Around the same time, researchers reported that a bacterium in “heavily manured” New Jersey soil produced streptomycin, an antibiotic that became the first effective treatment for tuberculosis. (The bacterium is now New Jersey’s official state microbe.)

Different microbes thrive at different latitudes, and Brady once dreamed of collecting dirt from Alaska to Argentina. Lately, however, he has been content to stay on campus. “There’s an endless number of local bacterial products we haven’t studied yet,” he told me. “There’s plenty of good stuff right beneath our feet.” He stomped the ground playfully.

We walked down a cobblestone path until we came to an open lawn. Pigeons congregated near a fountain; a duck waddled by. Brady cleared some leaves and plunged his shovel into the earth. After dropping a few scoops into the bucket, he handed me the shovel. “Your turn,” he said.

I looked around, wondering, absurdly, where I might hide if I were the next cure for cancer. When the bucket was half full, I wiped some raindrops from my face. “That should be more than enough,” Brady said. “One gram of soil has up to ten thousand different types of bacteria. You don’t need a lot. You just need to know what you’re looking for.”

Back in the lab, I scooped some soil into a tube and added a detergent that destroys bacteria’s cell walls and membranes, causing their DNA to spill out. Then Jan Burian, a researcher in Brady’s lab, showed me how to load the DNA onto a glass plate, which slid into a sequencer like a credit card into a payment terminal. “If we were sitting here ten or fifteen years ago, sequencing all this DNA might take weeks,” Burian told me. “Now it pretty much happens in real time.” He pulled up the machine’s output on a computer: an endless string of “G”s, “T”s, “C”s, and “A”s.

Bacteria produce numerous molecules that could become medicines, but most of them aren’t easily identified or synthesized with the technology that exists today. A small percentage of them, however, can be constructed by following instructions in the bacteria’s DNA. Burian helped me search the sequence for genes that looked familiar enough to be understandable but unfamiliar enough to produce novel compounds. We settled on a string of DNA that coded for seven linked amino acids, the same number found in vancomycin. Then Burian introduced me to Robert Boer, a synthetic chemist who would help me conjure our drug candidate.

What came next was intricately choreographed. You can’t just throw amino acids into a soup and hope that they bubble into a medicine, Boer said. Building them into a potential drug—in this case, a peptide—is like assembling IKEA furniture. There’s a specific sequence in which the parts need to be connected, with specific nuts and bolts. Before the amino acids could be fastened together, they had to be dissolved into a solution that would expose their chemical hardware.

“Nobody will speak to us in French, only high-pitched whistles and clicks—is it that obvious we’re tourists?”

Cartoon by Ellie Black

We mixed one amino-acid solution, which contained tyrosine, with another, which contained serine. After waiting half an hour for the first linkage to form, we repeated the process one amino acid at a time, until all seven were connected in the correct order. Finally, the liquid was evaporated until just a few milligrams of snowy-white powder remained. I dubbed our drug candidate “NY1,” for The New Yorker, and helped Boer pipette it onto a well plate. It would now spend a night with MRSA. I imagined the start of a microscopic cage match between our molecule and the bacteria.

When I returned the next day, the liquid in the plate was thick and cloudy. NY1 had been powerless against MRSA—the plate looked so grimy that I wondered if I’d accidentally created a bacterial superfood. Brady gave me a pep talk. “If you found one on your first try, we’d probably hire you,” he joked. “You’d be our lucky charm. And, when it comes to finding a drug, a little luck never hurts.” His team, he said, sometimes tests more than a hundred molecules per month. A tiny fraction might show antibiotic activity, and a tiny fraction of those perform well enough—and are nontoxic enough—to advance to animal testing and clinical trials. The reasons that one molecule succeeds while another fails are hard to predict. Near my failed drug candidate was another contender, which had come from a neighboring patch of dirt; its well was as clear as water. Whatever was in there, it had killed MRSA—and we had no idea how.

Tori Kinamon was a freshman gymnast at Brown University when her leg started to ache. The team’s athletic trainers suspected a muscle strain, but the pain sharpened even after ice and a massage. A few days later, Kinamon awoke feverish and sweating; her leg was swelling and felt like it was on fire. Finally, an MRI revealed an abscess along a muscle. She was started on antibiotics and rushed to the operating room, where surgeons scraped out pus and necrotic tissue—the consequence of a raging MRSA infection. The surgeons operated eight times in two weeks, filleting open the back of her leg with a two-foot incision from her glute to her calf. “I was so happy the days I would have to go under for surgery, because that meant that I didn’t have to deal with what was going on around me,” she told me. The surgeons narrowly managed to save her leg, but the infection did not clear for weeks. Vancomycin harmed her kidneys, so she had to be switched to daptomycin, another antibiotic that comes from a bacterium in soil. “I am a direct beneficiary of the fact that we had an extra antibiotic in the stockpile when vancomycin was too toxic for my body to handle,” Kinamon, who is now a surgery resident in Texas, said. “Some patients are left without options.”

The number of chemicals that theoretically could prove useful as drugs has been estimated at ten to the sixtieth power—a quantity greater than the number of atoms in the solar system. Some of these potential medicines can be found in nature. Others have already been discovered but we haven’t yet found their uses. Still others have never been imagined. In “The Drug Hunters: The Improbable Quest to Discover New Medicines,” Donald R. Kirsch and Ogi Ogas compare the pursuit of novel medicines to the search for meaning in “The Library of Babel,” a short story by Jorge Luis Borges in which the author envisions the universe as an infinite library. Each book contains random letters and punctuation marks, so most of the texts are nonsensical. But, because the library is unfathomably large, it also contains every conceivable story. “Every possible drug is contained somewhere in the vast theoretical library of chemical compounds,” Kirsch and Ogas write. Drug discovery is an effort to catalogue some small part of it.

In the mid-two-thousands, Stuart Schreiber, a chemist at the Broad Institute of M.I.T. and Harvard, set out to assemble an expansive library of chemicals that might have medical applications. Schreiber was frustrated that drug discovery was so tedious and unsystematic; his colleagues could study only the chemicals that they could make from scratch or buy from a commercial vender. He started by stocking up on simple chemicals from pharmaceutical companies and research organizations—“cheap stuff based on twentieth-century chemistry,” he told me. Next, he collected a large number of “natural products,” like the ones microbes make in dirt, that had a higher likelihood of proving useful as a drug. “Any chemist could take one look at the structures and tell you which molecule belonged to which group,” he said—synthetic or natural. “It was like telling a cat from a dog.” Then Schreiber gathered a group of chemists and had them invent molecules with the features of natural products—a process that they dubbed “diversity-oriented synthesis.” Some compounds dissolved in water; others clumped. Some reacted promiscuously with one another; others kept to themselves. “The new synthetic molecules were much more sophisticated,” Schreiber told me. “They were structurally quite similar to what’s found in nature.”

In the end, the chemists concocted a hundred thousand new molecules. The library, which is housed at the Center for the Development of Therapeutics (CDoT), grew to encompass nearly a million chemicals, including existing medications, drug candidates, and strange compounds with no known use. Many research labs and universities have compiled similar libraries. Since the late twentieth century, labs have increasingly used robots and automation to comb through the libraries, reasoning that, as scientists screened more chemicals for medical applications, the number of newly found drugs would inevitably increase. But this technique, which is known as high-throughput screening, has turned out to be less revolutionary than many once hoped. If a chemical library isn’t large and diverse enough, or if the selection process is largely random, the method tends to have low hit rates and produce many false positives. So far, no useful antibiotics have been found this way; in many screens, only about one per cent of molecules show activity against a bacterium. Many are similar to existing antibiotics or toxic to humans.

In 2012, a group of drug researchers warned that the number of new drugs approved in the U.S., per dollar invested in drug discovery, was falling by half every nine years, an eightyfold reduction in efficiency since 1950. They called their observation Eroom’s Law—an inversion of Moore’s Law, which says that the number of transistors per computer chip doubles every two years or so, driving down the cost of computers. Many drug developers, the researchers wrote, had fallen victim to “basic research–brute force bias.” They were powering through as many molecules as possible, more or less at random, on the off chance that something might work. The universe of potential drugs desperately needed a filing system.

In 2017, Jon Stokes, a Canadian postdoc in microbiology, joined the lab of James J. Collins, an M.I.T. professor of biological engineering. Stokes, who has long hair and glasses, giving him the look of a scientific David Foster Wallace, decorated his office with a poster of a skull and crossbones. “I think of myself as a professional poison discoverer,” he told me. “I want to kill things that kill patients.” (He added wryly, “I don’t support piracy.”) Stokes soon ordered two sets of chemicals from a commercial vender for twenty-five thousand dollars, with the aim of scouring them for antibiotic potential. One contained eight hundred natural products; the other was made up of around seventeen hundred drugs that the F.D.A. had already approved but that might have additional uses. They arrived in a Styrofoam box the size of a microwave, packed in dry ice. Stokes was preparing to screen each compound manually—pitting them against microbes on glass plates, much as I had done—when he sat in on a meeting about biological applications of artificial intelligence. “I had no idea what was going on,” Stokes recalled. “I wasn’t an A.I. guy.”