As the film opens, Joe Buck (Jon Voight), a young Texan working as a dishwasher, dresses in new cowboy clothing, packs a suitcase, and quits his job. He heads to New York City hoping to succeed as a male prostitute for women. Initially unsuccessful, he succeeds in bedding a well-to-do middle-aged New Yorker (Sylvia Miles), but Joe ends up giving her money.
Joe then meets Enrico Salvatore "Ratso" Rizzo (Dustin Hoffman), a street con man with a limp who takes $20 from Joe by offering to introduce him to a known pimp, who turns out to be a Bible thumper (John McGiver). Joe flees the encounter in pursuit of Ratso. Joe spends his days wandering the city and sitting in his hotel room. Soon broke, he is locked out of his hotel room and most of his belongings are impounded.
Joe tries to make money by agreeing to receive oral sex from a young man (Bob Balaban) in a movie theater. When Joe learns that he has no money, Joe threatens him and asks for his watch, but eventually lets him go. The following day, Joe spots Ratso and angrily shakes him down. Ratso offers to share the apartment in which he is squatting in a condemned building. Joe accepts reluctantly, and they begin a "business relationship" as hustlers. As they develop a bond, Ratso's health, which has never been good, grows steadily worse.
Joe's story is told through flashbacks throughout the film. His grandmother raises him after his mother abandons him, though his grandmother frequently neglects him as well. He also has a tragic relationship with Annie, a local girl. Ratso's back story comes through stories he tells Joe. His father was an illiterate Italian immigrant shoe-shiner, who worked in a subway station. He developed a bad back, and "coughed his lungs out from breathin' in that wax all day". Ratso learned shining from his father but won't stoop so low as to do so. He dreams of moving one day to Miami.
An unusual couple approach Joe and Ratso in a diner and hand Joe a flyer, inviting him to a party. They enter a Warhol-esque party scene (with Warhol superstars in cameos). Joe smokes a joint, thinking it's a normal cigarette and, after taking a pill someone offered, begins to hallucinate. He leaves the party with a socialite (Brenda Vaccaro), who agrees to pay $20 for spending the night with him, but Joe cannot perform. They play Scribbage together, and Joe shows his limited academic prowess. She teasingly suggests that Joe may be gay, and he is suddenly able to perform.
In the morning, the socialite sets up her friend as Joe's next customer, and it appears that his career is on its way. When Joe returns home, Ratso is bedridden and feverish. Ratso refuses medical help (because he has no money or health insurance) and begs Joe to put him on a bus to Florida. Desperate, Joe picks up a man in an amusement arcade (Barnard Hughes), and when things go wrong, robs the man when he tries to pay with a religious medallion instead of cash. With the cash he steals, Joe buys two bus tickets.
On the journey, Ratso's frail physical condition further deteriorates. At a rest stop, Joe buys new clothing for Ratso and himself, discarding his cowboy outfit. As they near Miami, Joe talks of getting a regular job, only to realize Ratso has died. The driver tells Joe there is nothing else to do, but continue on to Miami. The film closes with Joe, alone and afraid, seated with his arm around his dead friend.