The Bahamas, 1940
American artist Tom Hudson (George C. Scott) lives on the remote island of Bimini, in the western region of the Bahamas. At the dock, he finds two friends aboard his fishing boat, the Tortuga. As islander, named Joseph (Julius Harris) washes the deck, Eddy (David Hemmings), a British alcoholic, nurses his wounds from a fight the night before and brandishes a rifle that he won in a poker game. That evening, Tom celebrates the Queen Mother's birthday with a prostitute named Lil (Susan Tyrell), drinking and firing flares into the harbor.
The next day, Tom's three estranged sons, 19-year-old Tom, Jr. (Hart Bochner) from his first marriage, 14-year-old Dave (Michael-James Wixted) and 10-year-old Andy (Brad Savage) from his second marriage, arrive on the island, but he does not meet them at their seaplane; Eddy and Joseph drive the boys to their father's home.
Housebound during a rainstorm, Andy admires one of Tom's drawings, a portrait of Tom, Jr.'s mother, Audrey. When Andy asks why his father did not capture the likeness of his own mother, as well, Tom admits that he was not creative during his second marriage. That night, the boys bicker about their father as Tom listens from his studio. While Tom, Jr., has fond memories of his childhood with Tom and Audrey, Dave complains that their father physically abused his mother Joan during drunken altercations. However, Andy insists that their father has become a better man since the divorce. When Tom joins his sons in their room, a pillow fight ensues and Dave pummels his father.
With tensions eased in the coming days, the boys enjoy spear fishing in a reef as Tom and Eddy watch from the Tortuga. When a hammerhead shark threatens to attack, Tom shoots and misses, so Eddy kills the shark with his new machine gun. Andy is impressed by Eddy's heroism, although he notices that the man has a drinking problem, and Tom is dismayed that he was unable to defend his boys, himself. In the evening, the men see an explosion offshore and Eddy declares that a German U-boat has been torpedoed. The next day, a dead crewman washes onto the beach. Tom orders his sons to return to the house, then tells Joseph that Tom, Jr., is nearly old enough to fight in the war.
Sometime later, Tom admits to Tom, Jr., that he still loves his mother, Audrey. Later still, Tom tells his son that Jewish refugees are arriving on the island to escape the war in Europe.
During a deep-sea fishing excursion on the Tortuga, Dave hooks an enormous marlin and struggles with the line for hours. Although the boy is exhausted and bleeding, Tom coaches his son to keep up the fight. When the marlin gets away, Tom declares that he loves Dave unconditionally, and they embrace.
Back on the island, Tom, Jr., announces his decision to join the Royal Air Force in Canada. When the summer ends, Tom bids farewell to his sons as they board a seaplane bound for the U.S. At boarding school, Dave and Andy receive a letter from their father, who writes that the island is increasingly affected by the war. Tom professes his love for the sea, but reflects upon his isolation and promises to be more involved with boys' lives.
Sometime later, Tom finds Eddy at a local hotel, beaten in another fight. As Lil tends to Eddy's wounds, she asks Tom for a date, but he is distracted by the unexpected presence of his first wife, Audrey (Claire Bloom). As Tom escorts Audrey along the beach to his house, she announces her engagement to an army general, but admits that she still has feelings for Tom, and they kiss passionately.
At Tom's house, Audrey drinks to excess and Tom realizes that she has come to the island because Tom, Jr., was killed in battle. When she returns to the mainland, Tom wallows in anguish and decides to leave the island for good. Eddy offers to accompany him in his voyage back to America, but Tom disparages Eddy's alcoholism and declines.
As Tom and Joseph sail the Tortuga toward Florida, however, Eddy surprises them as a stowaway. The men soon spot the boat of a fellow islander, Captain Ralph (Gilbert Roland), smuggling Jewish refugees into Cuba. The vessel was attacked by a German U-boat the previous evening, so Tom and his friends rescue its passengers and resume their journey toward Cuba.
That night, the Tortuga is detained by the Cuban Coast Guard. As the Cubans snatch a female refugee, a gunfight ensues, leaving the Tortuga damaged. The following morning, the Tortuga is pursued again. Although Tom navigates an escape, Eddy dies from gunshot wounds and Tom casts his body overboard, lamenting the loss of his friend.
Sometime later, off the coast of Cuba, Tom delivers the refugees ashore, then triggers a gasoline explosion to detain the Coast Guard. Climbing aboard the Tortuga to get away, Tom is shot in the back. As Joseph declares his love for his fallen comrade, he reflects that Tom refuses to accept affection from others. In his final moments of life, Tom envisions himself and Audrey back at his island home, where he promises to love her eternally. He embraces young Dave and Andy, then leaves with the newly-reconfigured family as Joseph waves goodbye.