U.S. and Russian negotiators held their first security talks since Russia’s deployment of tens of thousands of troops to the Ukrainian border sparked fears of an invasion, but they left the Monday talks saying they failed to narrow their differences.

Although both sides described the discussions as businesslike, they remain at odds over Moscow’s demands that the North Atlantic Treaty Organization, the postwar military alliance of Western powers, halt its expansion in Europe, and the U.S.’s insistence that Russia remove more...

U.S. and Russian negotiators held their first security talks since Russia’s deployment of tens of thousands of troops to the Ukrainian border sparked fears of an invasion, but they left the Monday talks saying they failed to narrow their differences.

Although both sides described the discussions as businesslike, they remain at odds over Moscow’s demands that the North Atlantic Treaty Organization, the postwar military alliance of Western powers, halt its expansion in Europe, and the U.S.’s insistence that Russia remove more than 100,000 Russian troops arrayed near Ukraine’s border.

The troop buildup has prompted concerns that Russian President Vladimir Putin intends to invade a country he considers part of Russia’s sphere of influence or is generating a crisis to extract security concessions from the West.

Deputy Russian Foreign Minister Sergei Ryabkov said that Russia had “no intentions to attack Ukraine” and that the West needn’t fear “any kind of escalation.”

But many U.S. officials believe otherwise.

“We believe the threat of invasion is real,” White House national-security adviser Jake Sullivan said on “NBC Nightly News” after the talks concluded.

In a briefing with reporters after the Monday’s round concluded, Deputy Secretary of State Wendy Sherman, who led Washington’s side, said, “We’re just at the beginning, and we don’t know where all of this is headed quite yet.”

Mr. Ryabkov described the talks as “difficult, long, very professional, specific, without any attempts to embellish anything, to get around any sharp corners.”

But, he added, “the main questions are on hold, and we do not see the American side’s readiness to resolve them in a way that suits us.”

Ukrainian forces on the line of separation from Russian-backed rebels near Avdiivka, in the southeast region of Ukraine, on Saturday.

Photo: anatolii stepanov/Agence France-Presse/Getty Images

Few expected much progress from Monday’s talks in Geneva, because the two parties came to the meeting with starkly different goals. The U.S. hoped to ease tensions over Ukraine by involving the Russians in talks over intermediate-range missiles in Europe and the scope of military exercises. Russia, meanwhile, said its aim was to remake the post-Cold War security arrangements in Europe by halting NATO’s expansion, curtailing the alliance’s ties with Ukraine and parts of the former Soviet Union, and severely restricting military deployments on the territory of the alliance’s Eastern European members.

The differences were displayed immediately in the separate news conferences the two sides gave after the nearly eight-hour meeting.

“We will not allow anyone to slam closed NATO’s open-door policy, which has always been central to the NATO alliance,” Ms. Sherman said.

Mr. Ryabkov countered, “It is very important that Ukraine can never join NATO in the future. We need iron, legal obligations, not promises, but guarantees…This is a matter of Russia’s national security.”

An important objective for U.S. officials was to gauge whether Russia was ready to pursue a reduction of tensions on terms that NATO might accept or was keeping open its military options to attack Ukraine.

“It doesn’t sound like they have come out with an answer to that question,” said Angela Stent, a Brookings Institution fellow and former U.S. national intelligence officer for Russia and Eurasia.

“Nobody walked out of the talks. The question will be how much Russia insists on demands NATO says are nonnegotiable.”

Dimitri Simes,

president of the Center for the National Interest, a Washington think tank, said Mr. Ryabkov will convey his sense to the Kremlin on whether Russia can achieve its objectives with further talks.

A military buildup along the Ukrainian border is further straining ties between Russia and the U.S., after clashes over cybercrime, expulsions of diplomats and a migrant crisis in Belarus. WSJ explains what is deepening the rift between Washington and Moscow. Photo Composite/Video: Michelle Inez Simon The Wall Street Journal Interactive Edition

“The question is how this information will be interpreted in Moscow, in particular by Putin’s inner circle, in particular the siloviki—the people from the military and security services who have particular influence in making decisions at this point,” Mr. Simes said.

The U.S. and Russian sides also sketched out divergent timelines for further talks. Ms. Sherman said that arms control and other security issues are complicated and can’t be negotiated in a matter of weeks. Mr. Ryabkov said that Russian demands need to be dealt with urgently and “we can’t afford any additional delays.”

“We do expect to be in touch with the Russian Federation again in coming days to determine when and how this conversation will go forward,” State Department spokesman Ned Price said, referring to the bilateral talks between the U.S. and Russia.

Some of the security issues will also be taken up again in Brussels on Wednesday in a meeting of the NATO-Russia Council, a forum for consultations with Moscow. Ms. Sherman and Mr. Ryabkov will lead their nations’ delegations to the session. It will mark the first time that the council has met in more than two years.

NATO Secretary-General Jens Stoltenberg said after a Monday meeting with Ukraine Deputy Prime Minister Olga Stefanishyna that the organization was hoping for agreement on a way forward.

“I really hope there is a real will on both sides…to engage in a process that prevents a new armed conflict in Europe,” he said.

On Thursday, discussions will be held in Vienna at the Organization for Security and Cooperation in Europe, a multilateral forum that includes Russia and Ukraine.

Regional issues, including unrest in Kazakhstan and the deployment of Russian-led troops there, didn’t come up in the Monday talks, Ms. Sherman said.

The Biden administration’s willingness to discuss limits on intermediate-range missiles based on land is an important shift for Washington.

The Trump administration had rebuffed Mr. Putin’s proposals for a moratorium on the deployment of intermediate-range missiles in Europe, saying that such a step could preclude U.S. options while failing to lead to the elimination of the 9M729, a Russian ground-launched cruise missile the U.S. alleges violated the now-defunct Intermediate-range Nuclear Forces Treaty.

Russian officials have said Russia doesn’t plan to invade Ukraine, but they have also added that Mr. Putin will look into options prepared by his military experts if diplomacy fails or if the West continues what they call its “aggressive line.”

If Russia invades, U.S. officials are eyeing a range of options including leveraged sanctions on exports to Russia, such as semiconductors used in cutting-edge technology, as well as stepped-up military support to Ukraine, according to people familiar with the matter.

Write to Michael R. Gordon at michael.gordon@wsj.com, William Mauldin at william.mauldin@wsj.com and Ann M. Simmons at ann.simmons@wsj.com