The departure of the pope and his court was a damaging event for a city in the full swing of its development. The population fell to less than half, approximately 15,000, which remains nonetheless a respectable size for the Middle Ages. The city did hold on to a large part of its prestige. It remained an important commercial center, from which the Italians exerted their influence over all the south of France.

In 1432, the Avigonnais revolted. In effect, they wished to be governed by an eminent and influential person, and addressed themselves directly to the Pope in the hopes of regaining their lost status. Except, the pope nominated his own nephew, a man of no importance. The Avignonnais, deeply insulted, organized an uprising. The pope then nominated the Cardinal of Foix, but it was too late ; the Avigonnais reject the Cardinal's emissaries. They undergo another siege which lasted two months. A legate, holding extensive powers, charged with the administration of the pontifical possessions and the County of Venaissin, restored to Avignon the importance of a second Rome, "l'altera Roma". Certain cardinals of Avignon were quite celebrated, such as the Cardinal Julian de Rovere, legate from 1476 to 1503, before being elected pope with the name Jules II.

Nonetheless, political and economic decline was inevitable. At his death in 1481, the Count of Provence,Charles II left his estates to the king of France. Avignon and the County of Venaissin form hereafter an enclave within the French kingdom. The king his an earnest interventionist, whom Avignon must, more than ever, take into account. Francois I sojourned many times in Avignon, notably on his return from Marignan, and during the conflict which opposed him to Charles Quint.

The wars of religion spared Avignon, which remained faithful to the pope, but the County of Venaissin was the theater of many bloody battles. The Huguenots were solidly implanted in Orange and the neighboring Dauphiné. To insure the security of his estates, the pope was obliged to call on the French king Charles IX, who placed his cousin Charles of Burgundy, future king of the League, at the head of a delegation (from 1565 to 1590). This latter was represented by the Cardinal Georges d'Armagnac.

Since the 16th century, the legate was rarely present on the territories in his charge; it was his representative, the vice-legate who became de facto governor of Avignon and the County of Venaissin. After the wars of religion these positions were occupied only by Italians, the nomination submitted before the French king. In 1693, the suppression of the legate's office was the incontestable sign of the city's political decline.