One of my favorite bits in the OT recounts the story of the evil king Manasseh, and his heart-felt repentance:

2Chronicles 33:2–13 And he did what was evil in the sight of the LORD, according to the abominations of the nations whom the LORD drove out before the people of Israel. For he rebuilt the high places that his father Hezekiah had broken down, and he erected altars to the Baals, and made Asheroth, and worshiped all the host of heaven and served them. And he built altars in the house of the LORD, of which the LORD had said, “In Jerusalem shall my name be forever.” And he built altars for all the host of heaven in the two courts of the house of the LORD. And he burned his sons as an offering in the Valley of the Son of Hinnom, and used fortune-telling and omens and sorcery, and dealt with mediums and with necromancers. He did much evil in the sight of the LORD, provoking him to anger. And the carved image of the idol that he had made he set in the house of God, of which God said to David and to Solomon his son, “In this house, and in Jerusalem, which I have chosen out of all the tribes of Israel, I will put my name forever, and I will no more remove the foot of Israel from the land that I appointed for your fathers, if only they will be careful to do all that I have commanded them, all the law, the statutes, and the rules given through Moses.” Manasseh led Judah and the inhabitants of Jerusalem astray, to do more evil than the nations whom the LORD destroyed before the people of Israel. 

The LORD spoke to Manasseh and to his people, but they paid no attention. Therefore the LORD brought upon them the commanders of the army of the king of Assyria, who captured Manasseh with hooks and bound him with chains of bronze and brought him to Babylon. And when he was in distress, he entreated the favor of the LORD his God and humbled himself greatly before the God of his fathers. He prayed to him, and God was moved by his entreaty and heard his plea and brought him again to Jerusalem into his kingdom. Then Manasseh knew that the LORD was God.

The apocryphal Prayer of Manasseh - which Luther retained in his translation of the Bible into German, and which the Orthodox church still uses in certain liturgies - is so incredibly beautiful:

O Lord, God of our Fathers, God of Abraham, Isaac & Jacob
You who made the heaven and the earth…
You before whom all things tremble…
None can stand before your anger and your fury toward sin.

But unending and immeasurable are your promised mercies
Because you are the Lord, long-suffering, merciful and greatly compassionate…

You, O Lord, according to your gentle grace
Promised forgiveness to those who repent of their sins.

You have appointed grace for me… I who am a sinner

My sins exceed the number of the sand on the seashore
And on account of the multitude of my iniquities, I have no strength to lift up my eyes…

And now, behold, I am bending the knees of my heart before you 
And I plead for kindness

I have sinned, O Lord, I have sinned
And I certainly know my sins
I plead with you… forgive me, O Lord, forgive me
For you are the God of those who repent

Display your amazing grace in me…