Up into the southern side of the Caucasus Range, on the border between Georgia and Russia lies the disputed area of South Ossetia. It has declared itself an independent republic, although the majority of the world's countries consider it part of Georgia.
South Ossetia is mountainous all over, and most of it is at an altitude of 1000 m or higher. Highest point is Mount Khalatsa (3938 m), on the border with North Ossetia in Russia.
Alas, it appears it is currently impossible and unsafe to travel there due to political instability.