First time the Žusem castle is indirectly mentioned is in the year 1203 in a gift document of a knight, ministerial of Krško Poppo de Suzzenheima, and directly in 1364 as vest Suzzenhaim. In this year the castle was inherited by Counts of Celje from brothers Heinzel and Albreht. In 1404 the fortification wnet back in to posession of Hans of Žusem and his uncle Andrej. In 1437 Andrej had to pawn the castle to Counts of Celje. In 1480 the administration of the castle is taken over by Hans's uncle Bernard Lichtenberger. Until 1871 owners have changed and the last among them, L.Fieglmuller, left the castle to collapse in the mentioned year, so he would not have to pay taxes for the castle.
Also known information: in 1550 emperor Ferdinand allowed noble Franc Altenhaus to spend 400 fl. for construction works on the castle and add this amount tothe mortgage sum. In 1579 Danijel Kupič, town scribe of Celje and member of committee, which catalogued and valuing provincial prince's castles and estates, described Žusem in detail. He also mentiones that the chapel had cross arch from carved stone named Spigulj. In 1635 the castle was ransacked and demolished by rebelled peasants, in 1695 it was burned. Building master J.Satler from Celje valued the damage to be 2188 fl.

As most of Slovenian castles, which have preserved into the new era, Žusem is also looking like it was at the end of the 19^th century, as a result of many reconstruction and extension works. Against very fragmented castle core, that is almost fully ruined, three-floored six-axial residential wing was leaning, of which front corners were reinforced by imposing round towers. Ground floor, which was arched on the inside and was used as a cellar, was reinforced with renaissance arrow-loops with key-shape like, carved from monolith loops. Triangular foreheads abowe windows in the third floor had a totally mature late-renaissance features; stone window profiles are still preserved. At the east side of the castle a chapel was located, of which a small tower is visible on old images. According to description by Kupič from 1579, its architecture was gothic. Obviously visible in one of ruined walls in castle core is later reconstructed corner of primary romanesque castle; it is defined by carved corner stones which regulate fairly proper construction. According to preserved evidence and old images the primary castle was a fortified towered house. Path to the castle was masoned, protected with internal and external entrance tower.
(Quoted)

The castle is not hard to access, but some parts of the ruins are diffucult to reach. Walking on the castle can be dangerous, because filled up walls are partially very high and behind them usually lies a deep drop.