I'll tell you what I can gather as a non-native speaker - Russian familii have more or less the same sort of variation of origins as English surnames. Istomin in particular is a surname derived from a given/forename: Istoma.
That makes it a patronymic surname, which is one of the main sources of Russian surnames along with occupational names and toponyms. Russians (among others) still carry their direct patronyms separately, though, as a third name, basically in place of an English middle name, for those who have them. So patronymic family names are a sort of ancestral patronym, in addition to the person's actual patronym.
The most common Russian surnames are Ivanov ("Johnson"), Smirnov (from the old male forename Smirniy, peaceful, no obvious English equivalnent) and Kuznetsov (as naturalplastic explained!).
Just as naturalplastic says above, the -vich/-ovich/-evich (male), -ova/-eva (female) endings are the normal patronymic forms (son of... daughter of...).
So for Denis Olegovich Istomin's case, we have Denis, son of Oleg, one of the ancestral line of one Istoma.
Istoma is apparently recorded as a widespread male name in medieval Rus' of the 15th century onwards, roughly when surnames were adopted by Rus'an nobility. There doesn't seem to be total agreement between name origins researchers (who could expect there to to be?), but most commonly the forename Istoma is described as one given in Old Rus' to boys whose birth, or some other aspect of their character, was difficult, the name being derived from the slavonic verb istomit', which means roughly to wear out, exhaust or even torment. Later Russian, meanwhile, does actually have the noun istoma, which curiously is quite positive, meaning the feeling of complete pleasant exhaustion or weakness.
Most of the male forenames that surnames were originally derived from were, in fact, adjectives. The majority of the derived surnames take the grammatical from that answers the form "whose(pl.) are you?". That is, the grammatical case that has the function, roughly speaking, of the English preposition of. The possessive. This is the origin of the endings -ov/-ev/-in which are, basically, the genitive plural endings of adjectives for masculine/neuter: -ov/ev and feminine: -in. The sons of Smirniy, a masculine adjective, were those of Smirnijs = Smirnov. The sons of Istoma, with a feminine ending (-a), were those of Istomas = Istomin.
Apparently, and interestingly, the -in surname ending is relatively much more common compared to -ov/ev in Belarus compared to Russia.
A couple of more examples from very famous -in names:
Pushkin: of the Gunners (Pushka, gun, gunner, artilleryman).
Gagarin, those of the 'Loons' (Gagara, literally Loon (the bird). According to Russopedia, a forename given by a 15th century prince to his sons and spreading ever since, associated with black hair, long necks and a jollity reminiscent of the Loon (maybe similar to the English name Raven?) )
You strayed into a special interest of mine here, so thank you
Perhaps a native will appear in the future to be a bit more more specific. I hope in the meantime that this is interesting.
Вам удачи.