Video Gamez R kool

The number and variety of video games is humongous. As a relatively new medium, video game’s impressive growth attracts numerous scholars from different fields, though in a sometimes frustrating way. Many studies are focused on small scales and lack representativity, while others stay on an abstract level with their data handling.

So, what better way to make an interdisciplinary study of video games than to bring humanities, game studies and machine learning together?

With an enormous dataset of Steam reviews, we question the relevance of game genres, explore the possibility of classifying games based on their reviews, and study the resemblance between this classification and genres as well as user-defined tags. Continue reading for more details and other interesting statistics on Steam reviews!

Table of Contents

• Dataset
• Game embeddings
• Classification with K-means clustering
• Clusters and tags comparison
• A closer look at the reviews
• PCA and classification
• What about the users?
• Conclusion
• References

Dataset

For this project, we used a large-scale dataset of Steam reviews compiled in 2021, which is openly available on Kaggle¹.

Contents

This dataset consists of over reviews published on Steam, an online video game marketplace. In total, there are 21 million reviews, 10 million of which are in English, from over 300 different games. For each of these reviews, additional metadata is provided, such as the timestamp, the number of users who found that review helpful or funny, whether that game was received for free or during early-access, and other user-related information.

Source caveats

Because only a subset of all reviews from Steam were kept for this collection, potential biases were introduced. We asked the author for more information on the selection of the games. They said: “The games were picked by hand without stringent criteria. I did, however, pay attention to a few aspects and made an effort to include: both mainstream titles as well as indie titles, largely positively reviewed games as well as largely negatively reviewed games, and some older titles alongside newer titles, though you’ll find most of the dataset is comprised of games released in 2017 and later.”. They also added regarding the proportion of reviews collected for each selected game: “I attempted to collect all the reviews at the time (Jan. 2021) for every title, and as such, no reviews were left out intentionally. I did observe there were some reviews missing but I do not have an exact number. If I remember correctly it was usually something like 1% of a game’s reviews that was missing.”

Game tags

To complete our analysis, we gathered the official tags from Steam (also called genres) and the most popular user-defined tags. This procedure was made manually and some tags were either broken down into subparts or ignored to avoid synonyms while staying general. For example, FPS was broken down into ‘First person’ and ‘Shooter’, and ‘Tactics’ was ignored for the more frequent ‘Strategy’ was already taken. The decision to avoid/break the different tags or not was highly subjective but mostly done by one person to stay consistent. The games and their tags are publicly accessible here.

Game embeddings

Given the massive quantity of reviews, one key challenge is to condense all the information contained in a useful way. To do so, we tried to obtain numerical representations for each game, which we call game embeddings. Ideally, these embeddings would capture key information about the games and place conceptually similar games close together in the embedding space.

The process to obtain these embeddings is quite involved, although we can split it in two key steps: First, we take the textual content from all the English reviews and use it to train a Sent2vec² model to obtain 100-dimensional sentence embeddings. Then, we take the 1000 most helpful reviews for each of the 200 most reviewed games, feed them to the trained Sent2vec model, and average the resulting sentence embeddings for each game. This gives us a 100-dimensional embedding for each of the top 200 most reviewed games in our dataset. This process is summarized in the figure below.

The final game embeddings, and similarly obtained genre and tag embeddings, are publicly available, and can easily be visualized using the Tensorflow Embedding Projector. We heavily recommend taking a look at this dedicated page for more info.

When looking at these embeddings, we notice that similar games are usually very close together in the embedding space, and that games with specific tags are often grouped together. We find these embeddings to be quite satisfying, and use them throughout our analysis.

Classification with K-means clustering

We now try to see if we can use clustering on these embeddings to group similar games together. To do so, we pick K-Means, a simple yet effective technique for distance-based clustering. We used the Silhouette Coefficient to determine a good number of clusters, but we also chose a high number in order to get more specific clusters, which would be more interesting to analyze. After a bit of trial and error with different seeds and cluster numbers, we settled on a 20-cluster classification that produced fairly satisfying results.

The figure below shows the games which are part of each cluster. Use the menu to navigate to the cluster of your choice.

If you know your games well, you’ll see that the clustering worked quite well, aside from a few oddities.

Here are some clusters we found particularly satisfying:

• Cluster 3, which groups grand strategy wargames such as the Civilization and Total War series
• Cluster 19, which contains simulation games of all sorts
• Cluster 7, which comprises many games of different genres and visual styles, but which are all well-known for their strong co-op / multiplayer aspect

Cluster 5 is interesting as well, as it contains a lot of games, including several that don’t seem that similar, apart from the fact that they are linked by being multiplayer games. Maybe their belonging to the same cluster is a sign that the multiplayer component is the most salient part of the game, superseding the other aesthetic and gameplay features, in the players’ reviews. Alternatively, it could simply be that these games were hard to group with anything else, and their relative proximity in the embedding space resulted in a cluster (as with K-Means clustering, all games must belong to a cluster).

So, hypotheses can be found to explain the regrouping of games when more obvious aspects are lacking. It appears that we can effectively find a coherent classification using only the users’ reviews. To confidently verify the hypothesis that the 5th cluster gathers games because players pay more attention to the multiplayer feature of these games than the others, a thorough linguistic analysis of the reviews’ content is necessary.

Although this grouping in 20 clusters is slightly uncertain and not without outliers, it seems that the model was overall able to capture enough information from the users’ discourse to form a coherent understanding of each game.

Clusters and tags comparison

The classical criteria to classify games is the use of genres that describe part of their aesthetic or some gameplay features. Steam also uses tags that users can assign to games, and the more popular ones are displayed on the games’ pages on the platform. Do our clusters connect to the corresponding genres and tags of each game? To answer this question, we isolated for each cluster the genres and tags that were present in all of their games. One interesting thing to keep in mind is that the model for game embeddings had no information about the related tags during training outside the direct use of those words in the reviews, which is not highly significant. For example, the word “indie” does not even show up in the top 1000 most frequently used words, appearing in less than 0.3% of reviews.

The size indicates the number of games in each cluster, and the prototypes are the most central games in each cluster. Prototypes can be further analysed to try and create a classification system based on representative games, though this goes beyond the aim of our study. However, they can still be useful to better understand the composition of each cluster, as we tried to do in the previous section.

We see that most clusters effectively contain games with a couple of tags or genres in common. It is interesting to see the contrast between the tags and the diversity of some clusters. Looking at cluster 5 again, we find the common idea of multiplayer action, while the rest of the gameplay of these games is relatively varied. This reinforces our earlier hypothesis but is insufficient to verify it. Following it, we could interpret our clustering as a model selecting the defining feature(s) of each game to then place them in corresponding clusters. Once again, this idea should be further explored by looking at the reviews and find whether the characteristics associated with the genres and tags standing out for each cluster do in fact appear significantly more than other patterns in the players’ discourse.

Such results, with a more thorough analysis, could help us determine the most defining feature(s) of a game and classify genres and tags according to the value players assign to them in a game. This could help players orient themselves more precisely when looking for a specific type of experience, contrary to simply search for a tag or genre’s presence in a game, which does not indicate how important this characteristic will be in said game.

In addition to this overall suggestion that some genres and tags are more salient than others depending on the games, this table offers an interesting perspective on the genres and user-defined tags themselves. Indeed, at the exception of cluster 19, all genres defined by Steam are appearing in the user tags as well.

Supposedly, then, Steam users relate to the official genres. However, the games from clusters 2, 5, 9, 11, and 16 all share the same genre, Action. They can be distinguished by their tags, which are more precise. The same happens with the RPG genre in clusters 10 and 13. In cluster 13, the only tag shared by all games is RPG as well, while cluster 10 has several tags complementing it. Possibly, the games from cluster 13 could be considered as the most representative of the RPG genre, while cluster 10 regroups RPG games with other salient features. Although the official genres are generally included in the user-defined tags, they appear to be insufficient by themselves, especially the Action genre.

If we were to create a new categorisation system, then, these genres would have their share of information to give, but what is most striking here is how important and useful players’ opinions and perspectives can bring to the discussion.

A closer look at the reviews

Comparing four close games: Night in the Woods, Undertale, OneShot, and To the Moon

Four games appear close together in our embeddings: Night in the Woods, Undertale, OneShot, and To the Moon, with all of their distance to one another contained between 0.161 and 0.230. Although To the Moon is in cluster 4 and the others in cluster 8, they all have the other games as their top 3 (4 in the case of Night in the Woods) closest games. Consequently, they will still be considered as an ‘alternate’ smaller cluster for this analysis.

Night in the Woods

Night in the Woods is a single-player adventure game. It is a story-focused exploration game in which we play Mae, a character coming back to her hometown that she explores, meeting and interacting with non-playable characters.

Undertale

In Undertale, we play a human who fell in a hole and ends up exploring an unknown world, encountering villagers and ‘enemies’ we can choose to fight or spare, with our main aim being to come back to the surface and in our own world.

Oneshot

OneShot is a story in which we control a child from an outside perspective, acting as a sort of guardian for them. The game also has an interactive part with the operating system, which is praised in the reviews for its originality.

To the Moon

To the Moon is a puzzle and adventure game. You play as two doctors exploring an old man’s memories to alter them, so that he can die with his dying wish completed in his memories.

Looking at the content of these games’ reviews to try and find what patterns may have been found by our embedding method, several elements are striking. The four games’ reviews have frequent references to terms referring to:

• the story in a highly positive way, commenting on the quality of the writing
• emotions, with words such as ‘feelings’, ‘cry’, ‘sad’, ‘happy’
• comments on the games not being focused on gameplay
• remarks on the high quality of the soundtrack
• the word ‘experience’

The four of them have ‘Story rich’ and ‘Great Soundtrack’ in their top 3 user-defined tags. This is unsurprising: users reviewing the games and assigning them tags are most probably broadly the same people. Reading these four games’ reviews, then, there seems to be lexical patterns justifying the measurements made by the program.

Anomaly? Night in the Woods and VA-11 Hall-A: Cyberpunk Bartender Action

Night in the Woods is the only game of this cluster having a closer neighbour than the three others in this group: VA-11 Hall-A: Cyberpunk Bartender Action. It is a game in which the player incarnates a bartender preparing and serving drinks to different customers while listening to them in a cyberpunk setting.

VA-11 Hall-A: Cyberpunk Bartender Action

In this game you play a bartender at the eponymous VA-11 HALL-A, a small bar in a dystopian downtown which is said to attract the "most fascinating" of people. Gameplay consists of players making and serving drinks to bar attendees whilst listening to their stories and experiences.

Supposedly, this game’s reviews should have more in common with Night in the Woods than with the three other games. However, ‘Story Rich’ is the fourth user-defined tag, while ‘Great Soundtrack’ is absent of the list. Looking at the ten main tags for each game, nothing strikingly separates Night in the Woods and VA-11 Hall-A: Cyberpunk Bartender Action from Undertale, To the Moon and OneShot.

VA-11 Hall-A: Cyberpunk Bartender Action’s reviews contains numerous mentions of sex, sexuality and alcohol, and terms such as ‘waifu’/’weeb’ are recurrent. Although the reviews frequently refer to the story’s depth and how it affected the reviewer, terms such as ‘feelings’, ‘sad’ or ‘happy’ are mainly absent.

A possible link between this game and Night in the Woods could be mental health issues, as terms directly related to these appear in the latter’s review, and the former has mentions of alcoholism and personal or relational conflicts (in reference to the game’s content). VA-11 Hall-A: Cyberpunk Bartender Action’s bring more information, as there are frequent mentions of the game’s music as being great, as well as comments on beautiful visuals and a focus on storytelling. The writing is pointed at, though for being shallow and boring. These elements bring the game closer to the cluster previously observed but does not give any indication of a specific similarity with Night in the Woods (whose negative reviews mostly mention of the developer, Alec Holowka’s suicide).

VA-11 Hall-A: Cyberpunk Bartender Action can be understood as an anomaly when comparing these games’ reviews. While it is crucial to keep in mind that this analysis is based on a non-exhaustive overview and that striking resemblances between both games may have been overlooked, this suggests that the game embeddings are built on elements that are not as obvious to the human eye as topics and words choice, at least not only. There are most probably strong elements behind the game embeddings that are inaccessible to us.

Opposites: Undertale vs Insurgency: Sandstorm and Arma 3

Finally, let us compare the 4-games cluster with opposite ones, focusing on the two most distant from Undertale: Insurgency: Sandstorm (1.541), an action game according to Steam, and Arma 3 (1.526), classified as an action, simulation and strategy game. Both are online multiplayer games in which the players engage in shooting fights against other players.

Insurgency: Sandstorm

Insurgency: Sandstorm is a multiplayer tactical first-person shooter. It is set in an unnamed fictional Middle Eastern region, the game depicts a conflict between two factions: "Security" and "Insurgents".

Arma 3

ARMA 3 is an open-world, realism-based, military tactical shooter video game. The single-player campaign has the player take control of U.S. Army soldier Corporal Ben Kerry. In the single player mode, the player is placed in a variety of situations, from lone wolf infiltration missions to the commanding of large-scale armored operations. This game has a very active modding community.

Insurgency: Sandstorm’s reviews contain recurrent comments on the game’s performances, mentions of sound but as in realistic rather than beautiful/enjoyable (sound design vs soundtrack). A review stood out, describing how passionate another player was, and how this intense experience was deeply appreciated by the reviewer. This game’s reviews are obviously strikingly different from the games observed above. Similarly, Arma 3’s reviews contain mainly feedback on the game’s content, which is similar to Insurgency: Sandstorm and quite different from the four other games. However, the reviews contain mentions of ‘experience’, ‘depth’ and intense feelings - somewhat like the standing-out review from Insurgency: Sandstorm - that can be paralleled with the four-games cluster.

It is highly possible that terms such as ‘experience’ are way too frequent to be used as a way to distinguish games’ reviews (like ‘game’, for example); this would explain the similarities between those highly distanced games. While some potential explanation can be found, slight similarities do exist between strongly opposed games in the embeddings.

Embeddings: a black box

As the program that created the embeddings does not understand English but measures the distances on grounds unknown to us, it is hard to understand how the distances have been calculated and visualise why two games will be close or distant. Consequently, Night in the Woods, Undertale, OneShot and To the Moon may be connected on elements that are unrelated to the meaning of their reviews. However, there is a crucial lack of information in this analysis, as it is based on an overview of only a part of the reviews, especially the ones voted the most helpful, so there is a lack of representativity here. No strong conclusions can be drawn here, only hypotheses pointing out the difficulty to pinpoint the elements determining the distance between games.

PCA and classification

Sometimes, the Principal Component Analysis (PCA) decomposition can lead to interesting projection vectors. Out of curiosity, we tried to see if the PCA dimensions had similarities with the tags repartition. For each PCA dimension, we plotted the mean position of each tag (Once all games are projected, we calculate the mean position of all games possessing the tag). On the second plot, you can visualize each dimension and the position of all tagged games to get a better idea.

Repartition of tags along each PCA axes

It was expected, but individual axes do not correspond to individual tags and cannot be properly interpreted. Still, we can use these graphics to find the most representative axes for a specific tag and better visualize groups of games on the Embedding Projector. (Try components 1,2 and 4, and select color by User Story Rich to see most of the Story Rich games clumped together in a corner of the space.)

While that’s all we can get out of PCA on the game embeddings themselves, what happens if we use it in on our other embeddings? To find out, we merge the embeddings of the top user tags (20 of them) and top official Steam genres (9 of them), and use PCA to project them in a 2D space. The results are shown in the figure below (click on the legend to hide / reveal the user-defined tags and Steam genres).

2D PCA projection of user tags and Steam genre embeddings

This projection gives two interesting results:

• First, we notice that identical user tags and official Steam genres are usually quite close together, which aligns with our findings in the clustering section. However, some broad genres (e.g. action) are much further apart than more objective tags (e.g. indie games). This indicates that, while the genres / tags are mostly consistent, some of them might be too broad to properly categorize a game (e.g. action might not mean exactly the same thing for everyone), and that more fine-grained game genres are needed.
• Second, the two principal components seem to represent two important concepts: whether a game is single-player or multiplayer (horizontal axis), and the game’s audience: casual or serious (vertical axis). This may indicate that we can describe many aspects of a game, such as the gameplay, atmosphere and soundtrack, based on how strongly associated they are associated with each of these concepts. This may also help us understand how genres relate to each other.

What about the users?

As a bonus, let’s deep-dive into some stats obtained using the dataset’s metadata! The number of reviews per category is available by hovering over them.

Average rating of the games in our dataset

We can see that the average score of the games in our dataset is quite high. This probably due to the selection of popular games which tend to be well received.

Average rating of games received for free or not

This was one of the most surprising results we got. Apparently, on average (for the games in our dataset at least), receiving the game for free doesn’t make you want to recommend a game more.

Average rating of games played during early access or not

“Early access” is a term meaning that the game can be purchased when the game is still being developed, usually at a smaller price, so that the player can provide feedback on what should be fixed or improved.

From this result, we can see a small tendency to be more critical when a game is released in early access. They usually have many bugs and sometimes, a game changes so much during development that early adopters of a game will not like how it is evolving. In protest, they may write “Not recommended” reviews. This funding model also has some issues, as some game developers might take the money and abandon development, but those incidents are quite rare and did not occur for any of the games in our dataset.

Average rating from users that posted a certain number of reviews

Those results show that people will be more critical when posting more reviews until a certain point. One hypothesis would be that newcomers review their favourite games first, and then only after do they review the ones they dislike. This could be differentiated from “critics” (people who review many games), which probably have a more nuanced view on video games, and write reviews on all the games they play, regardless of whether they liked or not.

Average rating from users that own a certain number of games

We can see that the more someone owns games, the more likely they are to rate a game negatively. We can hypothesize that people that own a high amount of games are most likely avid players with plenty of video game experience. This might make them more critical of games, as, for each of the genres they like playing, they have much more games against which to compare.

Average rating from reviews with a certain number of words

The longer the reviews, the more critical they become. We also found that positive reviews had an average of 35 words, with the median length being 10 words. For negative reviews, we have an average of 76 words with a median length of 28 words. It seems that when users do not recommend a game, they usually provide a detailed review on what they disliked about it; this is clearly visible in the results.

Conclusion

We hope you enjoyed diving into this ocean of Steam reviews! This is a first step that needs more inputs from other fields such as linguistics, but the results so far are promising and we hope it gave you inspiration for further studies and maybe even write some code!

References