Metrics are just some quantities that you collect. This is one of those things that many people know about, but few go about and do it, and even fewer care to do it right. Let us first spend some serious time to really understand the need. Your understanding on this subject would truly affect your software. The following applies to networked games. The argument, however, can be extended to many different application domains.
In networked games, there are at least three interesting types of metrics. The performance of the infrastructure, the in-game economics and measurements of player actitivies. Each of the three concerns a differnt crowd of people, the software engineers vs. the game designers and folks on the business side of the fence.
From those three kinds of data, not only the team can tell something is working well or less efficiently, but also what contents are being used, what new contents should be created, even which players are causing too much trouble for the money the pay. :-)
The key is to take guessing work out of discussions and/or decision making. There is a lot of truth in saying "until you can measure it, you probably still don't have a satisfactory knowledge about it" and "if you can measure it, you can improve it". This argument is similar to that for testing.
For a large-scale project, you already know something about the development team and the testing team. In applications like networked game development, there is another peer group of the two, the tools team. Their job is to create and maintain the tools necessary to support both the development and testing team. Metric collection and analysis is a major task of the tools team, since it's non-trivial to gain a clear view into the inner workings a large-scale, distributed complex system. If the team has no way to focus their efforts on key, measurable issues and to visualize the overall scenario, it would be hard for the development and testing teams to be successful and efficient at the same time.
For example, in load testing, we need to gradually increase the load on the daily reference servers to identify any potential problems. The test cases are best as close to the real scenario as possible, though. Using the metrics system, one can collect real statistics about players, such as the beta testers, and come up with a description of an average player. Jittering the parameters within in a random way, you can then generate large numbers of similar but slightly different players for use in load tests. Please note though, how to create those random players in a reliable and meaningful way is a challenging task.
The tools team also supports game designers and marketing. Players would always seek the shortest path to achieve their goal. To wonder and figure things out is the journey of theirs in the immersive world. In this regard, having bugs in the code or a cheating player is one thing, having unbalanced inflationary economics is another. To have contents created at a high cost to go under utilized is an even bigger problem that hurts your chance of making money. How the team can find such contents and guide players to use them, or how the team can figure what players really like to do and create contents for such needs, is also important for business. The operation team also relies on the tools team, since otherwise what is there to monitor with.
To summarize, the metrics system works during development to stablize, scale and profile the system, and also after launch so that the system can be refined to work according to design or even to adapt the design.
Its importance goes without saying. So if you see a system without any metric collection natively built, you know what conclusion to draw.
The collection part is not hard to understand. You just log numbers. The key is to understand what to log, how expensive it is to log each entity and make sensible choices accordingly.
What to log? Player action or events such as time spent doing an activity, where they go most often, step by step move during a fight, who they are friends with, what the most favoriate assets are ... Just think of all the things that the big brother might want to do to us. You can do all of those to the players in the game (of course, I am outrageously simplifying the problem of overlooking the wealth of issues about privacy and, particularly, what's okay in the opinion of your players). This list can also include who makes most complaints, or spent most time on your 1-800 number.
There are also numbers about your game engine. At each time instance, what are the utilization of CPU, memory, network bandwidth, buffer size, number of failures, concurrent players, ...
You should also monitor the virtual economy just like Fed keeps a lid on the real U.S. economy. What are the sources of income, the spending pattern, various types of money sinks, rate of resource consumption, ... Most importantly, collect those numbers over time.
In essense, there are two categories of metrics, those that are computed from your player database and game database, and those recorded on a per event basis. The first category is the cleanest one to implement. By taking periodic snapshots of the databases and storing them, any type of analysis can be performed at a later time.
The event-based logs are not that easy to do, though. Is there a good design to facilitate this? Any ideas come to mind? (We just talked about this).
Before analyzing, one needs a set of reliable tools to generate reports at various levels of detail. This is the same task we have just discussed too. The reporting module is obviously reusable by both testing and metrics teams.
The unique part of the metrics team is a need to do to sift through the data and make sense of it. With GB's of data created on a daily basis, the metrics data quickly becomes overwhelming. Any single technique is not likely sufficient. A conglomerate of techniques, such as data mining and visualization, need to be integrated for effective use.