Metrics are always important and many software products do generate various metrics. Here we discuss a simple and elegant metric pattern that would be part of your organization's common library or utilities for effective reuse.
The requirement to support metrics quickly becomes cumbersome when you require the cartesian product of metrics of each item type with its possible states. As the required possible metric combinations explore, the simple traditional way of having a variable to keep track of each combination doesn't work and maintenance becomes nightmare. The suggested metric pattern will let you count anything without the explicit need of a corresponding variable by making it simple and elegant and will be part of your library.
For example, a DevOps engineer writes a monitor tool to track the health of colocated servers across geographic locations. If these servers are in 2 locations, say California and Nevada and 2 possible machine states, Up and Down, we will have 4 metrics for many machines are up/down in California/Nevada. Over time, the DevOps engineer would like to collect more granular information about the server like whether its reachable and how fast it is responding while the company is extending to more geographic locations. The cartesian product "m*n", i.e, the number of collocations multiplied by possible state of a machine, of possible metrics makes the code quickly unmanageable and unmaintainable.
Here we discuss a metric pattern which let you collect metrics thats simple, clean, and elegant that is reusable, maintainable and extensible.
First, we need a Counter interface that can simply count every possible state.
interface Counter<T> {
long increment(T key, long count);
long increment(T key);
long get(T key);
long put(T key, Long value);
Map<T, Long> getMetricsMap();
}
The above Counter interface can count any possible state. By having Counter<String> , we can count any thing TOMATOS to CARS. If the 'T' is an enum, then we have complete list of all possible states and is highly recommended for a clean design of the software. We can define a type called Result.
public enum Result {
PASS,
FAIL,
SKIP,
WARN,
EXCEPTION,
EXECUTION_TIME,
AVERAGEG_EXECUTION_TIME }
By defining Counter<Result>, we can count all possible result of a test. How many tests failed, succeed or skipped.
Next, say we want to categorize tests based on their priority like P0, P1 and P2 to track metrics. Now we want to count how many P1 test cases failed or how many P2 test cases skipped and whats the average execution time of P3 test cases. Lets define a Priority interface that has all possible priorities of a test like following.
public enum Priority {
PO,
P1,
P2,
P3,
P4,
NONE }
Now we define a Implementation class for Counter<T> interface that handles both test type and possible result.
public static class CounterImpl<T> implements Counter<T> {
private Map<T, Long> mapMetrics = new TreeMap<T, Long>();
private static final Long ZERO = new Long(0);
public long increment(T key, long count) {
if (!getMetricsMap().containsKey(key)) {
getMetricsMap().put(key, ZERO);
}
getMetricsMap().put(key, getMetricsMap().get(key) + count);
return getMetricsMap().get(key);
}
public long increment(T key) {
return increment(key, 1);
}
public long get(T key) {
long retValue = 0;
if (getMetricsMap().containsKey(key)) {
retValue = getMetricsMap().get(key);
}
return retValue;
}
public long put(T key, Long value) {
getMetricsMap().put(key, value);
return value;
}
public Map<T, Long> getMetricsMap() {
return mapMetrics;
}
}
Thats it! We are ready to collect all possible metrics for the cartesian product of 6 test types x 7 possible test results, or 42 metrics with a data structure of map of counters like,
Map<Priority, Counter<Result>>
Counter and CounterImpl will be your common library for your organization. What all you require is enums like Priority and Result (or strings) to count cartesian metrics with this simple and elegant pattern.
Your code for collecting metrics is like following:
Map<Priority, Counter<Result>> mapPriorityCounters = new ConcurrentHashMap<Priority, Counter<Result>>();
for (Priority Priority : Priority.values()) {
mapPriorityCounters.put(Priority, new CounterImpl<Result>());
}
mapPriorityCounters.get(Priority.P2).increment(Result.PASS);
mapPriorityCounters.get(Priority.P2).increment(Result.PASS);
mapPriorityCounters.get(Priority.P3).increment(Result.SKIP, 12);
mapPriorityCounters.get(Priority.P4).increment(Result.FAIL, 2);
mapPriorityCounters.get(Priority.P4).increment(Result.FAIL, 2);
for (int i = 0; i < 5; i++) {
mapPriorityCounters.get(Priority.P1).increment(Result.PASS);
mapPriorityCounters.get(Priority.P1).increment(Result.EXECUTION_TIME, 3);
long totalExecutionTime = mapPriorityCounters.get(Priority.P1).get(Result.EXECUTION_TIME);
long count = mapPriorityCounters.get(Priority.P1).get(Result.PASS);
mapPriorityCounters.get(Priority.P1).put(Result.AVERAGEG_EXECUTION_TIME,
new Long(totalExecutionTime / count));
}
for (Map.Entry<Priority, Counter<Result>> PriorityEntry : mapPriorityCounters.entrySet()) {
for (Map.Entry<Result, Long> counterEntry : PriorityEntry.getValue().getMetricsMap().entrySet()) {
System.out.println(PriorityEntry.getKey() + "_" + counterEntry.getKey() + ":" + counterEntry.getValue());
}
}
The result will be:
P3_PASS:12
P4_FAIL:4
P2_PASS:2
P1_PASS:5
P1_EXECUTION_TIME:15
P1_AVERAGEG_EXECUTION_TIME:3
As demonstrated in the above code snippet, this metric pattern can be used beyond counting to handle statical measures like averages.
You can refer/download the code of this pattern at
Metric Pattern Java Reference Code.
We have used this pattern in Search Science (for tracking Fields, Factors, Models, Profiles), monitoring (Whether desired application version available across colos), sonar compliance (sonar metrics like coverage, violations) and generating java properties files.
A simple and elegant solution.
WHEN: An important question is when must the automation be triggered? How quickly would one know the failures? For example, when there is a code change (post-commit CI) or a code change request (pre-commit CI), the CI automation will kick-off. Monitoring will be done at regular intervals. The software component's version is checked on production machines. This automation check will be implemented at deployment as well at regular intervals. When an off-line production machine with an older version-ed component is added back, the versions do mismatch. A "Change", "Regular Interval" and/or both is required for optimal utilization of automation resources.