One of the consistent challenges with research and analysis in Afghanistan that isn’t based on 3 days of guided SOCOM tours (looking at you, @MaxBoot) is that the data is often incomplete. This is due to a variety of factors, from questionable methodology (specifically the discussion about an Asia Foundation survey), to the data being produced by those with a vested interest in the outcome (Yay! Kagans!), to people flat out making stuff up. That last one broke me for a while. Little fed up with great stories turning out to be someone’s big fat lie.
Joshua Foust has recently written on the nearly complete lack of measurable metrics to assess progress in the conflict in Afghanistan, and I’ve written here about the fact that even the Army’s main COIN guy has no idea how to measure COIN success. What makes it difficult is when ISAF, which is leading this race toward the drain, puts out information that others use to analyze progress, only to learn that ISAF’s own information shouldn’t be used to demonstrate how well ISAF is doing. Even though the authors of the study at that link mention some 30 times the fact that the data set was flawed.
Fortunately for ISAF, folks are pretty happy to take their word for it.
Nope, that you can’t make up. That just happened.
Due to a recent post over at Registan, which details yet another LTG Caldwell attempt to convince these aren’t the metrics we’re looking for, and that we’re all stuck in 2009, I started pondering what kind of manipulation of data that ISAF might be actually doing in order to demonstrate progress. This is because at one point in the briefing LTG Caldwell mentions the recent record number of “southern Pashtun” recruits in the ANA. Or folks that just claim Pashtun ancestry but may not be able to find the south of Afghanistan on the map. Whichever. He’s got his bump, albeit at the cost of transparency.
So I started asking some folks who know stuff (like @strickvl and @joshuafoust) about the recent release by ISAF (via @ISAFMedia) of their most recent “Enemy Initiated Attacks” data. Back in September the release of UNAMA’s quarterly report showing a 39% increase in attacks overall in Afghanistan was in stark contrast with what was being communicated by ISAF at that point. In fact, it generated a back and forth on Twitter that made for interesting reading mainly because @ISAFMedia was behaving like an actual human. @joshuafoust was doing what he generally does after having spent a couple of minutes around ISAF and its minions: asking questions.
To sum up, Foust’s main concern was that ISAF seemed to be defining violence quite a bit more narrowly than the UN was in order to paint a rosier picture about what was going on in Afghanistan. @ISAFMedia responded with:
The following are a couple of highlights from the Twitter conversation that followed.
Which, is completely true.
Except for 2008.
Oh, and 2010.
It’s not until 2011, when there has been the most pressure politically to demonstrate progress in Afghanistan that ISAF seems to have stopped reporting on the full spectrum of IED incidents, which could potentially lead to artificially reducing the number of violence indicators by just…ignoring them.
In an April 2008 progress report on Afghanistan, on page 4, ISAF separated IED events into the following categories:
4) Turned In (by Local Nationals)
In May of 2009 ISAF released this “Unclassified Metrics” report, where they broke the categories down a little differently:
2) ISAF Finds
3) ANSF Finds
4) LN (Local National) Turn-Ins
In an April of 2010 report to Congress, on page 41 ISAF actually breaks down the IEDs further:
1) Premature Detonation
2) Own Goal
3) Mine Strike
4) Mine Found/Cleared
5) IED Hoax
6) IED Found/Cleared
7) IED Explosion
Apparently this report was prepped by a hockey/soccer fan with a sense of humor: Category #2, “Own Goal,” was defined thus:
“Own Goal” describes an event where the insurgent inadvertently detonates the mine or IED and kills himself.
So up until a year ago, ISAF was releasing this information and separating it into various categories to account more accurately for all the IEDs, not just the ones that went “boom.” Oh, and as a sidenote: yes, those are all uploaded to Scribd, but the 2008 and 2011 reports I pulled directly off of ISAF’s media site. The rest I had to dig for, but it’s not like they’re working too hard at hiding the fact that they’re making the numbers dance like Ben Stiller in Zoolander.
Which brings us to the most recent 2011 report, where on page 5 of the 6 page document, we’re left with one category: “Executed IED Attacks.” Interestingly enough, ISAF does point out the fact that IED attacks for the same period (January to September) are 6% higher than they were a year prior, in 2011. They just don’t bother mentioning any other kind of IED events.
So without any explanation, ISAF, in its public reporting of IEDs, the greatest threat to the security of the Afghan population, no longer considers IEDs an indicator of violence unless there’s an actual IED attack. Which attacks, unfortunately, are the cause of 55% of all civilian casualties (CIVCAS).
Editor’s Note: In looking at the recent release as well as this one from the end of September, in the September release it states that 70% of all CIVCAS was attributable to IEDs. In the October release it’s down to 55%. There doesn’t seem to be any explanation for the 15% decrease in civilian casualties from month to month. Might have been a reporting error, but no acknowledgement as such from ISAF PAO that I can find.
While it might seem at first glance to be splitting hairs on data points, what this kind of shift in reporting does is focus attention away from the threat of IEDs. By doing so, ISAF in its reporting ignores the significant impact on multiple levels that just planting an IED has on the local population and its freedom of movement. Fortunately, ISAF is doing a fine job of letting the locals dig up the IEDs themselves.
While many of the compounds were used by insurgents as ambush sites and areas containing IED defensive belts, the families continue to clear known IED belts through controlled burns and plowing.
I’m not much of an agricultural specialist, but it’s got to be a little disruptive when you have to clear out the IEDs before you can start your planting and plowing.
Beyond that impact, however, is the greater concern that ISAF, by shifting its metrics to account for violence, is able to report reductions in levels of that violence that aren’t really reductions at all. In fact, this shuffling of the data is in stark contrast to the sort of transparency ISAF claims it wants in its operations.
A lack of transparency goes right to the heart of the issue, Dussault said.
Ma’am, I could not agree more. By continuing the direction in which it is heading in its releases to the public, ISAF appears to be trying to make the facts fit the message, rather than the other way around.
- @ISAFMedia Hates My Face, or We’re ISAF, and We’re Here to Kill You (findingmytribe.wordpress.com)
- LTG Caldwell Updates the World on His Version of Afghanistan (findingmytribe.wordpress.com)
- Taliban and Nato-led forces engage in war of words on Twitter (guardian.co.uk)
- More Signs ISAF May Not Understand Social Media (findingmytribe.wordpress.com)