Finding the silver(ish) lining in Afghanistan's Emerald City

Making the Facts Fit the Message, or, The First Casualty of COIN is Metrics

One of the consistent challenges with research and analysis in Afghanistan that isn’t based on 3 days of guided SOCOM tours (looking at you, @MaxBoot) is that the data is often incomplete. This is due to a variety of factors, from questionable methodology (specifically the discussion about an Asia Foundation survey), to the data being produced by those with a vested interest in the outcome (Yay! Kagans!), to people flat out making stuff up. That last one broke me for a while. Little fed up with great stories turning out to be someone’s big fat lie.

Joshua Foust has recently written on the nearly complete lack of measurable metrics to assess progress in the conflict in Afghanistan, and I’ve written here about the fact that even the Army’s main COIN guy has no idea how to measure COIN success. What makes it difficult is when ISAF, which is leading this race toward the drain, puts out information that others use to analyze progress, only to learn that ISAF’s own information shouldn’t be used to demonstrate how well ISAF is doing. Even though the authors of the study at that link mention some 30 times the fact that the data set was flawed.

Fortunately for ISAF, folks are pretty happy to take their word for it.

Nope, that you can’t make up. That just happened.

Due to  a recent post over at Registan, which details yet another LTG Caldwell attempt to convince these aren’t the metrics we’re looking for, and that we’re all stuck in 2009, I started pondering what kind of manipulation of data that ISAF might be actually doing in order to demonstrate progress. This is because at one point in the briefing LTG Caldwell mentions the recent record number of “southern Pashtun” recruits in the ANA. Or folks that just claim Pashtun ancestry but may not be able to find the south of Afghanistan on the map. Whichever. He’s got his bump, albeit at the cost of transparency.

So I started asking some folks who know stuff (like @strickvl and @joshuafoust) about the recent release by ISAF (via @ISAFMedia) of their most recent “Enemy Initiated Attacks” data. Back in September the release of UNAMA’s quarterly report showing a 39% increase in attacks overall in Afghanistan was in stark contrast with what was being communicated by ISAF at that point. In fact, it generated a back and forth on Twitter that made for interesting reading mainly because @ISAFMedia was behaving like an actual human. @joshuafoust was doing what he generally does after having spent a couple of minutes around ISAF and its minions: asking questions.

To sum up, Foust’s main concern was that ISAF seemed to be defining violence quite a bit more narrowly than the UN was in order to paint a rosier picture about what was going on in Afghanistan. @ISAFMedia responded with:

The following are a couple of highlights from the Twitter conversation that followed.

http://twitter.com/#!/joshuafoust/statuses/119380086133755904

Which, is completely true.

Except for 2008.

And 2009.

Oh, and 2010.

It’s not until 2011, when there has been the most pressure politically to demonstrate progress in Afghanistan that ISAF seems to have stopped reporting on the full spectrum of IED incidents, which could potentially lead to artificially reducing the number of violence indicators by just…ignoring them.

In an April 2008 progress report on Afghanistan, on page 4, ISAF separated IED events into the following categories:

1)      Detonated

2)      Discovered

3)      Pre-Detonated

4)      Turned In (by Local Nationals)

In May of 2009 ISAF released this “Unclassified Metrics” report, where they broke the categories down a little differently:

1)      Detonations

2)      ISAF Finds

3)      ANSF Finds

4)      LN (Local National) Turn-Ins

In an April of 2010 report to Congress, on page 41 ISAF actually breaks down the IEDs further:

1)      Premature Detonation

2)      Own Goal

3)      Mine Strike

4)      Mine Found/Cleared

5)      IED Hoax

6)      IED Found/Cleared

7)      IED Explosion

Apparently this report was prepped by a hockey/soccer fan with a sense of humor: Category #2, “Own Goal,” was defined thus:

“Own Goal” describes an event where the insurgent inadvertently detonates the mine or IED and kills himself.

Just classic.

So up until a year ago, ISAF was releasing this information and separating it into various categories to account more accurately for all the IEDs, not just the ones that went “boom.” Oh, and as a sidenote: yes, those are all uploaded to Scribd, but the 2008 and 2011 reports I pulled directly off of ISAF’s media site. The rest I had to dig for, but it’s not like they’re working too hard at hiding the fact that they’re making the numbers dance like Ben Stiller in Zoolander.

Which brings us to the most recent 2011 report,  where on page 5 of the 6 page document, we’re left with one category: “Executed IED Attacks.” Interestingly enough, ISAF does point out the fact that IED attacks for the same period (January to September) are 6% higher than they were a year prior, in 2011. They just don’t bother mentioning any other kind of IED events.

So without any explanation, ISAF,  in its public reporting of IEDs, the greatest threat to the security of the Afghan population, no longer considers IEDs an indicator of violence unless there’s an actual IED attack. Which attacks, unfortunately, are the cause of 55% of all civilian casualties (CIVCAS).

Editor’s Note: In looking at the recent release as well as this one from the end of September, in the September release it states that 70% of all CIVCAS was attributable to IEDs. In the October release it’s down to 55%. There doesn’t seem to be any explanation for the 15% decrease in civilian casualties from month to month. Might have been a reporting error, but no acknowledgement as such from ISAF PAO that I can find. 

While it might seem at first glance to be splitting hairs on data points, what this kind of shift in reporting does is focus attention away from the threat of IEDs. By doing so, ISAF in its reporting ignores the significant impact on multiple levels that just planting an IED has on the local population and its freedom of movement.  Fortunately, ISAF is doing a fine job of letting the locals dig up the IEDs themselves.

While many of the compounds were used by insurgents as ambush sites and areas containing IED defensive belts, the families continue to clear known IED belts through controlled burns and plowing.

I’m not much of an agricultural specialist, but it’s got to be a little  disruptive when you have to clear out the IEDs before you can start your planting and plowing.

Beyond that impact, however, is the greater concern that ISAF, by shifting its metrics to account for violence, is able to report reductions in levels of that violence that aren’t really reductions at all. In fact, this shuffling of the data is in stark contrast to the sort of transparency ISAF claims it wants in its operations.

A lack of transparency goes right to the heart of the issue, Dussault said.

Ma’am, I could not agree more. By continuing the direction in which it is heading in its releases to the public, ISAF appears to be trying to make the facts fit the message, rather than the other way around.

Related articles

Tagged as: , , , , , , , , ,

7 Responses »

  1. Hiya Dan. Thanks for your reply to my comment. As for my experience there, there is none. I’ve just been trying to keep up with what’s been going on in Afghanistan for the last 4-5 years by reading blogs and such. Granted, that only gives me a skewed view as most everyone writes from their own perspective. Regardless, I try to read all reports by everyone that has experience in the suck.

    Re Josh’s drift towards the cynical – I’ve read his blog for some time, and can understand a little bit about his shift to the negative. I think it came about due to the fact that his ideas and suggestions about some issues involving the HUMMIT (?) actions went by the wayside. I reckon cynicism would be anyone’s reaction to see good information not being utilized to the fullest extent. Though it does tend to cloud all remaining reactions to the news coming out of a place he once had high hopes for. It doesn’t help anyone at all to be constantly looking for the ‘cons’ instead of noting and touting the ‘pros’. Surely, there is a happy medium somewhere in the mix?

    I guess the way I look at things is this…Every little drop may not make an immediate improvement, but over time, all those drops will increase the level to where the bucket eventually overflows. We may not change life for all, but we’ve changed life for quite a few, yes?

    I hope I’ve not confused you as much as I feel at times. I do applaud you for your willingness to affect a change for the Afghan people. And I wish you much luck in your endeavors.

  2. I’d missed the ‘own goal’ category. Beautiful.

  3. So, you don’t think that having several different Generals in charge over the last several years hasn’t impacted any of these metrics?

    We’ve had what, at least 4 changes in command in just as many years, yes? Do not each and every commander put their own spin on what they think is important? Does not what the CinC want to see come into play in some way?

    Quite honestly, at the end of 2014 or thereabouts, If…someone gets their way, we’ll pull our troops out just so someone can say he kept his campaign promise.

    This is going to be a long drawn out struggle no matter the metrics used or not.

    Oh, Josh is a smart young man, but IMO became a bit cynical after having left Kapisa province several years ago.

    • Completely agree that the focal shifts are due to the shifts in command and political pressures. I mentioned that in the post, that the recent reshuffle of data is due to the political pressure to make this thing work.

      And, I don’t know what your experience here has been, but re: your comment on Josh — I came here 2 years ago convinced that I could actually help somehow. That our efforts here could affect some kind of glorious change for the people of Afghanistan. 2 years in, and here’s where I am: 1) I’m just starting to realize how little I know about this place, and 2) every day here is just another lap around the drain. We’ve set this up for failure on a startling variety of levels.

      So I can understand Josh’s drift toward the cynical: for myself, I don’t see “cynical” as a negative thing. It’s someone who’s smart enough to know when they’re being lied to, and who generally has a solution to the problems they’re pointing out. I know that’s true in my case, as I’m never going to say, “That’s all wrong…fix it,” without either offering a different approach that, generally, I’ve seen work successfully before.

Trackbacks

  1. So You Want to Be Paid to Sit at a Desk and Think About Things | colin cookman
  2. 5 Things I Learned This Week in Afghanistan: 04 May 2012 Edition | It's Always Sunny in Kabul
  3. 5 Things I Learned This Week in Afghanistan, 25 June 2012 Ed. | It's Always Sunny in Kabul

Gimme some of that feedback!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 2,702 other followers

%d bloggers like this: