Students at a U.S. military graduate school in California are mining social media with new methods that may change the way the armed forces collect intelligence overseas.
Students and researchers at the Naval Postgraduate School have tackled two projects that could begin the shift in the way intelligence is gathered. The first is a piece of software they wrote that harnesses the Twitter API (application programming interface) and the second is a project focusing on Syria that uses many social networks to look at U.S. policy options there, though civil liberties experts say the technology concerns them.
The software for Twitter, called the Dynamic Twitter Network Analysis (DTNA), is now being field-tested by three Defense Department units overseas to help gauge public opinion in some of the world's hot spots.
The software pulls in data from the public Twitter feed, then sorts it, live, by phrases, keywords or hashtags. The program is continuously updated, integrating a mapping feature and geo-tagged information. Intelligence officers could use DTNA to understand people's moods about a topic, or hopefully prevent or simply respond faster in any future U.S. embassy attacks.
The group's second project incorporates the DTNA software but also pulls in public information from Facebook, YouTube, Google and other sources to protect potential weapon-of-mass-destruction sites in Syria while the conflict there continues.
Watch an IDG News Service video of researchers at the Naval Postgraduate School's Common Operational Research Environment Lab explaining some of their research here.
The Syria project is spearheaded by an intelligence officer getting his master's degree at the school.
Army Major Seth Lucente set out to analyze Syria with social media because of the U.S. plan to keep from entering the Syrian civil war unless chemical weapon stockpiles are exposed to danger. That goal was spelled out by U.S. President Barack Obama in August, which sparked Lucente's idea for the project. (You can watch a video of the speech here.)
The methodology Lucente and researchers are using is called sentiment analysis. It's been around for about 10 years, used primarily by consumer-facing companies to pull public information on social media streams and analyze it for trends. But this is the first known use of sentiment analysis by the military.
"In the commercial world, everyone is doing it," said Bing Liu, a computer science professor who works on sentiment analysis and data mining at the University of Illinois at Chicago. "I'm not aware of work in the military. But I'm sure they're using it."
But Lucente says that was exactly the problem; intelligence methods are more antiquated than that. The military's standard intelligence-collection techniques still tend to mimic those of the Cold War, with satellite imagery and agents deployed to locations to collect information. Depending on the difficulty in accessing locations or groups, it can take a year or more to corroborate information. When analyzing mood, he says those steps simply take too long.
"Given the military culture and that warfare is usually attrition of military forces, the Army hasn't, in my perspective, tried to move to understand sentiment of population," said Lucente.
So he's designed the Syria project to harness real-time social media streams as a test for the sort of fast intelligence gathering he'd like to do. Lucente's project started out broadly by researching use of social media in Syria. He found that, rather than the Assad regime, opposition forces are most active online.
Researchers were aided by the fact that Syrian opposition forces have to rely on social media to get the word out on their activities, since they are not traditionally funded. That means there is a wealth of information on public Facebook groups and Twitter profiles, including photos and videos, all ripe for analysis.
"It was unusual because unlike conventional war, these organizations don't have funding or resources," Lucente said. "There are no secure communications radios."
What opposition forces have is a massive online presence, detailing their every move. The "Syrian Revolution 2011" Facebook page has more than 647,000 likes. The affiliated Twitter handle, where attacks, death tolls and sometimes troop movements are routinely broadcast, has more than 78,000 followers.
Though, "for the most part, they are getting themselves out there to teach people who they are and what they're fighting for," Lucente said.
Lucente says high-ranking U.S. military officials are surprised when he points out the extreme wealth of online information available on Syria. The most stunning thing to him is a Google map, updated every 24 hours by revolution forces, which he says would take roughly 100 U.S. intelligence officers to be able to update at the same pace using traditional methods. The map is scattered with pins, many of which have videos associated with air strikes, ground movement and other details, every day. (Take a look at the map yourself, found here.)
"It's a pretty wow-factor map," Lucente said. "They're really leveraging this [social media] stuff."
With the Syrian opposition activities targeted for sheer wealth of information for the project, Lucente narrowed the scope of his project to ask which areas of the country are most at risk for losing nuclear, biological or chemical weapons in the event that Syria's government falls. He and two CORE Lab researchers focused on a city called Homs, an important location with a major highway intersection hub. Lucente says it's in a key position for controlling the rest of the country because whichever group holds Homs controls the highways. Syrian websites that track deaths, like http://syrianshuhada.com, say that the Homs Province has the highest number of casualties.
Researchers then checked the Nuclear Threat Initiative, a nonprofit group founded to address risks related to weapons of mass destruction, for how many potential weapons-of-mass-destruction sites exist in the city. They found four: one chemical production site, a fertilizer company, an oil refinery and a uranium recovery plant.
To help protect those sites, the researchers culled through Facebook posts and YouTube videos to analyze opposition forces in the area. Their research culminated in a recommendation to talk with one particular Syrian opposition group near the city. Lucente proposed asking the Farouq Battalion, a group of men who fight for the Khalid bin Walid Battalion, to consider watching those four sites in case the Syrian government should fall. As a counterintelligence officer, Lucente is worried that if those locations are not protected, terrorists could theoretically seize them or black marketers could sell the material.
Rob Schroeder and Gregory Freeman, both research assistants at the CORE Lab, helped map and provide data visualization for the project. The DTNA software pulls Arabic and English information, which is already a step beyond much of the available, consumer-facing software that reads public opinion. But researchers say better foreign language use could crack social media analysis wide open.
"The major ridgeline to be crossed is going to be foreign language analysis," Schroeder said.
The project took two months to complete and since then, Lucente, Schroeder and Freeman have been in demand. The group presented the project to senior military leaders, who they've been asked not to name, and say they were interested in the project. Since making the initial presentation, the researchers have been asked to give the same brief more than a dozen times, all to high-ranking military members involved in intelligence and cyberwarfare.
If the Syria project and the Twitter software go on to establish models that the military deems successful, they could bring about a shift in the way U.S. military intelligence is gathered, increasing speed by focusing on publicly available social media streams.
While social media analytics for military applications interests social media data scientists, Internet freedom watchdogs are less quick to praise. They are worried about privacy breaches, even though mining the public information is legal.
"While technically it is legal to pull social media information, I don't know that it's always ethical, " said Eddan Katz, a visiting fellow for Access to Knowledge at the Information Society Project at Yale Law School.
Liu, of the University of Illinois at Chicago, said any expectation of privacy by social media users is naïve. Raw public data mining is already happening, perfected by corporations eager to learn what consumers think about their products. "You should expect someone is using it, with a computer system to mine something with it," he said.
Lucente is quick to point out that the project is not a matter of Big Brother spying, since the information is public and the military would not use it on U.S. citizens. Still, knowing projects out there exist that have the capability to do such in-depth analysis makes civil liberties proponents worried.
"There's no telling how it will be used, or stored, and that is something we want to watch," said Gregory Nojeim, the director of the Center for Democracy and Technology's Project on Freedom.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.