What’s under the canals of Venice? Old boats, tires and a few surprises

Image courtesy Ismar-Cnr.

Most visitors to Venice drift through the canals on gondolas taking selfies. But a group of researchers spent seven months puttering along pointing high-resolution multibeam echosounders into the waters instead. About 30 of them in all worked aboard the powerboat Litus, intent on mapping the Venice lagoon to gauge the effects of climate change on one of the world’s most improbable cities.

Research boat Litus, courtesy Ismar-CNR

While what’s under those gray-green waters isn’t exactly surprising — boat parts, old tires and containers — scientists say the underwater elevation mapping (that’s “bathymetry,” for the technically minded) comes at a critical time.

Old boats, tires and containers. Image courtesy Ismar-Cnr.

The last 100 years have radically altered the shape and ecological makeup of the lagoon, researchers say: for starters, salt marsh areas shrunk by half and underlying sediment has radically shifted. The “floating city” already struggles to stay above water in the spring and summer floods and relative sea level rise is expected to increase their frequency. The Mose system, with its 78 mobile gates that can hold back almost 10 feet of water, construction launched in 2003 and is said to be near completion in 2018.

Entrance to Malamocco port 1) Mose gate 2) 48-meter (157-foot) trench 3) the oil refinery canal. Image courtesy Ismar-Cnr.

“Before the Mose system begins to function, it was important to have a full picture of the bathymetry and currents of the tidal channels and inlets, which are the most dynamic portion of the lagoon,” researchers say in a paper published in “Nature.” They caution that the relatively rapid erosive process could threaten the stability of the “hard structures” (read: priceless palazzos) in the near future and should certainly be periodically monitored.

If you want to dig into the datasets, the scientists from research groups (Ismar-Cnr and Iim) have CC-licensed and made them available online with the paper.

A scour hole found where two channels meet. Image courtesy Ismar-Cnr.

“The data also allows us to identify areas with large dunes at the bottom and adjacent erosion sites that document the most dynamic points in the deep lagoon, where it’s important to cyclically repeat these studies to quantify the movement of sediments,” head of the study Fantina Madricardo says in the press release (translation mine.)

Part of the reason these Venice maps look so trippy (or alarming?) is due to the city’s curious geography, perching atop 118 islands bridged by canals. On most bathymetric maps, deeper waters are represented by soothing darker shades (green, blue, violet) and warmer colors (red, orange, yellow) represent shallower waters. A bathymetric map of the San Francisco Bay by comparison looks, well, a lot more soothing despite its notorious currents.

How to investigate your government through algorithms

Some kinds of reporting-by-the-numbers are anything but lazy. Take investigations looking into algorithms — examining the formulas used by the government to determine who is more likely to commit a crime or how likely your building is to have a fire inspection.

Speaking at the recent International Festival of Journalism, Nick Diakopoulos, assistant professor at the University of Maryland’s Philip Merrill College of Journalism and a member of its Human Computer Interaction Lab, gave a solid primer on how to get started.

He’s been studying the wider reach of algorithms in society, government and industry for about four years, coming at it from a computer science background as a “techie who worked my way into journalism.” Boyish, bespectacled and occasionally prone to professorial turns of phrase like “algorithmic accountability,” Diakopoulos offered a look into the numbers that shape our lives.

What they are

Photo: Jon Oropeza via Flickr. CC-licensed.

At the most basic level algorithms are like recipes, Diakopoulos says. They have ingredients, assembly instructions and a sequence or order for those instructions — your basic how-to method for doing something. Where the analogy falters, he says, is that unlike the sequence that results in a good plate of pasta al pomodoro, algorithms are decision-making formulas. “The crux of algorithmic power is how they make decisions, or have the potential to make decisions, potentially without any human involvement.”

These break down broadly into four types of decisions: prioritization, classification, association and filtering. Familiar examples include search engines top-ranking better sources of information; YouTube’s strainer for picking up copyrighted material; connecting terms—as in the slander lawsuit over Google’s auto-complete — and filtering, whereby some news sources rank higher than others.

Why you should care

Far from being impartial gatekeepers or shortcuts, algorithms are designed by humans — often with built-in bias that can shape our daily lives. They are deciding what schools kids attend, who gets released on parole and who your next date is.

“It’s time to start getting skeptical about algorithms,” Diakopoulos says. “It’s time to start asking questions to learn more about how these systems function and get more details on how they work.”

That’s where algorithmic accountability — pulling back the curtain on the formulas —  comes in. Diakopoulos cites a ProPublica investigation into software used in crime cases that asks a number of seemingly benign questions  — “What neighborhood do you live in?”  “What’s your education level?” “Are you in touch with your family?” — to arrive at a flight or future crime risk. Looking at the results in 7,000 cases, reporters discovered that the resulting “risk assessments” are not only biased against blacks but only slightly more accurate than a coin toss for predicting who will commit more crimes.

“Algorithmic accountability means investigating these systems and trying to understand how these quantifications affect people,”  he says. His team’s investigations have lead to articles including “How Google shapes the news you see about the candidates” and “Uber seems to offer better service in areas with more white people.” Continue reading

Crooked! Donald Trump’s most recent insults as a word cloud

UPDATE: The Times is still tracking the list of insults — as of January 2017 it grew to 305 — and added a visualization that shows the kinds of people and things most frequently insulted. (Spoiler alert: journalists and Democrats.)

The reporters at the New York Times combed through Republican presidential nominee Donald Trump’s Twitter feed for the most recent 250 insults to nations, people and random things – including a podium.

NYtimesThis is the kind of story that cries out for a visual representation – there has to be a better way to process the information than listing names of the people he insulted in alphabetical order and the tweets as quotes underneath them. What story does that tell?

Most commonly used words in Trump insults, by frequency.

Most commonly used words in Trump insults, by frequency. By Nicole Martinelli, via Wordle.

A quick word cloud will tell you that the most common insult for the straight-talking New Yorker is “crooked” (his go-to insult for rival Hillary Clinton) followed by “dishonest,” “bad,” and “failing.”

A couple of necessary caveats: this cloud was made with a tool called Wordle and the size of the word corresponds to the number of times it appears in the text. The text in the graphic was copied and pasted from the article on the NYT site without any additional weighting or manipulation. The program automatically cuts out common words (i.e. articles) but it would be interesting to see how the cloud shifts by cutting some filler words like “new” “news” “many” “another” etc.

Digital publishing gives public figures so many ways to broadcast a message – it’s our job as journalists to make sense of it. What would you trawl through other political figures tweets to understand?

Digital mapping finally blossoms for parks and rec

CC-licensed, thanks to KM on Flickr.

CC-licensed, thanks to KM on Flickr

If you went by the maps available today your smartphone, you’d probably think the era of paper maps went out with the Rubik’s Cube.

But the next time you’re in a garden or park, you might come across a group of volunteers huddled around an 11×17 black-and-white printout, hopelessly trying to verify what should be where, clumsily marking it up with pencils and colored highlighters. Their work trails back indoors where it adds to a pile of similar maps that have to be verified (or interpreted?) before changes are put into the data base. They are often printed out and verified again for accuracy.

The hateful verification map, SFBG.

A hateful paper verification map, SFBG.

That’s why around 40 people from zoos, gardens and parks from around the world came to a recent talk to hear how Steve Gensler, GIS manager of the San Francisco Botanical Gardens, and Veronica Nixon of the Desert Botanical Garden in Phoenix are using Collector for ArcGIS to maintaining botanical garden plant records at the ESRI User Conference.

Continue reading

The Associated Press Stylebook weighs in on data journalism

bye3nzmg6q355a3splxzCC-licensed, via hatalamas on Flickr.

If you write about tech, you’ll find the Associated Press Stylebook is a little bit like Dear Abby. By the time the bouffant-hair-and-matching-handbag set gets around to addressing an issue, it’s often already been answered by collective common sense.

Still, it’s nice to see the venerable news organization writing about data journalism in the same update where it finally relinquishes capitalizing the word internet.

The AP Stylebook entry on data journalism, added 2016-04-19, weighs in at just under 500 words.

It begins with six rules for evaluating a data set that range from the very basic (“What is the source?”) to the kind of deep dive that may prevent you from ever filing the story (“Is there a data dictionary or record layout document for the data set – which would describe the fields, types of data they contain and details and announcing detail as indicated?”) Side note: If you’re looking for an entire book of how to present data facts and figures for journalists, my favorite is still “The Wall Street journal guide to information graphics: the dos and don’ts of presenting data, facts, and figures” by Dona M Wong. [public library]

 

Screen Shot 2016-06-02 at 1.47.30 PM

The next section launches into the math of doing data journalism, a reminder that word people are often not numbers people. Or a reminder to all that, yeah, elementary school math is good to know.

“Avoid percentage and percent change comparisons from a small base. Rankings should include raw numbers to provide a sense of relative importance.
When comparing dollar amounts across time, be sure to adjust for inflation. When using averages (that is, adding together a group of numbers and dividing the sum by the quantity of numbers in the group), be wary of extreme, outlier values that may unfairly skew the result. It may be better to use the median (the middle number among all the numbers being considered) if there is a large difference between the average (mean) and the median.”

It heads into more advanced territory with a paragraph on causality, rounding numbers and sample size before winding up with a solid reminder for data-happy hacks: “Try not to include too many numbers in a single sentence or paragraph.”

Now we only have to wait and see how the Stylebook passes judgement on the proper abbreviation for “internet of things.”

Armchair mappers: help prepare for the next humanitarian crisis

Screen Shot 2015-05-07 at 5.05.35 PMKathmandu before and after OpenStreetMap’s humanitarian team dived in.

In just 48 hours after Nepal’s devastating earthquake, thousands of volunteers from around the world helped create maps that guided emergency response teams.

Many of these “digital humanitarians” came from OpenStreetMap, an open source mapping effort. OpenStreet Map launched  Humanitarian OpenStreetMap Team (HOT) in 2010 after the earthquake in Haiti, when the office safeguarding country’s maps pancaked in the 7.0 temblor.

_Schuyler Erle shows what happened to Haiti's mapping office post earthquake._

Schuyler Erle shows what happened to Haiti’s mapping office post earthquake.

Continue reading

Data points: visualization that means something [review]

Proving a point about data with the author's wedding photos.

Proving a point about data with the author’s wedding photos.

Nathan Yau is a self-appointed cicerone who shows the rest of us around the big, beautiful data visualization world.

A statistician by trade, the likeable, plain-speaking Yau runs Flowingdata.com. And, like all good tour guides, his job is to get you to think about what he’s presenting, not just drop your jaw at the sights.

In “Data Points: Visualization That Means Something,” [public library] Yau wants you to think about sample size — even if he has to show you with a jar of gumballs. To bring home the point about data representing real life, he takes out his own wedding photos. Later in the book, he manages to transform an eye-crossingly dull table of U.S. education stats into 29 different graphics to show how it’s done.
Continue reading

Where in the world can you still send a telegram? [Map]

Telegrams may have gone the way of the steam engine, but there are number of places around the world, from Japan to Mexico, still sending them.

The news about India shuttering its 162-year-old telegram service sounded like the last, labored puff of a country making progress into a bold new era.

So I wondered where people are still using them as a swift, inexpensive means to send condolences and well wishes on important occasions. (An outfit called – what else? – iTelegram took over from Western Union for the U.S., though I can’t remember ever sending or receiving a telegram here.)
Continue reading

Painful lessons in data journalism: scraping with Python

3932023011_32e5e18838_z

Lost in the woods. CC-licensed, Chris-Håvard Berge on Flickr.

Lost and found ads can be a good way to sniff out a story.

Take the ones on Craigslist about iPhones. There’s a woman who gained a husband in a quickie wedding at city hall but left her iPhone behind. Or a drunk college kid who dropped his phone on the passenger seat of a good samaritan who took him home.

Is there a bigger story about lost and stolen iPhones? To find out, I scraped all 50 states of Craigslist lost and found ads using Python and BeautifulSoup. If you want to check out or improve that code, it’s on GitHub. The full story (with charts and things!) is over at Cult of Mac.

The project required more fist clenching and eye straining than anticipated – even though writing a basic scraper for Craigslist is considered an easy-peasy programming project.

Let me just say it: as a novice Pythonista, I am challenged by nearly everything. I mean, command line interface, seriously? But I can get past that. I slogged through (and recommend) Learning Python the Hard Way, as well as finished some examples in Scraping for Journalists.
Continue reading